Skip to content

cmd/distpack: release archives don't include directory members #61862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jeffreytolar opened this issue Aug 8, 2023 · 9 comments
Closed

cmd/distpack: release archives don't include directory members #61862

jeffreytolar opened this issue Aug 8, 2023 · 9 comments
Assignees
@rsc
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.22

Comments

@jeffreytolar
Copy link

What version of Go are you using (go version)?

# /usr/local/go/bin/go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
# /usr/local/go/bin/go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY='go.ourcompany.com'
GONOSUMDB='go.ourcompany.com'
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE='go.ourcompany.com'
GOPROXY='https://our-internal-goproxy.ourcompany.com,https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2916500537=/tmp/go-build -gno-record-gcc-switches'

What did you do?

# curl -L --remote-name-all https://go.dev/dl/go{1.20,1.21.0}.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    73  100    73    0     0    327      0 --:--:-- --:--:-- --:--:--   328
100 95.2M  100 95.2M    0     0  73.2M      0  0:00:01  0:00:01 --:--:--  107M
100    75  100    75    0     0   2027      0 --:--:-- --:--:-- --:--:--  2027
100 63.3M  100 63.3M    0     0   117M      0 --:--:-- --:--:-- --:--:--  144M


# sha256sum go*.tar.gz
5a9ebcc65c1cce56e0d2dc616aff4c4cedcfbda8cc6f0288cc08cda3b18dcbf1  go1.20.linux-amd64.tar.gz
d0398903a16ba2232b389fb31032ddf57cac34efda306a0eebac34f0965a0742  go1.21.0.linux-amd64.tar.gz


# tar tzvf go1.20.linux-amd64.tar.gz | grep ' go/[^/]*/$'
drwxr-xr-x root/root         0 2023-01-30 21:48 go/api/
drwxr-xr-x root/root         0 2023-01-30 21:49 go/bin/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/doc/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/lib/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/misc/
drwxr-xr-x root/root         0 2023-01-30 21:49 go/pkg/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/src/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/test/


# tar tzvf go1.21.0.linux-amd64.tar.gz | grep ' go/[^/]*/$'
(empty)
# tar tzvf go1.21.0.linux-amd64.tar.gz | grep -c '/$'
0

What did you expect to see?

Directory entries in the release tarball

What did you see instead?

No directory entries in the release tarball

I'm not entirely sure if this is a bug or if it's an intentional change; we noticed it on hosts with a restricted umask, since the extracted directories didn't have world-read/execute permissions (unlike past releases).

I searched the issue tracker for "distpack", but didn't see anything that looked relevant.

This is presumably a consequence of 1635205 (switching to distpack for 1.21) / #24904?
@dmitshur
Copy link
Contributor

dmitshur commented Aug 8, 2023

The documentation of distpack.Archive suggests it was an intentional decision at the time:

Directories are implied by the files and not explicitly listed.

As does the second bullet point in #61513.

CC @rsc, @heschi.

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 8, 2023
@mknyszek mknyszek added this to the Backlog milestone Aug 8, 2023
@mknyszek mknyszek added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 8, 2023
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 8, 2023
@heschi
Copy link
Contributor

heschi commented Aug 8, 2023

Yeah, it was an intentional change. Because of the reproducibility push, changing back would be a bit troublesome. Offhand, it seems like honoring the host's umask is reasonable?

@jeffreytolar
Copy link
Author

Ok; it results in some odd behavior, however - GNU tar defaults to --preserve-permissions, which means it'll use the file modes in the tarball. With a umask of 027, this results in only the directories having the restrictive permissions.

E.g., on rhel8:

bash-4.4# curl -L --remote-name-all https://go.dev/dl/go{1.20,1.21.0}.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    73  100    73    0     0    304      0 --:--:-- --:--:-- --:--:--   305
100 95.2M  100 95.2M    0     0  72.8M      0  0:00:01  0:00:01 --:--:--  110M
100    75  100    75    0     0   1923      0 --:--:-- --:--:-- --:--:--  1923
100 63.3M  100 63.3M    0     0  91.4M      0 --:--:-- --:--:-- --:--:--  142M

bash-4.4# umask 027
bash-4.4# mkdir /usr/local/go{1.20,1.21.0}
bash-4.4# tar xzf go1.20.linux-amd64.tar.gz -C /usr/local/go1.20/
bash-4.4# tar xzf go1.21.0.linux-amd64.tar.gz -C /usr/local/go1.21.0/


bash-4.4# ls -ld /usr/local/go{1.20,1.21.0}/go
drwxr-xr-x. 10 root root 4096 Jan 30  2023 /usr/local/go1.20/go
drwxr-x---. 10 root root 4096 Aug  8 22:52 /usr/local/go1.21.0/go

bash-4.4# ls -ld /usr/local/go{1.20,1.21.0}/go/bin/go
-rwxr-xr-x. 1 root root 15563590 Jan 30  2023 /usr/local/go1.20/go/bin/go
-rwxr-xr-x. 1 root root 12403672 Aug  4 20:14 /usr/local/go1.21.0/go/bin/go

(Looks like rhel8's bsdtar has the same behavior)

@bernd bernd mentioned this issue Aug 9, 2023
Closed
@rsc rsc modified the milestones: Backlog, Go1.21.1 Aug 10, 2023
@rsc rsc self-assigned this Aug 10, 2023
@rsc
Copy link
Contributor

rsc commented Aug 10, 2023

We will fix this for Go 1.21.1. For supply chain reasons we can't reissue the 1.21.0 files.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/518335 mentions this issue: cmd/distpack: include directory entries in tar files

@rsc
Copy link
Contributor

rsc commented Aug 10, 2023

@gopherbot please backport go1.21

@gopherbot gopherbot mentioned this issue Aug 10, 2023
Closed
@gopherbot
Copy link
Contributor

Backport issue(s) opened: #61927 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@andrewkroh andrewkroh mentioned this issue Aug 10, 2023
Closed
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Aug 10, 2023
@dmitshur dmitshur modified the milestones: Go1.21.1, Go1.22 Aug 10, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/518835 mentions this issue: [release-branch.go1.21] cmd/distpack: include directory entries in tar files

gopherbot pushed a commit that referenced this issue Aug 11, 2023
@rsc @gopherbot
[release-branch.go1.21] cmd/distpack: include directory entries in ta…
9398951 
…r files

Various tools expect tar files to contain entries for directories.
I dropped them when writing cmd/distpack because they're not
strictly necessary and omitting them saves space, but it also
turns out to break some things, so add them back.

We will backport this to release-branch.go1.21 so that Go 1.21.1
will include the directory entries. We can't do anything about
Go 1.21.0 retroactively.

% tar tzvf go1.22rsc1.src.tar.gz | sed 10q
drwxr-xr-x  0 0      0           0 Aug 10 10:07 go/
-rw-r--r--  0 0      0        1337 Aug 10 10:07 go/CONTRIBUTING.md
-rw-r--r--  0 0      0        1479 Aug 10 10:07 go/LICENSE
-rw-r--r--  0 0      0        1303 Aug 10 10:07 go/PATENTS
-rw-r--r--  0 0      0        1455 Aug 10 10:07 go/README.md
-rw-r--r--  0 0      0         419 Aug 10 10:07 go/SECURITY.md
-rw-r--r--  0 0      0          42 Aug 10 10:07 go/VERSION
drwxr-xr-x  0 0      0           0 Aug 10 10:07 go/api/
-rw-r--r--  0 0      0        1142 Aug 10 10:07 go/api/README
-rw-r--r--  0 0      0       35424 Aug 10 10:07 go/api/except.txt
% tar tzvf go1.22rsc1.darwin-amd64.tar.gz | sed 10q
drwxr-xr-x  0 0      0           0 Aug 10 10:07 go/
-rw-r--r--  0 0      0        1337 Aug 10 10:07 go/CONTRIBUTING.md
-rw-r--r--  0 0      0        1479 Aug 10 10:07 go/LICENSE
-rw-r--r--  0 0      0        1303 Aug 10 10:07 go/PATENTS
-rw-r--r--  0 0      0        1455 Aug 10 10:07 go/README.md
-rw-r--r--  0 0      0         419 Aug 10 10:07 go/SECURITY.md
-rw-r--r--  0 0      0          42 Aug 10 10:07 go/VERSION
drwxr-xr-x  0 0      0           0 Aug 10 10:07 go/api/
-rw-r--r--  0 0      0        1142 Aug 10 10:07 go/api/README
-rw-r--r--  0 0      0       35424 Aug 10 10:07 go/api/except.txt
%

Fixes #61862.
Fixes #61927.

Change-Id: Iecd9ba893015295e88715b031b79a104236b9ced
Reviewed-on: https://go-review.googlesource.com/c/go/+/518335
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/518835
Auto-Submit: Russ Cox <rsc@golang.org>
@heschi
Copy link
Contributor

heschi commented Aug 11, 2023

Dry-run build looks good to me, including the signed Darwin archives.

@seankhliao seankhliao mentioned this issue Sep 6, 2023
Closed
@golang golang locked and limited conversation to collaborators Aug 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@rsc rsc

Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Milestone
Go1.22
Development

No branches or pull requests
6 participants
@rsc @mknyszek @dmitshur @gopherbot @heschi @jeffreytolar