cmd/distpack: release archives don't include directory members #61862

jeffreytolar · 2023-08-08T19:40:05Z

What version of Go are you using (`go version`)?

# /usr/local/go/bin/go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

# /usr/local/go/bin/go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY='go.ourcompany.com'
GONOSUMDB='go.ourcompany.com'
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE='go.ourcompany.com'
GOPROXY='https://our-internal-goproxy.ourcompany.com,https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2916500537=/tmp/go-build -gno-record-gcc-switches'

What did you do?

# curl -L --remote-name-all https://go.dev/dl/go{1.20,1.21.0}.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    73  100    73    0     0    327      0 --:--:-- --:--:-- --:--:--   328
100 95.2M  100 95.2M    0     0  73.2M      0  0:00:01  0:00:01 --:--:--  107M
100    75  100    75    0     0   2027      0 --:--:-- --:--:-- --:--:--  2027
100 63.3M  100 63.3M    0     0   117M      0 --:--:-- --:--:-- --:--:--  144M


# sha256sum go*.tar.gz
5a9ebcc65c1cce56e0d2dc616aff4c4cedcfbda8cc6f0288cc08cda3b18dcbf1  go1.20.linux-amd64.tar.gz
d0398903a16ba2232b389fb31032ddf57cac34efda306a0eebac34f0965a0742  go1.21.0.linux-amd64.tar.gz


# tar tzvf go1.20.linux-amd64.tar.gz | grep ' go/[^/]*/$'
drwxr-xr-x root/root         0 2023-01-30 21:48 go/api/
drwxr-xr-x root/root         0 2023-01-30 21:49 go/bin/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/doc/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/lib/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/misc/
drwxr-xr-x root/root         0 2023-01-30 21:49 go/pkg/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/src/
drwxr-xr-x root/root         0 2023-01-30 21:48 go/test/


# tar tzvf go1.21.0.linux-amd64.tar.gz | grep ' go/[^/]*/$'
(empty)
# tar tzvf go1.21.0.linux-amd64.tar.gz | grep -c '/$'
0

What did you expect to see?

Directory entries in the release tarball

What did you see instead?

No directory entries in the release tarball

I'm not entirely sure if this is a bug or if it's an intentional change; we noticed it on hosts with a restricted umask, since the extracted directories didn't have world-read/execute permissions (unlike past releases).

I searched the issue tracker for "distpack", but didn't see anything that looked relevant.

This is presumably a consequence of 1635205 (switching to distpack for 1.21) / #24904?

The text was updated successfully, but these errors were encountered:

dmitshur · 2023-08-08T19:46:45Z

The documentation of distpack.Archive suggests it was an intentional decision at the time:

Directories are implied by the files and not explicitly listed.

As does the second bullet point in #61513.

CC @rsc, @heschi.

heschi · 2023-08-08T21:13:50Z

Yeah, it was an intentional change. Because of the reproducibility push, changing back would be a bit troublesome. Offhand, it seems like honoring the host's umask is reasonable?

jeffreytolar · 2023-08-08T23:24:55Z

Ok; it results in some odd behavior, however - GNU tar defaults to --preserve-permissions, which means it'll use the file modes in the tarball. With a umask of 027, this results in only the directories having the restrictive permissions.

E.g., on rhel8:

bash-4.4# curl -L --remote-name-all https://go.dev/dl/go{1.20,1.21.0}.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    73  100    73    0     0    304      0 --:--:-- --:--:-- --:--:--   305
100 95.2M  100 95.2M    0     0  72.8M      0  0:00:01  0:00:01 --:--:--  110M
100    75  100    75    0     0   1923      0 --:--:-- --:--:-- --:--:--  1923
100 63.3M  100 63.3M    0     0  91.4M      0 --:--:-- --:--:-- --:--:--  142M

bash-4.4# umask 027
bash-4.4# mkdir /usr/local/go{1.20,1.21.0}
bash-4.4# tar xzf go1.20.linux-amd64.tar.gz -C /usr/local/go1.20/
bash-4.4# tar xzf go1.21.0.linux-amd64.tar.gz -C /usr/local/go1.21.0/


bash-4.4# ls -ld /usr/local/go{1.20,1.21.0}/go
drwxr-xr-x. 10 root root 4096 Jan 30  2023 /usr/local/go1.20/go
drwxr-x---. 10 root root 4096 Aug  8 22:52 /usr/local/go1.21.0/go

bash-4.4# ls -ld /usr/local/go{1.20,1.21.0}/go/bin/go
-rwxr-xr-x. 1 root root 15563590 Jan 30  2023 /usr/local/go1.20/go/bin/go
-rwxr-xr-x. 1 root root 12403672 Aug  4 20:14 /usr/local/go1.21.0/go/bin/go

(Looks like rhel8's bsdtar has the same behavior)

rsc · 2023-08-10T15:52:13Z

We will fix this for Go 1.21.1. For supply chain reasons we can't reissue the 1.21.0 files.

gopherbot · 2023-08-10T15:52:48Z

Change https://go.dev/cl/518335 mentions this issue: cmd/distpack: include directory entries in tar files

rsc · 2023-08-10T15:56:19Z

@gopherbot please backport go1.21

gopherbot · 2023-08-10T15:56:34Z

Backport issue(s) opened: #61927 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2023-08-11T17:39:21Z

Change https://go.dev/cl/518835 mentions this issue: [release-branch.go1.21] cmd/distpack: include directory entries in tar files

…r files Various tools expect tar files to contain entries for directories. I dropped them when writing cmd/distpack because they're not strictly necessary and omitting them saves space, but it also turns out to break some things, so add them back. We will backport this to release-branch.go1.21 so that Go 1.21.1 will include the directory entries. We can't do anything about Go 1.21.0 retroactively. % tar tzvf go1.22rsc1.src.tar.gz | sed 10q drwxr-xr-x 0 0 0 0 Aug 10 10:07 go/ -rw-r--r-- 0 0 0 1337 Aug 10 10:07 go/CONTRIBUTING.md -rw-r--r-- 0 0 0 1479 Aug 10 10:07 go/LICENSE -rw-r--r-- 0 0 0 1303 Aug 10 10:07 go/PATENTS -rw-r--r-- 0 0 0 1455 Aug 10 10:07 go/README.md -rw-r--r-- 0 0 0 419 Aug 10 10:07 go/SECURITY.md -rw-r--r-- 0 0 0 42 Aug 10 10:07 go/VERSION drwxr-xr-x 0 0 0 0 Aug 10 10:07 go/api/ -rw-r--r-- 0 0 0 1142 Aug 10 10:07 go/api/README -rw-r--r-- 0 0 0 35424 Aug 10 10:07 go/api/except.txt % tar tzvf go1.22rsc1.darwin-amd64.tar.gz | sed 10q drwxr-xr-x 0 0 0 0 Aug 10 10:07 go/ -rw-r--r-- 0 0 0 1337 Aug 10 10:07 go/CONTRIBUTING.md -rw-r--r-- 0 0 0 1479 Aug 10 10:07 go/LICENSE -rw-r--r-- 0 0 0 1303 Aug 10 10:07 go/PATENTS -rw-r--r-- 0 0 0 1455 Aug 10 10:07 go/README.md -rw-r--r-- 0 0 0 419 Aug 10 10:07 go/SECURITY.md -rw-r--r-- 0 0 0 42 Aug 10 10:07 go/VERSION drwxr-xr-x 0 0 0 0 Aug 10 10:07 go/api/ -rw-r--r-- 0 0 0 1142 Aug 10 10:07 go/api/README -rw-r--r-- 0 0 0 35424 Aug 10 10:07 go/api/except.txt % Fixes #61862. Fixes #61927. Change-Id: Iecd9ba893015295e88715b031b79a104236b9ced Reviewed-on: https://go-review.googlesource.com/c/go/+/518335 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-on: https://go-review.googlesource.com/c/go/+/518835 Auto-Submit: Russ Cox <rsc@golang.org>

heschi · 2023-08-11T20:34:21Z

Dry-run build looks good to me, including the signed Darwin archives.

mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 8, 2023

mknyszek added this to the Backlog milestone Aug 8, 2023

mknyszek added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 8, 2023

gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 8, 2023

bernd mentioned this issue Aug 9, 2023

Go 1.21.0 package does not install niemeyer/godeb#51

Closed

rsc modified the milestones: Backlog, Go1.21.1 Aug 10, 2023

rsc self-assigned this Aug 10, 2023

rsc added the release-blocker label Aug 10, 2023

gopherbot mentioned this issue Aug 10, 2023

cmd/distpack: release archives don't include directory members [1.21 backport] #61927

Closed

andrewkroh mentioned this issue Aug 10, 2023

GVM cannot install 1.21 andrewkroh/gvm#61

Closed

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Aug 10, 2023

dmitshur modified the milestones: Go1.21.1, Go1.22 Aug 10, 2023

gopherbot closed this as completed in 673d64a Aug 11, 2023

seankhliao mentioned this issue Sep 6, 2023

Installation error of 1.21.0 on Linux: wrong permissions after uncompress tar file. #62479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/distpack: release archives don't include directory members #61862

cmd/distpack: release archives don't include directory members #61862

jeffreytolar commented Aug 8, 2023

dmitshur commented Aug 8, 2023

heschi commented Aug 8, 2023

jeffreytolar commented Aug 8, 2023

rsc commented Aug 10, 2023

gopherbot commented Aug 10, 2023

rsc commented Aug 10, 2023

gopherbot commented Aug 10, 2023

gopherbot commented Aug 11, 2023

heschi commented Aug 11, 2023

cmd/distpack: release archives don't include directory members #61862

cmd/distpack: release archives don't include directory members #61862

Comments

jeffreytolar commented Aug 8, 2023

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

dmitshur commented Aug 8, 2023

heschi commented Aug 8, 2023

jeffreytolar commented Aug 8, 2023

rsc commented Aug 10, 2023

gopherbot commented Aug 10, 2023

rsc commented Aug 10, 2023

gopherbot commented Aug 10, 2023

gopherbot commented Aug 11, 2023

heschi commented Aug 11, 2023

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?