Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add longtest builders for linux-arm64 #49649

Closed
bcmills opened this issue Nov 17, 2021 · 45 comments
Closed

x/build: add longtest builders for linux-arm64 #49649

bcmills opened this issue Nov 17, 2021 · 45 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 17, 2021

According to https://golang.org/wiki/PortingPolicy#first-class-ports, linux/arm and linux/arm64 are currently “first class ports”.

However, we have no longtest builder coverage whatsoever for these architectures. If we really intend to support them on par with linux/386 and linux/amd64, they ought to have builders running the tests at the same level of depth.

(CC @golang/release, @thanm)

@bcmills bcmills added new-builder NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 17, 2021
@bcmills bcmills added this to the Unreleased milestone Nov 17, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 17, 2021
@toothrot
Copy link
Contributor

I have to do some math, but I am not sure that we currently have the hardware capacity for this in the immediate term.

@cagedmantis
Copy link
Contributor

@toothrot Now that we have linux-amd64 instances on GCP, do you see any reason why we wouldn't be able to add a longtest builder?

@heschi
Copy link
Contributor

heschi commented Nov 22, 2022

@cherrymui do you think you can take a look at this? For now we can only do arm64 because arm isn't working on GCE.

@cherrymui
Copy link
Member

Sure. I'll take a look.

@gopherbot
Copy link

Change https://go.dev/cl/452858 mentions this issue: dashboard: add linux-arm64-longtest builder

gopherbot pushed a commit to golang/build that referenced this issue Nov 23, 2022
For golang/go#49649.
Updates golang/go#53851.

Change-Id: Ifbfd2b089b909542d0568d529ba738e4b217cf5c
Reviewed-on: https://go-review.googlesource.com/c/build/+/452858
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@cherrymui
Copy link
Member

The linux-arm64-longtest builder is up and running. But there are some test failures. I'll look into them.

@cherrymui
Copy link
Member

    --- FAIL: TestScript/build_issue48319 (15.85s)
        script_test.go:134: 2022-11-23T20:27:16Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-3429322971/tmpdir1263247943/build_issue483194135768660
        script_test.go:154: 
            # Regression test for https://go.dev/issue/48319:
            # cgo builds should not include debug information from a stale GOROOT_FINAL. (0.000s)
            # This test is sensitive to cache invalidation,
            # so use a separate build cache that we can control. (0.000s)
            # Build a binary using a specific value of GOROOT_FINAL. (7.952s)
            # Now clean the cache and build using a different GOROOT_FINAL.
            # The resulting binaries should differ in their debug metadata. (7.411s)
            # Set GOROOT_FINAL back to the first value.
            # If the build is properly reproducible, the two binaries should match. (0.445s)
            > env GOROOT_FINAL=$WORK${/}goroot1
            > go build -o main.exe
            > cmp -q main.exe main1.exe
        script_test.go:154: FAIL: build_issue48319.txt:30: cmp -q main.exe main1.exe: main.exe and main1.exe differ

I looked into this. It is not that #48319 is regressed or anything to do with GOROOT_FINAL, but external linking looks nondeterministic. Somehow the (externally linked) output contains the path of a temporary file. I'll see if there is a way to prevent that.

The C toolchain on the builder is

gcc (Debian 10.2.1-6) 10.2.1 20210110
GNU ld (GNU Binutils for Debian) 2.35.2

@cherrymui
Copy link
Member

   --- FAIL: TestScript/version_buildvcs_git_gpg (0.04s)
        script_test.go:134: 2022-11-23T22:15:30Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-262463489/tmpdir1357961938/version_buildvcs_git_gpg2188942372
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.043s)
            > exec gpg --batch --passphrase '' --quick-generate-key gopher@golang.org
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-262463489/tmpdir1357961938/version_buildvcs_git_gpg2188942372/.gpupg/pubring.kbx' created
            gpg: error running '/usr/bin/gpg-agent': probably not installed
            gpg: failed to start agent '/usr/bin/gpg-agent': Configuration error
            gpg: can't connect to the agent: Configuration error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key gopher@golang.org: exit status 2

The builder has gpg command installed, but not gpg-agent. The test fails when gpg tries to run gpg-agent. Installing gpg-agent makes it pass.

@bcmills perhaps we could make the test tolerate missing gpg-agent?
@cagedmantis should we install gpg-agent on the builder image?
Besides, on our linux-amd64-longtest builder (AMD64, not ARM64), despite we try to install gpg in our script https://cs.opensource.google/go/x/build/+/master:env/linux-x86-bullseye/Dockerfile;l=47 , it seems the builder actually does NOT have the gpg binary.

@cherrymui
Copy link
Member

cherrymui commented Nov 23, 2022

I ran the tests manually on gomote. Besides the two failures above, and a flaky failure of #56574, another issue is in misc/reboot it runs out of disk space.

##### ../misc/reboot
Building Go cmd/dist using /tmp/workdir/go. (devel gomote.XXXXX linux/arm64)
Building Go toolchain1 using /tmp/workdir/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
go build cmd/compile/internal/noder: # cmd/compile/internal/noder
compile: writing output: write $WORK/b148/_pkg_.a: no space left on device
go tool dist: FAILED: /tmp/TestRepeatBootstrap2475739123/001/goroot/pkg/tool/linux_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

The machine seems to have rather small amount of disk space:

Filesystem      Size  Used Avail Use% Mounted on
overlay         5.7G  5.0G  642M  89% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/nvme0n1p1  5.7G  5.0G  642M  89% /etc/hosts

@cagedmantis would it be possible to increase the disk space? Thanks.

@gopherbot
Copy link

Change https://go.dev/cl/453656 mentions this issue: env/linux-arm64-bullseye: add GPG agent to the image

@cagedmantis
Copy link
Contributor

@cherrymui @bcmills adding gpg-agent on the linux-arm64-bullseye image changes the TestScript/version_buildvcs_git_gpg test failure output to:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-11-28T17:41:48Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-4248730206/tmpdir3252825733/version_buildvcs_git_gpg386675436
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.013s)
            > exec gpg --batch --passphrase '' --quick-generate-key gopher@golang.org
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-4248730206/tmpdir3252825733/version_buildvcs_git_gpg386675436/.gpupg/pubring.kbx' created
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key gopher@golang.org: exit status 2

@cagedmantis
Copy link
Contributor

I can roll the increase in disk space into another set of changes I am making. That will be mailed shortly.

@cherrymui
Copy link
Member

@cagedmantis thanks for the quick update!

I'm not very familiar with gpg. @bcmills do you recognize that failure mode? Thanks.

@bcmills
Copy link
Contributor Author

bcmills commented Nov 28, 2022

@bcmills perhaps we could make the test tolerate missing gpg-agent?

My understanding from https://www.gnupg.org/documentation/manuals/gnupg/GPG-Configuration-Options.html#GPG-Configuration-Options is that recent versions of gpg always require gpg-agent. I don't know why distro packagers aren't treating it as a hard dependency.

From that page:

--use-agent
--no-use-agent
This is dummy option. gpg always requires the agent.

So I think it's fine for the test to fail if gpg is installed but broken. (The user always has the option to either fix it or remove the broken tool.)

I'm not very familiar with gpg. @bcmills do you recognize that failure mode? Thanks.

I do not. I've mailed https://go.dev/cl/453775 to try to produce more useful output for debugging.

@gopherbot
Copy link

Change https://go.dev/cl/453775 mentions this issue: cmd/go: run the gpg command verbosely in TestScript/version_buildvcs_git_gpg

gopherbot pushed a commit that referenced this issue Nov 28, 2022
…git_gpg

Also update test helper programs to avoid the deprecated io/ioutil
package and fix minor formatting issues.

For #49649.

Change-Id: Id404acbb2795470420854d682f849d959d2080c0
Reviewed-on: https://go-review.googlesource.com/c/go/+/453775
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Nov 28, 2022
This change adds gpg-agent to the image used for linux-arm64-longtest.
TestScript/version_buildvcs_git_gpg is failing because it is missing.
This also adds gpg-agent and triggers a rebuild of the
linux-x86-bullseye image.

Upates golang/go#49649

Change-Id: I8adac58f21beeb7a1403d8d672b5a7409fd14f74
Reviewed-on: https://go-review.googlesource.com/c/build/+/453656
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@cagedmantis
Copy link
Contributor

The linux-x86-bullseye and linux-arm64-bullseye containers images have been updated with the changes in go.dev/cl/453656.

@gopherbot
Copy link

Change https://go.dev/cl/453955 mentions this issue: all: add additonal storage to linux-arm64-longtest builder

@heschi heschi changed the title x/build: add longtest builders for linux-arm and linux-arm64 x/build: add longtest builders for linux-arm64 Nov 29, 2022
@heschi
Copy link
Contributor

heschi commented Nov 29, 2022

Retitling to reflect the fact that we don't have short-term plans to add 32-bit ARM since it isn't available on GCP at the moment.

gopherbot pushed a commit to golang/build that referenced this issue Nov 29, 2022
This adds the ability to set the size of the root drive for a GCE
host machine. The linux-arm64-longtest boot disk has been increased
from 10GB to 20GB.

Updates golang/go#49649

Change-Id: I74ebf4bd533c91c8716b7df167afacbfdded3ed3
Reviewed-on: https://go-review.googlesource.com/c/build/+/453955
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@bcmills
Copy link
Contributor Author

bcmills commented Nov 29, 2022

Are 64-bit GCP machines capable of running 32-bit ARM binaries? (That is, could we test GOARCH=arm on an arm64 instance, the same way one can test GOARCH=386 on an amd64 machine?)

@cherrymui
Copy link
Member

I think the kernel can be configured to support 32-bit binaries or not. Perhaps ours is not. Not sure if we could choose a different kernel configuration.

@cagedmantis
Copy link
Contributor

@cherrymui I will reach out to troubleshoot with you once I work on it again.

@cagedmantis
Copy link
Contributor

The change that increases the disk space for the linux-arm64-longtest builder has been deployed to production.

@cherrymui
Copy link
Member

Thank you @cagedmantis !

@cagedmantis
Copy link
Contributor

cagedmantis commented Nov 30, 2022

TestScript/version_buildvcs_git_gpg continues to fail and the verbose flag isn't really giving us any new information:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.09s)
        script_test.go:134: 2022-11-30T19:55:13Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-761598066/tmpdir1312967028/version_buildvcs_git_gpg3191988827
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.090s)
            > exec gpg --batch --passphrase '' --quick-generate-key --verbose gopher@golang.org
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-761598066/tmpdir1312967028/version_buildvcs_git_gpg3191988827/.gpupg/pubring.kbx' created
            gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose gopher@golang.org: exit status 2

If we explicitly start gpg-agent before calling gpg we see the following error:

    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-11-30T19:20:43Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.005s)
            > exec gpg-agent --daemon
            [stderr]
            gpg-agent[99526]: directory '/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583/.gpupg/private-keys-v1.d' created
            gpg-agent[99526]: socket name '/tmp/workdir/tmp/cmd-go-test-2211690138/tmpdir4063638195/version_buildvcs_git_gpg1952539583/.gpupg/S.gpg-agent' is too long
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg-agent --daemon: exit status 2

This seems like the reason why we are seeing an error when gpg is starting gpg-agent. I believe the limit is 100 characters. The socket name used above is 110 characters. Does anybody have a preferred method for addressing this? Perhaps using a temp directory instead of something in the work directory?

@bcmills
Copy link
Contributor Author

bcmills commented Nov 30, 2022

Oof. That seems like an upstream bug in gpg-agent, which really ought to avoid assuming that it can create sockets in arbitrary locations. Given how fragile gpg seems to be, I wonder if we should just delete that test.

The test was added it CL 367034@andremarianiello, do you have any suggestions as to how we can make the gpg invocations more robust?

@cherrymui
Copy link
Member

The gpg failure is interesting. When I gomote ssh to the builder and run the test, it passes.

root@43856dcab278:/tmp/workdir/go/src# go test -run=TestScript/version_buildvcs_git_gpg cmd/go
ok  	cmd/go	2.719s

But if I run it without ssh, with gomote run, it fails

$ gomote run $VM go/bin/go test -run=TestScript/version_buildvcs_git_gpg cmd/go
# Streaming results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote356094641/cherryyz-linux-arm64-longtest-0.stdout"...
vcs-test.golang.org rerouted to http://127.0.0.1:46213
https://vcs-test.golang.org rerouted to https://127.0.0.1:44069
go test proxy running at GOPROXY=http://127.0.0.1:37133/mod
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
        script_test.go:134: 2022-12-01T18:39:44Z
        script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-585135258/tmpdir109437432/version_buildvcs_git_gpg800350627
        script_test.go:154: 
            # This test checks that VCS information is stamped into Go binaries even when
            # the current commit is signed and the use has configured git to display commit
            # signatures. (0.000s)
            # Create GPG key (0.005s)
            > exec gpg --batch --passphrase '' --quick-generate-key --verbose gopher@golang.org
            [stderr]
            gpg: keybox '/tmp/workdir/tmp/cmd-go-test-585135258/tmpdir109437432/version_buildvcs_git_gpg800350627/.gpupg/pubring.kbx' created
            gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
            gpg: error running '/usr/bin/gpg-agent': exit status 2
            gpg: failed to start agent '/usr/bin/gpg-agent': General error
            gpg: can't connect to the agent: General error
            gpg: agent_genkey failed: No agent running
            gpg: key generation failed: No agent running
        script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose gopher@golang.org: exit status 2
FAIL
FAIL	cmd/go	0.063s
FAIL
# Wrote results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote356094641/cherryyz-linux-arm64-longtest-0.stdout".
Error running run: unable to execute go/bin/go: rpc error: code = Unknown desc = command execution failed: exit status 1

Weird. Maybe the gpg command has to connect to a terminal, despite we pass --batch and it is not actually interactive?

Also, even after CL 453656, the x86 builder, linux-amd64-longtest, still does not have gpg command installed so the test would be skipped. Weird...

@andremarianiello
Copy link
Contributor

Oof. That seems like an upstream bug in gpg-agent, which really ought to avoid assuming that it can create sockets in arbitrary locations. Given how fragile gpg seems to be, I wonder if we should just delete that test.

The test was added it CL 367034@andremarianiello, do you have any suggestions as to how we can make the gpg invocations more robust?

I don't, unfortunately. Ideally we could run gpg with an agent, but it appears that gpg2 requires an agent to run. I also couldn't find a setting to control the max socket name length. I wouldn't bother me personally to remove the test, but it was requested (IIRC) by one of the reviewers of my MR, so you might want to ask for that person's opinion as well.

@bcmills
Copy link
Contributor Author

bcmills commented Dec 1, 2022

@andremarianiello

you might want to ask for that person's opinion as well.

I'm pretty sure that person was me. 🙃

@bcmills
Copy link
Contributor Author

bcmills commented Dec 1, 2022

@cherrymui, the difference between SSH and gomote run might have something to do with #56015 — maybe try running the test with -v in the SSH session to see what TMPDIR paths it's using?

@cagedmantis
Copy link
Contributor

@cherrymui I just rebuilt the linux-x86-bullseye image and tagged it as latest. I created and gomote ssh into an instance and can confirm gpg is installed. I must have made some sort of error when I ran through this procedure before.

@cherrymui
Copy link
Member

@bcmills thanks for the information!

SSH:

# go test -run=TestScript/version_buildvcs_git_gpg -v cmd/go
=== RUN   TestScript
vcs-test.golang.org rerouted to http://127.0.0.1:35423
https://vcs-test.golang.org rerouted to https://127.0.0.1:41031
go test proxy running at GOPROXY=http://127.0.0.1:36889/mod
=== RUN   TestScript/version_buildvcs_git_gpg
=== PAUSE TestScript/version_buildvcs_git_gpg
=== CONT  TestScript/version_buildvcs_git_gpg
    script_test.go:134: 2022-12-01T19:12:53Z
    script_test.go:136: $WORK=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854
    script_test.go:154: 
        PATH=/tmp/cmd-go-test-85347730/tmpdir2963332174/testbin:/tmp/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/workdir/go/bin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=arm64
        TESTGO_GOHOSTARCH=arm64
        GOCACHE=/tmp/workdir/gocache/
        GOCOVERDIR=
        GODEBUG=
        GOEXE=
        GOEXPERIMENT=
        GOOS=linux
        TESTGO_GOHOSTOS=linux
        GOPROXY=http://127.0.0.1:36889/mod
        GOPRIVATE=
        GOROOT=/tmp/workdir/go
        GOROOT_FINAL=
        GOTRACEBACK=system
        TESTGO_GOROOT=/tmp/workdir/go
        TESTGO_VCSTEST_HOST=127.0.0.1:35423
        TESTGO_VCSTEST_TLS_HOST=127.0.0.1:41031
        TESTGO_VCSTEST_CERT=/tmp/cmd-go-test-85347730/vcstest3018419858/cert.pem
        GOSUMDB=localhost.localdev/sumdb+00000c67+AcTrnkbUA+TU4heY3hkjiSES/DSQniBqIeQ/YppAUtK6
        GONOPROXY=
        GONOSUMDB=
        GOVCS=*:all
        devnull=/dev/null
        goversion=1.20
        CMDGO_TEST_RUN_MAIN=true
        GIT_TRACE_CURL=1
        GIT_TRACE_CURL_NO_DATA=1
        GIT_REDACT_COOKIES=o,SSO,GSSO_Uberproxy
        WORK=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854
        TMPDIR=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/tmp
        GOPATH=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath
        PWD=/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/src
        
        # This test checks that VCS information is stamped into Go binaries even when
        # the current commit is signed and the use has configured git to display commit
        # signatures. (0.000s)
        > [!git] skip
        [condition not met]
        > [!exec:gpg] skip
        [condition not met]
        > [short] skip
        [condition not met]
        > env GOBIN=$GOPATH/bin
        > env GNUPGHOME=$WORK/.gpupg
        > mkdir $GNUPGHOME
        > chmod 0700 $GNUPGHOME
        # Create GPG key (2.362s)
        > exec gpg --batch --passphrase '' --quick-generate-key --verbose gopher@golang.org
        [stderr]
        gpg: keybox '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/pubring.kbx' created
        gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
        gpg: waiting for the agent to come up ... (5s)
        gpg: connection to agent established
        gpg: writing self signature
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 [?]"
        gpg: writing key binding signature
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 [?]"
        gpg: writing public key to '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/pubring.kbx'
        gpg: /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/trustdb.gpg: trustdb created
        gpg: using pgp trust model
        gpg: key 00F57831E92D5A59 marked as ultimately trusted
        gpg: directory '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d' created
        gpg: writing to '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d/27144942722831C9413D64E600F57831E92D5A59.rev'
        gpg: RSA/SHA512 signature from: "00F57831E92D5A59 gopher@golang.org"
        gpg: revocation certificate stored as '/tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/.gpupg/openpgp-revocs.d/27144942722831C9413D64E600F57831E92D5A59.rev'
        > exec gpg --list-secret-keys --with-colons gopher@golang.org
        [stdout]
        sec:u:3072:1:00F57831E92D5A59:1669921973:1732993973::u:::scESC:::+:::23::0:
        fpr:::::::::27144942722831C9413D64E600F57831E92D5A59:
        grp:::::::::1721A19CCA165EF2A4A5EC0F0C5DFFF351AF6F5D:
        uid:u::::1669921973::BECC172B7C69C614EE8645F3D9D1A9776D8A0C9D::gopher@golang.org::::::::::0:
        ssb:u:3072:1:A55E7D2A642C0E1E:1669921973::::::e:::+:::23:
        fpr:::::::::3F4912A4EEB743DB04896097A55E7D2A642C0E1E:
        grp:::::::::B09AC1E76120C776ED01BCCEAE6F71B727CC0699:
        [stderr]
        gpg: checking the trustdb
        gpg: marginals needed: 3  completes needed: 1  trust model: pgp
        gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
        gpg: next trustdb check due at 2024-11-30
        > cp stdout keyinfo.txt
        > go run extract_key_id.go keyinfo.txt
        [stdout]
        00F57831E92D5A59> cp stdout keyid.txt
        # Initialize repo (0.127s)
        > cd repo/
        > exec git init
        [stdout]
        Initialized empty Git repository in /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/src/repo/.git/
        [stderr]
        hint: Using 'master' as the name for the initial branch. This default branch name
        hint: is subject to change. To configure the initial branch name to use in all
        hint: of your new repositories, which will suppress this warning, call:
        hint: 
        hint: 	git config --global init.defaultBranch <name>
        hint: 
        hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
        hint: 'development'. The just-created branch can be renamed via this command:
        hint: 
        hint: 	git branch -m <name>
        > exec git config user.email gopher@golang.org
        > exec git config user.name 'J.R. Gopher'
        > exec git config --add log.showSignature true
        > go run ../configure_signing_key.go ../keyid.txt
        # Create signed commit (0.037s)
        > cd a
        > exec git add -A
        > exec git commit -m 'initial commit' --gpg-sign
        [stdout]
        [master (root-commit) bae5d74] initial commit
         3 files changed, 9 insertions(+)
         create mode 100644 README
         create mode 100644 a/a.go
         create mode 100644 a/go.mod
        > exec git log
        [stdout]
        commit bae5d7449cabf06cc7a205907d3e5039280c15e0
        gpg: Signature made Thu Dec  1 19:12:55 2022 UTC
        gpg:                using RSA key 27144942722831C9413D64E600F57831E92D5A59
        gpg: Good signature from "gopher@golang.org" [ultimate]
        Author: J.R. Gopher <gopher@golang.org>
        Date:   Thu Dec 1 19:12:55 2022 +0000
        
            initial commit
        # Verify commit signature does not interfere with versioning (0.102s)
        > go install
        > go version -m $GOBIN/a
        [stdout]
        /tmp/cmd-go-test-85347730/tmpdir2963332174/version_buildvcs_git_gpg3142109854/gopath/bin/a: devel gomote.XXXXX
        	path	example.com/a
        	mod	example.com/a	(devel)	
        	build	-buildmode=exe
        	build	-compiler=gc
        	build	CGO_ENABLED=1
        	build	CGO_CFLAGS=
        	build	CGO_CPPFLAGS=
        	build	CGO_CXXFLAGS=
        	build	CGO_LDFLAGS=
        	build	GOARCH=arm64
        	build	GOOS=linux
        	build	vcs=git
        	build	vcs.revision=bae5d7449cabf06cc7a205907d3e5039280c15e0
        	build	vcs.time=2022-12-01T19:12:55Z
        	build	vcs.modified=false
        > stdout '^\tbuild\tvcs\.revision='
        matched: 	build	vcs.revision=bae5d7449cabf06cc7a205907d3e5039280c15e0
        > stdout '^\tbuild\tvcs\.time='
        matched: 	build	vcs.time=2022-12-01T19:12:55Z
        > stdout '^\tbuild\tvcs\.modified=false$'
        matched: 	build	vcs.modified=false
--- PASS: TestScript (0.02s)
    --- PASS: TestScript/version_buildvcs_git_gpg (2.63s)
PASS
ok  	cmd/go	2.681s

gomote run:

$ gomote run $VM go/bin/go test -run=TestScript/version_buildvcs_git_gpg -v cmd/go
# Streaming results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote3455930797/cherryyz-linux-arm64-longtest-0.stdout"...
=== RUN   TestScript
vcs-test.golang.org rerouted to http://127.0.0.1:42595
https://vcs-test.golang.org rerouted to https://127.0.0.1:40005
go test proxy running at GOPROXY=http://127.0.0.1:46363/mod
=== RUN   TestScript/version_buildvcs_git_gpg
=== PAUSE TestScript/version_buildvcs_git_gpg
=== CONT  TestScript/version_buildvcs_git_gpg
    script_test.go:134: 2022-12-01T19:13:59Z
    script_test.go:136: $WORK=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812
    script_test.go:154: 
        PATH=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/testbin:/tmp/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=arm64
        TESTGO_GOHOSTARCH=arm64
        GOCACHE=/tmp/workdir/gocache
        GOCOVERDIR=
        GODEBUG=
        GOEXE=
        GOEXPERIMENT=
        GOOS=linux
        TESTGO_GOHOSTOS=linux
        GOPROXY=http://127.0.0.1:46363/mod
        GOPRIVATE=
        GOROOT=/tmp/workdir/go
        GOROOT_FINAL=
        GOTRACEBACK=system
        TESTGO_GOROOT=/tmp/workdir/go
        TESTGO_VCSTEST_HOST=127.0.0.1:42595
        TESTGO_VCSTEST_TLS_HOST=127.0.0.1:40005
        TESTGO_VCSTEST_CERT=/tmp/workdir/tmp/cmd-go-test-1077973377/vcstest1961311730/cert.pem
        GOSUMDB=localhost.localdev/sumdb+00000c67+AcTrnkbUA+TU4heY3hkjiSES/DSQniBqIeQ/YppAUtK6
        GONOPROXY=
        GONOSUMDB=
        GOVCS=*:all
        devnull=/dev/null
        goversion=1.20
        CMDGO_TEST_RUN_MAIN=true
        GIT_TRACE_CURL=1
        GIT_TRACE_CURL_NO_DATA=1
        GIT_REDACT_COOKIES=o,SSO,GSSO_Uberproxy
        WORK=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812
        TMPDIR=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/tmp
        GOPATH=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/gopath
        PWD=/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/gopath/src
        
        # This test checks that VCS information is stamped into Go binaries even when
        # the current commit is signed and the use has configured git to display commit
        # signatures. (0.000s)
        > [!git] skip
        [condition not met]
        > [!exec:gpg] skip
        [condition not met]
        > [short] skip
        [condition not met]
        > env GOBIN=$GOPATH/bin
        > env GNUPGHOME=$WORK/.gpupg
        > mkdir $GNUPGHOME
        > chmod 0700 $GNUPGHOME
        # Create GPG key (0.005s)
        > exec gpg --batch --passphrase '' --quick-generate-key --verbose gopher@golang.org
        [stderr]
        gpg: keybox '/tmp/workdir/tmp/cmd-go-test-1077973377/tmpdir2489833868/version_buildvcs_git_gpg2267961812/.gpupg/pubring.kbx' created
        gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
        gpg: error running '/usr/bin/gpg-agent': exit status 2
        gpg: failed to start agent '/usr/bin/gpg-agent': General error
        gpg: can't connect to the agent: General error
        gpg: agent_genkey failed: No agent running
        gpg: key generation failed: No agent running
    script_test.go:154: FAIL: version_buildvcs_git_gpg.txt:14: exec gpg --batch --passphrase  --quick-generate-key --verbose gopher@golang.org: exit status 2
--- FAIL: TestScript (0.02s)
    --- FAIL: TestScript/version_buildvcs_git_gpg (0.01s)
FAIL
FAIL	cmd/go	0.055s
FAIL
# Wrote results from "cherryyz-linux-arm64-longtest-0" to "/var/folders/hn/13ch8f612p7_6sr0h3wzf_9r00bkwx/T/gomote3455930797/cherryyz-linux-arm64-longtest-0.stdout".
Error running run: unable to execute go/bin/go: rpc error: code = Unknown desc = command execution failed: exit status 1

It looks like SSH use /tmp based tmp dir, whereas gomote run uses /tmp/workdir/tmp. I wouldn't think this is a problem. But interestingly, on SSH, TMPDIR=/tmp/workdir/tmp go test -run=TestScript/version_buildvcs_git_gpg cmd/go fails. So it may be actually it! Not sure why gpg is sensitive to the tmp dir path, though...

@cherrymui
Copy link
Member

@cagedmantis yeah, it does have gpg now. Thanks!

@cherrymui
Copy link
Member

@cagedmantis with gpg installed on the linux-amd64-longtest builder, the test actually fails https://build.golang.org/log/31dc3ccbeed3498171be92216b02a7a6db44fb5a

Maybe we want to roll it back or skip the test...

@cagedmantis
Copy link
Contributor

I've tagged linux-x86-bullseye@sha256:b7509b8ff467dca29b1606e2171ce5dd6b92efa1aba479fa3fd41778aa6d2e69 (the previous version) as latest. This should prevent the linux-amd64-longtest and linux-386-longtest builders from failing. The GPG test failure should probably become its own issue. Creating a separate issue would facilitate backporting fixes if that becomes necessary.

@andremarianiello
Copy link
Contributor

I'm pretty sure that person was me. 🙃

WOMP, sorry! I'm fine with whatever you want to do haha!

@bcmills
Copy link
Contributor Author

bcmills commented Dec 1, 2022

@cherrymui, the difference in TMPDIR lengths is the likely explanation for the difference. The struct passed to the system call to create a Unix socket has an astonishingly short hard-coded length for the path.

@cherrymui
Copy link
Member

Split the gpg failure and the nondeterministic build failure to #57034 and #57035 . Thanks.

@cherrymui
Copy link
Member

@bcmills exactly! Thanks!
Added strace to gpg-agent, and

write(2, "gpg-agent[35726]: socket name '/"..., 141gpg-agent[35726]: socket name '/tmp/workdir/tmp/cmd-go-test-1432085988/tmpdir2462573141/version_buildvcs_git_gpg4118434835/.gpupg/S.gpg-agent) = 141
write(2, "' is too long\n", 14' is too long

(If we want to keep the test) is there something the test script could do to work around?

@cherrymui
Copy link
Member

With all the CLs/issues above the linux-arm64-longtest builder is now "ok". Thank you @cagedmantis @bcmills !

I'll watch it for a while and if it works reliably we can remove the "known issue".

@cherrymui
Copy link
Member

Tried a trybot run

  2022-12-06T01:19:33Z tests_complete took 9m4.037541154s; aggregate 41m57.48768334s; saved 32m53.450142186s
  2022-12-06T01:19:33Z finish_make_and_test after 10m16.3s
  2022-12-06T01:19:33Z done all tests passed

Seems reasonably fast, faster than the linux-arm-aws and windows-386-2008 builders.
Seems okay to turn it on as trybot for release branch.

@gopherbot
Copy link

Change https://go.dev/cl/456043 mentions this issue: dashboard: enable linux-arm64-longtest builder on 1.20+ release branches

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 19, 2023
@gopherbot
Copy link

Change https://go.dev/cl/504525 mentions this issue: internal/releasetargets: set LongTestBuilder for linux-arm64

gopherbot pushed a commit to golang/build that referenced this issue Jun 22, 2023
The linux/arm64 port is first class and a longtest builder was added
for it during the Go 1.20 development cycle. It's a mandatory trybot
on 1.20+ release branches and is working well.

The builder recently found a real problem in go.dev/issue/60825, which
has been resolved by now. Upgrade this builder from an advisory TryBot
to have more complete release test coverage.

For golang/go#40561.
For golang/go#49649.
For golang/go#29252.

Change-Id: I1765e820deb0d34b7bb9a72aba397d87c1369ced
Reviewed-on: https://go-review.googlesource.com/c/build/+/504525
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder
Projects
Archived in project
Development

No branches or pull requests

8 participants