x/benchmarks/sweet/cmd/sweet: TestSweetEndToEnd failing with gvisor build errors since CL 386698 #51445

bcmills · 2022-03-02T21:26:23Z

TestSweetEndToEnd is persistently failing on the linux-amd64-longtest and linux-amd64-perf builders since CL 386698 (attn @dr2chase, @prattmic, @aclements).

--- FAIL: TestSweetEndToEnd (181.32s)
    integration_test.go:128: no results produced.
    integration_test.go:150: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5
        [sweet] Benchmarks: gvisor
        [sweet] Setting up benchmark: gvisor
        [shell] git clone -b go https://github.com/google/gvisor /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
        [shell] git -C /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src checkout 6392b0f3bea052af0de9d95677233dd9e442dbd5
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/results-5/gvisor
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/tmp
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/benchmarks/gvisor
        [shell] /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/gvisor-bench
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src/runsc
        [shell] CGO_ENABLED=0 /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
        [sweet] error: build gvisor for go: exit status 1
        [sweet] error: failed to execute benchmarks, see log for details
    integration_test.go:151: exit status 1
FAIL
FAIL	golang.org/x/benchmarks/sweet/cmd/sweet	181.334s

greplogs --dashboard -md -l -e 'error: build gvisor for .*: exit status 1'

The text was updated successfully, but these errors were encountered:

prattmic · 2022-03-03T15:21:02Z

cc @mknyszek

prattmic · 2022-03-03T15:23:42Z

The secondary issue here is that we are apparently not logging the stderr output from the go command, so the actual error is missing.

prattmic · 2022-03-03T15:54:35Z

[sweet] Work directory: /tmp/gosweet2197573647
[sweet] Benchmarks: gvisor
[sweet] Setting up benchmark: gvisor
[shell] git clone -b go https://github.com/google/gvisor /tmp/gosweet2197573647/gvisor/src
[shell] git -C /tmp/gosweet2197573647/gvisor/src checkout 6392b0f3bea052af0de9d95677233dd9e442dbd5
[shell] mkdir -p /usr/local/google/home/mpratt/src/benchmarks/sweet/results/gvisor
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/bin
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/src
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/tmp
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/assets
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet/benchmarks/gvisor
[shell] GOROOT=/usr/local/google/home/mpratt/src/go/ /usr/local/google/home/mpratt/src/go/bin/go build -o /tmp/gosweet2197573647/gvisor/goroot/bin/gvisor-bench
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet
[shell] cd /tmp/gosweet2197573647/gvisor/src/runsc
[shell] CGO_ENABLED=0 GOROOT=/usr/local/google/home/mpratt/src/go/ /usr/local/google/home/mpratt/src/go/bin/go build -o /tmp/gosweet2197573647/gvisor/goroot/bin/runsc
package gvisor.dev/gvisor/runsc
        imports gvisor.dev/gvisor/runsc/cli
        imports gvisor.dev/gvisor/pkg/coverage
        imports gvisor.dev/gvisor/pkg/hostarch
        imports gvisor.dev/gvisor/pkg/gohacks: build constraints exclude all Go files in /tmp/gosweet2197573647/gvisor/src/pkg/gohacks
package gvisor.dev/gvisor/runsc
        imports gvisor.dev/gvisor/runsc/cli
        imports gvisor.dev/gvisor/runsc/cmd
        imports gvisor.dev/gvisor/pkg/sentry/control
        imports gvisor.dev/gvisor/pkg/sentry/fdimport
        imports gvisor.dev/gvisor/pkg/sentry/fs/host
        imports gvisor.dev/gvisor/pkg/sentry/kernel
        imports gvisor.dev/gvisor/pkg/goid: build constraints exclude all Go files in /tmp/gosweet2197573647/gvisor/src/pkg/goid
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet
[sweet] error: build gvisor for goroot: exit status 1
[sweet] error: failed to execute benchmarks, see log for details

This is failing due to https://go.dev/cl/388376 [1]. gVisor has negative go1.19 build tags due to a limitation in their static analysis tool. We'll need to disable this benchmark until gVisor is updated.

[1] I don't understand why this CL wasn't specifically tested on https://build.golang.org/?repo=golang.org%2fx%2fbenchmarks.

gopherbot · 2022-03-03T16:35:07Z

Change https://go.dev/cl/389754 mentions this issue: sweet: disable gvisor benchmark

gopherbot · 2022-03-03T17:01:41Z

Change https://go.dev/cl/389774 mentions this issue: sweet/common: include stderr in error string

Upstream Go tip has updated its version to 1.19, so we need to bump these build tags to allow tip testing. Ideally we'll get checklinkname (or equivalent) to check these remaining unsafe uses (like internal struct field offsets) soon. For golang/go#51445. PiperOrigin-RevId: 432215030

gVisor currently does not build with Go 1.19, so exclude it from "all", "default", and integration testing until it is fixed. It can still be requested explicitly with -run=gvisor. For golang/go#51445. Change-Id: Ib839899af169ff247aa59f264c7891e16784603c Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/389754 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

We use cmd.Output in order to capture stderr, but ExitError.Error doesn't actually include stderr in the string. Create a wrapper error that does so. For golang/go#51445. Change-Id: I5101389c758897e96faa74ef49e9800db1758586 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/389774 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>

prattmic · 2022-03-03T18:59:46Z

https://go.dev/cl/389754 fixes the builders by disabling the gvisor benchmark. Keeping this open to track reenabling the benchmarks once google/gvisor#7252 goes in.

Upstream Go tip has updated its version to 1.19, so we need to bump these build tags to allow tip testing. This CL bumps the minimum version in pkg/sync/runtime_amd64.s from 1.8 to 1.14. The offset was different prior to Go 1.14. Ideally we'll get checklinkname (or equivalent) to check these remaining unsafe uses (like internal struct field offsets) soon. For golang/go#51445. PiperOrigin-RevId: 433304124

gopherbot · 2022-03-08T22:31:39Z

Change https://go.dev/cl/390954 mentions this issue: sweet/harnesses: update gVisor for 1.19 compatibility

gopherbot · 2022-03-08T22:31:39Z

Change https://go.dev/cl/390955 mentions this issue: Revert "sweet: disable gvisor benchmark"

google/gvisor@adc7bb5 includes https://gvisor.dev/pr/7252, which allows gVisor to build with a Go 1.19 toolchain. For golang/go#51445. Change-Id: I9f352d1bdd2bf57a4dc91bac989d14d3a08db6c0 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/390954 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>

gopherbot · 2022-03-09T17:38:15Z

Change https://go.dev/cl/391194 mentions this issue: sweet: disable gvisor benchmark

bcmills · 2022-03-09T17:38:37Z

Unfortunately still failing at head (https://build.golang.org/log/65cac28815309e7fe290362018f8929ca2204c58):

--- FAIL: TestSweetEndToEnd (222.47s)
    integration_test.go:136: output for gvisor:
    integration_test.go:144: === Benchmark stdout+stderr ===
        
        Hello from Docker!
        This message shows that your installation appears to be working correctly.
        
        To generate this message, Docker took the following steps:
         1. The Docker client contacted the Docker daemon.
         2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
            (amd64)
         3. The Docker daemon created a new container from that image which runs the
            executable that produces the output you are currently reading.
         4. The Docker daemon streamed that output to the Docker client, which sent it
            to your terminal.
        
        To try something more ambitious, you can run an Ubuntu container with:
         $ docker run -it ubuntu bash
        
        Share images, automate workflows, and more with a free Docker ID:
         https://hub.docker.com/
        
        For more examples and ideas, visit:
         https://docs.docker.com/get-started/
        
        
        error: exit status 137
        
    integration_test.go:150: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5
        [sweet] Benchmarks: gvisor
        [sweet] Setting up benchmark: gvisor
        [shell] git clone -b go https://github.com/google/gvisor /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
        [shell] git -C /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src checkout adc7bb5e1baf4a7489e428e1fad756e5e2aa3410
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/results-5/gvisor
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/tmp
        [shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/benchmarks/gvisor
        [shell] /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/gvisor-bench
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src/runsc
        [shell] CGO_ENABLED=0 /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc
        [shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
        [shell] chmod 755 /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc
        [shell] unzip /tmp/go-sweet-assets/assets-v0.3.0.zip 'gvisor/*' -d /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets
        [sweet] Running benchmark gvisor for go: run 1
        [shell] /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/gvisor-bench -runsc /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc -assets-dir /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets -tmp /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/tmp -short
        [sweet] error: run benchmark gvisor for config go: exit status 1
        [sweet] error: failed to execute benchmarks, see log for details
    integration_test.go:151: exit status 1
FAIL
FAIL	golang.org/x/benchmarks/sweet/cmd/sweet	222.488s

mknyszek · 2022-03-09T17:41:02Z

This test only runs on the longtest builders, so generally modifications to Sweet should check longtest as a slowbot.

I have been considering making the test run on the usual trybots.

mknyszek · 2022-03-09T17:52:50Z

To be clear, I don't think anyone did anything wrong here (except maybe me, for marking this test as "long" 😅 ). I do think this is pretty good evidence to suggest this test should probably just run on the regular trybots. We have plenty of timeout headroom, though it will add a few minutes of latency to the trybots.

This reverts CL 390955. Reason for revert: benchmark is still failing on longtest builders. Updates golang/go#51445. Change-Id: I35a057933231e89e5b56810ff39c73f3702fd383 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/391194 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>

bcmills · 2022-03-09T20:36:01Z

This test only runs on the longtest builders, so generally modifications to Sweet should check longtest as a slowbot.

It may be possible to modify x/build/cmd/coordinator to do that; compare #42661, #43719.

aclements · 2022-04-06T16:09:20Z

What's the status on this? The x/benchmarks repo isn't failing at this point. Is the gvisor benchmark still disabled? If so, we should retitle this to make it about re-enabling this benchmark.

mknyszek · 2022-04-13T16:21:46Z

The current status is that after the previous fix (https://go.dev/cl/390954) there were still failures, so the benchmark was disabled again. This needs to be investigated again, I believe.

bcmills · 2022-05-13T14:53:19Z

I had labeled this as a release-blocker because the builders were failing, but now the benchmark is disabled and the builder is passing, so it won't mask any other regressions and from my perspective isn't as urgent.

Is it important to re-enable this benchmark for the Go 1.19 release, or should we remove the release-blocker label?

mknyszek · 2022-05-13T16:25:19Z

I don't think there's any particular reason fixing this should be tied to the release cycle. I think putting it in Unreleased makes sense.

prattmic · 2022-08-30T19:53:09Z

I've tested the latest gVisor. It is currently blocked on google/gvisor#7930, which causes us to see spurious SIGKILL exit status.

gopherbot · 2022-10-21T19:28:52Z

Change https://go.dev/cl/444758 mentions this issue: sweet: update and re-enable gvisor benchmark

prattmic · 2022-10-21T20:38:13Z

Now we are blocked on google/gvisor#8111. :(

gVisor currently does not build with Go 1.19, so exclude it from "all", "default", and integration testing until it is fixed. It can still be requested explicitly with -run=gvisor. For golang/go#51445. Change-Id: Ib839899af169ff247aa59f264c7891e16784603c