x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050

bcmills · 2022-02-07T15:50:52Z

#!watchflakes
post <- `\A[a-z0-9-]* at .*\n\n\n#####`

Builder logs normally include a :: Running /workdir/go/src/make.bash line at the start of the log that explicitly includes the builder environment:

linux-amd64-longtest at 248ad855b7d0e49839b7b4281d9e60e222368583

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-stretch-morecpu-rn0060fc6" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=1s" "GO_STAGE0_DL_DELAY=200ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64-longtest" "GO_TEST_SHORT=0" "GOROOT_BOOTSTRAP=/go1.4" "GO_TEST_TIMEOUT_SCALE=5" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PWD=/workdir/go/src"] in dir /workdir/go/src

From what I can tell, that is triggered by some step that sets ExecOpts.Debug. There are currently three call sites that do so in x/build/internal/buildgo/buildgo.go. Setting the Debug field causes the buildlet to emit the log line:
https://cs.opensource.google/go/x/build/+/master:cmd/buildlet/buildlet.go;l=892-895;drc=05d22632dd9ae9f87a48a12960617e223cecdf9c

However, some of the logs don't seem to include that information at all. It's not clear to me why that happens; I suspect it may be a side effect of sharding (or the lack thereof). That complicates debugging for failures: without the environment, it isn't always clear how to reproduce a failure.

CC @golang/release @prattmic @aclements

greplogs --dashboard -md -l -e '\A[a-z0-9-]* at .*\n\n\n#####' --since=2021-01-01

The text was updated successfully, but these errors were encountered:

dmitshur · 2022-02-07T15:58:47Z

I suspect it may be a side effect of sharding

I also suspect this is related. For example, see #51005, where the final build log doesn't contain the environment or the commands that were running. A copy of temporary build output while the build was in progress includes a trace of events, and seeing those makes it apparent the error from "go tool dist" came from an attempt to execute a subset of packages, i.e., test sharding.

So at least one case when this seems to happen is when an invocation to run a subset of tests via dist fails before the test execution begins.

bcmills · 2022-03-30T19:37:46Z

greplogs --dashboard -md -l -e '\A[a-z0-9-]* at .*\n\n\n#####' --since=2022-02-08

prattmic · 2022-03-30T19:59:52Z

Is this just a case of snapshotting allowing the builder to skip the make.bash step?

The logs look like :: Running /workdir/go/src/make.bash ..., which presumably comes from RunMake here. If st.useSnapshot() is true, then we won't run make.bash, and thus won't print the log.

The main execution of the test comes from here (or the same call in the other goroutine below), which executes go/bin/go without Debug: true (which makes sense since we never see logs about running go/bin/go (that I am aware of).

Two things I think we could do easily:

Log a message that the toolchain is copied from a snapshot (I imagine recording the exact environment that the snapshotted toolchain was built with is more complex).
Add Debug: true to st.runTestsOnBuildlet so that the go command execution also gets logging.

bcmills · 2022-04-29T21:00:38Z

Here's one where the missing environment header would be useful: https://build.golang.org/log/d8c2c0d1c990fa6474ce32c4dd66559668918c8d

The Connection timed out failure mode suggests maybe a slow HTTPS proxy, but I can't tell from the log whether this builder has a proxy configured in the environment.

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 7, 2022

gopherbot added this to the Unreleased milestone Feb 7, 2022

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 7, 2022

dmitshur changed the title ~~x/build: failures missing environment and bootstrapping headers~~ x/build/cmd/coordinator: failures missing environment and bootstrapping headers May 31, 2022

This was referenced Jun 14, 2022

runtime,cmd/compile: exit status 0xc0000374 (STATUS_HEAP_CORRUPTION) on windows-amd64-longtest #52647

Closed

cmd/dist: "unexpected stale targets" on linux-riscv64-unmatched since 2022-06-06 #53379

Closed

bcmills mentioned this issue Nov 29, 2023

cmd/cgo/internal/test: consistently failing with segmentation fault on linux-arm-aws since CL 545277 #64447

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050

x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050

bcmills commented Feb 7, 2022 •

edited

dmitshur commented Feb 7, 2022

bcmills commented Mar 30, 2022

prattmic commented Mar 30, 2022

bcmills commented Apr 29, 2022 •

edited

x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050

x/build/cmd/coordinator: failures missing environment and bootstrapping headers #51050

Comments

bcmills commented Feb 7, 2022 • edited

dmitshur commented Feb 7, 2022

bcmills commented Mar 30, 2022

prattmic commented Mar 30, 2022

bcmills commented Apr 29, 2022 • edited

bcmills commented Feb 7, 2022 •

edited

bcmills commented Apr 29, 2022 •

edited