x/build: "unexpected stale targets" on plan9-arm #49691

bcmills · 2021-11-19T19:10:21Z

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

greplogs --dashboard -md -l -e '(?ms)\Aplan9-.*go tool dist: unexpected stale targets' --since=2021-10-09

2021-11-05T18:20:07-53bab19/plan9-arm
2021-11-02T20:59:34-79024cf/plan9-arm
2021-11-02T18:35:22-0b37bd9/plan9-arm
2021-11-02T05:35:11-6f327f7/plan9-arm
2021-10-29T22:29:31-c812b97/plan9-arm
2021-10-28T20:43:02-e741e2f/plan9-arm
2021-10-26T05:05:24-23fdd7f/plan9-arm
2021-10-25T18:46:45-f47335e/plan9-arm
2021-10-25T17:02:16-52b10ab/plan9-arm
2021-10-20T19:30:13-3ff39c5/plan9-arm
2021-10-20T19:29:12-4320949/plan9-arm
2021-10-19T18:43:12-d944984/plan9-arm
2021-10-19T18:18:37-9820602/plan9-arm
2021-10-19T08:44:22-1e49210/plan9-arm
2021-10-14T01:51:22-276fb27/plan9-arm
2021-10-13T16:39:23-3986e5c/plan9-arm
2021-10-12T21:15:50-3d051ba/plan9-arm
2021-10-10T15:11:26-d2d21d9/plan9-arm
2021-10-09T23:29:26-b99abf3/plan9-arm

@millerresearch, is this possibly due to a filesystem issue on the builder? I'm not sure how else we would be ending up with mismatched build IDs.

The text was updated successfully, but these errors were encountered:

millerresearch · 2021-11-19T20:15:11Z

The error message is not that helpful for the uninitiated. What exactly does "build ID mismatch" mean? Is the staleness algorithm documented somewhere?

It can sometimes happen that make.rc is run on one builder, the result is saved as a snapshot and used to run 'dist test' on another builder. If the two builders are not configured exactly the same, is that likely to result in stale targets? I would hope not, because the snapshot contains all the source, object code, and toolchain, so it should be self-consistent in any environment. But it's hard to be certain without a definition of what "stale" is supposed to mean.

In some of the logs, there is clearly some other filesystem-related failure first (some bits of the tree have disappeared), which is likely what triggers the stale target report. But in other examples, the stale target report seems to be the first or only thing that has gone wrong.

bcmills · 2021-11-19T22:09:45Z

The build ID is a hash of the inputs to the build:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/exec.go;l=216-401;drc=4f73fd05a91a9b8ceced6b7f89d35f363c414ec8

bcmills · 2021-11-19T22:11:46Z

The hash inputs include any relevant flags, tool versions, and whatever directory information will end up in the binary (which is itself a function of GOROOT_FINAL, GOROOT, and the working directory for the build.)

bcmills · 2021-11-19T22:21:01Z

In the latest failure I see this line:

HASH[link cmd/dist]: "GOROOT=/tmp/workdir-rpi31/go\n"

Do all of the builders end up with the same working directory (/tmp/workdir-rpi31)? If not, then that could be the cause.

In #33598 we found that the darwin builders were sensitive to the PWD environment variable: their GOROOT was in a symlinked directory, so without a correct PWD they ended up reporting the underlying directory rather than the original path. Is there a similar consideration for Plan 9?

millerresearch · 2021-11-21T20:31:27Z

There are two sets of plan9-arm Raspberry Pi builders, run by @0intro and myself, running slightly different scripts. Mine all use /boot/workdir/go as GOROOT, but David's use a different directory on each host because they're sharing the same server filesystem. @0intro, may I suggest that you add a bind in your script to use the same GOROOT as mine, and that should prevent stale target failures when the build and tests are run on different hosts.

That won't fix all cases, though. For example in this one, the tests are running immediately after building the whole system on the same machine, and getting stale dependencies on vendor/golang.org/x/text/unicode/norm. How is that possible? Is there something tricky about how the vendor subtree is built?

0intro · 2021-11-21T20:42:35Z

@0intro, may I suggest that you add a bind in your script to use the same GOROOT as mine, and that should prevent stale target failures when the build and tests are run on different hosts.

Yes, I'll try that.

bcmills · 2022-12-07T14:45:29Z

No occurrences since June. Probably fixed?

greplogs --dashboard -md -l -e '(?ms)\Aplan9-.*go tool dist: unexpected stale targets' --since=2022-01-01
2022-06-22T16:18:44-92c9b81/plan9-386-0intro
2022-06-22T02:40:04-be0b2a3/plan9-386-0intro
2022-06-21T23:17:35-f2c7e78/plan9-386-0intro
2022-06-21T22:05:17-ab422f2/plan9-386-0intro
2022-06-21T19:48:04-66685fb/plan9-386-0intro
2022-05-26T14:42:13-581a63c/plan9-arm
2022-05-23T21:58:55-3fa5ab0/plan9-386-0intro
2022-03-07T16:24:54-cc9d3f5/plan9-arm

gopherbot · 2023-03-08T18:18:19Z

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-02-27 12:43 plan9-arm go@85b55c79 (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [std] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE encoding/csv: build ID mismatch
	STALE encoding/json: build ID mismatch
	STALE expvar: stale dependency: log
	STALE flag: build ID mismatch
	STALE go/ast: stale dependency: go/token
	STALE go/build: stale dependency: go/token
	STALE go/constant: stale dependency: go/token
	STALE go/doc: stale dependency: go/token
	STALE go/doc/comment: build ID mismatch
...
	STALE time/tzdata: build ID mismatch
	STALE vendor/golang.org/x/net/http/httpguts: stale dependency: log
	STALE vendor/golang.org/x/net/http/httpproxy: stale dependency: log
	STALE vendor/golang.org/x/net/http2/hpack: build ID mismatch
	STALE vendor/golang.org/x/net/idna: stale dependency: log
	STALE vendor/golang.org/x/net/nettest: stale dependency: internal/syscall/execenv
	STALE vendor/golang.org/x/text/secure/bidirule: stale dependency: log
	STALE vendor/golang.org/x/text/transform: build ID mismatch
	STALE vendor/golang.org/x/text/unicode/bidi: stale dependency: log
	STALE vendor/golang.org/x/text/unicode/norm: stale dependency: vendor/golang.org/x/text/transform

— watchflakes

gopherbot · 2023-05-23T13:28:21Z

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-05-23 11:36 plan9-arm go@5322e66a (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/compile: stale dependency: go/build

— watchflakes

gopherbot · 2023-08-23T02:40:39Z

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-08-22 22:22 plan9-arm go@291a32aa (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/compile/internal/gc: stale dependency: cmd/compile/internal/importer
	STALE cmd/compile/internal/importer: build ID mismatch
	STALE cmd/compile/internal/noder: stale dependency: cmd/compile/internal/importer
	STALE cmd/compile/internal/pkginit: stale dependency: cmd/compile/internal/importer

— watchflakes

gopherbot · 2023-09-15T22:30:30Z

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "plan9-arm" && `unexpected stale targets`

2023-09-15 17:51 plan9-arm go@6192f461 (log)

go tool dist: unexpected stale targets reported by /boot/workdir/go/pkg/tool/plan9_arm/go_bootstrap list -gcflags="" -ldflags="" for [cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE cmd/go: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/bug: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/clean: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/envcmd: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/fix: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/generate: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/list: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/modcmd: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/modget: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/run: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/test: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/toolchain: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/vet: stale dependency: cmd/go/internal/work
	STALE cmd/go/internal/work: build ID mismatch
	STALE cmd/go/internal/workcmd: stale dependency: cmd/go/internal/work

— watchflakes

bcmills added arch-arm Issues solely affecting the 32-bit arm architecture. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Plan9 labels Nov 19, 2021

bcmills added this to the Backlog milestone Nov 19, 2021

bcmills mentioned this issue Nov 19, 2021

all: plan9-arm builder failing frequently with a variety of errors #49338

Open

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 19, 2021

bcmills closed this as completed Dec 7, 2022

gopherbot reopened this Mar 8, 2023

bcmills mentioned this issue Mar 16, 2023

proposal: all: add plan9/arm64 port #57540

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: "unexpected stale targets" on plan9-arm #49691

x/build: "unexpected stale targets" on plan9-arm #49691

bcmills commented Nov 19, 2021 •

edited

millerresearch commented Nov 19, 2021 •

edited

bcmills commented Nov 19, 2021 •

edited

bcmills commented Nov 19, 2021

bcmills commented Nov 19, 2021

millerresearch commented Nov 21, 2021

0intro commented Nov 21, 2021

bcmills commented Dec 7, 2022

gopherbot commented Mar 8, 2023

gopherbot commented May 23, 2023

gopherbot commented Aug 23, 2023

gopherbot commented Sep 15, 2023

x/build: "unexpected stale targets" on plan9-arm #49691

x/build: "unexpected stale targets" on plan9-arm #49691

Comments

bcmills commented Nov 19, 2021 • edited

millerresearch commented Nov 19, 2021 • edited

bcmills commented Nov 19, 2021 • edited

bcmills commented Nov 19, 2021

bcmills commented Nov 19, 2021

millerresearch commented Nov 21, 2021

0intro commented Nov 21, 2021

bcmills commented Dec 7, 2022

gopherbot commented Mar 8, 2023

gopherbot commented May 23, 2023

gopherbot commented Aug 23, 2023

gopherbot commented Sep 15, 2023

bcmills commented Nov 19, 2021 •

edited

millerresearch commented Nov 19, 2021 •

edited

bcmills commented Nov 19, 2021 •

edited