Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/dist: "unexpected stale targets" on linux-riscv64-unmatched since 2022-06-06 #53379

Closed
bcmills opened this issue Jun 14, 2022 · 10 comments
Closed

cmd/dist: "unexpected stale targets" on linux-riscv64-unmatched since 2022-06-06 #53379

bcmills opened this issue Jun 14, 2022 · 10 comments
Assignees
@mengzhuo
Labels
arch-riscv Issues solely affecting the riscv64 architecture. Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Backlog

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 14, 2022

greplogs -l -e '(?ms)\Alinux-riscv64-unmatched.*unexpected stale targets'
2022-06-10T23:30:35-55590f3/linux-riscv64-unmatched
2022-06-10T22:10:29-ff3db8d/linux-riscv64-unmatched
2022-06-09T18:17:39-91019cc/linux-riscv64-unmatched
2022-06-08T21:56:02-13f6be2/linux-riscv64-unmatched
2022-06-07T18:47:38-f3e051a/linux-riscv64-unmatched
2022-06-07T16:07:58-d2630aa/linux-riscv64-unmatched
2022-06-06T19:03:15-95547ae/linux-riscv64-unmatched

(attn @mengzhuo)
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-riscv Issues solely affecting the riscv64 architecture. labels Jun 14, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Jun 14, 2022

Curiously, all of the above failures are also missing the environment and bootstrapping headers (#51050), but maybe that's just a byproduct of the interaction between the staleness check and the way cmd/coordinator shards the tests?

@bcmills
Copy link
Contributor Author

bcmills commented Jun 14, 2022

With this being a reverse builder, at first I thought it might be related to the recent cmd/go module indexing changes.

However, these failures started well before CL 410821 was merged, so I don't immediately see how that's possible.

@bcmills
Copy link
Contributor Author

bcmills commented Jun 14, 2022

The other likely causes of unexpected stale targets that I can think of are:

@gopherbot
Copy link

Change https://go.dev/cl/412175 mentions this issue: dashboard: add a known issue for linux-riscv64-unmatched

@bcmills bcmills mentioned this issue Jun 14, 2022
Closed
gopherbot pushed a commit to golang/build that referenced this issue Jun 14, 2022
@gopherbot
dashboard: add a known issue for linux-riscv64-unmatched
59e7a6b 
For golang/go#53379.

Change-Id: I758dfac4bf794de151d9c5105954e68224d955b7
Reviewed-on: https://go-review.googlesource.com/c/build/+/412175
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@mengzhuo
Copy link
Contributor

mengzhuo commented Jun 15, 2022 •
edited

Thanks for reporting this ( I forget to submit this issue ).
After CL 411475 (different builders failed twice in dashboard).
I had cleared all temp files and rebuilt stage0 on unmatched01 and unmatched057.
All builders using /tmp/ (ramdisk) as it's workdir so I think filesystem corruption shouldn't be the problem.

@mengzhuo mengzhuo self-assigned this Jun 16, 2022
@gopherbot
Copy link

Change https://go.dev/cl/413334 mentions this issue: cmd/buildlet/stage0: define env for linux-riscv64-unmatched

@mengzhuo
Copy link
Contributor

mengzhuo commented Jun 22, 2022 •
edited

The root cause might be my env setting to identify three builders in the pool.
All failed builders are running the bot by setup different WORKDIR and rm -rf $WORKDIR after bot complete their job.

gopherbot pushed a commit to golang/build that referenced this issue Jun 23, 2022
@mengzhuo @gopherbot
cmd/buildlet/stage0: define env for linux-riscv64-unmatched
6da0eb5 
This CL sets up the environment for `host-linux-riscv64-unmatched`

For golang/go#53379

Change-Id: I6b24dabb9407cb98e095c555001adc5a022d1d66
Reviewed-on: https://go-review.googlesource.com/c/build/+/413334
Auto-Submit: Meng Zhuo <mzh@golangcn.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
@bcmills
Copy link
Contributor Author

bcmills commented Jun 28, 2022

@mengzhuo, is this fixed? (If so, could you send a CL to remove the KnownIssue in x/build?)

greplogs -l -e '(?ms)\Alinux-riscv64-unmatched.*unexpected stale targets' --since=2022-06-11

(0 matching logs)

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 28, 2022
@bcmills bcmills added this to the Backlog milestone Jun 28, 2022
@gopherbot
Copy link

Change https://go.dev/cl/414934 mentions this issue: dashboard: drop known issue for linux-riscv64-unmatched

@mengzhuo
Copy link
Contributor

@mengzhuo, is this fixed? (If so, could you send a CL to remove the KnownIssue in x/build?)

greplogs -l -e '(?ms)\Alinux-riscv64-unmatched.*unexpected stale targets' --since=2022-06-11

(0 matching logs)

Done, I've submitted cl 414934.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. Builders x/build issues (builders, bots, dashboards) and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 29, 2022
@golang golang locked and limited conversation to collaborators Jun 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mengzhuo mengzhuo

Labels
arch-riscv Issues solely affecting the riscv64 architecture. Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@mengzhuo @dmitshur @bcmills @gopherbot