x/build: linux-arm trybot/slowbot failure #35628
Comments
|
I suspect this was "broken" by https://go-review.googlesource.com/c/build/+/205603 which upgraded the ARM environment (where tests run) to Buster, but the compilation half on x86 was unchanged. I say "broken" in quotes because this was never a problem until slowbots which let you do this again. But it used to work with trybots on by default a few years ago, so it does work if we configure both halves the same. Will fix. |
|
Thanks @bradfitz |
|
Hmm, this isn't as obvious as I'd hoped. Both environments are Debian buster: Cross-compilation: Linux-arm on Scaleway: @rsc, you probably remember me debugging similar issues in the past. I'd love some better tooling to help debug these sorts of issues. It doesn't affect many people, though. But if we could do this sort of cross-compilation (make.bash in one place, run tests elsewhere) for more builders (riscv, mips, arm, etc) we could get much better throughput from our slower builders. But I'm always scared of doing it due to issues like this. |
|
Ping. Still seeing this and it means we can't run the trybots on arm targets: https://storage.googleapis.com/go-build-log/986a3d00/linux-arm_a2dcef07.log |
|
Unassigning @bradfitz since I don't think he's working on |
|
In the mean time, I have been using gomote for ARM. Using android-arm is another option, which is not the same as linux-arm but also has many similarities. |
cagedmantis
added
the
NeedsInvestigation
|
This cost me a chunk of time this morning. Can we disable linux/arm slowbots until this is fixed? Or have gopherbot emit a warning and reference to this issue when it recognizes linux/arm? |
|
If this cannot be fixed soon, can we just disable/reject linux-arm for trybot? And maybe use android-arm for "TRY=arm"? The failure has confused people several times (e.g. https://go-review.googlesource.com/c/go/+/248684). |
|
I've closed a newer #40872, which seems to be the same issue.
Based on #40872 (comment), this is closer to being resolved now. It seems reasonable to reject/ignore it during TRY= slowbot requests until then. |
|
Change https://golang.org/cl/249420 mentions this issue: |
dmitshur
changed the title
The current linux-arm builder is known to have trouble when used as a SlowBot. Start warning about it when the builder is requested via the TRY= SlowBot UI. I've considered also removing or disabling the "arm" SlowBot alias, but that would make it easier to miss that there's an issue, since SlowBots don't warn about unknown builders: If you specify an unknown TRY= token, it'll just ignore it and won't report an error. We can consider making further changes as this situation evolves. The goal here is to start notifying about a known problem sooner. For golang/go#35628. For golang/go#40872. Change-Id: Ibc1205720c44ec4823c632c04fc2f887368258c1 Reviewed-on: https://go-review.googlesource.com/c/build/+/249420 Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
|
Change https://golang.org/cl/270517 mentions this issue: |
The linux-arm-aws builder was initially labeled with a known issue because it was experimental. The builder has been tested and is no longer considered experimental. Fixes golang/go#41867 Updates golang/go#40872 Updates golang/go#35628 Change-Id: I61f43f2c2651c26d3f5d4db01b779686ddb6a92b Reviewed-on: https://go-review.googlesource.com/c/build/+/270517 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
|
With the addition of the linux-arm-aws builder, I believe this issue can be closed. |
|
I think there's a little more to do here. The "linux-arm" builder currently has KnownIssue set to 35628 (see builders.go#L1830), which is this issue. To close this issue, we should either modify that builder to point to another open issue, resolve its SlowBot failure problem, or remove the builder. |
|
I think it is better for |
|
Change https://golang.org/cl/303230 mentions this issue: |
What version of Go are you using (
go version)?tip (d856e05)
What did you do?
Run linux-arm trybot on tip (with a dummy change, CL https://go-review.googlesource.com/c/go/+/207440)
It failed with https://storage.googleapis.com/go-build-log/09ec0769/linux-arm_4add410f.log
I also got a few failures on a different CL (https://go-review.googlesource.com/c/go/+/207350)
https://storage.googleapis.com/go-build-log/3c197532/linux-arm_4a2eca13.log
https://storage.googleapis.com/go-build-log/3c197532/linux-arm_e2868c28.log
https://storage.googleapis.com/go-build-log/3472bd16/linux-arm_0d7459e0.log
I cannot reproduce it with gomote. Also the linux-arm builder is happy.
I think it may be related to that the trybot does a cross-compilation and then ships the binaries to the ARM machine.
cc @bradfitz
The text was updated successfully, but these errors were encountered: