Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: linux-arm-arm5spacemonkey persistently broken #29177

Closed
bcmills opened this issue Dec 11, 2018 · 8 comments
Closed

x/build: linux-arm-arm5spacemonkey persistently broken #29177

bcmills opened this issue Dec 11, 2018 · 8 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Dec 11, 2018

The linux-arm-arm5spacemonkey builder has been broken for as far back as I dared to page through the logs. It seems to be the only all-red (#22689) builder in the “first-class ports” view (#27689).

As far as I can tell, it's a configuration issue: for some reason the builder sets the Go version to builder host-linux-armel-cross instead of a more realistic version string, and that somehow causes the API check to fail.

See previously #28041. (CC @bradfitz, @onionjake, @zeebo, @dmitshur )

Sample: https://build.golang.org/log/32dafc31f49b66f75902acd5f0d61cb8f6277de6

:: Running /bin/bash with args ["/bin/bash" "-c" "cd $WORKDIR/go/src && ./make.bash && cd .. && mv bin/*_*/* bin && rmdir bin/*_* && rm -rf pkg/linux_amd64 pkg/tool/linux_amd64 pkg/bootstrap pkg/obj"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-armel-cross-rn8aed9e5" "IN_KUBERNETES=1" "META_BUILDLET_BINARY_URL=http://storage.googleapis.com/go-builder-data/buildlet.linux-amd64" "META_BUILDLET_HOST_TYPE=host-linux-armel-cross" "KUBERNETES_PORT_443_TCP_PORT=443" "KUBERNETES_PORT_443_TCP_ADDR=10.3.240.1" "KUBERNETES_SERVICE_HOST=10.3.240.1" "KUBERNETES_SERVICE_PORT=443" "KUBERNETES_SERVICE_PORT_HTTPS=443" "KUBERNETES_PORT=tcp://10.3.240.1:443" "KUBERNETES_PORT_443_TCP=tcp://10.3.240.1:443" "KUBERNETES_PORT_443_TCP_PROTO=tcp" "DEBIAN_FRONTEND=noninteractive" "GOROOT_BOOTSTRAP=/go1.4" "GOOS=linux" "GOARCH=arm" "CC_FOR_TARGET=arm-linux-gnueabi-gcc" "HOME=/root" "WORKDIR=/tmp/workdir" "CGO_ENABLED=1" "GOARM=5"] in dir /tmp/workdir

Building Go cmd/dist using /go1.4.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go toolchain1 using /go1.4.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, linux/amd64.
Building packages and commands for target, linux/arm.
[…]
##### API check
Error running API checker: exit status 1
Go version is "builder host-linux-armel-cross", ignoring -next /home/builder/stage0scratch/workdir/go/api/next.txt
+pkg bytes, func ReplaceAll([]uint8, []uint8, []uint8) []uint8
[…]
+pkg syscall, type RawSockaddrUnix struct
exit status 1
2018/11/26 19:20:01 Failed: exit status 1
2018/11/26 19:20:07 FAILED
@gopherbot gopherbot added this to the Unreleased milestone Dec 11, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Dec 11, 2018
@cherrymui
Copy link
Member

For the record, if I remembered correctly, that builder is special in that it does cross compilation. The toolchain and the standard library is built on AMD64 machine(s) then shipped to the ARMv5 machine(s) to run the tests. When @rsc implemented the build cache, it would never work in this setting (failure was probably toolchain2 and toolchain3 were different). (And it is not possible to run everything on the ARMv5 machine because it is too slow.) So we decided to set up a toolchain version, to bypass certain checks.

@mvdan
Copy link
Member

mvdan commented Dec 12, 2018

Seems like a duplicate of #24754, initially titled cmd/api: failing on linux-arm builder due to non-devel Go version. Now it has a title for a proposed fix, as far as I understand.

@onionjake
Copy link

I'm happy to help if there is something we can do on the builders to make this easier.

@ALTree ALTree mentioned this issue Jan 2, 2019
Closed
@onionjake
Copy link

Is this resolved now? Or is this timeout issue masking it?

This is from one of the latest test runs (https://build.golang.org/log/5137b5ef408a5c5a56d13ae902fe833f501d1e12):

Test "cgo_test" ran over 20m0s limit (20m0.040245806s)

@gopherbot
Copy link

Change https://golang.org/cl/162540 mentions this issue: cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE

gopherbot pushed a commit to golang/build that referenced this issue Feb 14, 2019
@bradfitz
cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEO…
34b995b 
…UT_SCALE

So slow builders can finish their cgo tests, etc.

Updates golang/go#29177 etc

Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352
Reviewed-on: https://go-review.googlesource.com/c/162540
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bcmills
Copy link
Contributor Author

bcmills commented Feb 28, 2019

This builder is still persistently broken. Now it appears to be ignoring the next file in the API check.

From https://build.golang.org/log/4728fbb67eb490e9f3df8c5553e16c14fc7cfaf2:

##### API check
Error running API checker: exit status 1
Go version is "builder host-linux-armel-cross", ignoring -next /home/builder/stage0scratch/workdir/go/api/next.txt
+pkg errors, func As(error, interface{}) bool
+pkg errors, func Caller(int) Frame
+pkg errors, func Is(error, error) bool
+pkg errors, func Opaque(error) error
+pkg errors, func Unwrap(error) error
+pkg errors, method (Frame) Format(Printer)
+pkg errors, type Formatter interface { Error, FormatError }
+pkg errors, type Formatter interface, Error() string
+pkg errors, type Formatter interface, FormatError(Printer) error
+pkg errors, type Frame struct
+pkg errors, type Printer interface { Detail, Print, Printf }
+pkg errors, type Printer interface, Detail() bool
+pkg errors, type Printer interface, Print(...interface{})
+pkg errors, type Printer interface, Printf(string, ...interface{})
+pkg errors, type Wrapper interface { Unwrap }
+pkg errors, type Wrapper interface, Unwrap() error
+pkg text/scanner, const AllowNumberbars = 1024
+pkg text/scanner, const AllowNumberbars ideal-int
+pkg text/scanner, const GoTokens = 2036
exit status 1

@mvdan
Copy link
Member

mvdan commented Feb 28, 2019

@bcmills #24754 talks precisely about the API check and how the version name in that builder makes the test fail. So I still think this is a duplicate.

@bcmills
Copy link
Contributor Author

bcmills commented Feb 28, 2019

Agreed. Also #24789.

@bcmills bcmills closed this as completed Feb 28, 2019
@bcmills bcmills mentioned this issue Jun 3, 2019
Closed
@golang golang locked and limited conversation to collaborators Feb 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@onionjake @mvdan @bcmills @gopherbot @cherrymui