Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/dist: request to increase user higher test timeout scale for ARM #43002

Open
rbalint opened this issue Dec 4, 2020 · 5 comments
Open

cmd/dist: request to increase user higher test timeout scale for ARM #43002

rbalint opened this issue Dec 4, 2020 · 5 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@rbalint
Copy link

rbalint commented Dec 4, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.7 linux/arm

Does this issue reproduce with the latest release?

Most likely (see #42798), but we carry carry the fix in Ubuntu so I'm not absolutely sure.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
go version go1.14.7 linux/arm
root@tender-starling:~# go env
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.14"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.14/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build283516640=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran a test on armhf:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/armhf/g/golang-gopkg-square-go-jose.v2/20200706_025803_f6666@/log.gz

What did you expect to see?

Test passing, like on all other architectures.

What did you see instead?

Internal test timeout:

=== RUN TestOpaqueKeyRoundtripJWE
panic: test timed out after 10m0s

Also reported as https://bugs.launchpad.net/ubuntu/+source/golang-gopkg-square-go-jose.v2/+bug/1893640 and fixed in Ubuntu by bumping arm's timeout scale to 3 from 2.

I had hard time resulting only a failure to get cross-company CLA signed, thus I'm not filing a PR, but the change is trivial. Please consider applying it.

The reasoning is that most developers use (faster) amd64 machines setting the test timeouts, but test may run on slower arm ones and also different tests may suffer different relative slowdown on the arm CPUs.

Due to those two varying factors it is safer to allow a higher timeout scale on arm to avoid tests timing out.

@cherrymui
Copy link
Member

You can set the environment variable GO_TEST_TIMEOUT_SCALE to increase timeout for all tests.

@egonelbre
Copy link
Contributor

Unlikely that it's related to 42798, since the code doesn't seem to be using float slices.

@egonelbre
Copy link
Contributor

It seems that golang.org/x/crypto/pbkdf2.Key is the hotspot in that test (with arm).

@rbalint
Copy link
Author

rbalint commented Dec 4, 2020

Thank you for the quick replies. The problem was not unique, we observed timeouts in multiple packages. Also the armhf test runners are slower machines than the ones running test for other architectures and I expect this to be typical. Since the timeout scale default is already set by architecture I think relaxing it for arm makes sense. Or there could be different default scales for 64 bit and 32 bit arm architectures.

@ALTree
Copy link
Member

ALTree commented Dec 4, 2020

For reference: #10314 (cmd/dist: default timeout value is too short for weaker CPUs), and #16106. Solved by introducing the GO_TEST_TIMEOUT_SCALE variable @cherrymui mentioned, in 7579867.

Instead of increasing timeoutScale or skipping tests, this CL
introduces an environment variable $GO_TEST_TIMEOUT_SCALE that
could be set to manually set a larger timeoutScale for those
machines/builders.

@cagedmantis cagedmantis changed the title Please user higher test timeout scale for ARM cmd/dist: request to increase user higher test timeout scale for ARM Dec 4, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 4, 2020
@cagedmantis cagedmantis added this to the Backlog milestone Dec 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants