x/build/cmd/release: linux-armv6l release tests aren't passing #43202

dmitshur · 2020-12-15T22:16:45Z

As part of the plan in #40561 (comment), the linux-armv6l binary release has changed from using the "linux-arm" builder (on Scaleway) to the "linux-arm-aws" builder in CL 276034. Previously, tests were skipped for that target. Since the "linux-arm-aws" builder had more resource allocated, we tried to remove the test skip:

 {
+	GoQuery: ">= go1.16beta1",
 	OS:      "linux",
 	Arch:    "arm",
-	Builder: "linux-arm",
-	Goarm:   6, // for compatibility with all Raspberry Pi models.
-	// The tests take too long for the release packaging.
-	// Much of the time the whole buildlet times out.
-	SkipTests: true,
+	Builder: "linux-arm-aws",
+	Goarm:   6, // For compatibility with all Raspberry Pi models.
 },

The tests are not passing when using a recent tip commit with CL 276454 applied on top.

This can be reproduced if your account has permissions needed to run releasebot (documented here) with:

# (The release command uses builders to create a release artifact locally.
#  It does not publish anything, so it's safe to run for testing needs.)
$ release -target=linux-armv6l -version=go1.16beta123 -watch \
          -rev=456cd66e5d73d6971ab40ce6c57b1c9cf9444e1f  # or -rev=master for latest commit

[...]
ok  	cmd/internal/src	0.012s
ok  	cmd/internal/sys	0.029s
ok  	cmd/internal/test2json	0.103s
--- FAIL: TestPIESize (0.00s)
    --- FAIL: TestPIESize/TestPieSize-external (10.99s)
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/pieexternal -buildmode=pie -ldflags=-linkmode=external pie.go]
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/exeexternal -buildmode=exe -ldflags=-linkmode=external pie.go]
        elf_test.go:411: real size difference 0xfffffffffffffd87, expected 0xfffffffffffffb58
        elf_test.go:414: PIE unexpectedly large: got difference of 18446744073709550983 (2693752 - 2692792), expected difference 18446744073709550424
FAIL
FAIL	cmd/link	18.954s
ok  	cmd/link/internal/benchmark	0.043s
--- FAIL: TestRuntimeTypeAttrExternal (1.54s)
    dwarf_test.go:1030: DWARF type offset was 0x3b50+0x744c8, but test program said 0xa47018
FAIL
FAIL	cmd/link/internal/ld	6.289s
ok  	cmd/link/internal/loader	0.041s
--- FAIL: TestExternalLinkerCgoExec (3.04s)
    nm_test.go:191: want 0x638140 address for main.main symbol, but have 0xb4140
    nm_test.go:191: want 0x6efa8c address for main.testdata symbol, but have 0x16ba8c
    nm_test.go:191: want 0x638164 address for main.testfunc symbol, but have 0xb4164
FAIL
FAIL	cmd/nm	4.917s
ok  	cmd/objdump	6.293s
ok  	cmd/pack	2.633s
ok  	cmd/trace	0.248s
ok  	cmd/vet	11.428s
FAIL
go tool dist: Failed: exit status 1

Full Log

Building Go cmd/dist using /usr/local/go-bootstrap. (go1.15.1 linux/arm)
Building Go toolchain1 using /usr/local/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/arm.

##### Testing packages.
ok  	archive/tar	0.055s
ok  	archive/zip	0.462s
ok  	bufio	0.164s
ok  	bytes	5.752s
ok  	compress/bzip2	0.091s
ok  	compress/flate	1.125s
ok  	compress/gzip	0.119s
ok  	compress/lzw	0.069s
ok  	compress/zlib	0.739s
ok  	container/heap	0.025s
ok  	container/list	0.012s
ok  	container/ring	0.053s
ok  	context	0.041s
ok  	crypto	0.003s
ok  	crypto/aes	0.025s
ok  	crypto/cipher	0.021s
ok  	crypto/des	0.020s
ok  	crypto/dsa	0.021s
ok  	crypto/ecdsa	0.250s
ok  	crypto/ed25519	0.168s
ok  	crypto/elliptic	0.062s
ok  	crypto/hmac	0.037s
ok  	crypto/internal/subtle	0.003s
ok  	crypto/md5	0.036s
ok  	crypto/rand	0.017s
ok  	crypto/rc4	0.040s
ok  	crypto/rsa	0.202s
ok  	crypto/sha1	0.009s
ok  	crypto/sha256	0.024s
ok  	crypto/sha512	0.026s
ok  	crypto/subtle	0.021s
ok  	crypto/tls	2.004s
ok  	crypto/x509	2.473s
ok  	database/sql	0.659s
ok  	database/sql/driver	0.022s
ok  	debug/dwarf	0.049s
ok  	debug/elf	0.059s
ok  	debug/gosym	0.044s
ok  	debug/macho	0.005s
ok  	debug/pe	0.035s
ok  	debug/plan9obj	0.018s
ok  	embed/internal/embedtest	0.011s
ok  	encoding/ascii85	0.019s
ok  	encoding/asn1	0.023s
ok  	encoding/base32	0.047s
ok  	encoding/base64	0.028s
ok  	encoding/binary	0.003s
ok  	encoding/csv	0.031s
ok  	encoding/gob	0.038s
ok  	encoding/hex	0.004s
ok  	encoding/json	0.193s
ok  	encoding/pem	0.015s
ok  	encoding/xml	0.023s
ok  	errors	0.011s
ok  	expvar	0.024s
ok  	flag	0.115s
ok  	fmt	0.073s
ok  	go/ast	0.005s
ok  	go/build	1.414s
ok  	go/constant	0.037s
ok  	go/doc	0.053s
ok  	go/format	0.021s
ok  	go/importer	0.156s
ok  	go/internal/gccgoimporter	0.020s
ok  	go/internal/gcimporter	3.207s
ok  	go/internal/srcimporter	8.038s
ok  	go/parser	0.031s
ok  	go/printer	0.273s
ok  	go/scanner	0.011s
ok  	go/token	0.028s
ok  	go/types	2.441s
ok  	hash	0.021s
ok  	hash/adler32	0.012s
ok  	hash/crc32	0.032s
ok  	hash/crc64	0.027s
ok  	hash/fnv	0.004s
ok  	hash/maphash	0.062s
ok  	html	0.003s
ok  	html/template	0.127s
ok  	image	0.111s
ok  	image/color	0.034s
ok  	image/draw	0.070s
ok  	image/gif	0.487s
ok  	image/jpeg	0.168s
ok  	image/png	0.062s
ok  	index/suffixarray	1.468s
ok  	internal/cpu	0.006s
ok  	internal/fmtsort	0.020s
ok  	internal/poll	0.150s
ok  	internal/profile	0.007s
ok  	internal/reflectlite	0.089s
ok  	internal/singleflight	0.017s
ok  	internal/trace	0.075s
ok  	internal/unsafeheader	0.004s
ok  	internal/xcoff	0.013s
ok  	io	0.052s
ok  	io/fs	0.020s
ok  	io/ioutil	0.014s
ok  	log	0.006s
ok  	log/syslog	1.217s
ok  	math	0.028s
ok  	math/big	2.042s
ok  	math/bits	0.014s
ok  	math/cmplx	0.042s
ok  	math/rand	0.411s
ok  	mime	0.034s
ok  	mime/multipart	0.208s
ok  	mime/quotedprintable	0.033s
ok  	net	14.877s
ok  	net/http	11.030s
ok  	net/http/cgi	0.647s
ok  	net/http/cookiejar	0.029s
ok  	net/http/fcgi	0.039s
ok  	net/http/httptest	0.085s
ok  	net/http/httptrace	0.025s
ok  	net/http/httputil	0.065s
ok  	net/http/internal	0.012s
ok  	net/http/pprof	5.139s
ok  	net/internal/socktest	0.003s
ok  	net/mail	0.039s
ok  	net/rpc	0.020s
ok  	net/rpc/jsonrpc	0.006s
ok  	net/smtp	0.029s
ok  	net/textproto	0.010s
ok  	net/url	0.013s
ok  	os	1.039s
ok  	os/exec	0.988s
ok  	os/signal	2.205s
ok  	os/user	0.004s
ok  	path	0.005s
ok  	path/filepath	0.011s
ok  	plugin	0.020s
ok  	reflect	0.344s
ok  	regexp	0.164s
ok  	regexp/syntax	0.579s
ok  	runtime	55.410s
ok  	runtime/debug	0.058s
ok  	runtime/internal/atomic	0.109s
ok  	runtime/internal/math	0.026s
ok  	runtime/internal/sys	0.006s
ok  	runtime/metrics	0.016s
ok  	runtime/pprof	8.553s
ok  	runtime/race	0.021s
ok  	runtime/trace	0.831s
ok  	sort	0.104s
ok  	strconv	0.425s
ok  	strings	5.003s
ok  	sync	0.483s
ok  	sync/atomic	0.186s
ok  	syscall	0.183s
ok  	testing	1.260s
ok  	testing/fstest	0.010s
ok  	testing/iotest	0.008s
ok  	testing/quick	0.042s
ok  	text/scanner	0.007s
ok  	text/tabwriter	0.017s
ok  	text/template	0.049s
ok  	text/template/parse	0.014s
ok  	time	2.447s
ok  	unicode	0.011s
ok  	unicode/utf16	0.021s
ok  	unicode/utf8	0.021s
ok  	cmd/addr2line	6.385s
ok  	cmd/api	9.669s
ok  	cmd/asm/internal/asm	1.364s
ok  	cmd/asm/internal/lex	0.025s
ok  	cmd/compile	11.035s
ok  	cmd/compile/internal/gc	28.827s
ok  	cmd/compile/internal/logopt	0.164s
ok  	cmd/compile/internal/ssa	0.568s
ok  	cmd/compile/internal/syntax	0.012s
ok  	cmd/compile/internal/test	0.024s
ok  	cmd/compile/internal/types	0.014s
ok  	cmd/cover	2.363s
ok  	cmd/doc	0.094s
ok  	cmd/fix	3.569s
ok  	cmd/go	7.115s
ok  	cmd/go/internal/auth	0.034s
ok  	cmd/go/internal/cache	0.079s
ok  	cmd/go/internal/fsys	0.041s
ok  	cmd/go/internal/generate	0.015s
ok  	cmd/go/internal/get	0.089s
ok  	cmd/go/internal/imports	0.035s
ok  	cmd/go/internal/load	0.061s
ok  	cmd/go/internal/lockedfile	0.175s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.045s
ok  	cmd/go/internal/modconv	0.016s
ok  	cmd/go/internal/modfetch	0.034s
ok  	cmd/go/internal/modfetch/codehost	0.053s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.050s
ok  	cmd/go/internal/modload	0.047s
ok  	cmd/go/internal/mvs	0.028s
ok  	cmd/go/internal/par	0.053s
ok  	cmd/go/internal/renameio	0.030s
ok  	cmd/go/internal/search	0.012s
ok  	cmd/go/internal/str	0.028s
ok  	cmd/go/internal/test	0.035s
ok  	cmd/go/internal/txtar	0.021s
ok  	cmd/go/internal/vcs	0.033s
ok  	cmd/go/internal/web	0.054s
ok  	cmd/go/internal/work	0.028s
ok  	cmd/gofmt	0.055s
ok  	cmd/internal/archive	0.818s
ok  	cmd/internal/buildid	0.333s
ok  	cmd/internal/dwarf	0.009s
ok  	cmd/internal/edit	0.022s
ok  	cmd/internal/goobj	0.027s
ok  	cmd/internal/moddeps	1.699s
ok  	cmd/internal/obj	0.026s
ok  	cmd/internal/obj/arm64	0.130s
ok  	cmd/internal/obj/ppc64	0.171s
ok  	cmd/internal/obj/riscv	0.100s
ok  	cmd/internal/obj/s390x	0.017s
ok  	cmd/internal/obj/x86	4.505s
ok  	cmd/internal/objabi	0.005s
ok  	cmd/internal/pkgpath	0.077s
ok  	cmd/internal/src	0.012s
ok  	cmd/internal/sys	0.029s
ok  	cmd/internal/test2json	0.103s
--- FAIL: TestPIESize (0.00s)
    --- FAIL: TestPIESize/TestPieSize-external (10.99s)
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/pieexternal -buildmode=pie -ldflags=-linkmode=external pie.go]
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/exeexternal -buildmode=exe -ldflags=-linkmode=external pie.go]
        elf_test.go:411: real size difference 0xfffffffffffffd87, expected 0xfffffffffffffb58
        elf_test.go:414: PIE unexpectedly large: got difference of 18446744073709550983 (2693752 - 2692792), expected difference 18446744073709550424
FAIL
FAIL	cmd/link	18.954s
ok  	cmd/link/internal/benchmark	0.043s
--- FAIL: TestRuntimeTypeAttrExternal (1.54s)
    dwarf_test.go:1030: DWARF type offset was 0x3b50+0x744c8, but test program said 0xa47018
FAIL
FAIL	cmd/link/internal/ld	6.289s
ok  	cmd/link/internal/loader	0.041s
--- FAIL: TestExternalLinkerCgoExec (3.04s)
    nm_test.go:191: want 0x638140 address for main.main symbol, but have 0xb4140
    nm_test.go:191: want 0x6efa8c address for main.testdata symbol, but have 0x16ba8c
    nm_test.go:191: want 0x638164 address for main.testfunc symbol, but have 0xb4164
FAIL
FAIL	cmd/nm	4.917s
ok  	cmd/objdump	6.293s
ok  	cmd/pack	2.633s
ok  	cmd/trace	0.248s
ok  	cmd/vet	11.428s
FAIL
go tool dist: Failed: exit status 1

The release command sets some environment variables that may not be set when post-submit tests run on the linux-arm-aws builder, perhaps that's relevant?

if b.Goarm > 0 {
	env = append(env, fmt.Sprintf("GOARM=%d", b.Goarm))
	env = append(env, fmt.Sprintf("CGO_CFLAGS=-march=armv%d", b.Goarm))
	env = append(env, fmt.Sprintf("CGO_LDFLAGS=-march=armv%d", b.Goarm))
}

CC @golang/release, @ianlancetaylor.

The text was updated successfully, but these errors were encountered:

cherrymui · 2020-12-15T23:12:36Z

I can confirm that it is indeed those flags. On the builder, it passes without CGO_LDFLAGS,

root@13fe5f0e6a12:/workdir/go/src# ../bin/go test cmd/nm
ok  	cmd/nm	1.117s

and fails with CGO_LDFLAGS,

root@13fe5f0e6a12:/workdir/go/src# CGO_LDFLAGS=-march=armv6 ../bin/go test cmd/nm
--- FAIL: TestExternalLinkerCgoExec (0.61s)
    nm_test.go:191: want 0x7df4cc address for main.main symbol, but have 0xa74cc
    nm_test.go:191: want 0x88ca5c address for main.testdata symbol, but have 0x154a5c
    nm_test.go:191: want 0x7df4f0 address for main.testfunc symbol, but have 0xa74f0
FAIL
FAIL	cmd/nm	1.008s
FAIL

I'll look into it.

gopherbot · 2020-12-15T23:39:30Z

Change https://golang.org/cl/278592 mentions this issue: cmd/link: pass arch-specific flags to external linker when testing supported flag

cagedmantis · 2021-06-10T14:57:12Z

We've encountered the same set of errors while testing the linux-arm-aws against the tip of the release-branch.go1.15 branch. I'm going to request a backport issue in hopes that it will fix the issue.

cagedmantis · 2021-06-10T14:59:16Z

@gopherbot please consider this for backport to 1.15

gopherbot · 2021-06-10T14:59:29Z

Backport issue(s) opened: #46684 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Dec 15, 2020

dmitshur added this to the Go1.16 milestone Dec 15, 2020

gopherbot closed this as completed in a318d56 Dec 16, 2020

cagedmantis mentioned this issue Jun 10, 2021

x/build: remove Scaleway builders #45066

Closed

gopherbot mentioned this issue Jun 10, 2021

x/build/cmd/release: linux-armv6l release tests aren't passing [1.15 backport] #46684

Closed

golang locked and limited conversation to collaborators Jun 10, 2022

gopherbot added the FrozenDueToAge label Jun 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/cmd/release: linux-armv6l release tests aren't passing #43202

x/build/cmd/release: linux-armv6l release tests aren't passing #43202

dmitshur commented Dec 15, 2020

cherrymui commented Dec 15, 2020

gopherbot commented Dec 15, 2020

cagedmantis commented Jun 10, 2021 •

edited

cagedmantis commented Jun 10, 2021

gopherbot commented Jun 10, 2021

x/build/cmd/release: linux-armv6l release tests aren't passing #43202

x/build/cmd/release: linux-armv6l release tests aren't passing #43202

Comments

dmitshur commented Dec 15, 2020

cherrymui commented Dec 15, 2020

gopherbot commented Dec 15, 2020

cagedmantis commented Jun 10, 2021 • edited

cagedmantis commented Jun 10, 2021

gopherbot commented Jun 10, 2021

cagedmantis commented Jun 10, 2021 •

edited