Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: unrecognized failures on freebsd-amd64-13_0 #56812

Open
gopherbot opened this issue Nov 17, 2022 · 10 comments
Open

build: unrecognized failures on freebsd-amd64-13_0 #56812

gopherbot opened this issue Nov 17, 2022 · 10 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@gopherbot
Copy link 

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""

Issue created automatically to collect these failures.

Example (log):

panic: test timed out after 5m0s
running tests:
	TestMSAN (5m0s)
	TestMSAN/msan_fail (4m48s)

syscall.Syscall6(0x214, 0x0, 0x900d, 0xc00007acb4, 0x18, 0x0, 0x0)
	/tmp/workdir/go/src/syscall/asm_unix_amd64.s:39 +0x5 fp=0xc00007ac60 sp=0xc00007ac58 pc=0x4745e5
os.wait6(0xc00030b1a0?, 0xc00007ad48?, 0xc00030b140?)
	/tmp/workdir/go/src/os/wait6_freebsd64.go:18 +0x4d fp=0xc00007acd0 sp=0xc00007ac60 pc=0x495ecd
os.(*Process).blockUntilWaitable(0xc0003025a0)
	/tmp/workdir/go/src/os/wait_wait6.go:20 +0x33 fp=0xc00007ad00 sp=0xc00007acd0 pc=0x495f53
os.(*Process).wait(0xc0003025a0)
	/tmp/workdir/go/src/os/exec_unix.go:22 +0x28 fp=0xc00007ad60 sp=0xc00007ad00 pc=0x491428
os.(*Process).Wait(...)
	/tmp/workdir/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000308dc0)
	/tmp/workdir/go/src/os/exec/exec.go:890 +0x45 fp=0xc00007adc8 sp=0xc00007ad60 pc=0x4e5ca5
os/exec.(*Cmd).Run(0x40?)
	/tmp/workdir/go/src/os/exec/exec.go:590 +0x39 fp=0xc00007ade8 sp=0xc00007adc8 pc=0x4e47b9
os/exec.(*Cmd).CombinedOutput(0xc000308dc0)
	/tmp/workdir/go/src/os/exec/exec.go:1005 +0xa8 fp=0xc00007ae10 sp=0xc00007ade8 pc=0x4e6608
misc/cgo/testsanitizers_test.TestMSAN.func1(0xc000082ea0)
	/tmp/workdir/go/misc/cgo/testsanitizers/msan_test.go:72 +0x2c5 fp=0xc00007af70 sp=0xc00007ae10 pc=0x5355e5
testing.tRunner(0xc000082ea0, 0xc0000da190)

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 17, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""
2022-11-14 18:35 freebsd-amd64-13_0 go@74b6a220 TestMSAN (log) 
panic: test timed out after 5m0s
running tests:
	TestMSAN (5m0s)
	TestMSAN/msan_fail (4m48s)

syscall.Syscall6(0x214, 0x0, 0x900d, 0xc00007acb4, 0x18, 0x0, 0x0)
	/tmp/workdir/go/src/syscall/asm_unix_amd64.s:39 +0x5 fp=0xc00007ac60 sp=0xc00007ac58 pc=0x4745e5
os.wait6(0xc00030b1a0?, 0xc00007ad48?, 0xc00030b140?)
	/tmp/workdir/go/src/os/wait6_freebsd64.go:18 +0x4d fp=0xc00007acd0 sp=0xc00007ac60 pc=0x495ecd
os.(*Process).blockUntilWaitable(0xc0003025a0)
	/tmp/workdir/go/src/os/wait_wait6.go:20 +0x33 fp=0xc00007ad00 sp=0xc00007acd0 pc=0x495f53
os.(*Process).wait(0xc0003025a0)
	/tmp/workdir/go/src/os/exec_unix.go:22 +0x28 fp=0xc00007ad60 sp=0xc00007ad00 pc=0x491428
os.(*Process).Wait(...)
	/tmp/workdir/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000308dc0)
	/tmp/workdir/go/src/os/exec/exec.go:890 +0x45 fp=0xc00007adc8 sp=0xc00007ad60 pc=0x4e5ca5
os/exec.(*Cmd).Run(0x40?)
	/tmp/workdir/go/src/os/exec/exec.go:590 +0x39 fp=0xc00007ade8 sp=0xc00007adc8 pc=0x4e47b9
os/exec.(*Cmd).CombinedOutput(0xc000308dc0)
	/tmp/workdir/go/src/os/exec/exec.go:1005 +0xa8 fp=0xc00007ae10 sp=0xc00007ade8 pc=0x4e6608
misc/cgo/testsanitizers_test.TestMSAN.func1(0xc000082ea0)
	/tmp/workdir/go/misc/cgo/testsanitizers/msan_test.go:72 +0x2c5 fp=0xc00007af70 sp=0xc00007ae10 pc=0x5355e5
testing.tRunner(0xc000082ea0, 0xc0000da190)

— watchflakes

@bcmills
Copy link
Contributor

bcmills commented Nov 17, 2022

Duplicate of #56805

@bcmills bcmills marked this as a duplicate of #56805 Nov 17, 2022
@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""
2023-01-20 09:27 freebsd-amd64-13_0 go@f53137f9 (log) 
go: error obtaining buildID for go tool compile: fork/exec /tmp/workdir/go/pkg/tool/freebsd_amd64/compile: no such file or directory

go tool dist: FAILED: go list -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std: exit status 1

— watchflakes

@gopherbot gopherbot reopened this Jan 25, 2023
@bcmills
Copy link
Contributor

bcmills commented Jan 31, 2023

That last one is weird.

@golang/freebsd, any idea what would cause $GOROOT/pkg/tool/freebsd_amd64/compile to suddenly disappear?

@paulzhol
Copy link
Member

paulzhol commented Feb 1, 2023

That last one is weird.

@golang/freebsd, any idea what would cause $GOROOT/pkg/tool/freebsd_amd64/compile to suddenly disappear?

@bcmills are there any syslogs from that host?

We're doing out, err := exec.Command("/sbin/mount", "-u", "-o", "async,noatime", "/").CombinedOutput()
https://github.com/golang/build/blob/d3662e6622f8b507bed9ffd7d1685f3c1d54ce1c/cmd/buildlet/buildlet.go#L1768
in the buildlet, I think this also includes /tmp where $GOROOT is installed.

I can't remember if there's an explicit tmpfs mount in the fstab. Also we don't have the rc.conf setting to enable tmpmfs="YES" setting (which enables a ramdisk formatted with UFS). So maybe somehow broke the local UFS2 filesystem?

@bcmills
Copy link
Contributor

bcmills commented Feb 1, 2023

are there any syslogs from that host?

Unfortunately for builders in ephemeral VMs we only have the test logs. (#49165 suggests that we should also collect core files, but to my knowledge nobody has a plan to collect system logs.)

@paulzhol
Copy link
Member

paulzhol commented Feb 1, 2023

I found an old kernel bug which might be related:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238663
async mount followed by writes but then a remount ro. We probably don't remount root or even just /tmp in the tests.
The underlying cause was a suspected firmware bug in the TRIM impl. of the drive.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""
2024-02-07 21:58 freebsd-amd64-13_0 tools@5fcc6273 go@dd31ad7e (log) 
../../../../../pkg/mod/honnef.co/go/tools@v0.4.6/config/config.go:13:2: github.com/BurntSushi/toml@v1.2.1: read "http://gk3-services-nap-jetxd907-8ae6730a-8g6b.c.symbolic-datum-552.internal:30157/github.com/%21burnt%21sushi/toml/@v/v1.2.1.zip": unexpected EOF

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""
2024-02-08 14:31 freebsd-amd64-13_0 benchmarks@adbfebb2 go@6cbe522f (log) 
driver/driver_unix.go:16:2: golang.org/x/sys@v0.17.0: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/golang.org/x/sys/@v/v0.17.0.mod": dial tcp 10.128.0.6:30157: i/o timeout
sweet/benchmarks/bleve-index/main.go:19:2: github.com/blevesearch/bleve@v1.0.14: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/github.com/blevesearch/bleve/@v/v1.0.14.mod": dial tcp 10.128.0.6:30157: i/o timeout
sweet/benchmarks/bleve-index/main.go:20:2: github.com/blevesearch/bleve@v1.0.14: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/github.com/blevesearch/bleve/@v/v1.0.14.mod": dial tcp 10.128.0.6:30157: i/o timeout
third_party/bleve-bench/mapping.go:25:2: github.com/blevesearch/bleve@v1.0.14: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/github.com/blevesearch/bleve/@v/v1.0.14.mod": dial tcp 10.128.0.6:30157: i/o timeout
../../../../pkg/mod/gitlab.com/golang-commonmark/markdown@v0.0.0-20211110145824-bf3e522c626a/entity.go:7:8: gitlab.com/golang-commonmark/html@v0.0.0-20191124015941-a22733972181: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/gitlab.com/golang-commonmark/html/@v/v0.0.0-20191124015941-a22733972181.zip": dial tcp 10.128.0.6:30157: i/o timeout
../../../../pkg/mod/gitlab.com/golang-commonmark/markdown@v0.0.0-20211110145824-bf3e522c626a/linkify.go:10:2: gitlab.com/golang-commonmark/linkify@v0.0.0-20191026162114-a0c2df6c8f82: Get "http://gk3-services-nap-jetxd907-8f582974-r2wj.c.symbolic-datum-552.internal:30157/gitlab.com/golang-commonmark/linkify/@v/v0.0.0-20191026162114-a0c2df6c8f82.zip": dial tcp 10.128.0.6:30157: i/o timeout

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "freebsd-amd64-13_0" && pkg == "" && test == ""
2024-03-13 17:06 freebsd-amd64-13_0 telemetry@f301f316 go@6d229889 (log) 
../../../../../pkg/mod/cloud.google.com/go/iam@v1.1.3/apiv1/iampb/iam_policy.pb.go:28:2: google.golang.org/genproto/googleapis/api@v0.0.0-20231016165738-49dd2c1f3d0b: reading http://gk3-services-nap-jetxd907-8f582974-oya3.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/googleapis/api/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod: 500 Internal Server Error
	server response: Get "https://proxy.golang.org/google.golang.org/genproto/googleapis/api/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod": read tcp 10.102.128.204:47738->74.125.126.141:443: read: connection reset by peer
../../../../../pkg/mod/cloud.google.com/go/iam@v1.1.3/apiv1/iampb/policy.pb.go:27:2: google.golang.org/genproto@v0.0.0-20231016165738-49dd2c1f3d0b: reading http://gk3-services-nap-jetxd907-8f582974-oya3.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod: 500 Internal Server Error
	server response: Get "https://proxy.golang.org/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod": read tcp 10.102.128.204:47738->74.125.126.141:443: read: connection reset by peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/genproto@v0.0.0-20231016165738-49dd2c1f3d0b: reading http://gk3-services-nap-jetxd907-8f582974-oya3.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod: 500 Internal Server Error
	server response: Get "https://proxy.golang.org/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod": read tcp 10.102.128.204:47738->74.125.126.141:443: read: connection reset by peer
../../../../../pkg/mod/cloud.google.com/go/cloudtasks@v1.12.4/apiv2/cloud_tasks_client.go:38:2: google.golang.org/genproto@v0.0.0-20231016165738-49dd2c1f3d0b: reading http://gk3-services-nap-jetxd907-8f582974-oya3.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod: 500 Internal Server Error
	server response: Get "https://proxy.golang.org/google.golang.org/genproto/@v/v0.0.0-20231016165738-49dd2c1f3d0b.mod": read tcp 10.102.128.204:47738->74.125.126.141:443: read: connection reset by peer

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@paulzhol @bcmills @gopherbot