Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

all: test failures with device not configured #58768

Open
bcmills opened this issue Feb 27, 2023 · 7 comments
Open

all: test failures with device not configured #58768

bcmills opened this issue Feb 27, 2023 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 27, 2023 

#!watchflakes
post <- test == "" && log ~ `: device not configured`
@bcmills
Copy link
Contributor Author

bcmills commented Feb 27, 2023

(Pulled out from #57995 (comment); attn @paulzhol.)

@bcmills bcmills added this to the Backlog milestone Feb 27, 2023
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 27, 2023
@bcmills bcmills mentioned this issue Feb 27, 2023
Closed
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- test == "" && log ~ `: device not configured`
2023-02-27 00:12 freebsd-arm-paulzhol go@af6bb3f9 crypto/tls [build] (log) 
can't create $WORK/b084/_pkg_.a: open $WORK/b084/_pkg_.a: device not configured

— watchflakes

@paulzhol
Copy link
Member

paulzhol commented Mar 3, 2023 •
edited

This is specific to my builder setup I think. It has this sysctl set kern.iscsi.fail_on_disconnection=1.

kern.iscsi.fail_on_disconnection
	     Controls the behavior after an iSCSI connection has been dropped
	     due to network problems.  When set	to 1, a	dropped	connection
	     causes the	iSCSI device nodes to be destroyed.  After reconnect-
	     ing, they will be created again.  By default, the device nodes
	     are left intact.  While the connection is down all	input/output
	     operations	are suspended, to be retried after the connection is
	     reestablished.

Otherwise, on the rare occasion the iSCSI connection does drop for some reason - the builder just hangs.

I didn't find any indicative logs on the iSCSI host around 2023-02-27 00:12 (these disconnects are standard after-build reboots of the builder).

Feb 26 01:08:59 stori kernel: WARNING: <IP> (iqn.1994-09.org.freebsd:nanobsd-rpi3): connection error; dropping connection
...
Feb 26 22:44:47 stori syslogd: last message repeated 1 times
Feb 26 22:44:47 stori syslogd: last message repeated 1 times
Feb 27 01:24:58 stori kernel: WARNING: <IP> (iqn.1994-09.org.freebsd:nanobsd-rpi3): connection error; dropping connection
Feb 27 01:24:58 stori syslogd: last message repeated 1 times
Feb 27 02:26:11 stori syslogd: last message repeated 1 times
Feb 27 02:26:11 stori syslogd: last message repeated 1 times
Feb 27 03:27:06 stori syslogd: last message repeated 1 times
Feb 27 03:27:06 stori syslogd: last message repeated 1 times
Feb 27 03:59:22 stori syslogd: last message repeated 1 times
Feb 27 03:59:22 stori kernel: WARNING: <IP> (iqn.1994-09.org.freebsd:nanobsd-rpi3): waiting for CTL to terminate 14 tasks
Feb 27 03:59:22 stori kernel: WARNING: <IP> (iqn.1994-09.org.freebsd:nanobsd-rpi3): tasks terminated
Feb 27 04:19:25 stori kernel: WARNING: <IP> (iqn.1994-09.org.freebsd:nanobsd-rpi3): connection error; dropping connection
Feb 27 04:19:25 stori syslogd: last message repeated 1 times
Feb 27 05:01:33 stori syslogd: last message repeated 1 times
Feb 27 05:08:29 stori syslogd: last message repeated 3 times

(Times are UTC+2)

@paulzhol paulzhol closed this as completed Mar 3, 2023
@paulzhol paulzhol added the Builders x/build issues (builders, bots, dashboards) label Mar 3, 2023
@gopherbot gopherbot reopened this Apr 10, 2023
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- test == "" && log ~ `: device not configured`
2023-04-07 18:10 freebsd-arm-paulzhol net@3d5a8ee7 go@22c1d18a x/net/http2 [build] (log) 
can't create $WORK/b215/_pkg_.a: open $WORK/b215/_pkg_.a: device not configured

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- test == "" && log ~ `: device not configured`
2023-04-21 14:43 freebsd-arm-paulzhol go@ed832ed3 (log) 
error writing long arguments to response file: open /tmp/workdir-host-freebsd-arm-paulzhol/tmp/args1519101681: device not configured
unexpected fault address 0xcdb78
fatal error: fault
[signal SIGBUS: bus error code=0x3 addr=0xcdb78 pc=0xcdb78]

goroutine 1 [running]:
panic during panic
SIGBUS: bus error
PC=0x71c40 m=3 sigcode=3

go tool dist: signal: bus error

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- test == "" && log ~ `: device not configured`
2023-04-17 18:44 freebsd-arm-paulzhol go@2068d565 runtime [build] (log) 
can't create $WORK/b009/_pkg_.a: open $WORK/b009/_pkg_.a: device not configured
go: unlinkat /tmp/workdir-host-freebsd-arm-paulzhol/tmp/go-build2035123384: device not configured
go tool dist: FAILED: /tmp/workdir-host-freebsd-arm-paulzhol/go/pkg/tool/freebsd_arm/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
/tmp/workdir-host-freebsd-arm-paulzhol/go/src/make.bash: line 226:  1073 Segmentation fault      ./cmd/dist/dist bootstrap -a $vflag $GO_DISTFLAGS "$@"

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- test == "" && log ~ `: device not configured`
2023-12-14 19:59 freebsd-arm-paulzhol go@7c282ba1 cmd/compile/internal/ssa [build] (log) 
can't create $WORK/b121/_pkg_.a: open $WORK/b121/_pkg_.a: device not configured

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Status: Done
Milestone
Backlog
Development

No branches or pull requests
3 participants
@paulzhol @bcmills @gopherbot