Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

all: failures with no space left on device on netbsd-.*-bsiegert builders #56879

Open
gopherbot opened this issue Nov 21, 2022 · 10 comments
Open

all: failures with no space left on device on netbsd-.*-bsiegert builders #56879

gopherbot opened this issue Nov 21, 2022 · 10 comments
Assignees
@bsiegert
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Milestone
Unreleased

Comments

@gopherbot
Copy link

gopherbot commented Nov 21, 2022 •
edited by bcmills 

#!watchflakes
post <- builder ~ `.*-bsiegert` && (`no space left on device` || `no file system`)

Issue created automatically to collect these failures.

Example (log):

PASS
testing: can't write /var/gobuilder/buildlet/tmp/go-build431852774/b606/testlog.txt: write /var/gobuilder/buildlet/tmp/go-build431852774/b606/testlog.txt: no space left on device

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 21, 2022
@gopherbot gopherbot added this to the Unreleased milestone Nov 21, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/website/internal/redirect" && test == ""
2022-11-18 18:55 netbsd-arm64-bsiegert website@5934f13c go@bda02354 x/website/internal/redirect (log) 
PASS
testing: can't write /var/gobuilder/buildlet/tmp/go-build431852774/b606/testlog.txt: write /var/gobuilder/buildlet/tmp/go-build431852774/b606/testlog.txt: no space left on device

— watchflakes

@bcmills
Copy link
Contributor

bcmills commented Nov 21, 2022 •
edited

The no space left on device error indicates a problem with the builder itself.
(attn @bsiegert)

@bcmills bcmills changed the title x/website/internal/redirect: unrecognized failures all: failures with no space left on device on netbsd-.*-bsiegert builders Nov 21, 2022
@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) OS-NetBSD labels Nov 21, 2022
@bcmills bcmills mentioned this issue Nov 21, 2022
Open
@bsiegert
Copy link
Contributor

As far as I can tell, these were all on the same machine (fluebrig.bentsukun.ch). It turns out that the machine has been so stable that its buildlet log file grew to use all available storage :)

I deleted the old logs (and some other stuff) and rebooted the machine. This should no longer happen, please reopen if it does.

@gopherbot gopherbot reopened this Dec 27, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder ~ `.*-bsiegert` && (`no space left on device` || `no file system`)
2022-12-22 18:04 netbsd-arm-bsiegert go@c61d322d (log) 
go: creating work dir: mkdir /var/gobuilder/buildlet/tmp/go-build864668300: no space left on device

go tool dist: FAILED: go list -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std: exit status 1
2022-12-22 18:22 netbsd-arm-bsiegert tools@6546d82b go@41eb70ae internal/goversion [build] (log) 
vet: failed to write analysis facts: open $WORK/b197/vet.out: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b190/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b198/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b199/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b189/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b205/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b204/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b203/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b202/vet.cfg: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b201/vet.cfg: no space left on device
...
go build os/user: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b253/: no space left on device
go build golang.org/x/tools/cmd/getgo/server: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b254/: no space left on device
go build golang.org/x/tools/cmd/go-contrib-init: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b257/: no space left on device
go build hash/adler32: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b263/: no space left on device
go build debug/dwarf: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b264/: no space left on device
go build internal/pkgbits: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b267/: no space left on device
go build go/internal/srcimporter: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b268/: no space left on device
go build expvar: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b272/: no space left on device
go build encoding/gob: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b274/: no space left on device
go build golang.org/x/tools/godoc/analysis: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b275/: no space left on device
2022-12-22 18:22 netbsd-arm-bsiegert tools@6546d82b go@41eb70ae x/tools/go/gcexportdata [build] (log) 
can't create $WORK/b188/_pkg_.a: open $WORK/b188/_pkg_.a: no space left on device
go build golang.org/x/tools/godoc/vfs: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b277/: no space left on device
go build golang.org/x/tools/godoc/redirect: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b280/: no space left on device
go build index/suffixarray: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b279/: no space left on device
go build golang.org/x/tools/playground: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b285/: no space left on device
go build internal/profile: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b287/: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b081/vet.cfg: no space left on device
go build golang.org/x/tools/internal/fastwalk: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b291/: no space left on device
go build golang.org/x/tools/cmd/goyacc: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b301/: no space left on device
open /var/gobuilder/buildlet/tmp/go-build3430108209/b305/vet.cfg: no space left on device
...
go build golang.org/x/tools/internal/event/tag: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b702/: no space left on device
go build golang.org/x/tools/internal/fakenet: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b706/: no space left on device
go build golang.org/x/sync/errgroup: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b718/: no space left on device
go build golang.org/x/tools/internal/fuzzy: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b712/: no space left on device
go build golang.org/x/tools/internal/gocommand: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b721/: no space left on device
go build golang.org/x/tools/internal/xcontext: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b748/: no space left on device
go build golang.org/x/tools/internal/stack: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b733/: no space left on device
go build golang.org/x/tools/internal/apidiff: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b763/: no space left on device
go build golang.org/x/tools/internal/typesinternal_test: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b766/: no space left on device
go build golang.org/x/tools/internal/persistent: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b752/: no space left on device
2022-12-22 18:22 netbsd-arm-bsiegert tools@6546d82b go@41eb70ae x/tools/internal/testenv [build] (log) 
can't create $WORK/b210/_pkg_.a: open $WORK/b210/_pkg_.a: no space left on device
go build golang.org/x/tools/present: mkdir /var/gobuilder/buildlet/tmp/go-build3430108209/b770/: no space left on device
2022-12-22 18:22 netbsd-arm-bsiegert tools@6546d82b go@41eb70ae x/tools/blog.test [build] (log) 
/var/gobuilder/buildlet/go/pkg/tool/netbsd_arm/link: resize output file failed: truncate $WORK/b086/blog.test: no space left on device

— watchflakes

@bsiegert
Copy link
Contributor

This is the 32-bit builder machine. It had some trouble with a broken USB storage. I replaced it and rebooted, let's hope that this works.

I am in the process of adding more monitoring (with Prometheus) to detect these.

@gopherbot gopherbot reopened this Apr 7, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder ~ `.*-bsiegert` && (`no space left on device` || `no file system`)
2023-03-07 14:28 netbsd-arm64-bsiegert benchmarks@2652dad2 go@99001c46 (log) 
-> unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
-> unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/protobuf/@v/v1.27.1.zip: open /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/testprotos/legacy/bug1052/bug1052.pb.go: input/output error
-> unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/go.opencensus.io/@v/v0.23.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/go.opencensus.io@v0.23.0/zpages/internal: input/output error
go: writing go.mod cache: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/github.com/ianlancetaylor/demangle/@v: input/output error
go: writing go.mod cache: open /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/genproto/@v/v0.0.0-20210909211513-a8c4777a87af.mod692210092.tmp: input/output error
go: writing go.mod cache: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/golang.org/x/tools: input/output error
go: writing go.mod cache: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/gitlab.com/opennota: input/output error
go: writing go.mod cache: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/github.com/russross: input/output error
go: writing go.mod cache: open /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/appengine/@v/v1.6.6.mod319451327.tmp: input/output error
go: writing go.mod cache: open /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20201110031124-69a78807bb2b.mod607820298.tmp: no space left on device
...
../../../../pkg/mod/google.golang.org/api@v0.60.0/transport/grpc/dial.go:31:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/call_option.go:37:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/google.golang.org/api@v0.60.0/transport/grpc/dial.go:26:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/google.golang.org/api@v0.60.0/transport/grpc/dial.go:27:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/google.golang.org/api@v0.60.0/transport/grpc/dial.go:28:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/cloud.google.com/go@v0.97.0/iam/iam.go:33:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/apierror/apierror.go:41:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/grpc/@v/v1.40.0.zip: mkdir /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/grpc@v1.40.0/xds/internal/balancer/clusterimpl: input/output error
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/apierror/apierror.go:42:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/protobuf/@v/v1.27.1.zip: open /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/testprotos/legacy/bug1052/bug1052.pb.go: input/output error
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/apierror/internal/proto/error.pb.go:30:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/protobuf/@v/v1.27.1.zip: open /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/testprotos/legacy/bug1052/bug1052.pb.go: input/output error
../../../../pkg/mod/cloud.google.com/go/storage@v1.18.2/storage.go:55:2: unzip /scratch/gobuilder/buildlet/gopath/pkg/mod/cache/download/google.golang.org/protobuf/@v/v1.27.1.zip: open /scratch/gobuilder/buildlet/gopath/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/testprotos/legacy/bug1052/bug1052.pb.go: input/output error

— watchflakes

@bsiegert
Copy link
Contributor

Fixed by rebooting the machine.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder ~ `.*-bsiegert` && (`no space left on device` || `no file system`)
2023-06-12 16:55 netbsd-arm64-bsiegert text@3a7a2557 go@eaa4b1a6 runtime/debug [build] (log) 
can't create $WORK/b162/_pkg_.a: open $WORK/b162/_pkg_.a: no space left on device
vendor/golang.org/x/net/http2/hpack: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b153/: no space left on device
mime: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b154/: no space left on device
mime/quotedprintable: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b156/: no space left on device
net/http/internal: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b158/: no space left on device
vendor/golang.org/x/text/unicode/norm: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b150/: no space left on device
golang.org/x/text/internal/language: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b165/: no space left on device
internal/goarch: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b006/vet.cfg: no space left on device
internal/abi: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b005/vet.cfg: no space left on device
internal/unsafeheader: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b008/vet.cfg: no space left on device
...
os: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b032/vet.cfg: no space left on device
encoding/binary: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b052/vet.cfg: no space left on device
fmt: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b002/vet.cfg: no space left on device
internal/saferio: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b053/vet.cfg: no space left on device
flag: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b046/vet.cfg: no space left on device
go/build/constraint: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b059/vet.cfg: no space left on device
path/filepath: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b058/vet.cfg: no space left on device
internal/bisect: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b069/vet.cfg: no space left on device
internal/godebug: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b068/vet.cfg: no space left on device
context: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b072/vet.cfg: no space left on device
2023-06-12 16:55 netbsd-arm64-bsiegert text@3a7a2557 go@eaa4b1a6 x/text/encoding/charmap [build] (log) 
can't create $WORK/b251/_pkg_.a: open $WORK/b251/_pkg_.a: no space left on device
regexp/syntax: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b064/vet.cfg: no space left on device
os/exec: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b071/vet.cfg: no space left on device
internal/platform: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b074/vet.cfg: no space left on device
internal/goroot: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b070/vet.cfg: no space left on device
regexp: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b063/vet.cfg: no space left on device
hash: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b081/vet.cfg: no space left on device
text/tabwriter: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b077/vet.cfg: no space left on device
internal/lazyregexp: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b062/vet.cfg: no space left on device
log/internal: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b085/vet.cfg: no space left on device
...
vendor/golang.org/x/text/transform: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b148/vet.cfg: no space left on device
vendor/golang.org/x/crypto/chacha20: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b141/vet.cfg: no space left on device
net/http/internal/ascii: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b159/vet.cfg: no space left on device
vendor/golang.org/x/crypto/chacha20poly1305: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b140/vet.cfg: no space left on device
internal/sysinfo: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b161/vet.cfg: no space left on device
golang.org/x/text/internal/tag: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b166/vet.cfg: no space left on device
golang.org/x/text/transform: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b168/vet.cfg: no space left on device
golang.org/x/text/unicode/rangetable: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b170/vet.cfg: no space left on device
encoding/pem: open /scratch/gobuilder/buildlet/tmp/go-build2774581813/b129/vet.cfg: no space left on device
golang.org/x/sys/execabs: mkdir /scratch/gobuilder/buildlet/tmp/go-build2774581813/b207/: no space left on device
2023-06-12 16:55 netbsd-arm64-bsiegert text@3a7a2557 go@eaa4b1a6 x/text/encoding/japanese [build] (log) 
can't create $WORK/b262/_pkg_.a: open $WORK/b262/_pkg_.a: no space left on device

— watchflakes

@gopherbot gopherbot reopened this Jun 13, 2023
@bsiegert
Copy link
Contributor

Rebooted the machine; the cable to the hard drive had become loose.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- builder ~ `.*-bsiegert` && (`no space left on device` || `no file system`)
2023-12-11 20:30 netbsd-arm64-bsiegert go@78dcb532 internal/godebugs.test [build] (log) 
can't create $WORK/b095/_pkg_.a: open $WORK/b095/_pkg_.a: no space left on device
2023-12-11 20:30 netbsd-arm64-bsiegert go@78dcb532 internal/godebug_test [build] (log) 
can't create $WORK/b090/_pkg_.a: open $WORK/b090/_pkg_.a: no space left on device

— watchflakes

@gopherbot gopherbot reopened this Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bsiegert bsiegert

Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Projects
Test Flakes
Status: Done
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bsiegert @bcmills @gopherbot