Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: frequent dragonfly-amd64 failures with "No space left on device" since 2021-12-07 #50538

Closed
bcmills opened this issue Jan 10, 2022 · 10 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages) Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Jan 10, 2022

# golang.org/x/website/cmd/golangorg.test
/tmp/workdir-host-dragonfly-amd64-master/go/pkg/tool/dragonfly_amd64/link: running gcc failed: exit status 1
/usr/libexec/binutils234/elf/ld.gold: fatal error: $WORK/b330/golangorg.test: No space left on device
collect2: error: ld returned 1 exit status

FAIL	golang.org/x/website/cmd/golangorg [build failed]

It's not clear to me whether this is due to the Go buildlet failing to clear temporary data between runs, or due to some other programs running on the host filling up /tmp.

greplogs --dashboard -md -l -e '(?ms)\Adragonfly-amd64.*: No space left on device\n'

2022-01-10T04:23:42-5dfff35-6df0957/dragonfly-amd64
2022-01-09T18:43:51-5dfff35-2639f2f/dragonfly-amd64
2022-01-08T00:24:25-5dfff35-90860e0/dragonfly-amd64
2022-01-07T21:20:23-5dfff35-9de1ac6/dragonfly-amd64
2022-01-07T18:40:16-0b28845-f1596d7/dragonfly-amd64
2022-01-07T15:48:28-0b28845-9de1ac6/dragonfly-amd64
2022-01-07T06:34:04-f4f7323-11b28e7/dragonfly-amd64
2022-01-07T02:37:20-f4f7323-40afced/dragonfly-amd64
2022-01-07T02:32:39-f4f7323-2bb7f6b/dragonfly-amd64
2022-01-07T02:32:03-f4f7323-ab4556a/dragonfly-amd64
2022-01-07T01:36:17-f4f7323-c1e7c51/dragonfly-amd64
2022-01-07T00:15:59-f4f7323-07525e1/dragonfly-amd64
2022-01-07T00:02:57-f4f7323-c295137/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-9de1ac6/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-042548b/dragonfly-amd64
2022-01-06T23:39:43-e63aea0-042548b/dragonfly-amd64
2022-01-06T22:12:11-e63aea0-b9cae6f/dragonfly-amd64
2022-01-06T21:38:59-e63aea0-61014f0/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-9de1ac6/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-10f1ed1/dragonfly-amd64
2022-01-06T19:42:27-b511507-10f1ed1/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-b5bfaf4/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-ab70d7c/dragonfly-amd64
2022-01-06T00:26:47-4a88b4c-b5bfaf4/dragonfly-amd64
2022-01-05T21:22:03-4a88b4c-2b39d86/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-ab70d7c/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-002283e/dragonfly-amd64
2022-01-05T15:54:37-7a18f71-2c58bb2/dragonfly-amd64
2022-01-05T09:50:29-2aac4c0-2c58bb2/dragonfly-amd64
2022-01-05T01:53:39-2aac4c0-301db3f/dragonfly-amd64
2022-01-04T22:27:20-2aac4c0-f154f8b/dragonfly-amd64
2022-01-04T22:05:15-2aac4c0-e39ab9b/dragonfly-amd64
2022-01-04T20:37:27-2aac4c0-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-1242f43/dragonfly-amd64
2022-01-04T19:30:38-995577f-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-1242f43/dragonfly-amd64
2022-01-04T16:25:19-bc5b8e0-95b240b/dragonfly-amd64
2022-01-03T23:45:12-f07bd53-95b240b/dragonfly-amd64
2022-01-02T14:27:43-f07bd53-c886143/dragonfly-amd64
2022-01-02T14:27:32-f07bd53-d181885/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-6178d25/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-1242f43/dragonfly-amd64
2021-12-29T20:20:32-56b50be-6178d25/dragonfly-amd64
2021-12-29T19:23:11-56b50be-8a306e2/dragonfly-amd64
2021-12-29T17:53:46-56b50be-91e7821/dragonfly-amd64
2021-12-29T04:10:07-ee35583-91e7821/dragonfly-amd64
2021-12-29T01:44:55-ee35583-a78532a/dragonfly-amd64
2021-12-28T17:18:11-ee35583-b357b05/dragonfly-amd64
2021-12-23T20:03:38-faa0890-b357b05/dragonfly-amd64
2021-12-23T19:58:13-faa0890-0e4b878/dragonfly-amd64
2021-12-23T18:58:36-faa0890-af3b8cf/dragonfly-amd64
2021-12-23T17:27:50-faa0890-ed766b6/dragonfly-amd64
2021-12-22T23:31:40-faa0890-d2ce939/dragonfly-amd64
2021-12-22T20:47:09-faa0890-8cfcee1/dragonfly-amd64
2021-12-22T18:43:55-faa0890-0f3becf/dragonfly-amd64
2021-12-22T00:11:21-91908ba-90fb5a4/dragonfly-amd64
2021-12-21T23:09:59-91908ba-ebac50e/dragonfly-amd64
2021-12-21T22:12:49-91908ba-a1ce550/dragonfly-amd64
2021-12-21T19:17:52-dc59b70-a1ce550/dragonfly-amd64
2021-12-21T17:58:16-dc59b70-2d1d548/dragonfly-amd64
2021-12-21T15:58:14-983ac79-2d1d548/dragonfly-amd64
2021-12-21T14:25:10-49c3839-de690c2/dragonfly-amd64
2021-12-21T14:25:10-49c3839-2d1d548/dragonfly-amd64
2021-12-21T03:55:43-d4a18cb-2d1d548/dragonfly-amd64
2021-12-21T03:54:36-d4a18cb-b5e0654/dragonfly-amd64
2021-12-21T01:51:23-d4a18cb-4dfbb89/dragonfly-amd64
2021-12-21T01:10:17-d4a18cb-e087949/dragonfly-amd64
2021-12-21T00:00:05-d4a18cb-1555062/dragonfly-amd64
2021-12-20T23:46:23-c471c5c-1555062/dragonfly-amd64
2021-12-20T23:32:14-c471c5c-6713b5d/dragonfly-amd64
2021-12-20T15:13:56-4e7a786-7c94355/dragonfly-amd64
2021-12-20T15:13:38-4e7a786-a2004de/dragonfly-amd64
2021-12-19T20:16:45-4e7a786-87b2a54/dragonfly-amd64
2021-12-17T16:05:31-4e7a786-c5fee93/dragonfly-amd64
2021-12-17T03:59:09-4e7a786-5299390/dragonfly-amd64
2021-12-17T03:28:06-4e7a786-3c49583/dragonfly-amd64
2021-12-17T02:00:14-4e7a786-55a25aa/dragonfly-amd64
2021-12-17T01:59:32-4e7a786-63077bf/dragonfly-amd64
2021-12-17T00:14:53-4e7a786-2e6e9df/dragonfly-amd64
2021-12-16T23:52:17-4e7a786-0c24038/dragonfly-amd64
2021-12-16T22:41:31-4e7a786-1387b5e/dragonfly-amd64
2021-12-16T22:39:00-4e7a786-b1a53ec/dragonfly-amd64
2021-12-16T22:15:57-4e7a786-d01cd8f/dragonfly-amd64
2021-12-16T22:15:22-4e7a786-d93677a/dragonfly-amd64
2021-12-16T21:08:09-4e7a786-ae695cd/dragonfly-amd64
2021-12-16T21:02:37-4e7a786-ea26ce7/dragonfly-amd64
2021-12-16T15:44:40-4e7a786-1d3a5b4/dragonfly-amd64
2021-12-16T14:37:07-4e7a786-7f23145/dragonfly-amd64
2021-12-15T14:48:19-0c791db-b1c7703/dragonfly-amd64
2021-12-15T00:33:55-18bc0f9-9d0ca26/dragonfly-amd64
2021-12-14T22:42:47-18bc0f9-1540239/dragonfly-amd64
2021-12-08T17:24:46-0dd6a27-ac7e950/dragonfly-amd64
2021-12-07T03:44:02-0dd6a27-dc65c48/dragonfly-amd64
2021-02-18T17:57:36-a76efea/dragonfly-amd64-5_8

CC @golang/release; attn @tuxillo

@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. labels Jan 10, 2022
@bcmills bcmills added this to the Go1.18 milestone Jan 10, 2022
@bcmills bcmills added OS-Dragonfly Soon This needs to be done soon. (regressions, serious bugs, outages) labels Jan 10, 2022
@bcmills bcmills modified the milestones: Go1.18, Backlog Jan 10, 2022
@tuxillo
Copy link
Contributor

tuxillo commented Jan 10, 2022

This is what I can see in the builder:

root@gobuilder:~ # du -hs /tmp/*
 88K    /tmp/gobuilder.log
1.3G    /tmp/workdir-host-dragonfly-amd64-master
root@gobuilder:~ # df -h /tmp
Filesystem   Size   Used  Avail Capacity  Mounted on
tmpfs       1967M  1363M   603M    69%    /tmp

@bcmills
Copy link
Contributor Author

bcmills commented Jan 12, 2022

<2GiB seems awfully small for a /tmp partition. Any chance it could be increased?

Barring that, any idea what changed around Dec. 7 to cause this builder to start running over the limit? It looks like the failures overwhelmingly occur when linking golang.org/x/website/cmd/golangorg.test or tests closely before or after that one, but there isn't an obvious culprit CL –- it isn't even clear to me whether this is due to a change in x/website proper, or in the size of the linker inputs or outputs due to some other change in the toolchain.

@bcmills bcmills modified the milestones: Backlog, Go1.18 Jan 12, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Jan 12, 2022

Marking as release-blocker for Go 1.18 via #11811. This is clearly a mid-cycle regression of some sort, and it's not obvious to me what the root cause is or how it can be mitigated.

@tuxillo
Copy link
Contributor

tuxillo commented Jan 12, 2022

We did not do anything on that VM, in fact it's up for 70 days and running a kernel from Oct, 17th:

root@gobuilder:~ # uname -a
DragonFly gobuilder.muscles.dragonflybsd.org 6.1-DEVELOPMENT DragonFly v6.1.0.477.gc2e50-DEVELOPMENT #0: Sun Oct 17 10:27:56 UTC 2021     root@pkgbox64.dragonflybsd.org:/usr/obj/usr/src/sys/X86_64_GENERIC  x86_64
root@gobuilder:~ # uptime
 8:25AM  up 70 days,  9:39, 1 user, load averages: 5.79, 4.98, 4.47

The /tmp size can be increased indeed. I was also thinking in upgrading to latest bleeding-edge but I'll make no changes until you tell me to.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 12, 2022

Since the dragonfly builder is reverse, @dmitshur suggested that you could go ahead and increase the /tmp partition on this builder and we can investigate the size increases independently (for example, on a GCE builder).

@bcmills
Copy link
Contributor Author

bcmills commented Jan 12, 2022

I would suggest not upgrading the OS image at this point in the cycle, though. (Early in the 1.19 development window would give more time to chase down any regressions that may occur.)

@tuxillo
Copy link
Contributor

tuxillo commented Jan 13, 2022

@bcmills ok, i'll stop the builder, increase the /tmp partition and restart the VM. I won't be upgrading until you guys tell me to do so. With regards to a GCE builder, I still have to create images for GCE, unless somebody beats me to it.

@toothrot
Copy link
Contributor

@tuxillo Thanks! It looks like the builder has been missing for a few days. How is this going?

@tuxillo
Copy link
Contributor

tuxillo commented Jan 19, 2022

Sorry! I forgot to start the builder again.

It has now 4GB in /tmp:

root@gobuilder:~ # df -h /tmp
Filesystem   Size   Used  Avail Capacity  Mounted on
tmpfs       4096M   758M  3338M    19%    /tmp

@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022

Looks like these failures are no longer occurring. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages) Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Development

No branches or pull requests

4 participants