-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: frequent dragonfly-amd64 failures with "No space left on device" since 2021-12-07 #50538
Comments
This is what I can see in the builder:
|
<2GiB seems awfully small for a /tmp partition. Any chance it could be increased? Barring that, any idea what changed around Dec. 7 to cause this builder to start running over the limit? It looks like the failures overwhelmingly occur when linking |
Marking as release-blocker for Go 1.18 via #11811. This is clearly a mid-cycle regression of some sort, and it's not obvious to me what the root cause is or how it can be mitigated. |
We did not do anything on that VM, in fact it's up for 70 days and running a kernel from Oct, 17th:
The /tmp size can be increased indeed. I was also thinking in upgrading to latest |
Since the |
I would suggest not upgrading the OS image at this point in the cycle, though. (Early in the 1.19 development window would give more time to chase down any regressions that may occur.) |
@bcmills ok, i'll stop the builder, increase the /tmp partition and restart the VM. I won't be upgrading until you guys tell me to do so. With regards to a GCE builder, I still have to create images for GCE, unless somebody beats me to it. |
@tuxillo Thanks! It looks like the builder has been missing for a few days. How is this going? |
Sorry! I forgot to start the builder again. It has now 4GB in /tmp:
|
Looks like these failures are no longer occurring. Thanks! |
It's not clear to me whether this is due to the Go buildlet failing to clear temporary data between runs, or due to some other programs running on the host filling up
/tmp
.greplogs --dashboard -md -l -e '(?ms)\Adragonfly-amd64.*: No space left on device\n'
2022-01-10T04:23:42-5dfff35-6df0957/dragonfly-amd64
2022-01-09T18:43:51-5dfff35-2639f2f/dragonfly-amd64
2022-01-08T00:24:25-5dfff35-90860e0/dragonfly-amd64
2022-01-07T21:20:23-5dfff35-9de1ac6/dragonfly-amd64
2022-01-07T18:40:16-0b28845-f1596d7/dragonfly-amd64
2022-01-07T15:48:28-0b28845-9de1ac6/dragonfly-amd64
2022-01-07T06:34:04-f4f7323-11b28e7/dragonfly-amd64
2022-01-07T02:37:20-f4f7323-40afced/dragonfly-amd64
2022-01-07T02:32:39-f4f7323-2bb7f6b/dragonfly-amd64
2022-01-07T02:32:03-f4f7323-ab4556a/dragonfly-amd64
2022-01-07T01:36:17-f4f7323-c1e7c51/dragonfly-amd64
2022-01-07T00:15:59-f4f7323-07525e1/dragonfly-amd64
2022-01-07T00:02:57-f4f7323-c295137/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-9de1ac6/dragonfly-amd64
2022-01-06T23:48:15-f4f7323-042548b/dragonfly-amd64
2022-01-06T23:39:43-e63aea0-042548b/dragonfly-amd64
2022-01-06T22:12:11-e63aea0-b9cae6f/dragonfly-amd64
2022-01-06T21:38:59-e63aea0-61014f0/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-9de1ac6/dragonfly-amd64
2022-01-06T20:42:09-e63aea0-10f1ed1/dragonfly-amd64
2022-01-06T19:42:27-b511507-10f1ed1/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-b5bfaf4/dragonfly-amd64
2022-01-06T01:14:01-58a5fe9-ab70d7c/dragonfly-amd64
2022-01-06T00:26:47-4a88b4c-b5bfaf4/dragonfly-amd64
2022-01-05T21:22:03-4a88b4c-2b39d86/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-ab70d7c/dragonfly-amd64
2022-01-05T19:39:16-4a88b4c-002283e/dragonfly-amd64
2022-01-05T15:54:37-7a18f71-2c58bb2/dragonfly-amd64
2022-01-05T09:50:29-2aac4c0-2c58bb2/dragonfly-amd64
2022-01-05T01:53:39-2aac4c0-301db3f/dragonfly-amd64
2022-01-04T22:27:20-2aac4c0-f154f8b/dragonfly-amd64
2022-01-04T22:05:15-2aac4c0-e39ab9b/dragonfly-amd64
2022-01-04T20:37:27-2aac4c0-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-95b240b/dragonfly-amd64
2022-01-04T19:56:33-2b23ec4-1242f43/dragonfly-amd64
2022-01-04T19:30:38-995577f-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-95b240b/dragonfly-amd64
2022-01-04T17:38:42-b24dbb6-1242f43/dragonfly-amd64
2022-01-04T16:25:19-bc5b8e0-95b240b/dragonfly-amd64
2022-01-03T23:45:12-f07bd53-95b240b/dragonfly-amd64
2022-01-02T14:27:43-f07bd53-c886143/dragonfly-amd64
2022-01-02T14:27:32-f07bd53-d181885/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-6178d25/dragonfly-amd64
2021-12-29T23:18:47-f07bd53-1242f43/dragonfly-amd64
2021-12-29T20:20:32-56b50be-6178d25/dragonfly-amd64
2021-12-29T19:23:11-56b50be-8a306e2/dragonfly-amd64
2021-12-29T17:53:46-56b50be-91e7821/dragonfly-amd64
2021-12-29T04:10:07-ee35583-91e7821/dragonfly-amd64
2021-12-29T01:44:55-ee35583-a78532a/dragonfly-amd64
2021-12-28T17:18:11-ee35583-b357b05/dragonfly-amd64
2021-12-23T20:03:38-faa0890-b357b05/dragonfly-amd64
2021-12-23T19:58:13-faa0890-0e4b878/dragonfly-amd64
2021-12-23T18:58:36-faa0890-af3b8cf/dragonfly-amd64
2021-12-23T17:27:50-faa0890-ed766b6/dragonfly-amd64
2021-12-22T23:31:40-faa0890-d2ce939/dragonfly-amd64
2021-12-22T20:47:09-faa0890-8cfcee1/dragonfly-amd64
2021-12-22T18:43:55-faa0890-0f3becf/dragonfly-amd64
2021-12-22T00:11:21-91908ba-90fb5a4/dragonfly-amd64
2021-12-21T23:09:59-91908ba-ebac50e/dragonfly-amd64
2021-12-21T22:12:49-91908ba-a1ce550/dragonfly-amd64
2021-12-21T19:17:52-dc59b70-a1ce550/dragonfly-amd64
2021-12-21T17:58:16-dc59b70-2d1d548/dragonfly-amd64
2021-12-21T15:58:14-983ac79-2d1d548/dragonfly-amd64
2021-12-21T14:25:10-49c3839-de690c2/dragonfly-amd64
2021-12-21T14:25:10-49c3839-2d1d548/dragonfly-amd64
2021-12-21T03:55:43-d4a18cb-2d1d548/dragonfly-amd64
2021-12-21T03:54:36-d4a18cb-b5e0654/dragonfly-amd64
2021-12-21T01:51:23-d4a18cb-4dfbb89/dragonfly-amd64
2021-12-21T01:10:17-d4a18cb-e087949/dragonfly-amd64
2021-12-21T00:00:05-d4a18cb-1555062/dragonfly-amd64
2021-12-20T23:46:23-c471c5c-1555062/dragonfly-amd64
2021-12-20T23:32:14-c471c5c-6713b5d/dragonfly-amd64
2021-12-20T15:13:56-4e7a786-7c94355/dragonfly-amd64
2021-12-20T15:13:38-4e7a786-a2004de/dragonfly-amd64
2021-12-19T20:16:45-4e7a786-87b2a54/dragonfly-amd64
2021-12-17T16:05:31-4e7a786-c5fee93/dragonfly-amd64
2021-12-17T03:59:09-4e7a786-5299390/dragonfly-amd64
2021-12-17T03:28:06-4e7a786-3c49583/dragonfly-amd64
2021-12-17T02:00:14-4e7a786-55a25aa/dragonfly-amd64
2021-12-17T01:59:32-4e7a786-63077bf/dragonfly-amd64
2021-12-17T00:14:53-4e7a786-2e6e9df/dragonfly-amd64
2021-12-16T23:52:17-4e7a786-0c24038/dragonfly-amd64
2021-12-16T22:41:31-4e7a786-1387b5e/dragonfly-amd64
2021-12-16T22:39:00-4e7a786-b1a53ec/dragonfly-amd64
2021-12-16T22:15:57-4e7a786-d01cd8f/dragonfly-amd64
2021-12-16T22:15:22-4e7a786-d93677a/dragonfly-amd64
2021-12-16T21:08:09-4e7a786-ae695cd/dragonfly-amd64
2021-12-16T21:02:37-4e7a786-ea26ce7/dragonfly-amd64
2021-12-16T15:44:40-4e7a786-1d3a5b4/dragonfly-amd64
2021-12-16T14:37:07-4e7a786-7f23145/dragonfly-amd64
2021-12-15T14:48:19-0c791db-b1c7703/dragonfly-amd64
2021-12-15T00:33:55-18bc0f9-9d0ca26/dragonfly-amd64
2021-12-14T22:42:47-18bc0f9-1540239/dragonfly-amd64
2021-12-08T17:24:46-0dd6a27-ac7e950/dragonfly-amd64
2021-12-07T03:44:02-0dd6a27-dc65c48/dragonfly-amd64
2021-02-18T17:57:36-a76efea/dragonfly-amd64-5_8
CC @golang/release; attn @tuxillo
The text was updated successfully, but these errors were encountered: