Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/internal/gcimporter: frequent "out of memory" failures on 32-bit builders #45931

Closed
bcmills opened this issue May 3, 2021 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented May 3, 2021

It's not clear to me whether these test failures are due to a problem in gcimporter itself, or due to memory pressure from concurrently-running tests for another package, but the failure rate for the x/tools repo on the linux-arm-scaleway builder especially is quite high, and the windows-386-2008 failure rate is at least high enough that I noticed it on a TryBot run (https://storage.googleapis.com/go-build-log/7f9febd4/windows-386-2008_2222fa56.log).

Perhaps gcimporter_test needs a lower concurrency limit on 32-bit platforms?

CC @stamblerre @griesemer

2021-05-03T16:38:18-062bf4e/linux-arm-scaleway
2021-05-03T16:36:24-3e17c62/linux-arm-scaleway
2021-04-30T20:08:34-7a6108e/linux-arm-scaleway
2021-04-30T19:49:23-edbe9be/linux-arm-scaleway
2021-04-30T16:06:43-9b9633e/linux-arm-scaleway
2021-04-30T14:46:28-291330a/linux-arm-scaleway
2021-04-29T19:34:21-aec1372/linux-arm-scaleway
2021-04-29T13:06:21-28c1392/linux-arm-scaleway
2021-04-28T21:52:02-800adbe/linux-arm-scaleway
2021-04-28T01:46:37-16b25d2/linux-arm-scaleway
2021-04-27T20:55:22-7c72a84/linux-arm-scaleway
2021-04-27T19:53:45-d0768c9/linux-arm-scaleway
2021-04-27T15:36:10-6397a11/linux-arm-scaleway
2021-04-27T13:49:53-7b9993c/linux-arm-scaleway
2021-04-27T12:58:28-fe1c548/linux-arm-scaleway
2021-04-27T04:02:17-735ed62/linux-arm-scaleway
2021-04-26T20:11:56-e3dc99f/linux-arm-scaleway
2021-04-26T16:33:32-cf354b6/linux-arm-scaleway
2021-04-26T15:43:18-7c93484/linux-arm-scaleway
2021-04-23T21:15:18-f74a669/linux-arm-scaleway
2021-04-23T19:11:18-e435455/linux-arm-scaleway
2021-04-23T17:35:25-f7e8e24/linux-arm-scaleway
2021-04-22T17:05:18-f946a15/linux-arm-scaleway
2021-04-22T04:15:51-4934781/linux-arm-scaleway
2021-04-22T02:57:26-b3e5b99/linux-arm-scaleway
2021-04-20T19:23:16-e74674a/linux-arm-scaleway
2021-04-20T15:21:37-a13dbf1/linux-arm-scaleway
2021-04-19T19:59:27-10909d8/linux-arm-scaleway
2021-04-19T17:36:27-1dce19d/linux-arm-scaleway
2021-04-16T22:42:28-07295ca/linux-arm-scaleway
2021-04-16T18:23:08-fe50371/linux-arm-scaleway
2021-04-15T23:21:34-cb5dc85/linux-arm-scaleway
2021-04-15T21:16:26-799b682/linux-arm-scaleway
2021-04-14T19:50:58-d1362d7/linux-arm-scaleway
2021-04-14T19:20:17-59a2b45/linux-arm-scaleway
2021-04-12T17:45:26-2140cce/linux-arm-scaleway
2021-04-12T17:45:26-2140cce/windows-386-2008
2021-04-09T16:31:56-ec686a2/linux-arm-scaleway
2021-04-08T20:55:59-0243799/linux-arm-scaleway
2021-04-08T19:22:05-be791d0/linux-arm-scaleway
2021-04-08T02:08:45-b261fe9/linux-arm-scaleway
2021-04-07T17:38:53-e78b40c/linux-arm-scaleway
2021-04-07T10:44:45-3ac76b8/linux-arm-scaleway
2021-04-06T19:05:28-c602466/linux-arm-scaleway
2021-04-06T19:05:28-c602466/netbsd-386-9_0
2021-04-06T13:38:10-35a9159/linux-arm-scaleway
2021-04-06T04:24:37-5d33438/linux-arm-scaleway
2021-04-05T22:17:18-11c3f83/linux-arm-scaleway
2021-03-12T18:55:53-8e4f4c8/windows-386-2008
2021-03-03T21:54:20-376db57/windows-386-2008

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 3, 2021
@bcmills bcmills added this to the Backlog milestone May 3, 2021
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label May 3, 2021
@bcmills bcmills added the Testing An issue that has been verified to require only test changes, not just a test failure. label May 3, 2021
@findleyr
Copy link
Contributor

findleyr commented May 3, 2021

It's not clear to me whether these test failures are due to a problem in gcimporter itself, or due to memory pressure from concurrently-running tests for another package

It may be the latter, considering that we're not seeing equivalent OOMing in go/internal/gcimporter.

Note that the Scaleway builders are slated for deprecation:
#45066

@bcmills
Copy link
Contributor Author

bcmills commented May 3, 2021

Note that the Scaleway builders are slated for deprecation:

Yep, but AFAIK the windows-386 builders aren't. 😅

@gopherbot
Copy link

Change https://golang.org/cl/322409 mentions this issue: internal/testenv: update linux-arm to linux-arm-scaleway

gopherbot pushed a commit to golang/tools that referenced this issue May 25, 2021
CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the
x/tools ExitIfSmallMachine guard was not updated accordingly, resulting
in consistent failures for some tests.

For golang/go#45931
For golang/go#46183
Updates golang/go#32834

Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
rinchsan pushed a commit to rinchsan/gosimports that referenced this issue May 30, 2021
CL 303230 renamed the linux-arm builder to linux-arm-scaleway, but the
x/tools ExitIfSmallMachine guard was not updated accordingly, resulting
in consistent failures for some tests.

For golang/go#45931
For golang/go#46183
Updates golang/go#32834

Change-Id: Ief5f17fc61cb38cf6b7cf63b6cd5e64f9d56261e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/322409
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@bcmills
Copy link
Contributor Author

bcmills commented Nov 2, 2021

Looks like this failure mode went away with the linux-arm-scaleway builder. (I don't know why we're not observing faillures on windows-386.)

greplogs --dashboard -md -l -e '(?ms)out of memory.*FAIL\s+golang\.org/x/tools/go/internal/gcimporter' --since=2021-05-04

2021-05-25T04:04:01-98f8936-c89f122/linux-arm-scaleway
2021-05-24T22:27:31-71e666b-c89f122/linux-arm-scaleway
2021-05-24T21:23:15-71e666b-3380b18/linux-arm-scaleway
2021-05-24T21:23:15-71e666b-32b73ae/linux-arm-scaleway
2021-05-24T20:57:01-d5449c0-32b73ae/linux-arm-scaleway
2021-05-24T20:38:17-d5449c0-15d9d4a/linux-arm-scaleway
2021-05-24T19:06:05-88a9bcc-873401d/linux-arm-scaleway
2021-05-24T18:10:39-88a9bcc-b836106/linux-arm-scaleway
2021-05-24T18:03:47-88a9bcc-a22e317/linux-arm-scaleway
2021-05-24T18:03:47-88a9bcc-305fa95/linux-arm-scaleway
2021-05-24T16:47:51-e64a37c-4356e7e/linux-arm-scaleway
2021-05-24T16:41:39-e64a37c-52d7033/linux-arm-scaleway
2021-05-24T16:41:39-e64a37c-305fa95/linux-arm-scaleway
2021-05-24T15:03:18-4061312-52d7033/linux-arm-scaleway
2021-05-23T03:57:00-4061312-cca23a7/linux-arm-scaleway
2021-05-23T03:57:00-4061312-305fa95/linux-arm-scaleway
2021-05-22T00:51:17-1e0c960-cca23a7/linux-arm-scaleway
2021-05-21T23:46:07-1e0c960-f87194c/linux-arm-scaleway
2021-05-21T23:19:31-1e0c960-217f5dd/linux-arm-scaleway
2021-05-21T22:39:16-2275bb5-217f5dd/linux-arm-scaleway
2021-05-21T22:34:38-2275bb5-3380b18/linux-arm-scaleway
2021-05-21T21:07:03-2275bb5-3c65644/linux-arm-scaleway
2021-05-21T20:38:43-2275bb5-76b2d6a/linux-arm-scaleway
2021-05-21T18:55:18-2275bb5-e4d7525/linux-arm-scaleway
2021-05-21T17:49:01-2275bb5-4fb10b2/linux-arm-scaleway
2021-05-21T16:51:34-2275bb5-3148694/linux-arm-scaleway
2021-05-21T13:21:00-2275bb5-7e63c8b/linux-arm-scaleway
2021-05-21T03:21:56-2275bb5-831573c/linux-arm-scaleway
2021-05-20T20:37:02-2275bb5-baa934d/linux-arm-scaleway
2021-05-20T20:37:02-2275bb5-baa934d/windows-386-2008
2021-05-20T18:25:17-2275bb5-9636878/linux-arm-scaleway
2021-05-20T18:25:17-2275bb5-7c692cc/linux-arm-scaleway
2021-05-20T17:40:58-3063790-7c692cc/linux-arm-scaleway
2021-05-20T17:06:05-3063790-ce9a3b7/linux-arm-scaleway
2021-05-20T16:41:09-3063790-f8be906/linux-arm-scaleway
2021-05-20T16:09:44-3063790-def5360/linux-arm-scaleway
2021-05-20T15:46:45-3063790-f07e4da/linux-arm-scaleway
2021-05-20T15:46:45-3063790-c7248a0/linux-arm-scaleway
2021-05-20T14:08:36-0886cdd-f07e4da/linux-arm-scaleway
2021-05-20T13:19:43-46e69bf-f07e4da/linux-arm-scaleway
2021-05-20T01:22:28-46e69bf-a8d8591/linux-arm-scaleway
2021-05-20T00:02:15-46e69bf-9636878/linux-arm-scaleway
2021-05-19T23:21:03-46e69bf-6c1c055/linux-arm-scaleway
2021-05-19T23:21:03-46e69bf-44a6805/linux-arm-scaleway
2021-05-19T23:21:03-46e69bf-04cd717/linux-arm-scaleway
2021-05-19T23:19:09-f803486-6c1c055/linux-arm-scaleway
2021-05-19T16:08:23-49064d2-6c1c055/linux-arm-scaleway
2021-05-19T16:08:23-49064d2-04cd717/linux-arm-scaleway
2021-05-19T15:20:08-a0f4b7b-6c1c055/linux-arm-scaleway
2021-05-19T14:12:35-a0f4b7b-658b5e6/linux-arm-scaleway
2021-05-19T14:12:35-a0f4b7b-44a6805/linux-arm-scaleway
2021-05-19T14:12:23-f451690-658b5e6/linux-arm-scaleway
2021-05-19T14:12:23-f451690-04cd717/linux-arm-scaleway
2021-05-19T13:04:45-17b3466-658b5e6/linux-arm-scaleway
2021-05-19T01:09:20-17b3466-15a374d/linux-arm-scaleway
2021-05-18T22:36:55-17b3466-eeadce2/linux-arm-scaleway
2021-05-18T20:04:57-17b3466-6d2ef2e/linux-arm-scaleway
2021-05-18T18:21:53-17b3466-690a8c3/linux-arm-scaleway
2021-05-18T18:21:53-17b3466-44a6805/linux-arm-scaleway
2021-05-18T18:21:53-17b3466-04cd717/linux-arm-scaleway
2021-05-18T15:00:55-6da3d7a-690a8c3/linux-arm-scaleway
2021-05-18T07:50:25-6da3d7a-8b0901f/linux-arm-scaleway
2021-05-18T02:39:04-6da3d7a-5e191f8/linux-arm-scaleway
2021-05-18T02:12:20-6da3d7a-bfe3573/linux-arm-scaleway
2021-05-18T02:12:20-6da3d7a-04cd717/linux-arm-scaleway
2021-05-17T22:19:03-8f301ca-bfe3573/linux-arm-scaleway
2021-05-17T17:18:20-8f301ca-b1aff42/linux-arm-scaleway
2021-05-17T17:18:20-8f301ca-44a6805/linux-arm-scaleway
2021-05-17T16:02:12-09ab05b-b1aff42/linux-arm-scaleway
2021-05-15T02:39:08-09ab05b-ce92a20/linux-arm-scaleway
2021-05-14T19:27:25-09ab05b-02699f8/linux-arm-scaleway
2021-05-14T16:42:01-09ab05b-3d324f1/linux-arm-scaleway
2021-05-14T15:35:28-09ab05b-a938e52/linux-arm-scaleway
2021-05-14T14:37:54-09ab05b-d137b74/linux-arm-scaleway
2021-05-14T07:53:46-09ab05b-c925e15/linux-arm-scaleway
2021-05-14T00:56:52-09ab05b-12d383c/linux-arm-scaleway
2021-05-13T21:16:06-09ab05b-b4833f7/linux-arm-scaleway
2021-05-13T18:59:27-09ab05b-7a7624a/linux-arm-scaleway
2021-05-13T17:35:42-09ab05b-cde2d85/linux-arm-scaleway
2021-05-13T17:35:42-09ab05b-04cd717/linux-arm-scaleway
2021-05-13T15:33:24-cd1be5d-cde2d85/linux-arm-scaleway
2021-05-13T14:52:20-cd1be5d-2a61b3c/linux-arm-scaleway
2021-05-13T13:20:04-cd1be5d-a63cded/linux-arm-scaleway
2021-05-13T13:20:04-cd1be5d-a63cded/netbsd-arm-bsiegert
2021-05-13T13:20:04-cd1be5d-04cd717/linux-arm-scaleway
2021-05-13T12:32:21-57c3a74-a63cded/linux-arm-scaleway
2021-05-13T09:12:33-57c3a74-0fa2302/linux-arm-scaleway
2021-05-12T22:01:29-57c3a74-6db7480/linux-arm-scaleway
2021-05-12T22:01:29-57c3a74-04cd717/linux-arm-scaleway
2021-05-12T20:20:11-9dfac01-6db7480/linux-arm-scaleway
2021-05-12T17:32:39-9dfac01-f93b951/linux-arm-scaleway
2021-05-12T16:42:30-9dfac01-3b321a9/linux-arm-scaleway
2021-05-12T16:42:30-9dfac01-04cd717/linux-arm-scaleway
2021-05-12T15:23:09-be4aaae-0388670/linux-arm-scaleway
2021-05-12T02:04:57-be4aaae-1a0ea1a/linux-arm-scaleway
2021-05-11T21:22:08-be4aaae-9995c6b/linux-arm-scaleway
2021-05-11T20:42:14-be4aaae-9b84814/linux-arm-scaleway
2021-05-11T20:42:14-be4aaae-04cd717/linux-arm-scaleway
2021-05-11T18:22:54-cd1d088-9b84814/linux-arm-scaleway
2021-05-11T17:48:41-cd1d088-2520e72/linux-arm-scaleway
2021-05-11T17:48:41-cd1d088-04cd717/linux-arm-scaleway
2021-05-11T14:51:35-2db0265-2520e72/linux-arm-scaleway
2021-05-11T14:29:16-9cddb0e-2520e72/linux-arm-scaleway
2021-05-11T14:29:16-9cddb0e-07d8cba/linux-arm-scaleway
2021-05-11T14:29:16-9cddb0e-04cd717/linux-arm-scaleway
2021-05-11T02:46:21-79d39ff-326a792/linux-arm-scaleway
2021-05-10T23:42:56-79d39ff-5c48951/linux-arm-scaleway
2021-05-10T23:22:37-79d39ff-dc50683/linux-arm-scaleway
2021-05-10T23:22:37-79d39ff-07d8cba/linux-arm-scaleway
2021-05-10T23:22:37-79d39ff-04cd717/linux-arm-scaleway
2021-05-10T23:21:19-fa05545-dc50683/linux-arm-scaleway
2021-05-10T21:57:30-5a66778-dc50683/linux-arm-scaleway
2021-05-10T21:56:33-0185c7e-dc50683/linux-arm-scaleway
2021-05-10T21:56:33-0185c7e-07d8cba/linux-arm-scaleway
2021-05-10T21:56:33-0185c7e-04cd717/linux-arm-scaleway
2021-05-10T19:19:34-ecb7392-dc50683/linux-arm-scaleway
2021-05-10T18:10:43-ecb7392-73d5aef/linux-arm-scaleway
2021-05-10T17:21:32-ecb7392-d9e068d/linux-arm-scaleway
2021-05-10T15:53:43-ecb7392-deb3403/linux-arm-scaleway
2021-05-10T15:11:50-ecb7392-2870259/linux-arm-scaleway
2021-05-10T15:11:50-ecb7392-04cd717/linux-arm-scaleway
2021-05-10T13:16:56-f05e912-2870259/linux-arm-scaleway
2021-05-09T17:07:22-f05e912-c14ecac/linux-arm-scaleway
2021-05-09T02:47:29-f05e912-5203357/linux-arm-scaleway
2021-05-08T17:03:18-f05e912-b38b1b2/linux-arm-scaleway
2021-05-08T14:59:49-f05e912-68327e1/linux-arm-scaleway
2021-05-08T05:57:31-f05e912-4c8f48e/linux-arm-scaleway
2021-05-08T05:15:48-f05e912-9d0819b/linux-arm-scaleway
2021-05-08T03:27:59-f05e912-b211fe0/linux-arm-scaleway
2021-05-07T21:35:41-f05e912-f24eac4/linux-arm-scaleway
2021-05-07T21:01:09-f05e912-3980c4d/linux-arm-scaleway
2021-05-07T20:56:39-f05e912-07d8cba/linux-arm-scaleway
2021-05-07T18:14:25-c0140e8-af6123a/linux-arm-scaleway
2021-05-07T17:43:31-c0140e8-b44c78b/linux-arm-scaleway
2021-05-07T16:08:24-c0140e8-9bfa6f7/linux-arm-scaleway
2021-05-07T14:16:52-c0140e8-a0eb662/linux-arm-scaleway
2021-05-07T03:55:29-c0140e8-f5423ea/linux-arm-scaleway
2021-05-07T02:45:26-c0140e8-832c70e/linux-arm-scaleway
2021-05-07T01:42:23-c0140e8-a11a171/linux-arm-scaleway
2021-05-06T20:30:00-c0140e8-4dbad79/linux-arm-scaleway
2021-05-06T19:28:34-c0140e8-90d6bbb/linux-arm-scaleway
2021-05-06T19:28:34-c0140e8-04cd717/linux-arm-scaleway
2021-05-06T18:08:01-d1ea2c7-402f177/linux-arm-scaleway
2021-05-06T17:09:23-d1ea2c7-51ff3a6/linux-arm-scaleway
2021-05-06T16:28:56-d1ea2c7-5f9fe47/linux-arm-scaleway
2021-05-06T16:00:55-d1ea2c7-6c591f7/linux-arm-scaleway
2021-05-06T15:33:43-d1ea2c7-54e20b5/linux-arm-scaleway
2021-05-06T13:39:39-d1ea2c7-d2fd503/linux-arm-scaleway
2021-05-06T12:33:02-d1ea2c7-ba0f8ce/linux-arm-scaleway
2021-05-06T03:15:21-d1ea2c7-87ffba3/linux-arm-scaleway
2021-05-06T03:15:21-d1ea2c7-1108cbe/linux-arm-scaleway
2021-05-06T03:14:34-08a4f34-87ffba3/linux-arm-scaleway
2021-05-06T03:14:34-08a4f34-1108cbe/linux-arm-scaleway
2021-05-06T02:57:18-f4a4129-1108cbe/linux-arm-scaleway
2021-05-06T02:21:55-dd255f2-1108cbe/linux-arm-scaleway
2021-05-05T22:35:07-dd255f2-cf73f1a/linux-arm-scaleway
2021-05-05T21:37:16-1949673-cf73f1a/linux-arm-scaleway
2021-05-05T21:11:25-68c6cab-ba7cac4/linux-arm-scaleway
2021-05-05T21:05:09-68c6cab-87ffba3/linux-arm-scaleway
2021-05-05T20:58:39-68c6cab-cf73f1a/linux-arm-scaleway
2021-05-05T20:44:55-68c6cab-15557af/linux-arm-scaleway
2021-05-05T19:17:46-250398d-543e098/linux-arm-scaleway
2021-05-05T18:14:49-250398d-784ef4c/linux-arm-scaleway
2021-05-05T16:20:51-250398d-ce04f86/linux-arm-scaleway
2021-05-05T16:20:51-250398d-18e666b/linux-arm-scaleway
2021-05-05T04:48:30-7cab0ef-9e0facd/linux-arm-scaleway
2021-05-05T01:48:39-7cab0ef-caf4c94/linux-arm-scaleway
2021-05-05T01:45:45-7cab0ef-4df662f/linux-arm-scaleway
2021-05-04T23:35:34-f03daea-137be77/linux-arm-scaleway
2021-05-04T22:12:42-f03daea-62a87f6/linux-arm-scaleway
2021-05-04T21:16:16-f03daea-cd28ca3/linux-arm-scaleway
2021-05-04T20:50:35-f03daea-d19e549/linux-arm-scaleway
2021-05-04T20:41:53-f03daea-ce04f86/linux-arm-scaleway
2021-05-04T20:18:55-f03daea-10a082a/linux-arm-scaleway
2021-05-04T19:12:24-f03daea-c0a7ecf/linux-arm-scaleway
2021-05-04T19:12:24-f03daea-8c3d217/linux-arm-scaleway

@bcmills bcmills closed this as completed Nov 2, 2021
@golang golang locked and limited conversation to collaborators Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

3 participants