Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls/internal/regtest: TestGCDetails is flaky on darwin #44099

Closed
findleyr opened this issue Feb 4, 2021 · 9 comments
Closed

x/tools/gopls/internal/regtest: TestGCDetails is flaky on darwin #44099

findleyr opened this issue Feb 4, 2021 · 9 comments
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@findleyr
Copy link
Contributor

findleyr commented Feb 4, 2021

Recent example:
https://build.golang.org/log/840ef7221cf0147181be16e21a553166c3a9c546

(or click any of the many red buttons in the darwin columns at https://build.golang.org/?repo=golang.org%2fx%2ftools#short :( )

This feature is os-dependent, so it would not be surprising if this is a real bug. Unfortunately, I ran it on gomote and was not able to repro after dozens of attempts. @stamblerre also ran it locally on darwin 1000x and was not able to repro.

CC @heschik

@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Feb 4, 2021
@findleyr findleyr changed the title x/tools/gopls/regtests: x/tools/gopls/regtests: TestGCDetails is flaky on darwin Feb 4, 2021
@gopherbot gopherbot added this to the Unreleased milestone Feb 4, 2021
@findleyr findleyr changed the title x/tools/gopls/regtests: TestGCDetails is flaky on darwin x/tools/gopls/internal/regtest: TestGCDetails is flaky on darwin Feb 4, 2021
@bcmills
Copy link
Contributor

bcmills commented Feb 4, 2021

Is there anything Darwin-specific about the failure mode? I'm seeing failures on many of the slow or low-memory builders, too, so it may depend on timing or memory pressure.

2021-02-03T19:28:29-ed97357/darwin-amd64-10_15
2021-02-03T19:28:29-ed97357/darwin-amd64-nocgo
2021-02-03T19:28:29-ed97357/plan9-arm
2021-02-02T23:37:55-e7dfe02/darwin-amd64-10_14
2021-02-02T23:37:55-e7dfe02/darwin-amd64-10_15
2021-02-02T23:37:55-e7dfe02/netbsd-arm64-bsiegert
2021-02-02T23:37:55-e7dfe02/plan9-arm
2021-02-02T20:40:51-51ce837/darwin-amd64-10_15
2021-02-02T20:40:51-51ce837/netbsd-arm64-bsiegert
2021-02-02T17:43:53-5659e49/darwin-amd64-10_14
2021-02-02T17:43:53-5659e49/darwin-amd64-10_15
2021-02-02T17:43:53-5659e49/netbsd-arm64-bsiegert
2021-02-02T16:03:30-ddc05f8/openbsd-386-62
2021-02-02T11:32:59-c3402e3/darwin-amd64-10_14
2021-02-02T11:32:59-c3402e3/netbsd-arm64-bsiegert
2021-02-01T22:49:46-2ab2386/netbsd-arm64-bsiegert
2021-02-01T17:36:10-9b8df07/darwin-amd64-10_14
2021-02-01T17:36:10-9b8df07/linux-ppc64-buildlet
2021-02-01T17:36:10-9b8df07/netbsd-arm64-bsiegert
2021-02-01T17:10:16-f1f686b/darwin-amd64-10_15
2021-02-01T17:07:00-d8a2a07/darwin-amd64-10_14
2021-02-01T17:07:00-d8a2a07/darwin-amd64-10_15
2021-02-01T17:07:00-d8a2a07/darwin-amd64-nocgo
2021-02-01T17:07:00-d8a2a07/openbsd-386-62
2021-02-01T16:52:01-19db92e/darwin-amd64-nocgo
2021-02-01T16:52:01-19db92e/linux-ppc64-buildlet
2021-02-01T16:52:01-19db92e/netbsd-arm64-bsiegert
2021-01-29T18:11:47-0cef57b/darwin-amd64-10_14
2021-01-29T18:11:47-0cef57b/darwin-amd64-10_15
2021-01-29T18:11:47-0cef57b/darwin-amd64-nocgo
2021-01-29T18:11:47-0cef57b/dragonfly-amd64
2021-01-29T18:11:47-0cef57b/netbsd-arm64-bsiegert
2021-01-29T18:11:47-0cef57b/plan9-arm
2021-01-28T15:45:56-db4c57d/darwin-amd64-10_14
2021-01-28T15:45:56-db4c57d/darwin-amd64-nocgo
2021-01-28T15:45:56-db4c57d/netbsd-arm64-bsiegert
2021-01-28T15:45:56-db4c57d/plan9-arm
2021-01-28T15:26:31-f871472/plan9-arm
2021-01-26T13:52:46-c2bea79/dragonfly-amd64
2021-01-26T13:52:46-c2bea79/plan9-arm
2021-01-26T00:28:59-514964b/dragonfly-amd64
2021-01-26T00:28:59-514964b/plan9-arm
2021-01-25T18:56:18-68bf78a/plan9-arm
2021-01-25T18:11:30-4922717/plan9-arm
2021-01-22T20:33:18-2972602/plan9-arm
2021-01-22T19:37:57-e13398c/plan9-arm
2021-01-22T17:08:14-cf1022a/plan9-arm
2021-01-22T00:09:48-87bc10f/plan9-arm
2021-01-21T18:28:06-bec622c/plan9-arm
2021-01-20T16:51:16-7e51fbd/plan9-arm
2021-01-19T22:25:03-fe37c9e/plan9-arm
2021-01-19T19:51:17-a46736d/plan9-arm
2021-01-15T20:22:50-e0d2015/plan9-arm
2021-01-14T06:55:38-d78b04b/plan9-arm
2021-01-14T06:03:00-1bdb73f/plan9-arm
2021-01-13T18:03:00-f964368/plan9-arm
2021-01-12T23:06:58-8b4aab6/plan9-arm
2021-01-12T21:35:07-9a6582c/plan9-arm
2021-01-12T21:35:00-f618651/plan9-arm
2021-01-12T21:27:48-7646fae/plan9-arm
2021-01-12T21:27:35-45115c1/plan9-arm
2021-01-12T18:33:07-1e6ecd4/plan9-arm
2021-01-11T22:19:46-d33bae4/plan9-arm
2021-01-08T19:58:28-1b1bb64/plan9-386-0intro
2021-01-08T19:58:28-1b1bb64/plan9-arm
2020-12-30T16:32:07-4a19ffb/plan9-arm
2020-12-30T15:44:40-ef3185b/netbsd-arm64-bsiegert
2020-12-30T15:44:40-ef3185b/plan9-arm
2020-12-29T22:18:35-b841374/plan9-arm
2020-12-29T01:39:31-929a849/plan9-arm
2020-12-29T01:23:49-fbbba25/plan9-arm
2020-12-28T20:48:37-84d76fe/plan9-arm
2020-12-28T18:51:49-0661ca7/plan9-arm
2020-12-28T17:52:27-13ff221/plan9-arm
2020-12-28T17:44:24-0f6027f/plan9-arm
2020-12-28T16:22:55-34cd474/plan9-arm
2020-12-26T21:56:59-b1c9089/plan9-arm
2020-12-24T04:30:29-2b0845d/plan9-arm
2020-12-24T00:32:54-57089f8/plan9-arm
2020-12-23T23:03:58-834755c/plan9-arm
2020-12-23T22:53:30-bdbb3c9/plan9-arm
2020-12-23T22:32:11-3e0a2b7/plan9-arm
2020-12-23T20:03:49-f6952e4/plan9-arm
2020-12-23T17:49:54-9cbb1ef/plan9-arm
2020-12-23T01:07:50-3fa0e8f/plan9-arm
2020-12-22T16:32:15-f2e330f/plan9-arm
2020-12-21T20:10:19-1965356/linux-ppc64-buildlet
2020-12-21T20:10:19-1965356/plan9-arm
2020-12-21T19:58:07-b57d1c5/linux-ppc64-buildlet
2020-12-18T02:47:24-ae774e9/linux-ppc64-buildlet
2020-12-18T02:47:24-ae774e9/plan9-arm
2020-12-17T23:51:54-5b06639/linux-ppc64-buildlet
2020-12-17T23:51:54-5b06639/plan9-arm
2020-12-17T23:10:36-11a5667/plan9-arm
2020-12-17T22:11:02-5b43ef9/linux-ppc64-buildlet
2020-12-17T16:56:54-008e477/plan9-arm
2020-12-17T16:18:45-48e5bd1/linux-ppc64-buildlet
2020-12-17T16:18:45-48e5bd1/linux-ppc64le-buildlet
2020-12-15T19:20:05-fa10ef0/linux-ppc64-buildlet
2020-12-11T18:50:31-d93e913/plan9-arm
2020-12-11T02:55:43-abf6a1d/netbsd-arm64-bsiegert
2020-12-11T02:55:43-abf6a1d/plan9-arm
2020-12-10T16:13:38-b71f123/netbsd-arm64-bsiegert
2020-12-08T23:30:53-a543418/plan9-arm
2020-12-08T22:50:42-ef0c635/netbsd-arm64-bsiegert
2020-12-08T21:18:28-de58e7c/netbsd-arm64-bsiegert
2020-12-08T19:40:06-5737bec/netbsd-arm64-bsiegert
2020-12-08T18:36:58-cc33081/windows-arm-zx2c4
2020-12-08T06:23:17-e652b2f/netbsd-arm-bsiegert
2020-12-08T06:23:17-e652b2f/windows-arm-zx2c4
2020-12-08T00:26:38-66f9315/netbsd-arm-bsiegert
2020-12-08T00:26:38-66f9315/windows-arm-zx2c4
2020-12-07T20:43:33-a835c87/netbsd-arm-bsiegert
2020-12-07T20:43:33-a835c87/windows-arm-zx2c4
2020-12-07T16:54:04-05664e2/linux-ppc64le-buildlet
2020-12-06T23:03:34-368bee8/netbsd-arm-bsiegert
2020-12-06T23:03:34-368bee8/windows-arm-zx2c4
2020-12-04T22:23:52-6543527/netbsd-arm-bsiegert
2020-12-04T22:23:52-6543527/plan9-arm
2020-12-04T22:23:52-6543527/windows-arm-zx2c4
2020-12-04T21:35:57-ab68579/darwin-amd64-nocgo
2020-12-04T21:35:57-ab68579/netbsd-arm64-bsiegert
2020-12-04T21:17:34-9a0e0bb/netbsd-arm-bsiegert
2020-12-04T21:17:34-9a0e0bb/netbsd-arm64-bsiegert
2020-12-04T16:22:04-73cf035/netbsd-arm64-bsiegert
2020-12-04T16:22:04-73cf035/windows-arm-zx2c4
2020-12-04T06:28:50-5457889/netbsd-arm-bsiegert
2020-12-04T06:28:50-5457889/windows-arm-zx2c4
2020-12-04T05:15:55-214da9f/netbsd-arm-bsiegert
2020-12-03T23:01:54-3949734/windows-arm-zx2c4
2020-12-02T20:03:35-bef1c47/netbsd-arm-bsiegert
2020-12-02T20:03:35-bef1c47/plan9-arm
2020-12-02T10:05:33-7534955/netbsd-arm-bsiegert
2020-12-02T10:05:33-7534955/plan9-arm
2020-12-02T10:05:33-7534955/windows-arm-zx2c4
2020-12-01T21:08:46-92771a2/darwin-amd64-10_15
2020-12-01T21:08:46-92771a2/linux-ppc64-buildlet
2020-12-01T21:08:46-92771a2/netbsd-arm-bsiegert
2020-12-01T21:08:46-92771a2/plan9-arm
2020-12-01T21:08:46-92771a2/windows-arm-zx2c4
2020-12-01T05:30:39-1937afe/netbsd-arm-bsiegert
2020-12-01T05:30:39-1937afe/windows-arm-zx2c4
2020-12-01T03:00:18-7470481/windows-arm-zx2c4
2020-12-01T01:59:40-7e522c8/linux-ppc64-buildlet
2020-11-30T22:00:05-fd5f293/windows-386-2008
2020-11-30T22:00:05-fd5f293/windows-arm-zx2c4
2020-11-30T18:27:53-ccae4fb/plan9-arm
2020-11-30T17:56:50-e7a5458/netbsd-arm-bsiegert
2020-11-30T17:56:50-e7a5458/openbsd-386-64
2020-11-25T23:11:58-b5590de/windows-arm-zx2c4
2020-11-25T22:00:14-2e53cd6/openbsd-386-64
2020-11-24T20:20:34-299f270/linux-386-clang
2020-11-24T20:20:34-299f270/netbsd-amd64-9_0
2020-11-24T17:38:11-208275b/freebsd-386-11_2
2020-11-24T17:38:11-208275b/plan9-arm
2020-11-24T17:38:01-325b08a/plan9-arm
2020-11-24T16:59:54-ed677e9/plan9-arm
2020-11-24T11:59:21-2c860bd/plan9-arm
2020-11-24T00:57:43-911501b/plan9-arm
2020-11-23T15:28:13-45586dd/netbsd-arm-bsiegert
2020-11-13T20:20:37-1643af1/netbsd-arm-bsiegert
2020-11-11T22:45:57-41a3a58/dragonfly-amd64
2020-11-11T13:33:15-69daaf9/dragonfly-amd64
2020-11-10T20:14:00-7099162/dragonfly-amd64
2020-11-10T19:54:50-092357f/dragonfly-amd64
2020-11-10T18:16:08-f6c1dd6/dragonfly-amd64
2020-11-10T12:42:07-079ba7b/dragonfly-amd64
2020-11-09T18:20:53-3db8fd2/darwin-amd64-10_15
2020-11-05T00:16:34-bc3cf28/plan9-arm
2020-11-03T23:54:15-b653051/plan9-arm
2020-11-02T21:20:25-f46e424/plan9-arm
2020-10-31T02:16:30-582c62e/plan9-arm
2020-10-30T20:42:49-4fc0492/plan9-arm
2020-10-30T01:04:31-2feb2bb/netbsd-arm-bsiegert
2020-10-30T01:04:31-2feb2bb/plan9-arm
2020-10-29T19:59:06-061905c/plan9-arm
2020-10-29T19:53:23-c86e623/plan9-arm
2020-10-29T13:53:53-690a3c2/plan9-arm
2020-10-28T22:47:54-2c11599/plan9-arm
2020-10-28T02:59:01-8cd080b/plan9-arm
2020-10-27T23:31:11-0b86805/darwin-amd64-10_15
2020-10-27T23:31:11-0b86805/plan9-arm
2020-10-27T18:00:23-8dabb74/plan9-arm
2020-10-26T22:31:36-e84cfc6/plan9-arm
2020-10-23T17:41:41-c8cfbd0/plan9-arm
2020-10-23T15:00:57-2f4fa18/plan9-arm
2020-10-22T21:58:48-ffe8bce/plan9-arm
2020-10-22T21:19:33-13b3b30/plan9-arm
2020-10-22T03:59:29-9cf592e/plan9-arm
2020-10-21T20:36:10-6652d1f/plan9-arm
2020-10-21T17:10:30-d105bfa/plan9-arm
2020-10-21T17:08:55-2cbe144/plan9-arm
2020-10-21T12:24:55-2be66b6/plan9-arm
2020-10-21T12:08:33-c12dc66/plan9-arm
2020-10-21T00:02:07-d49c4ed/netbsd-arm-bsiegert
2020-10-21T00:02:07-d49c4ed/plan9-arm
2020-10-19T17:57:15-b894a32/plan9-arm
2020-10-19T16:07:06-0a3dccc/darwin-amd64-10_15
2020-10-17T00:14:24-6003fad/plan9-arm
2020-10-14T23:16:27-1610a49/plan9-arm
2020-10-13T20:10:25-64a9e34/plan9-arm
2020-10-13T17:44:05-d88ec18/plan9-arm
2020-10-13T05:33:47-2db1cd7/plan9-arm
2020-10-13T01:55:53-ec925d8/plan9-arm
2020-10-12T19:26:20-5bd0538/plan9-arm
2020-10-11T14:58:50-ed2f502/plan9-arm
2020-10-09T16:22:40-fcf8212/plan9-arm
2020-10-09T01:09:51-9ab7e51/plan9-arm
2020-10-09T01:01:41-7ddb464/plan9-arm
2020-10-08T18:49:44-d01b322/plan9-arm
2020-10-08T17:44:24-ffec978/plan9-arm
2020-10-08T02:52:39-9df6960/plan9-arm
2020-10-07T03:26:33-0806396/plan9-arm
2020-10-07T01:32:53-d5f20aa/plan9-arm
2020-10-06T22:15:05-454bc3d/plan9-arm
2020-10-06T18:39:45-23a3aa1/plan9-arm
2020-10-06T17:02:58-90a82dd/plan9-arm
2020-10-05T18:50:03-576e169/plan9-arm
2020-09-30T20:43:29-2e5f0cf/plan9-arm
2020-09-30T20:10:28-a6f32d1/plan9-arm
2020-09-30T19:40:55-a44386f/plan9-arm
2020-09-30T16:38:20-66e72d0/linux-ppc64le-buildlet
2020-09-30T16:38:20-66e72d0/plan9-arm
2020-09-29T17:13:17-ffa3839/plan9-arm

@findleyr
Copy link
Contributor Author

findleyr commented Feb 4, 2021

@bcmills Yes, that is certainly possible. Darwin in particular has, in our experience, highly variable timing (we think due to kernel locking). It's odd that I couldn't reproduce on gomote though, given the frequency of flakes -- perhaps I just got lucky.

For now, let me double the timeout for this one test case to see if the flakes reduce.

@gopherbot
Copy link

Change https://golang.org/cl/289690 mentions this issue: gopls/internal/regtest/codelens: increase the timeout on TestGCDetails

@gopherbot
Copy link

Change https://golang.org/cl/289692 mentions this issue: gopls/internal/regtest: only run in 'singleton' mode on certain GOOS

@stamblerre stamblerre modified the milestones: Unreleased, gopls/unplanned Feb 4, 2021
gopherbot pushed a commit to golang/tools that referenced this issue Feb 4, 2021
Experiment with increasing the timeout for TestGCDetails, given how its
flakiness correlates with slow builders.

For golang/go#44099

Change-Id: I27a8732256a77c3d9ce2a1b1f75ce5d0fbc11e67
Reviewed-on: https://go-review.googlesource.com/c/tools/+/289690
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
gopherbot pushed a commit to golang/tools that referenced this issue Feb 4, 2021
The gopls regtests have highly variable performance on certain operating
systems, we believe due to kernel contention. Flakes are a problem, and
it's better to have coverage of our default execution mode than no
coverage at all.

For golang/go#44099
For golang/go#42789

Change-Id: Ie9f3f93bb950930401aac3ce55cdffb889d0f0e1
Reviewed-on: https://go-review.googlesource.com/c/tools/+/289692
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/291232 mentions this issue: gopls/internl/regtest: skip known flakes

@gopherbot
Copy link

Change https://golang.org/cl/291232 mentions this issue: gopls/internal/regtest: skip known flakes

gopherbot pushed a commit to golang/tools that referenced this issue Feb 12, 2021
This test is flaking at a relatively high rate on darwin and freebsd.
Skipping it is better than having a build dashboard with many known
failures.

Perhaps we could limit this skip to certain GOOS, but for now let's skip
it entirely until we understand the problem better.

For golang/go#44099

Change-Id: I58703b2db0e5768f75758080d07f9d29b8b5d661
Reviewed-on: https://go-review.googlesource.com/c/tools/+/291232
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
@stamblerre stamblerre added this to To Do in gopls on-deck Feb 28, 2021
@gopherbot
Copy link

Change https://golang.org/cl/304169 mentions this issue: internal/lsp: hold the gc details lock when storing diagnostics

gopherbot pushed a commit to golang/tools that referenced this issue Mar 25, 2021
At long last, with Pontus's help reproing on Github actions, we have
tracked down the race to the gc_details diagnostics.

Since toggling gc_details does not increment the snapshot ID, we have to
use careful locking to ensure that the gc_details diagnostics we store
are consistent with the current state of the gc_details toggle.

Updates golang/go#44099
Fixes golang/go#44826

Change-Id: I7b9108a829c98a84360c9012c1b60f4990839b5a
Reviewed-on: https://go-review.googlesource.com/c/tools/+/304169
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/305409 mentions this issue: gopls/internal/regtest: re-enable the gc_details regtest for -short

gopherbot pushed a commit to golang/tools that referenced this issue Mar 30, 2021
My hope is that CL 304169 resolved this flake, but we'll have to see.

For golang/go#44099

Change-Id: Iac82dc24167daacd0361eac1a5567048a7ecf11c
Reviewed-on: https://go-review.googlesource.com/c/tools/+/305409
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@findleyr
Copy link
Contributor Author

This is no longer flaking.

gopls on-deck automation moved this from To Do to Done Apr 26, 2021
@golang golang locked and limited conversation to collaborators Apr 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
No open projects
Development

No branches or pull requests

4 participants