Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls/internal/regtest/completion: sandbox cleanup errors with "directory not empty" in TestPostfixSnippetCompletion #50707

Closed
bcmills opened this issue Jan 20, 2022 · 6 comments
Assignees
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Jan 20, 2022

closing test runner: errors closing the test runner:
	error(s) cleaning sandbox: cleaning modcache: <nil>; removing files: unlinkat /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/gopls-regtest-664105203/TestPostfixSnippetCompletion/singleton: directory not empty
FAIL	golang.org/x/tools/gopls/internal/regtest/completion	13.493s

greplogs --dashboard -md -l -e 'error\(s\) cleaning sandbox.*directory not empty'

2022-01-20T01:03:13-7c251d6-e7d5857/darwin-amd64-10_15
2022-01-13T20:54:45-3f6aab1-9de1ac6/darwin-amd64-10_15
2022-01-07T05:34:58-351aaa6-40afced/netbsd-amd64-9_0-n1
2021-12-15T20:26:19-c6ae451-07ed86c/darwin-arm64-11_0-toothrot
2021-11-29T20:12:13-6e52f51-f598e29/netbsd-386-9_0-n2d
2021-11-25T04:02:39-cb80a01-45bae64/darwin-amd64-10_14
2021-11-13T03:33:55-49ce184-c239790/darwin-arm64-11_0-toothrot
2021-11-06T16:43:43-0c60b7c-3544082/linux-amd64-longtest
2021-11-03T20:51:16-84e69e7-88407a8/darwin-arm64-11_0-toothrot
2021-11-03T19:38:32-012c320-88407a8/darwin-amd64-10_14
2021-11-02T15:33:01-4d06874-da7173a/darwin-arm64-11_0-toothrot
2021-10-29T21:19:39-a2be0cd-994049a/darwin-amd64-10_14
2021-10-20T07:21:22-4ea6123-d9421ce/darwin-arm64-11_0-toothrot
2021-10-15T14:09:01-98f6e03-2ac3bdf/darwin-amd64-10_14
2021-10-01T15:56:03-6f5fd9b-04242ac/darwin-amd64-10_14
2021-09-21T18:37:00-0d12d39-552410f/darwin-arm64-11_0-toothrot
2021-08-15T19:56:18-a55d515-717894c/darwin-arm64-11_0-toothrot
2021-08-11T15:16:17-5f5a173-fa6aa87/darwin-amd64-10_14
2021-08-06T19:38:52-d529aec-fa6aa87/darwin-amd64-10_14
2021-08-03T15:00:47-3395cb0-37c117f/darwin-arm64-11_0-toothrot
2021-07-26T20:19:17-4ad98e9-ed8cbbc/darwin-amd64-10_14
2021-07-15T13:35:41-6e9046b-21a04e3/darwin-arm64-11_0-toothrot
2021-07-14T22:42:09-0cf4e27-bc51e93/darwin-arm64-11_0-toothrot
2021-07-10T00:32:22-5b540d3-fb052db/darwin-arm64-11_0-toothrot
2021-07-08T21:02:30-64bd808-296ddf2/darwin-arm64-11_0-toothrot
2021-07-02T16:14:50-7edcfe5-cb4cd9e/darwin-arm64-11_0-toothrot
2021-07-01T08:39:23-20dafe5-cb4cd9e/netbsd-386-9_0
2021-06-14T21:46:05-4b484fb-0fd20ed/dragonfly-amd64
2021-06-11T15:30:39-490eac8-7677616/darwin-amd64-10_14
2021-06-04T20:34:05-1225b6f-831f937/darwin-arm64-11_0-toothrot
2021-05-24T18:10:39-88a9bcc-b836106/darwin-arm64-11_0-toothrot
2021-05-21T23:46:07-1e0c960-f87194c/netbsd-arm64-bsiegert
2021-05-19T16:08:23-49064d2-04cd717/darwin-arm64-11_0-toothrot
2021-05-19T14:12:35-a0f4b7b-44a6805/netbsd-386-9_0
2021-05-19T14:12:35-a0f4b7b-04cd717/darwin-arm64-11_0-toothrot
2021-05-03T20:05:58-19b1717-9f34703/darwin-arm64-11_0-toothrot
2021-04-28T01:46:37-16b25d2-06c9756/freebsd-arm64-dmgk
2021-04-27T18:54:49-6397a11-0ae9c3b/darwin-amd64-10_14
2021-04-27T15:36:10-6397a11-f12cf76/freebsd-386-11_4
2021-04-20T16:44:18-a13dbf1-109d758/dragonfly-amd64
2021-04-07T17:38:53-e78b40c-b3064b6/darwin-arm64-11_0-toothrot
2021-04-06T20:22:15-c602466-f5efa5a/linux-s390x-ibm

(CC @findleyr @muirdm)

@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Jan 20, 2022
@gopherbot gopherbot added this to the Unreleased milestone Jan 20, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Jan 20, 2022

The error comes from x/tools/internal/lsp/fake/sandbox.go:
https://cs.opensource.google/go/x/tools/+/master:internal/lsp/fake/sandbox.go;l=270;drc=384460091cd7916680a7c5252c587572c601197e

However, it appears that the directory for which the error occurs always refers to TestPostfixSnippetCompletion. I suspect a synchronization error in that test: perhaps some process or goroutine started by the test is continuing to write to the directory while the Sandbox is attempting to remove it.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 20, 2022

This is a release-blocker via #11811; it is causing frequent failures in the builders, especially for darwin/amd64 and darwin/arm64 (which are first-class ports).

(If the root cause isn't readily apparent, I suggest that we skip TestPostfixSnippetCompletion to reduce test-failure noise until someone has time to investigate.)

@bcmills bcmills modified the milestones: Unreleased, Go1.18 Jan 20, 2022
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 20, 2022
@gopherbot
Copy link

Change https://golang.org/cl/379735 mentions this issue: gopls/internal/regtest/completion: temporarily skip TestPostfixSnippetCompletion

@findleyr
Copy link
Contributor

Thanks. I'll look into this with urgency.

I'd really llke to understand this failure mode, as this test is not doing a lot of IO: it doesn't even use any on-disk Go files, just a go.mod. All Go state exists only in in-memory buffers.

Looking at it briefly, I noticed that the test is repeatedly "opening" the same buffer, which I would expect to panic but doesn't (we should change that). However, I wouldn't expect that to cause this failure.

Disabling cleanup leaves the sandbox in the following state. None of this state should change following initial workspace load.

> find .
.
./work
./work/go.mod
./gopath
./gopath/pkg
./gopath/pkg/mod
./gopath/pkg/mod/cache
./gopath/pkg/mod/cache/lock
./proxy

gopherbot pushed a commit to golang/tools that referenced this issue Jan 20, 2022
…tCompletion

TestPostfixSnippetCompletion appears to produce racy writes to the
filesystem, and the failure mode does not seem to be
platform-specific. This skips the test to reduce noise until someone
has the time to investigate in more depth.

For golang/go#50707

Change-Id: I31c3aaf0cdb92ae2323cc8a926d3775d1433a9ff
Reviewed-on: https://go-review.googlesource.com/c/tools/+/379735
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@bcmills bcmills removed this from the Go1.18 milestone Jan 20, 2022
@findleyr
Copy link
Contributor

findleyr commented Jul 8, 2022

CC @adonovan

Looked into this today, and it revealed a memory leak in gopls. When we create a processEnv for running goimports, we acquire the snapshot: https://cs.opensource.google/go/x/tools/+/master:internal/lsp/cache/imports.go;l=146;drc=53ead67a981c04bcacdd4f593330c43ee9285578

But the process env is associated with the view, and so this snapshot is never released.

As a result, we hold on to whichever snapshot was used to create the process env. As time passes the data referenced by this snapshot becomes staler and staler, and is increasingly duplicated. In very brief testing this leak seemed significant.

@findleyr findleyr self-assigned this Jul 8, 2022
@gopherbot
Copy link

Change https://go.dev/cl/416594 mentions this issue: internal/lsp: wait for snapshot shutdown before shutting down the server

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

3 participants