Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: TempDir RemoveAll cleanup failures with "The process cannot access the file because it is being used by another process." #51442

Closed
bcmills opened this issue Mar 2, 2022 · 14 comments
Labels
NeedsFix The path to resolution is known, but the work has not been done. OS-Windows release-blocker
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Mar 2, 2022

--- FAIL: TestRWithNonexistentFile (0.83s)
    testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\gopher\AppData\Local\Temp\1\TestRWithNonexistentFile1599077814\001\pack.exe: The process cannot access the file because it is being used by another process.
FAIL
FAIL	cmd/pack	4.916s

This can probably be fixed by expanding the set of errors handled by the retry loop added for #50051.

greplogs --dashboard -md -l -e 'TempDir RemoveAll cleanup: .* The process cannot access the file because it is being used by another process\.'

2022-02-25T23:38:11-01e522a/windows-amd64-2012
2021-10-28T18:10:50-2bc8ed8/windows-amd64-2012
2021-10-14T18:43:59-765c911/windows-amd64-2012
2021-09-30T19:56:27-70b1a45/windows-amd64-2012
2021-09-17T19:32:44-74e384f/windows-amd64-2012
2021-03-09T08:19:14-b70a2bc/windows-386-2008
2021-03-09T08:19:14-b70a2bc/windows-amd64-2008
2021-03-09T08:19:14-b70a2bc/windows-amd64-2012
2021-03-09T08:19:14-b70a2bc/windows-amd64-2016
2021-03-09T08:19:14-b70a2bc/windows-amd64-longtest
2021-03-09T08:19:14-b70a2bc/windows-amd64-race
2021-03-09T08:19:14-b70a2bc/windows-arm-zx2c4
2021-03-09T07:44:18-437d229/windows-amd64-2008
2021-03-09T07:44:18-437d229/windows-amd64-2012
2021-03-09T07:44:18-437d229/windows-amd64-race
2021-03-09T07:44:18-437d229/windows-arm-zx2c4
2021-03-09T02:20:55-618b66e/windows-amd64-2008
2021-03-09T02:20:55-618b66e/windows-amd64-2012
2021-03-09T02:20:55-618b66e/windows-amd64-2016
2021-03-09T02:20:55-618b66e/windows-amd64-race
2021-03-09T02:20:55-618b66e/windows-arm-zx2c4
2021-03-08T23:13:52-b6def6a/windows-amd64-2008
2021-03-08T23:13:52-b6def6a/windows-amd64-2012
2021-03-08T23:13:52-b6def6a/windows-amd64-2016
2021-03-08T23:13:52-b6def6a/windows-amd64-race
2021-03-08T20:41:06-a08adda/windows-386-2008
2021-03-08T20:41:06-a08adda/windows-amd64-2008
2021-03-08T20:41:06-a08adda/windows-amd64-2016
2021-03-08T20:41:06-a08adda/windows-amd64-longtest
2021-03-08T20:41:06-a08adda/windows-amd64-race
2021-03-08T20:03:01-bd37284/windows-386-2008
2021-03-08T20:03:01-bd37284/windows-amd64-2008
2021-03-08T20:03:01-bd37284/windows-amd64-2012
2021-03-08T20:03:01-bd37284/windows-amd64-2016

@bcmills bcmills changed the title testing: TempDir RemoveAll cleanup failures with "The process cannot access the file because it is being used by another process.| testing: TempDir RemoveAll cleanup failures with "The process cannot access the file because it is being used by another process." Mar 2, 2022
@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 2, 2022
@bcmills bcmills added this to the Go1.19 milestone Mar 2, 2022
@bcmills bcmills self-assigned this Mar 2, 2022
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Mar 2, 2022
@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Mar 2, 2022
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 2, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Mar 2, 2022

(This error message is 0x20 ERROR_SHARING_VIOLATION.)

@gopherbot
Copy link

Change https://go.dev/cl/388916 mentions this issue: testing: include ERROR_SHARING_VIOLATION in Windows cleanup retries

@bhasden
Copy link

bhasden commented May 18, 2022

Apologies for commenting on a closed issue. However, I wanted to ask if there is any chance this makes its way out prior to 1.19?

@ianlancetaylor
Copy link
Contributor

@bhasden Are you encountering this yourself? What Go version and what GOOS/GOARCH? Thanks.

@bcmills
Copy link
Contributor Author

bcmills commented May 18, 2022

FWIW, this fix would be easy to backport to Go 1.18, but somewhat less trivial to backport to Go 1.17 (it would also need CL 371296, which is a bit more invasive).

There is also a workaround that you can apply for your own specific tests independent of the Go version: you can defer (or t.Cleanup) your own function that iterates os.RemoveAll on the directory returned by TempDir until it is successfully removed.

@bhasden
Copy link

bhasden commented May 18, 2022

@bhasden Are you encountering this yourself? What Go version and what GOOS/GOARCH? Thanks.

I believe I'm running into this same issue in Golang 1.18.1 and 1.18.2 (only two versions I've tested) with some casbin tests that are loading a model configuration file. It's only reproducible in Windows. Team members on non-Windows machines are not running into the issue.

go env output:
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\BrianHasden\AppData\Local\go-build
set GOENV=C:\Users\BrianHasden\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\BrianHasden\go\pkg\mod
set GONOPROXY=######
set GONOSUMDB=######
set GOOS=windows
set GOPATH=C:\Users\BrianHasden\go
set GOPRIVATE=######
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\BrianHasden\go\go1.18.2
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\BrianHasden\go\go1.18.2\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18.2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=######
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\BRIANH~1\AppData\Local\Temp\go-build2379612813=/tmp/go-build -gno-record-gcc-switches

@ianlancetaylor
Copy link
Contributor

@gopherbot Please open backport to 1.18.

At least one person is encountering this problem in 1.18. There are workarounds but they are awkward. The fix would be harder to backport to earlier releases.

@gopherbot
Copy link

Backport issue(s) opened: #52986 (for 1.18).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link

Change https://go.dev/cl/407877 mentions this issue: [release-branch.go1.18] testing: include ERROR_SHARING_VIOLATION in Windows cleanup retries

@rsc rsc unassigned bcmills Jun 22, 2022
gopherbot pushed a commit that referenced this issue Jul 25, 2022
…indows cleanup retries

Fixes #52986
Updates #51442
Updates #50051

Change-Id: I1bfbc08c907077467fd50febbec6299a9b73af41
Reviewed-on: https://go-review.googlesource.com/c/go/+/388916
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit eeb9f09)
Reviewed-on: https://go-review.googlesource.com/c/go/+/407877
Reviewed-by: Nooras Saba‎ <saba@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
nao1215 added a commit to nao1215/gup that referenced this issue Sep 18, 2022
@i4ki
Copy link

i4ki commented Apr 13, 2023

I'm still experiencing this issue in go1.20.3.

There is also a workaround that you can apply for your own specific tests independent of the Go version: you can defer (or t.Cleanup) your own function that iterates os.RemoveAll on the directory returned by TempDir until it is successfully removed.

@bcmills this workaround still works.

It looks like the 2 seconds timeout in the removeAll workaround of the testing package is not enough...
https://cs.opensource.google/go/go/+/master:src/testing/testing.go;l=1248-1276;drc=master

@ianlancetaylor
Copy link
Contributor

@i4ki This issue has been closed for almost a year, and we think it is fixed. If you are seeing a problem with 1.20, please open a new issue with reproduction details. It may be the same problem, but it may be a new one. Thanks.

@rasa
Copy link

rasa commented May 4, 2023

I'm experiencing the same error in syncthing's test suite on go 1.20.3:

go test -r TestHandleFile ./...
...
--- FAIL: TestHandleFile (1.60s)
    testing.go:1206: TempDir RemoveAll cleanup: remove C:\Users\ross\AppData\Local\Temp\TestHandleFile1557195908\001: The process cannot access the file because it is being used by another process.
FAIL
FAIL    github.com/syncthing/syncthing/lib/model        1.821s
...
--- FAIL: TestHandleFileWithTemp (1.45s)
    testing.go:1206: TempDir RemoveAll cleanup: remove C:\Users\ross\AppData\Local\Temp\TestHandleFileWithTemp2880978547\001: The process cannot access the file because it is being used by another process.
FAIL
FAIL    github.com/syncthing/syncthing/lib/model        1.578s

@rasa
Copy link

rasa commented May 4, 2023

Syncthing is solving this by rewriting the code to use an emulated file system in syncthing/syncthing#8889

@bcmills
Copy link
Contributor Author

bcmills commented May 4, 2023

@rasa, as far as we know the retry loop that was added for this issue works.
It is likely that your test is actually leaking an open file when the test returns.
If you can definitely rule that out, please file a new issue.

@golang golang locked as resolved and limited conversation to collaborators May 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
NeedsFix The path to resolution is known, but the work has not been done. OS-Windows release-blocker
Projects
None yet
Development

No branches or pull requests

6 participants