Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document if the go test and build cache is threadsafe #40461

Open
alvaroaleman opened this issue Jul 28, 2020 · 13 comments
Open

cmd/go: document if the go test and build cache is threadsafe #40461

alvaroaleman opened this issue Jul 28, 2020 · 13 comments
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@alvaroaleman
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.15rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

What did you do?

go help cache

What did you expect to see?

A sentence telling me if the go build and test cache is safe for parallel use by multiple invocation of the go binary

What did you see instead?

@davecheney
Copy link
Contributor

I don’t think it is

@ALTree
Copy link
Member

ALTree commented Jul 29, 2020

I know for sure that cleaning the build cache (go cache -clean) while building something is not safe. This is discussed in #31948, which also functions as a TODO for possibly making it safe.

@ALTree ALTree changed the title Document if the go test and build cache is threadsafe cmd/go: document if the go test and build cache is threadsafe Jul 29, 2020
@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 29, 2020
@ALTree
Copy link
Member

ALTree commented Jul 29, 2020

cc @bcmills @jayconrod

@alvaroaleman
Copy link
Author

I know for sure that cleaning the build cache (go cache -clean) while building something is not safe.

Okay but what about multiple parallel go build or go test invocations?

@jayconrod
Copy link
Contributor

Okay but what about multiple parallel go build or go test invocations?

This is safe on most file systems. Files are added to the cache atomically, and a global file lock is held when files are evicted. If the file system does not support file locking or atomic renaming, then it may not be safe.

@jayconrod jayconrod added this to the Backlog milestone Aug 3, 2020
@bcmills
Copy link
Contributor

bcmills commented Aug 3, 2020

FWIW, the build and test cache does not use file locking, and I don't think it relies on atomic renaming either. (Instead, it writes the expected lengths in metadata files, and overwrites and appends to partial files in-place rather than truncating or deleting them.)

@bcmills
Copy link
Contributor

bcmills commented Aug 3, 2020

I take that back: we use file-locking in one place in the test cache, to store the expiration date for stale tests.

// Read testcache expiration time, if present.
// (We implement go clean -testcache by writing an expiration date
// instead of searching out and deleting test result cache entries.)
if dir := cache.DefaultDir(); dir != "off" {
if data, _ := lockedfile.Read(filepath.Join(dir, "testexpire.txt")); len(data) > 0 && data[len(data)-1] == '\n' {

@networkimprov
Copy link

@alvaroaleman are you hoping to do simultaneous go build/go test runs of the same project, or different ones, or partially overlapping ones?

@alvaroaleman
Copy link
Author

@networkimprov the question came up in the context of "We prepopulate and then use one cache per machine that runs many concurrent builds"

@networkimprov
Copy link

If prepopulated, is your cache read-only?

@alvaroaleman
Copy link
Author

If prepopulated, is your cache read-only?

Its not my cache and I believe its not

@zimmski

This comment has been minimized.

@bcmills

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

8 participants