isn't this sort of necessary if go doesn't track the contents of external files?
eg if you modify testdata, without modtime it wouldn't know to rerun the test

@mvdan Would it be possible for you to first run some script that sets the mtime of the relevant files to the last commit time of that file?

see also #22593

@seankhliao You're right that any opened files don't currently get their content tracked in the test cache inputs:

	} else if info.Mode().IsRegular() {
		// Because files might be very large, do not attempt
		// to hash the entirety of their content. Instead assume
		// the mtime and size recorded in hashWriteStat above
		// are good enough.
		//
		// To avoid problems for very recent files where a new
		// write might not change the mtime due to file system
		// mtime precision, reject caching if a file was read that
		// is less than modTimeCutoff old.
		if time.Since(info.ModTime()) < modTimeCutoff {
			return cache.ActionID{}, errFileTooNew
		}
	}

However, I think that the logic is somewhat flawed and could be improved. For example, we're already getting the file information, so if a file is small (say, less than 256KiB), we could read and hash its contents rather than using its modified time for the cache input.

This would automatically solve almost all git clone problems I have with CI; the majority of the testdata files I use are plaintext and very small, like https://github.com/cue-lang/cue/blob/7d89acd5c44bca0c756f76089436ee9913818b03/cmd/cue/cmd/testdata/script/cmd_dir.txtar. Go's test cache would be less prone to unfortunate misses, and I don't think it would get noticeably slower.

Would it be possible for you to first run some script that sets the mtime of the relevant files to the last commit time of that file?

I could, yes, but having to do that for every project is quite a lot of work. Plus, I didn't even notice that CI on multiple of my projects was slow due to this bug for years. I think it's unfortunate for Go's test cache to not work well on CI systems by default.

There's also the problem with scripting CI in general. The majority of these projects test on Linux, Mac, and Windows. Even writing a POSIX Shell or Bash script to run on all of these is rather tricky.

We could also try to convince third parties like git clone or actions/checkout to leave reproducible modified times on files, but I don't think that is likely to happen.

It looks like there is a third-party git-restore-mtime tool (written in Python, which I would think would be portable enough) that does the obvious thing — setting the mtime to match the last author timestamp or commit timestamp of each file. But I agree that it would be better not to rely on that, since there are all kinds of ways an mtime may be changed.

We looked at using git-restore-mtime for our CI, but that script is quite slow, wiping out many of the benefits of caching. We are currently using a modified version of this script that improves the handling of directories.

We looked at using git-restore-mtime for our CI, but that script is quite slow, wiping out many of the benefits of caching. We are currently using a modified version of this script that improves the handling of directories.

@oschwald, would you possibly be able to update the link in your comment? We are currently exploring options for test caching in CI/CD.

@bradfitz pointed this out to me. As others noted, we use the mtime as an approximation to content changes. Otherwise if you have a file being read by a test that, say, contains a "0", and you change it to contain "1" and run go test again, we will not notice the file is changed, say it's a match, and then incorrectly reuse the cached entry. The alternative would be to reread all the files and compute content hashes, but we decided that could be too expensive. For example a test might open a large binary database but only use a small amount of the file. In that case the test would run quickly but the cache validation would run much slower than the test.

I'm not sure the solutions being proposed here are any better. In particular I don't think it makes sense to special-case small files and still have a different behavior for large files.

In particular I don't think it makes sense to special-case small files and still have a different behavior for large files.

I agree it's not a great solution, but it was the best I could come up with in terms of a compromise that should work across all systems.

Perhaps there is something we could use for modern filesystems that isn't entirely portable but would give better results for the majority of users?

The only alternative that comes to mind is to lobby VCS tools such as git to default git clone and git checkout to use the VCS modified times when creating or updating files. In an ideal world that would happen and then there would be nothing for us to do but wait for most users to update, but in reality I think it's unlikely to happen - there's nothing wrong per se with their current approach.

I think hashing the content of small files is a good, pragmatic solution. It's simple and easy to explain, and in practice it will largely fix this issue while avoiding a bad performance hit from reading large test files.

I looked at a large, private code base and found that about 2.5% of testdata fixtures are larger than 256 KiB, and <1% of all packages depended on large testdata files (or about 7.5% of packages that have some testdata files).

Maybe we could do something like the following:

• Cache content hashes of files in testdata/ and store modtime next to those hashes.
• When modtime of those files does not match, recompute the content hash.
• Use the content hash (without the modtime) to determine if the test result is cached.

I think it is important to note we don't extend this to arbitrary files. It really would just be testdata/, which is probably in VCS [and in my maybe too simple mental model suggests it is not too bad to hash/detect if it changed]. This would commit to hashing testdata/ contents and this might be moderately wasteful for the cases rsc mentioned.

I have written a CLI, which modify mtimes of files based on sha256. Nasty solution, but it works https://github.com/slsyy/mtimehash