2020-02-20T05:04:39-49f8d45/freebsd-arm64-dmgk
2020-02-20T00:34:12-06fa4c9/linux-386-sid
2020-02-19T21:59:50-e5f5607/linux-386-387
2020-02-19T21:58:05-635f4b9/linux-386

Hm, the only change we made to this test is adding the t.Parallel() call. I'll see if I can figure out what the problem is.

I vaguely recall that a few years ago several of us undertook an effort to speed up all.bash. That included sprinkling a bunch of t.Parallel calls. Several of those had to get backed out, particularly in the runtime and cmd/go, because they caused flakiness. In some cases it was scheduling delays, in others assumptions that no one else shared resources we were tracking (fds), in others it was mutating global testing state like hooks.

Which doesn’t shed much light in this case; it is just to say “just adding t.Parallel” is known to be a common cause of flakiness. :)

Yup, that gives me more confidence that it was the right call to convert tests to the script framework rather than put in t.Parallel. I'll remove this particular t.Parallel at if I can't figure out what's going on by EOD.

A hypothesis: perhaps the filesystem can't keep up during the parallel portion of the test, and certain kernels fail with ETXTBSY when attempting to execute a binary that has not yet been flushed to the filesystem?

I would be inclined to add a df.Sync() at the end of the copyFile helper function to see if that helps.

Change https://golang.org/cl/220317 mentions this issue: cmd/go: in tests, add sync before closing file in copyFile helper

Okay, let's see if it helps. Sent a CL to add the Sync in.

Well, there goes that theory:
https://build.golang.org/log/60bf5a1fb2bf300903733c248c3ea30f3f4bec52

At least the negative signal was immediate! 😅

Oh, sweet! This is a symptom of #22315.

Actually, it's not. That issue is talking about a bug that occurs despite O_CLOEXEC, but we're not even setting that yet. Let's try adding O_CLOEXEC to the os.OpenFile call.

alright, another CL on its way!

Reading #22315 in more detail, I suspect that it will still apply.

A more extreme solution might be to have TestExecutableGOROOT fork off a separate subprocess to copy the executable file. When that subprocess exits, we will know that the file does not remain open on any thread.

A less extreme solution might be to move the copyFile call ahead of the call to tg.parallel().

I think we can ditch the Sync, too.

I've decided to go for the most extreme solution: to just convert the test to the script framework.

Change https://golang.org/cl/220319 mentions this issue: cmd/go: convert TestExecutableGOROOT to the script framework

I wonder whether that will even work — I don't think the cp command in script_test.go currently executes in a subprocess...