Yea, looks related. And as with that one, probably a pain to reproduce, but I'll try.

Hasn't occurred since 2021-11-24. I also tried to reproduce on gomote but couldn't reproduce.

The interesting part is that the binary runs successfully with no error, just didn't print anything. This may be different from #49681.

Maybe we need fflush or something to flush the output? I would think we don't need that as it exits normally. But I'm not sure I know enough about C buffered I/O on Windows.

This hasn't happened again. Unfortunately, it was rare enough that that doesn't tell us much:

$ greplogs --dashboard -l -e 'FAIL: TestVectoredHandlerDontCrashOnLibrary' -since 2021-01-01 | findflakes -paths
First observed 5430203 09 Nov 20:10 2021 (2604 commits ago)
Last observed  9e7600d 24 Nov 20:56 2021 (658 commits ago)
51% chance failure is still happening
0.10% failure probability (3 of 1947 commits)

Still no more occurrences. I'm going to try stress testing this.

git checkout 9e7600d
gopool create -setup 'gomote push $VM && gomote run $VM go/src/make.bat' windows-amd64-longtest
stress2 -p 8 -fail FAIL gopool run gomote run \$VM go/bin/go test runtime -run TestVectoredHandlerDontCrashOnLibrary -v -count 1

It may be necessary to run the full runtime test, but I'll try just the failing test first.

I was able to reproduce this at commit 9e7600d, the last commit we saw a dashboard failure at. But it takes a long time. Over the course of about 20 hours with 8 gomotes, I got 4 failures out of 17,040 runs (plus 2,813 buildlet flakes :( ).

I've added a bunch of trace prints to the test program and will see if that sheds any light on this.

It seems I can only create 3 windows-amd64-longtest builders at a time, though I'm not sure where that limit is coming from. I'm switching to windows-amd64-2016, which is the same image, just on a smaller VM. Hopefully I can still reproduce there.

(Edited to clarify: on the builders, the test only runs with longtest, so we wouldn't expect to see it on windows-amd64-2016, but since I'm running the test manually, that doesn't matter.)

This will probably surprise no one, but it seems adding tracing made this no longer reproducible. I've done 21,665 runs with no failures.

Now I'm not sure what to do.

Now I'm not sure what to do.

Given @cherrymui's insight in #49959 (comment), and combining that with C best practices (https://wiki.sei.cmu.edu/confluence/display/c/FIO23-C.+Do+not+exit+with+unflushed+data+in+stdout+or+stderr), I would say maybe we should add an explicit fflush or fclose to the test program and close this issue as “possibly fixed”.

If we see any more of these failures (on our own builders or reported by users) after that, we can always either reopen this issue or open a followup issue to investigate further.

I'll try the fflush.

I tried reproducing it a different way that didn't cause nearly as many problems with the builders. Over the course of 20 hours I got 3 failures out of 231,000 iterations of the test.

cat >i49959 <<EOF
#!/bin/bash

set -ex

VM=$(gomote create windows-amd64-2016)
function cleanup() {
    gomote destroy $VM
}
trap cleanup EXIT
gomote push $VM
gomote run $VM go/src/make.bat
gomote run $VM go/bin/go test -c -o ../src/runtime/runtime.test.exe runtime
for i in {1..100}; do
    gomote run $VM go/src/runtime/runtime.test.exe -test.run TestVectoredHandlerDontCrashOnLibrary -test.v
done
EOF
stress2 -p 20 -timeout 20m -fail FAIL ./i49959

Looks like Cherry was right. 0 failures after 250,000 iterations with the fflush.

Change https://golang.org/cl/380494 mentions this issue: runtime: call fflush before exiting in C test