The windows-amd64-race builder is already using n1-highcpu-16 machine type (16 vCPUs, 14.4 GB mem), same as the windows-amd64-longtest builder. So, there isn't an easy opportunity to increase the RAM of the -race builder to match that of -longtest builder.

One difference between them is that the -race builder is using Windows server 2008, while -longtest is using Windows server 2016. Would it be a good first step to update the -race builder to a newer version of Windows? Or is it considered a benefit that it's testing with an older version of Windows?

I think it makes sense to upgrade the -race builder to the latest OS image, since it's really checking for bugs on the Go side anyway.

(I doubt that will fix this issue, but it's a fine first step to eliminate a variable.)

Change https://golang.org/cl/235419 mentions this issue: dashboard: update windows-amd64-race builder to Windows Server 2016 image

My team sees a similar error very frequently when running lots of tests sequentially from Makefiles. Curiously adding a few seconds delay between tests appears to improve the stability (although, without hard statistics we might be fooling ourselves). Simply rerunning the tests is often enough for them to succeed.

This is using go version go1.14.2 windows/amd64 on Microsoft Windows [Version 10.0.18363.836], and my impression is that the situation has become worse over time. That could be either or both of newer releases of golang or Windows. I wonder if a Windows expert knows more details of how Windows allocates memory and/or address space, and whether there is a better way for golang, or golang users, to interact with Windows.

This issue is a real nuisance for me and my team. We are very willing to help investigate and/or try potential fixes if anyone needs. Thanks!

Update at 2020-06-22: We have discovered that golang executables running out of memory is specific to executing our long "make" runs under powershell. Executing the same "make" runs under cmd.exe works correctly.

IWBN to understand the root cause though.

I've looked at the last 2 weeks of results (this is after the builder change in CL 235419), and I'm not spotting OOM-related failures. There are frequent failures on windows-amd64-race builder (and other race builders), but many of all seem to be due to #39865.

Only one OOM over the past year, but it's recent enough that I don't see any intervening changes that would have fixed it. 🤔

greplogs --dashboard -md -l -e '(?ms)\Awindows-amd64-race.*fatal error: out of memory.*^FAIL\s+golang\.org/x/tools' --since=2021-01-01

2021-12-01T18:48:35-3c63f30-6c4bf8f/windows-amd64-race

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)