Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: VirtualFree of 0 bytes failed with errno=487 (ERROR_INVALID_ADDRESS) #35465

Closed
heschi opened this issue Nov 8, 2019 · 10 comments
Closed
Labels
FrozenDueToAge OS-Windows release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages)
Milestone

Comments

@heschi
Copy link
Contributor

heschi commented Nov 8, 2019

Funny looking trybot failure on 32-bit Windows: https://storage.googleapis.com/go-build-log/0c8f6cc0/windows-386-2008_c0fd3e00.log

cc @mknyszek maybe?

@heschi heschi added release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages) labels Nov 8, 2019
@mknyszek
Copy link
Contributor

mknyszek commented Nov 8, 2019

The 0 bytes thing is a red herring, the code tries to decommit successively smaller chunks because of how windows mapping works.

It's almost certainly related to my code somehow, and maybe related to #35445.

@mknyszek
Copy link
Contributor

mknyszek commented Nov 8, 2019

On second thought... probably not related to #35445. The page size on 386 is 4 KiB.

@mknyszek mknyszek self-assigned this Nov 8, 2019
@bcmills bcmills added OS-Windows NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 8, 2019
@bcmills bcmills added this to the Go1.14 milestone Nov 8, 2019
@heschi
Copy link
Contributor Author

heschi commented Nov 12, 2019

@mknyszek
Copy link
Contributor

@heschik have you been seeing this often? Do you have more examples? I've been unable to reproduce thus far, and also I don't see it happening at all in CI which leads me to believe it's fairly rare.

I have one suspicion: the background scavenger is scavenging unreserved memory. The evidence here is that:

  • The failure is ERROR_INVALID_ADDRESS (though MSDN won't tell me under what conditions different error codes are set).
  • The page to be scavenged is at the very top of a palloc chunk, which would be the first thing the scavenger sees.

There is a check that guards against this though, so if this is indeed the case, that must be erroneously passing. But if that check is erroneously passing, then it's possible to allocate into unmapped memory, which is not a failure mode I've seen yet (though certainly possible).

@heschi
Copy link
Contributor Author

heschi commented Nov 12, 2019

@mknyszek
Copy link
Contributor

Ah, excellent. Thank you @heschik.

This supports my hypothesis, though it's curious that it's the same test every time. It's probably just quite good at tickling some edge case.

I was able to reproduce once in a gomote (unfortunately it dumped the output into my terminal window and I lost all the useful info) but it's timing out now. Now that I can reproduce though it should be easier to get to the bottom of this.

@mknyszek
Copy link
Contributor

I think I found the problem. Will push a fix in a bit.

@gopherbot
Copy link

Change https://golang.org/cl/206978 mentions this issue: runtime: check summary before scavenging in fast path

@mknyszek
Copy link
Contributor

@heschik confirmed this fixes the problem in the x/tools test (10 consecutive runs and no failure; the tests in that package take a while to run). It's a bit wild to me how consistently that test was able to trigger this, but it was nice to have a consistent reproducer. :)

@mknyszek mknyszek removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 13, 2019
@ianlancetaylor
Copy link
Contributor

@golang golang locked and limited conversation to collaborators Nov 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge OS-Windows release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages)
Projects
None yet
Development

No branches or pull requests

5 participants