Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: failures with "cannot map pages in arena address space" on freebsd-amd64-race since 2021-11-09 #49687

Closed
bcmills opened this issue Nov 19, 2021 · 10 comments
Closed

runtime: failures with "cannot map pages in arena address space" on freebsd-amd64-race since 2021-11-09 #49687

bcmills opened this issue Nov 19, 2021 · 10 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone
Go1.18

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 19, 2021 

fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw({0x9f747c?, 0x0?})
	/tmp/workdir/go/src/runtime/panic.go:992 +0x71
runtime.sysMap(0x0, 0x40ccc9?, 0xc000611358?)
	/tmp/workdir/go/src/runtime/mem_bsd.go:76 +0xa5
runtime.(*mheap).grow(0xdd4320, 0xc000611200?)
	/tmp/workdir/go/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0xdd4320, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
	/tmp/workdir/go/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
	/tmp/workdir/go/src/runtime/asm_amd64.s:469 +0x49

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-01-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race
2021-11-09T00:08:42-b76863e-5344dca/freebsd-amd64-race
@bcmills
Copy link
Contributor Author

bcmills commented Nov 19, 2021

I suspect that this may be due to memory exhaustion; as noted in #49347 (comment), the freebsd-amd64-race builder has a higher CPU-to-RAM ratio than most of the other -race builders, which may cause its GOMAXPROCS to be fairly aggressive for the amount of available memory.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 19, 2021
@bcmills bcmills added this to the Go1.18 milestone Nov 19, 2021
@bcmills bcmills added okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels Nov 19, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Dec 6, 2021

Have there been any new instances of this? I can't find any on the main dashboard (I haven't downloaded the subrepo logs).

@bcmills
Copy link
Contributor Author

bcmills commented Dec 6, 2021

None since Nov. 19, including the subrepos.

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-11-20

@bcmills
Copy link
Contributor Author

bcmills commented Dec 6, 2021

Actually, hang on. I think there's something wrong with my regexp.

@bcmills
Copy link
Contributor Author

bcmills commented Dec 6, 2021

Ah, it was a time-zone difference. There are indeed no occurrences since Nov. 19 UTC.

greplogs --dashboard -md -l -e 'cannot map pages in arena address space' --since=2021-11-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race

@mknyszek
Copy link
Contributor

mknyszek commented Dec 6, 2021 •
edited

Since this was due to memory exhaustion (it looks like that to me too -- the OS won't let the runtime commit any more pages) and hasn't happened since Nov 19th, @bcmills what do you think about putting this in WaitingForInfo?

We can also try increasing decreasing the CPU-to-RAM ratio as you suggest, but I get the impression that this doesn't even get into any -race code. That is, unless I'm misunderstanding and the go command is actually built with -race. If that's the case, this may also be totally mitigated by https://go.devl/cl/333529 which imports a new TSAN runtime that uses a lot less memory (though I'm gonna guess that's coming in 1.19 at this point).

@cherrymui cherrymui removed the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Dec 14, 2021
@cherrymui
Copy link
Member

Any update on this? Does it happen again? (pinging as this is a release blocker)

the go command is actually built with -race

I think you're right, the go command is not built with -race flag.

Perhaps other processes that are built with -race use more memory and cause the go command fail to map the address?

@aclements
Copy link
Member

As far as I can tell this hasn't happened again (though I only have a partial image of the subrepo logs). This seems like just random memory exhaustion, especially given the builder config. I think we should just close this.

@aclements
Copy link
Member

The error reporting could definitely be better here (like, what was errno?). I think we should just improve the error reporting and close this. If it recurs, then we can reopen with more information.

@gopherbot
Copy link

Change https://golang.org/cl/379354 mentions this issue: runtime: print error if mmap fails

jproberts pushed a commit to jproberts/go that referenced this issue Jun 21, 2022
@ianlancetaylor @jproberts
runtime: print error if mmap fails
8119564 
Fixes golang#49687

Change-Id: Ife7f64f4c98449eaff7327e09bc1fb67acee72c9
Reviewed-on: https://go-review.googlesource.com/c/go/+/379354
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@golang golang locked and limited conversation to collaborators Jan 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Milestone
Go1.18
Development

No branches or pull requests
5 participants
@mknyszek @aclements @bcmills @gopherbot @cherrymui