Skip to content

runtime: use MADV_REMOVE on Linux #61688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
113xiaoji opened this issue Aug 1, 2023 · 5 comments
Closed

runtime: use MADV_REMOVE on Linux #61688

113xiaoji opened this issue Aug 1, 2023 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@113xiaoji
Copy link

Starting with Go 1.12, the default configuration utilizes MADV_DONTNEED, but this doesn't guarantee that the operating system will immediately reclaim the memory. In containerized environments where resources are scarce and limits are set relatively low, there's still a high likelihood of running into an Out Of Memory (OOM) situation.

To address this issue, an option can be added to support MADV_REMOVE, allowing the reclaimed memory to be promptly returned to the operating system. This solution is particularly suited to scenarios that are sensitive to memory usage but not as concerned with efficiency.

For more detailed information on this subject, you can refer to the Linux man-pages: Linux man-pages.
@gopherbot gopherbot added this to the Proposal milestone Aug 1, 2023
@seankhliao seankhliao changed the title proposal: src/runtime add MADV_REMOVE on Linux proposal: runtime: use MADV_REMOVE on Linux Aug 1, 2023
@seankhliao
Copy link
Member

cc @golang/runtime

@113xiaoji
Copy link
Author

cc @golang/runtime

thank you

@ianlancetaylor
Copy link
Member

There is no API change here so taking out of the proposal process.

@ianlancetaylor ianlancetaylor changed the title proposal: runtime: use MADV_REMOVE on Linux runtime: use MADV_REMOVE on Linux Aug 1, 2023
@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. compiler/runtime Issues related to the Go compiler and/or runtime. and removed Proposal labels Aug 1, 2023
@ianlancetaylor ianlancetaylor removed this from the Proposal milestone Aug 1, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Aug 1, 2023

I don't think MADV_REMOVE applies in this circumstance. From the man pages you linked (emphasis is mine):

The specified address range must be mapped shared and writable.

All the Go runtime's mappings for the heap are private anonymous mappings. This madvise flag seems geared specifically toward memory-mapped files.

I'm not sure if this does exactly what you want, but on Linux platforms that don't support MADV_DONTNEED or MADV_FREE, the runtime will remap the pages (not unmap, just map over them again). In any case, I think it's unlikely that we'll add a new knob to select which madvise to use. The existing GODEBUG is already kind of an accident: it was only added because switching to MADV_FREE specifically caused problems.

@mknyszek
Copy link
Contributor

mknyszek commented Aug 2, 2023

In triage now, I don't think this is actionable, unless I'm misunderstanding the requirements of MADV_REMOVE. Closing for now.

@mknyszek mknyszek closed this as not planned Won't fix, can't repro, duplicate, stale Aug 2, 2023
@golang golang locked and limited conversation to collaborators Aug 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mknyszek @ianlancetaylor @gopherbot @seankhliao @113xiaoji