Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: runtime: mechanism for monitoring system memory #37452

Closed
NatoBoram opened this issue Feb 25, 2020 · 4 comments
Closed

proposal: runtime: mechanism for monitoring system memory #37452

NatoBoram opened this issue Feb 25, 2020 · 4 comments

Comments

@NatoBoram
Copy link

NatoBoram commented Feb 25, 2020

I want to run a program that orchestrates other programs that require a lot of memory in a constrained environment like a virtual machine or a IoT device, but can also run on a developer's machine with more sizable memory and a different OS. The problem is that I have no way to adjust the program's behaviour according to the available resources as not to bring my systems to their knees.

What did you do?

Use a lot of memory.

What did you expect to see?

Have a way to query the currently available RAM so I can adjust the program's behaviour accordingly.

What did you see instead?

runtime.MemStats to monitor the currently running program's memory usage.

Related

#21816

@gopherbot gopherbot added this to the Proposal milestone Feb 25, 2020
@ianlancetaylor ianlancetaylor changed the title proposal: os: mechanism for monitoring system memory proposal: runtime: mechanism for monitoring system memory Feb 25, 2020
@ianlancetaylor
Copy link
Contributor

See also #33060.

@rsc
Copy link
Contributor

rsc commented Feb 26, 2020

Issue #29696 is meant to help with using too much memory, by letting you specify the max heap and get a callback when you are using too much memory, to shed load. That definitely helps with "monitor the currently running program's memory usage".

If "query the currently available RAM" means the RAM on the surrounding operating system, that's much harder. You can find out how much RAM the OS has, but you don't know how much should be used by other programs, nor how much should really be left for the OS's file cache to avoid thrashing on file access or even executing different pages of an executable. I'm not sure we're going to solve that ("tell me how much RAM I can use").

Closing as a duplicate of #29696.

@NatoBoram
Copy link
Author

letting you specify the max heap

I do not want to limit my own use of memory, I want to delay launching other tasks in entirely different programs according to currently available memory.

If I limit my program to use an arbitrary amount of RAM, then it's impossible to have a single executable that can behave itself correctly on both a Rasberry Pi and a developer's machine. I could add a configuration, but it's absolutely useless if I'm orchestrating entirely different programs on the same machine.

You can find out how much RAM the OS has, but you don't know how much should be used by other programs

Simple enough : Whatever they want. I'm not here to dictate how other programs behave, I'm here to dictate my own behaviour according to available resources. If I choke my systems by using too much RAM, then it's my fault and no one else's.

You don't know how much should really be left for the OS's file cache to avoid thrashing on file access or even executing different pages of an executable.

That's my problem, not the standard library's. It's nice to have high-level suggestions like that, but it's unnecessary.

I'm not sure we're going to solve that "tell me how much RAM I can use".

I'm not asking to solve "tell me how much RAM I can use", I'm asking for "tell me how much RAM is in use". The other problem probably can't be resolved by Go itself, but rather by individual programmers in their individual programs for their individual use cases.

Closing as a duplicate of #29696.

"add way for applications to respond to GC backpressure" has nothing to do with "mechanism for monitoring system memory".

I want to know if I can launch a task before launching it.

@NatoBoram
Copy link
Author

NatoBoram commented Feb 27, 2020

I found that an appropriate solution was to use mem.VirtualMemory() from github.com/shirou/gopsutil or host.Memory() from github.com/elastic/go-sysinfo.

There's a few things that should honestly be in the standard library. If Go is to be used as a server-side language, then it should be able to be optimized for a server environment. This includes being able to easily query available resources and react accordingly. It's all fun and games to process our 200 GiB of data concurrently, but it's not useful if it crashes the host. By Googling MEMORYSTATUSEX golang site:github.com, I found multiple packages that implements that :

@golang golang locked and limited conversation to collaborators Feb 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

4 participants