Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os: expose accessible physical system memory size #21816

Closed
pbnjay opened this issue Sep 9, 2017 · 8 comments
Closed

proposal: os: expose accessible physical system memory size #21816

pbnjay opened this issue Sep 9, 2017 · 8 comments

Comments

@pbnjay
Copy link
Contributor

pbnjay commented Sep 9, 2017

Currently runtime.NumCPU() int exposes the number of cpu cores, which is easily useful for limiting concurrent cpu-bound goroutines. It would be similarly useful if there were a runtime.TotalMemory() uint64 call to help limit memory-intensive goroutines from swapping. I think semantics similar to NumCPU are fine - just detect total memory during startup and do not attempt to track available memory.

An example use case where this would be awesome: I have multiple terabytes of data and want to do all-against-all comparisons. By dividing the data into pairwise batches that fit into memory I can process it more efficiently without needing to pre-process or configure based on what system it's running on. Also my development system uses a different OS and has significantly less memory than my production systems (e.g. using a linux-specific package would make testing and development more difficult).

FWIW, this would also provide a platform-independent way to implement #21795 in userspace.

@gopherbot gopherbot added this to the Proposal milestone Sep 9, 2017
@davecheney
Copy link
Contributor

davecheney commented Sep 9, 2017 via email

@pbnjay
Copy link
Contributor Author

pbnjay commented Sep 9, 2017

An argument could be made that NumCPU doesn't "need" to be in the stdlib as well, but its utility is clear. CPUs directly relate to constraints on concurrency. Memory likewise relates to constraints on allocation/garbage collection. These are both pretty integral to Go yet memory limits don't get first-class status like cpu cores.

Personally I'd really like to avoid any java-style "-Xmx" mess without resorting to shell wrappers.

I'm happy to prototype - I submitted this proposal to initiate discussion.

@as
Copy link
Contributor

as commented Sep 9, 2017

In your use case, runtime.TotalMemory would provide the "detected total memory." However, if we are not providing the available memory, the metric is not useful in practice. To me it suggests that NumCPUs and TotalMemory are not the same. The user does not control what and how many CPUs are used, but might have control over indirect allocation. If all NumCPUs cant be used, the user never sees that. If TotalMemory cant be used, the runtime has a fatal error after a make().

@ianlancetaylor
Copy link
Contributor

I don't understand how to do this in a portable and useful way.

For example, on GNU/Linux, the "total memory available" depends on what other processes on the system are doing. There is no single value for "total memory available" over time.

@pbnjay
Copy link
Contributor Author

pbnjay commented Sep 9, 2017

Not available (I agree that is too difficult), just what is physically installed. E.g. what is the hard limit before overcommitting/swapping.

I've started a prototype here: https://github.com/pbnjay/memory

@pbnjay pbnjay changed the title proposal: runtime: expose physical system memory size proposal: os: expose accessible physical system memory size Sep 11, 2017
@pbnjay
Copy link
Contributor Author

pbnjay commented Sep 11, 2017

My prototype is reasonably complete now, tested on multiple GOOS/GOARCH. I concede package runtime is probably the wrong place for it, but would like to suggest os if this has any chance of seeing the light of day. Again, I don't need/want currently available memory, only what my code has a possibility of addressing without swapping.

@as I think you're forgetting about virtual memory on modern operating systems. I can easily make() a 32GB allocation on my 16GB laptop without a fatal error. When I try to use it, it will swap to disk. If I'm using a lot of that memory all the time, the system will start thrashing, which is exactly the situation I'd like to avoid (using this function's information).

@rsc
Copy link
Contributor

rsc commented Oct 16, 2017

/cc @RLH @aclements

@aclements
Copy link
Member

Like others, I'm concerned that this ignores multi-tenancy. There's nothing inherently wrong with the API, of course; it just doesn't seem that useful because of this. On most of my systems, using anything like the total available memory for one process would bring the system to its knees. :) One situation where this may be more useful is in a container-ized environment where you're more isolated from other tenants, but then it seems like you should be querying this information from the container manager (maybe your prototype effectively does this? I'm not sure how container memory limits affect Sysinfo. And some container managers allow much more complicated memory policies).

Certainly if we were to add this to std, it shouldn't go in runtime. The comparison to runtime.NumCPU is rather misleading. They're fundamentally different resources: time is fungible, space is not. That's why it's okay for GOMAXPROCS to default to the total number of CPUs, but not okay for the memory manager to just claim all available RAM. Plus, I'm not positive, but I suspect runtime.NumCPU mainly exists from back when GOMAXPROCS defaulted to 1 and it was often useful to explicitly set it to the number of CPUs. I suspect if we were to redo the runtime API we would leave that out.

This seems like a good package and if it's useful in certain environments, that's great. But it's not clear to me that it passes the bar for std.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants
@davecheney @rsc @pbnjay @aclements @ianlancetaylor @as @gopherbot and others