Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: implement SysUnused on Windows #5584

Closed
gopherbot opened this issue May 29, 2013 · 21 comments
Closed

runtime: implement SysUnused on Windows #5584

gopherbot opened this issue May 29, 2013 · 21 comments
Milestone

Comments

@gopherbot
Copy link

by raheelgup:

After discussions on golang-nuts I am opening the issue :
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/7V_zICSxngo

As per my observations if a Go Program touches "X" MB or GB in Ram usage and
even frees it by either the function exiting or you making the length of a slice to 0
again, the heap usage within the go program goes down after the GC. But the Ram it took
from the OS is never ever returned. If the go program touches 1 GB in use, it will never
return to the OS even though the MemStats report that the Alloc memory might be in 1-2
MBs. The remaining memory is held up by Go till the program quits. 

To top it off, lets say if a program asks for 150 MB by making a byte slice of 150 MB,
Go will ask nearly 2-3 times from the OS and hold it. Then lets say you remake the slice
to a size of 0. When the size is made to 0, the memory usag reported by the
Memstats.Alloc might be somewhere in KBs and the MemStats.Sys is nearly 300-450MB. When
you remake the slice to a size of 150 MB, Go might (sometimes) still ask for more memory
from the OS even though the it has 300-450 MB of unused memory. Eventually the
application crashes saying it couldnt allocate more memory.  

I have posted the codes here and anyone can test it. I found this behaviour on x86 and
x86_64 versions of CentOS 6 and Windows 7.

What steps will reproduce the problem?
Please run the code attached here and visit the URL : http://127.0.0.1:8080

What is the expected output?
Ram should have been freed and returned to the OS and also only the required amount of
memory should have been used by the Go program instead of 2-3 times the amount of Ram. 

What do you see instead?
In the command prompt (windows) or shell (linux) you will see the OS will keep
allocating more ram with each request and after 10-20 requests the application will
crash.

Which compiler are you using (5g, 6g, 8g, gccgo)?
CGO

Which operating system are you using?
Windows and Linux 32 and 64 bit.

Which version are you using?  (run 'go version')
1.1

Attachments:

  1. test1.zip (5129 bytes)
@rsc
Copy link
Contributor

rsc commented Jun 3, 2013

Comment 1:

runtime: possible memory leak

@davecheney
Copy link
Contributor

Comment 3:

This sample program shows no leak. The OP is mistaken about the capabilities of the Go
runtime and does not understand that Windows has no facility to return pages to the
Operating System.

Status changed to Invalid.

@alexbrainman
Copy link
Member

Comment 4:

Dave, can you, please, explain what your description of "Windows has no facility to
return pages to the Operating System" means?
Alex

@davecheney
Copy link
Contributor

Comment 5:

Sorry, to be more explicit, Go on windows/amd64 and windows/386 does not implement
runtime.SysUnused.
Arguably other statuses for this issue would be 'WorkingAsIntended' or 'Unfortunate'

@alexbrainman
Copy link
Member

Comment 6:

Well then it is not working as intended. It is unfortunate, but we should fix it. Let's
come up with a plan to fix that instead of marking this issue as invalid.
Alex

@davecheney
Copy link
Contributor

Comment 7:

Fair enough, SGTM.

Status changed to Accepted.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 9:

Implementing SysUnused should be a call to VirtualAllocEx with MEM_RESET. (We already
make calls with MEM_RESERVE and MEM_COMMIT in other functions.)

Labels changed: added priority-later, go1.2, removed priority-triage.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 10:

Issue #4960 has been merged into this issue.

@dvyukov
Copy link
Member

dvyukov commented Jul 30, 2013

Comment 11:

FTR, MEM_RESET is supported starting from Windows 8.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 12:

Labels changed: added feature.

@alexbrainman
Copy link
Member

Comment 13:

MEM_RESET will help because Windows memory manager does not need to write the contents
of memory page to the disk. But the problem remains, because the MEM_RESET page still
remains part of "committed" memory. "committed" memory is a finite resource. From the
top of my head, I cannot tell you how amount of "committed" memory affects Windows
memory manager, but I suspect it does.
What would need to change in go runtime for us to be able to de-committ memory?
I suspect we won't be able to de-committ memory, we should use MEM_RESET then. I am
happy to do it. But it is low on my todo list.
Alex
PS: Why VirtualAllocEx, not VirtualAlloc?
PSS: I bleive MEM_RESET is available on every Windows OS. Even on w2k. "MEM_RESET is
supported starting from Windows 8" - do you have a reference to that?

@rsc
Copy link
Contributor

rsc commented Jul 31, 2013

Comment 14:

I was mistaken; VirtualAlloc (non-Ex) is fine too.

@alexbrainman
Copy link
Member

Comment 15:

What would be good way to test windows SysUnused implementation?
Alex

@dvyukov
Copy link
Member

dvyukov commented Aug 9, 2013

Comment 16:

Say you have 4GB of RAM. Write a program that allocates and frees 3GB. Start the
program, wait till the 3GB will be marked as unused. Start another instance, wait till
the 3GB will be marked as unused. And so on. If you can start 10 such programs w/o
killing the machine with swapping, it works as expected.
You can use debug.FreeOSMemory to not wait for 5 minutes.

@dvyukov
Copy link
Member

dvyukov commented Aug 9, 2013

Comment 17:

Wrt "MEM_RESET is supported starting from Windows 8", it seems I confused it with
MEM_RESET_UNDO.

@dvyukov
Copy link
Member

dvyukov commented Aug 10, 2013

Comment 18:

Alex, is it "started" by you?

@alexbrainman
Copy link
Member

Comment 19:

No, you can have a go at it.
Alex

@dvyukov
Copy link
Member

dvyukov commented Aug 11, 2013

Comment 20:

Owner changed to @dvyukov.

Status changed to Started.

@dvyukov
Copy link
Member

dvyukov commented Aug 14, 2013

Comment 21:

This issue was closed by revision 4e76abb.

Status changed to Fixed.

@gopherbot
Copy link
Author

Comment 22 by mgspross:

I'm experiencing a similar issue (FreeOSMemory/garbage collection is not returning
memory to the OS consistently) on a 64-bit Windows 7 machine, particularly with the
go1.3.3 windows/386 release (but also to varying degrees with other releases).
I've attached a simple test program to demonstrate the problem. It waits 5 seconds,
allocates ~95MB of memory via a slice, sets it to nil, and periodically calls
FreeOSMemory to try to return the memory to the OS.
By watching the "Private Working Set" column for the program in the Windows task manager
while it is running, you can see the memory usage starts out small (as to be expected),
increases to include the 95MB of allocated memory, and then decreases (or doesn't...)
after FreeOSMemory is called, depending on the go version used.
The amount of memory actually returned to the OS varies with different go releases as
well (I tested the most recent releases). Here are my findings, running memtest.go with
"go run memtest.go" on various go releases:
 * go1.3.2 windows/386: The memory is never returned to the OS.
 * go1.3.2 windows/amd64: This works correctly. The memory is released and the memory usage returns to close to what it was when the program started (memory in-use was ~1.5MB after FreeOSMemory was called).
 * go1.3.3 windows/386: The memory is never returned to the OS.
 * go1.3.3 windows/amd64: This works correctly (memory in-use was ~1.5MB after FreeOSMemory was called).
 * go1.4rc1 windows/386: Some memory is returned, bu not all of it: ~13.5MB was still in use after FreeOSMemory was called.
 * go1.4rc1 windows/amd64: Some memory is returned, but not all of it: ~7.6MB was still in use after FreeOSMemory was called.
These results were all repeatable which each respective release.
I kept the program very simple, and didn't bother allocating a larger amount of memory
because it is was still able to clearly demonstrate the differences in behavior across
the different go builds tested (on my machine at least).
I haven't tested yet with other allocation sizes, and I'm not sure if some of this
behavior is expected.
My results seem to indicate that 1.3.2 windows/amd64 and 1.3.3 windows/amd64 do the best
job of returning as much memory as possible back to the OS, as the memory usage returns
very nearly to the amount of memory in-use when the test program is first started (prior
to allocating the slice).
I'm no expert on Windows memory management, but I thought it was interesting that the
386 and amd64 builds behave so differently, and wondered whether running the 386 build
on an amd64 machine might be a factor. I haven't be able to test the 386 build on a
32-bit Windows installation as of yet, but I wanted to try that as well for comparison,
and still can if that would be helpful.
P.S. I'm using the 386 build because my the program I'm writing uses interacts with
legacy 32-bit Windows COM libraries, so I can't use the amd64 build in my current
situation.

Attachments:

  1. memtest.go (647 bytes)

@davecheney
Copy link
Contributor

Comment 23:

This issue has been marked as fixed in 2013. Please open a new issue.

@rsc rsc added this to the Go1.2 milestone Apr 14, 2015
@rsc rsc removed the go1.2 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants