Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: GC & debug.FreeOSMemory calls don't return memory to the OS #45000

Closed
AlexBlack772 opened this issue Mar 14, 2021 · 2 comments
Closed

runtime: GC & debug.FreeOSMemory calls don't return memory to the OS #45000

AlexBlack772 opened this issue Mar 14, 2021 · 2 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@AlexBlack772
Copy link

AlexBlack772 commented Mar 14, 2021 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.15.5 linux/amd64
CentOS Linux release 7.7.1908 (Core)
kernel-3.10.0-1062.el7.x86_64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/var/www/virtual/service.extractor/service/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build181628151=/tmp/go-build -gno-record-gcc-switches"

What did you do?

A programm allocates some memory and than release it.

https://play.golang.org/p/Ljr0c6zRasB

What did you expect to see?

The programm releases allocated memor and after runtime.GC() & debug.FreeOSMemory() calls I expect that values of VmSize/VmData drop, but nothing happen.

What did you see instead?

Values of VmSize/VmData remains the same

-- (comment) the programm started

START VmSize:725Mb VmLck:0 VmPin:0 VmRSS:1 VmData:717 VmStk:0 VmExe:0 VmLib:1

-- (comment) the programm allocated memory for buffers

  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:21  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1

END
END
....
END

-- (comment) the programm released memory and shoud return it to the system

  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1

I tried to use "env GODEBUG=madvdontneed=1 ./test2" but it didn't help, it seems that kernel 3.10.0 always use MADV_DONTNEED and GODEBUG=madvdontneed=1 changes nothing

pmap also saw allocated memory

[root@localhost go]# pmap -x  5008
5008:   ./test2
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     828     672       0 r-x-- test2
00000000004cf000     776     396       0 r---- test2
0000000000591000     100      68      40 rw--- test2
00000000005aa000     204      64      64 rw---   [ anon ]
0000000002089000     132       4       4 rw---   [ anon ]
000000c000000000  655360     480     480 rw---   [ anon ]
00007f3e3a1d8000   20964   18100   18100 rw---   [ anon ]
00007f3e3b651000       4       0       0 -----   [ anon ]
00007f3e3b652000    8192       8       8 rw---   [ anon ]
00007f3e3be52000       4       0       0 -----   [ anon ]
00007f3e3be53000    8192       8       8 rw---   [ anon ]
00007f3e3c653000       4       0       0 -----   [ anon ]
00007f3e3c654000    8192       8       8 rw---   [ anon ]
00007f3e3ce54000       4       0       0 -----   [ anon ]
00007f3e3ce55000   44100    2148    2148 rw---   [ anon ]
00007f3e3f966000  263680       0       0 -----   [ anon ]
00007f3e4fae6000       4       4       4 rw---   [ anon ]
00007f3e4fae7000  293564       0       0 -----   [ anon ]
00007f3e61996000       4       4       4 rw---   [ anon ]
00007f3e61997000   36692       0       0 -----   [ anon ]
00007f3e63d6c000       4       4       4 rw---   [ anon ]
00007f3e63d6d000    4068       0       0 -----   [ anon ]
00007f3e64166000    1804     264       0 r-x-- libc-2.17.so
00007f3e64329000    2048       0       0 ----- libc-2.17.so
00007f3e64529000      16      16      16 r---- libc-2.17.so
00007f3e6452d000       8       8       8 rw--- libc-2.17.so
00007f3e6452f000      20      12      12 rw---   [ anon ]
00007f3e64534000      92      56       0 r-x-- libpthread-2.17.so
00007f3e6454b000    2044       0       0 ----- libpthread-2.17.so
00007f3e6474a000       4       4       4 r---- libpthread-2.17.so
00007f3e6474b000       4       4       4 rw--- libpthread-2.17.so
00007f3e6474c000      16       4       4 rw---   [ anon ]
00007f3e64750000     136     108       0 r-x-- ld-2.17.so
00007f3e6477f000     512     416     416 rw---   [ anon ]
00007f3e647ff000     512       0       0 -----   [ anon ]
00007f3e6487f000       4       4       4 rw---   [ anon ]
00007f3e64880000     508       0       0 -----   [ anon ]
00007f3e648ff000     396      44      44 rw---   [ anon ]
00007f3e64970000       4       4       4 rw---   [ anon ]
00007f3e64971000       4       4       4 r---- ld-2.17.so
00007f3e64972000       4       4       4 rw--- ld-2.17.so
00007f3e64973000       4       4       4 rw---   [ anon ]
00007ffd8d92f000     132      20      20 rw---   [ stack ]
00007ffd8d9c6000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1353356   22948   21420


      		
    

      
  





        


            

              
            

        

      


      
    








      

    



      

  
          

  

    
  

  


    

        @mdlayher
mdlayher




      changed the title
Golang 1.16 runtime.GC() & debug.FreeOSMemory() calls don't return memory to the OS

runtime: GC & debug.FreeOSMemory calls don't return memory to the OS


    Mar 14, 2021

  










      

  
      



  

  @cherrymui





  


    

      

  

    

        
    
    

  


      
          
    
            Copy link

      
    

  


  

      

      

  
  Member


      

  


  

    

  





      


        


  
    

      
          
cc @aclements @mknyszek

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  

    
  

  


    

        @cherrymui
cherrymui




        added
  the 

  NeedsInvestigation

  Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
 label


    Mar 15, 2021

  





  

  

    
  

  


    

        @cherrymui
cherrymui




        added this to the   Backlog milestone


    Mar 15, 2021

  










      

  
      



  

  @mknyszek





  


    

      

  

    

        
    
    

  


      
          
    
            Copy link

      
    

  


  

      

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
@AlexBlack772 VmSize is virtual memory footprint and for a Go application it always grows monotonically; the Go runtime never unmaps heap memory. It appears that VmData is just VmSize with a few things subtracted out (https://ewx.livejournal.com/579283.html), so I expect it to have ~the same effect. In generally, address space is cheap on modern systems and does not reflect actual physical memory resources used. When we return memory to the OS, we mark that memory as "unused" to the OS and it's up to the OS to take that memory back. The result is that VmRSS goes down.


VmRSS is the actual physical memory footprint, and Linux won't actually take that memory back until your system is under memory pressure in Go 1.15, because we use MADV_FREE by default (as you point out, though, Linux 3.10.0 doesn't have MADV_FREE, so we fall back on MADV_DONTNEED anyway). In Go 1.16 and above, GODEBUG=madvdontneed=1 is the default.


Tentatively closing this issue as WAI. Please reopen if you feel this is in error.

      		
    

  





        


            

              
            

        

      


      

            
  

    
  
  

  

  



    












      

  
          

  


  

  

    
  


  

      

    @cherrymui
cherrymui



    mentioned this issue
    
      Mar 18, 2021
    





  


  

      
  
  Closed

  







  




  

  

    
  

  


    

        @golang
golang




      locked and limited conversation to collaborators


    Mar 15, 2022

  





  











  
  

    

      

  





  




  
  

             

  
      Sign up for free
      to subscribe to this conversation on GitHub.
      Already have an account?
      Sign in.


  


  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    FrozenDueToAge

  

    NeedsInvestigation

  Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.








      

  

    

        

    Projects
  


        




    None yet




  



      

  

    
  

    Milestone
  


      
    
  
    Backlog
  





    
      
          


  

    
      

        
    

    Development
  




              
No branches or pull requests







    
  




        
      

    

  
      

  

    

      4 participants
    

    

        
          @mknyszek 
        
          @gopherbot 
        
          @cherrymui 
        
          @AlexBlack772