Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os: possible memory leak in os.ReadFile() #55041

Closed
g0194776 opened this issue Sep 13, 2022 · 7 comments
Closed

os: possible memory leak in os.ReadFile() #55041

g0194776 opened this issue Sep 13, 2022 · 7 comments

Comments

@g0194776
Copy link

g0194776 commented Sep 13, 2022 •
edited

What version of Go are you using (go version)?

1.17.2

Does this issue reproduce with the latest release?

I've tested in golang 1.18.6, it also has the same problem.

What operating system and processor architecture are you using (go env)?

[root@10 ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 4
cpu cores	: 8
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 5
cpu cores	: 8
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 6
cpu cores	: 8
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 63
model name	: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
stepping	: 2
microcode	: 0xffffffff
cpu MHz		: 2299.997
cache size	: 25600 KB
physical id	: 0
siblings	: 8
core id		: 7
cpu cores	: 8
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni cx16 hypervisor lahf_lm pti
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips	: 4599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

[root@10 ~]# uname -a
Linux 10.202.42.22 5.2.14-1.el7.elrepo.x86_64 #1 SMP Tue Sep 10 10:50:19 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@10 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

What did you do?

My program is running in the container.

I've met a strange memory leak problem with os.ReadFile(). My program will read some binary file data by soft-link(e.g. /proc/TARGET-PID/exe). After several days, I had found that my program process RSS memory usage is increasing and the pprof heap shows that only far smaller than RSS memory is used. The biggest part of memory usage just points to os.ReadFile().

profile003 (1)

As profile003 has shown, currently the heap just has 384MB but the target process RSS is 691.45MB.

Here is the heap dump.

heap.out.zip

I also manually set GODEBUG=madvdontneed=1 to speed RSS memory collection up, but it didn't work.

What did you expect to see?

All the garbage memory can be collected.

What did you see instead?

Target process RSS is increasing, and the memory read from os.ReadFile() could not be GC.
@D1CED
Copy link

D1CED commented Sep 13, 2022

This is totally expected given that GOGC=100 is set (the default). This means that the Go runtime keeps the process heap at about double the size of all reachable objects (live heap). You can set GOGC to a lesser value but then CPU load will increase.

The Go team has recently published a really nice piece of documentation with interactive views explaining the workings of the garbage collector https://go.dev/doc/gc-guide.

@mvdan
Copy link
Member

mvdan commented Sep 13, 2022

What @D1CED said. Also, see https://golang.org/wiki/Questions.

@mvdan mvdan closed this as not planned Won't fix, can't repro, duplicate, stale Sep 13, 2022
@g0194776
Copy link
Author

@D1CED @mvdan
Thanks for your help. I know what you said, but apparently, it's different. That was one of my cases, In my other case, the RSS was up to 3.4GB, and the heap usage was still 500~600MB.

I would like to know how can I debug it, or what kind of evidence should I upload.

@mvdan
Copy link
Member

mvdan commented Sep 13, 2022

Please see https://github.com/golang/go/wiki/Questions if you're looking for help to debug what's going on. You can always come back to the issue tracker when you've got a memory leak that can be reproduced.

@g0194776
Copy link
Author

@mvdan Okay, I really need your help. I will reproduce it after the process RSS over 3GB and generate a pprof heap dump uploading here.

@Jayuda
Copy link

Jayuda commented Feb 19, 2023

Do you solved this issue ?
I have this problems too. On go 1.19

@g0194776
Copy link
Author

Not yet. @Jayuda

@golang golang locked as resolved and limited conversation to collaborators Mar 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Jayuda @g0194776 @mvdan @D1CED