Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 #38827

Closed
BorisKozo opened this issue May 3, 2020 · 7 comments
Closed

runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 #38827

BorisKozo opened this issue May 3, 2020 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows

Comments

@BorisKozo
Copy link

BorisKozo commented May 3, 2020 •
edited

Note that I read carefully through #37927 and #35112 but it seems those issues were handled already.

What version of Go are you using (go version)?

Upgrade 1.13.10 -> 1.14.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Windows 10 (reproduces on other Windows versions as well)

What did you do?

Run my application memory benchmark

What did you expect to see?

Same or better results as in 1.13.10

What did you see instead?

Worse results

I have to add that our application is resource critical so every increase in memory or CPU usage raises eyebrows immediately.

I compiled our application with both GO 1.13.10 and 1.14.2 and the CPU and memory behavior are inconsistent between the two versions as the later consumes more memory (enough to trigger the benchmark failure). The behavior of the memory release is also different (see charts below) as there is no visible de-allocation of memory in 1.14.2.
I also tried changing the GOGC to various values but even at 10 the memory did not reach the behavior of 1.13.10 and CPU usage was considerable

go_13_GC_100

Initial run compiled with GO 1.13.10

Same code was compiled and run on the same machine with GOGC values as follows:
go_14_GC_10

GOGC = 10

go_14_GC_50

GOGC = 50

go_14_GC_100

GOGC = 100

go_14_GC_200

GOGC = 200

go_14_GC_500

GOGC = 500

in Issue #37525 @mknyszek mentions that another issue will be opened but I could not find the new issue.

Thanks.
@BorisKozo BorisKozo changed the title Increase in memory usage and behavior upgrading from 1.13.10 to 1.14.2 Increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 May 3, 2020
@agnivade
Copy link
Contributor

agnivade commented May 3, 2020 •
edited

Would you be able to share the output of setting GODEBUG=gctrace=1,gcpacertrace=1 in 1.13.10 vs 1.14.2 ? That would help us a lot in the investigation. Thanks.

@aclements @mknyszek

@agnivade agnivade changed the title Increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 runtime: increase in memory usage and different GC behavior upgrading from 1.13.10 to 1.14.2 May 3, 2020
@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 3, 2020
@BorisKozo
Copy link
Author

Hi, I generated the traces:
1.13.10
trace_1_13_10.txt

and
1.14.2
trace_1_14_2.txt

Look at the chart for 1.14.2, doesn't appear to be releasing anything and I am running with GOGC=100

2020-05-03_1554

@mknyszek
Copy link
Contributor

mknyszek commented May 4, 2020

Regarding memory release, #38617 brought our attention to a bug on Windows where in Go 1.14 the background scavenger wasn't running at all (the synchronous/eager scavenger and debug.FreeOSMemory should still work though). It's not clear to me yet that this is exactly the same problem, but try patching Go with https://golang.org/cl/229997 and see if it helps. Based on the traces you provided, I could see it helping.

Even with that patch, you may not be seeing the same log as in Go 1.13.10 with GODEBUG=gctrace=1 because we moved the scavenge trace under its own flag, which prints once per GC cycle. The flag for Go 1.14+ is GODEBUG=scavtrace=1.

@networkimprov
Copy link

@gopherbot add OS-Windows

@BorisKozo
Copy link
Author

@mknyszek I compiled master at a1ffbe9
and it seems the problem has been resolved.
go_14_fix

Thank you very much!
Which release is this fix planned to?

@networkimprov
Copy link

networkimprov commented May 4, 2020 •
edited

I posted a backport request. If there's no intervening security update, it will land in 1.14.3.

EDIT: #38856

@BorisKozo
Copy link
Author

BorisKozo commented May 4, 2020 via email

@golang golang locked and limited conversation to collaborators May 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@networkimprov @mknyszek @agnivade @BorisKozo @gopherbot