Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase of memory usage after upgrading from go1.8 to go2.2 #66541

Closed
arturg-acoustic opened this issue Mar 26, 2024 · 7 comments
Closed

Increase of memory usage after upgrading from go1.8 to go2.2 #66541

arturg-acoustic opened this issue Mar 26, 2024 · 7 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@arturg-acoustic
Copy link

arturg-acoustic commented Mar 26, 2024 •
edited

Go version

go1.22

Output of go env in your module/workspace:

GOARCH=amd64 GOOS=linux 
      "GODEBUG" : "disablethp=1, gctrace=1"

What did you do?

I have set "GODEBUG" : "disablethp=1, gctrace=1" in lambda environment
but it does not seem to have any effect. Unfortunately I can't change the linux provided.al2023 image, as it is not an option using serverless lambda.

What did you see happen?

My lambda memory consumption increased from 80-90mb to 700mb+

What did you expect to see?

I expected to see the same level of memory consumption.
@randall77
Copy link
Contributor

go2.2 does not exist.
Please include the complete output of go env.
I don't know what a "lambda environment" is, so some additional context would help.

@randall77 randall77 added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 26, 2024
@thediveo
Copy link

AWS environment, so probably an AWS problem.

@pablodz
Copy link

pablodz commented Mar 27, 2024

never seen go2.2

@arturg-acoustic
Copy link
Author

it's running on AWS lambda (serverless) on provided.al2023 image.
it's go 1.22

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Mar 27, 2024
@randall77
Copy link
Contributor

"GODEBUG" : "disablethp=1, gctrace=1"

The space in that string might be preventing gctrace=1 from being recognized.

@udhos
Copy link

udhos commented Mar 29, 2024

@arturg-acoustic Did you try GOMEMLIMIT ?

I am hitting a somewhat similar issue.
A new proxy application built with Go 1.22.1 increases memory usage at a slow constant rate.
This is my troubleshooting history:

1 - Requests are not served at constant rate but memory increases at slow constant rate.
2 - Profiling did not reveal a culprit for memory leak.
3 - Failed to reproduce it on staging environments.
4 - Attempted "disablethp": It did improve memory usage but did not fix the slow constant growth.
5 - Now I am experimenting with GOMEMLIMIT to restrict memory usage to a reasonable limit.
6 - If GOMEMLIMIT fails, next attempt would be to rebuild the application with 1.19 toolchain.

@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@udhos @randall77 @thediveo @gopherbot @seankhliao @pablodz @arturg-acoustic