Why do you think that GC does not happen? In the output I see "num-gc: 15002".

@ianlancetaylor After 15002 GC, it never happen again. You can see the last gc timestamp:
last-gc: 2023-10-17 03:15:28.72404389 +0000 UTC

Each major Go release is supported until there are two newer major releases.
I noticed that you mentioned using go1.19.0, will this be reproduced in the latest version?

@qiulaidongfeng , the issue happened in our production environment. I can not reproduce it in local environment. Now we do not have the plan to upgrade the golang version. It seems that the next-gc value is calucated by the following function.

func (c *gcControllerState) heapGoalInternal() (goal, minTrigger uint64)

From the metrics go_memstats_next_gc_bytes – shows the target heap size of the next GC cycle. We can see that the value spike from 300Mi to int64 max value

Even if I run gc manually, the allocated heap memory was reduced but it still was not returned to OS. It seems that when issue happens, runtime scavenge still can not work as well.

It is possible that debug.SetGCPercent(-1) is called somewhere.

@WangLeonard , in our code, we do not call SetGCPercent(-1)

Can anyone have any comments? Thanks a lot.

It's going to be hard to fix this without having a way to reproduce it.
If you can, it would help to pepper (*gcControllerState) heapGoalInternal with checks of all the values it is using to compute the bad value. For instance,

goal = c.gcPercentHeapGoal.Load()
if goal > 1<<60 {
    print("c.gcPercentHeapGoal is too large", goal)
    throw("exiting")
}

Similarly for all the other pieces of info it accesses. That might give us a clue about where a bad value is coming from.

I think I find the root cause for this issue in our environment. Actually it has nothing with golang GC code.
We use a tool gops (https://github.com/google/gops) for debug purpose. And we open a tcp port for the gops agent.
There is a potential security flaw. Anyone can send tcp packet to the port to trigger the gops command. If by change, the first byte of the data is 0x10 (defined by gops), it will trigger SetGCPercent() unexpectedly.
Thanks everyone.