@ALTree Can we get any fix in coming releases? or we should wait till Go1.10? Because my program already running in many production devices almost 10k devices and all are getting impact very badly.
Appreciate if it can be validated and fixed at earliest. Thanks.

@Durgababu unfortunately go1.9 is pretty much done at this point (we're deep in the code freeze period) so it's unlikely a fix for this problem (assuming this is a valid issue) will be included in go1.9.

See this wiki page for a description of the go release cycle.

In any case I haven't investigated this (it's not my area of expertise, and I don't want to risk missing something trivial and/or giving you a wrong diagnosis). That's why I tagged it NeedsInvestigation.

Tentatively cc'ing @aclements.

Here is a simpler version of the program to demonstrate the issue:

package main

import (
        "fmt"
        "os"
        "os/exec"
        "runtime"
        "time"
        "strconv"
)

func main() {
        for {
                output, err := exec.Command("ps", "-p", strconv.Itoa(os.Getpid()), "-o", "rss").Output()
                fmt.Printf("%s %v", output, err)

                time.Sleep(time.Second)
                runtime.GC()
        }
}

Thanks @zegl

Interesting. Using the shorter program from @zegl I do see the RSS size slowly rising with 1.8, but with 1.9 it is stable. I do not know what changed in this area.

@Durgababu could you try your real program with 1.9 to see if you still see the problem?

@ianlancetaylor Where can i get Go1.9? i am still seeing Go1.8.3 in https://golang.org/dl/
can you please provide go Go1.9 source?
can i use Go1.9beta?

@Durgababu You can use the following to get the latest Go 1.9 Beta as a separate command

$ go get golang.org/x/build/version/go1.9beta2
$ go1.9beta2 download

After that you can use go1.9beta2 as you would use the normal go tool.

@nussjustin
Great
Now i am not getting any leak with 1.9
When we can get Go1.9 stable version?

@Durgababu go1.9 release is scheduled for the first week of August.

Sounds like this is (will be) fixed in 1.9; thanks for confirming, @Durgababu. Closing.

@josharian, I am still getting leak in below 2 cases

1. when i commented runtime.gc() and run above simple code given by @zegl (leak is not immediate but after some time)

2. when i use cmd.Start() and cmd.Wait() instead of cmd.Output().
   Can you please reopen this issue?
   below is the sample program to demonstrate.

package main
import (
        "fmt"
        "os"
        "os/exec"
        //"runtime"
        "time"
        "strconv"
)
func main() {	
       index := 10
	for {
		cmd := exec.Command("sleep", "1")
		err := cmd.Start()
		if err != nil {
			fmt.Println(err)
		}
		err = cmd.Wait()
		fmt.Printf("Command finished with error: %v\n", err)
		index++
		if index > 10 {
			output, err := exec.Command("ps", "-p", strconv.Itoa(os.Getpid()), "-o", "rss").Output()
			fmt.Printf("%s %v", output)
			time.Sleep(time.Second)
			index = 0
		}
		//runtime.GC()
        }
} 

The latest code from @Durgababu (https://play.golang.org/p/ntUIyGh8sb) is slowly leaking for me too.

Version: go version go1.9beta2 linux/amd64

Can't reproduce with Go 1.9 Beta 2 on Linux and Mac.

Running the program for ~60m RSS never went above 7084 and stays constant now at 5020. The scavenger correctly releases the unused memory every few minutes. A memory profile also shows no retained memory.

Using a modified version (this version needs Go 1.9 for Duration.Truncate) of the program that also logs runtime statistics and running with GODEBUG=gctrace=1, I see the allocated memory correctly decreasing after each GC.

@Durgababu @zegl How long have you run the program? Have you tried looking at the GC trace output?

@nussjustin ,
So we should call runtime.GC() explicitly? i am not calling GC() in my code.
when i start the program memory is like 960-1000, but after some time it is increasing to 7000 and there after memory is not increasing much but still increasing . But it is taking 1000 to 7000 and not coming back why?
i ran the program for more than 1 hour.

Hi,
i am using this code(https://play.golang.org/p/BZWuAwcn4p) as per my business logic requirement(as i need to run different commands with pipe and need to capture output)

initially when program starts, RSS was 2000(2MB~), after 30-45 minutes, it increased to 32000(35MB~) and still keep on increasing....

You are leaking a goroutine. The goroutine created in line 64 will never exit as timer.C will never be closed and thus the goroutine holds onto some variables forever.

If you replace

timer := time.NewTimer(time.Millisecond * time.Duration(cmdTimeout))
go func(timer *time.Timer, cmd *exec.Cmd) {
	for _ = range timer.C {
		stdinWrite.Close()
		stdoutRead.Close()
		stderrRead.Close()
		errKill := cmd.Process.Signal(os.Kill)
		if errKill != nil {
			AgentLog.Error("Failed to stop the timer, error info : ", errKill)
		}
	}
}(timer, cmd)

with this

timer := time.AfterFunc(time.Millisecond * time.Duration(cmdTimeout), func() {
	stdinWrite.Close()
	stdoutRead.Close()
	stderrRead.Close()
	errKill := cmd.Process.Signal(os.Kill)
	if errKill != nil {
		AgentLog.Error("Failed to stop the timer, error info : ", errKill)
	}
})

both the goroutine and the variables can be garbage collected.

The reason for this (as mentioned at the beginning) is that timer.C will never be closed, since a time.Timer can be reused, and a range loop over a channel runs until the channel is closed (or break is used). By using time.AfterFunc the code is not only much clearer, but also the callback won't even be called and no goroutine created when Stop() is called before the time elapsed

@nussjustin
Thanks for the quick help.
Now it seems to be good. Initially it raised memory from 5MB to 15MB after that it is stable for last 30 min. Let me test some time and confirm.
But with above fix, it is looking far better than earlier.

@nussjustin
i ran the program for 4-5 hours with Go1.9beta2. seems no further increment in memory from 15MB.
Thanks a lot.

Looks like this was settled as a bug in the user's code, so I'm going to close the issue again.