Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exec.Command cannot get all stdout message #38268

Closed
liesauer opened this issue Apr 6, 2020 · 9 comments
Closed

exec.Command cannot get all stdout message #38268

liesauer opened this issue Apr 6, 2020 · 9 comments

Comments

@liesauer
Copy link

liesauer commented Apr 6, 2020

What version of Go are you using (go version)?

go version go1.13.4 darwin/amd64

What did you do?

try to run git command using golang and get the realtime output.

package main
 
import (
    "fmt"
    "io"
    "log"
    "os/exec"
    "syscall"
)
 
func output(reader io.ReadCloser) error {
    buf := make([]byte, 1024)
    for {
        num, err := reader.Read(buf)
        if err != nil && err != io.EOF {
            return err
        }
        if num > 0 {
            fmt.Printf("%s", string(buf[:num]))
        }
    }
    return nil
}
 
func execute() error {
    cmd := exec.Command("git", "clone", "https://github.com/dotnet/runtime.git")
 
    stdout, _ := cmd.StdoutPipe()
    stderr, _ := cmd.StderrPipe()
 
    if err := cmd.Start(); err != nil {
        log.Printf("Error executing command: %s......\n", err.Error())
        return err
    }
 
    go output(stdout)
    go output(stderr)
 
    if err := cmd.Wait(); err != nil {
        log.Printf("Error waiting for command execution: %s......\n", err.Error())
        return err
    }

    log.Printf("end");
 
    return nil
}
 
func main() {
    execute()
}

What did you expect to see?

Cloning into 'xxx'...
remote: Enumerating objects: 43, done.
remote: Counting objects: 100% (43/43), done.
remote: Compressing objects: 100% (43/43), done.
Receiving objects:   0% (805/1567998), 356.01 KiB | 219.00 KiB/s

remote message and progress is missing.

What did you see instead?

Cloning into 'xxxx'...

and nothing more

@ccbrown
Copy link

ccbrown commented Apr 6, 2020

This looks racey. You might just need to wait for the goroutines to finish printing their output before returning from main.

@ccbrown
Copy link

ccbrown commented Apr 6, 2020

Also:

It is thus incorrect to call Wait before all reads from the pipe have completed.

@ianlancetaylor
Copy link
Member

Per above, doesn't seem to be a bug in Go here. The program needs to delay the call to Wait until it has finished reading from stdout and stderr.

@liesauer
Copy link
Author

liesauer commented Apr 6, 2020

@ianlancetaylor i don't think so. the program is still running and keep reading from stdout, if i change my command to any other command, it works properly. it just looks like git doesn't send the remote: xxxxxx to stdout directly. i wrote a sleep program to simulate the delay output, and i can say that it definitely not a "delay issue".

@liesauer
Copy link
Author

liesauer commented Apr 6, 2020

have you try running my code above?

@ccbrown
Copy link

ccbrown commented Apr 6, 2020

Your code doesn't compile due to an unused import.

But even if you fix that, running your code is pretty pointless since it's doing something that the docs explicitly say you shouldn't do. Code always needs to be well-formed to be a meaningful reproduction of an issue.

This is well-formed:

package main

import (
	"io"
	"log"
	"os"
	"os/exec"
	"sync"
)

func execute() error {
	cmd := exec.Command("git", "clone", "https://github.com/dotnet/runtime.git")

	stdout, _ := cmd.StdoutPipe()
	stderr, _ := cmd.StderrPipe()

	if err := cmd.Start(); err != nil {
		log.Printf("Error executing command: %s......\n", err.Error())
		return err
	}

	var wg sync.WaitGroup

	wg.Add(1)
	go func() {
		defer wg.Done()
		io.Copy(os.Stdout, stdout)
	}()

	wg.Add(1)
	go func() {
		defer wg.Done()
		io.Copy(os.Stderr, stderr)
	}()

	wg.Wait()

	if err := cmd.Wait(); err != nil {
		log.Printf("Error waiting for command execution: %s......\n", err.Error())
		return err
	}

	log.Printf("end")

	return nil
}

func main() {
	execute()
}

Also note:

The output of git clone depends on whether or not git thinks it's being run in a terminal. The output will not be the same as if you run the git clone command in your terminal directly. See the --progress flag in git clone --help.

Your output function ignores io.EOF, which causes it to loop infinitely.

There is an easier way to pipe to your stdout/stderr.

package main

import (
	"log"
	"os"
	"os/exec"
)

func execute() error {
	cmd := exec.Command("git", "clone", "https://github.com/dotnet/runtime.git", "--progress")
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	if err := cmd.Start(); err != nil {
		log.Printf("Error executing command: %s......\n", err.Error())
		return err
	}

	if err := cmd.Wait(); err != nil {
		log.Printf("Error waiting for command execution: %s......\n", err.Error())
		return err
	}

	return nil
}

func main() {
	execute()
}

@liesauer
Copy link
Author

liesauer commented Apr 6, 2020

@ccbrown first of all, thanks for pointing out the issues of my code! after testing a couple of times, my conclusion is it may be a git issue, because if i run the git clone command under terminal directly, i don't need to set --progress explicitly, --process is on by default(that is what disturbs me). after adding --progress explicitly, all the codes above can access the realtime output exactly the same as in the terminal now(although my code have some issues). but i am taking the second way of yours, it is more easier and clearer obviously. thanks!

@Rwing
Copy link

Rwing commented Sep 25, 2020

@liesauer I had the same problem in another language today, and I've found a clear explanation here, FYI https://stackoverflow.com/questions/35700994/how-to-capture-git-clone-output-with-fork-exec

@liesauer
Copy link
Author

@liesauer I had the same problem in another language today, and I've found a clear explanation here, FYI https://stackoverflow.com/questions/35700994/how-to-capture-git-clone-output-with-fork-exec

right, but i don't understand why git outputs the details to stderr.

@golang golang locked and limited conversation to collaborators Sep 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants