Skip to content

os/exec: panic after execute Start #19918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ayanamist opened this issue Apr 11, 2017 · 18 comments
Closed

os/exec: panic after execute Start #19918

ayanamist opened this issue Apr 11, 2017 · 18 comments

Comments

@ayanamist
Copy link

ayanamist commented Apr 11, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.5 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build059832473=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

        cmd := exec.Command("some command")
	cmd.Stderr = ioutil.Discard
	stdout, err := cmd.StdoutPipe()
	if err != nil {
		return "", "", err
	}
	if err := cmd.Start(); err != nil {  // main.go:186
		if cmd.Process != nil {
			cmd.Process.Kill()
		}
		return "", "", err
	}
        .......
func panicRecoverWrap(f func()) func() {
	return func() {
		defer func() {
			if err := recover(); err != nil {
				log.Printf("panic %v: %s", err, debug.Stack()) // util.go:28
			}
		}()
		f()
	}
}

What did you expect to see?

Code execute without panic, maybe some errors

What did you see instead?

2017/04/11 08:00:41.089750 util.go:28: panic runtime error: invalid memory address or nil pointer dereference: goroutine 137 [running]:
runtime/debug.Stack(0xc42004f8c8, 0x67aa60, 0xc42000c100)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x79
main.panicRecoverWrap.func1.1()
        /home/foobar/.GOPATH/src/foobar/util.go:28 +0x76
panic(0x67aa60, 0xc42000c100)
        /usr/local/go/src/runtime/panic.go:458 +0x243
runtime.SetFinalizer(0x676c40, 0xc42000e420, 0x0, 0x0)
        /usr/local/go/src/runtime/mfinal.go:303 +0x84
os.(*file).close(0xc42000e420, 0x6cd67f, 0x1e)
        /usr/local/go/src/os/file_unix.go:146 +0xe0
os.(*File).Close(0xc42003c9a8, 0xc42004fcc0, 0xc420124630)
        /usr/local/go/src/os/file_unix.go:132 +0x33
os/exec.(*Cmd).closeDescriptors(0xc42009e580, 0xc4200c4bc0, 0x3, 0x4)
        /usr/local/go/src/os/exec/exec.go:262 +0x49
os/exec.(*Cmd).Start(0xc42009e580, 0x7e3820, 0xc42003c9a0)
        /usr/local/go/src/os/exec/exec.go:365 +0x442
created by main.main.func3
        /home/foobar/.GOPATH/src/foobar/main.go:186 +0x1cc
@odeke-em
Copy link
Member

@ayanamist would you mind posting a full/complete program, say in a playground snippet that can
be re-runnable by anyone?
Currently your code snippet doesn't match up with the execution lines in the crash report,
so it is hard to traceback to what tripped out.
Thanks.

@ayanamist
Copy link
Author

@odeke-em I have modified the description to add pointed line number in the line comment.
The full program is very complicated, but above provided code has been running for months without panic and i don't use any unsafe package methods and any third party package.
The original program restarts after panic and running well now, and at the same time it runs the same code provided above every 5 minutes on other thousands machines.
I think it maybe related with some race conditions?

@davecheney
Copy link
Contributor

davecheney commented Apr 11, 2017 via email

@ayanamist
Copy link
Author

@davecheney I add -race after found this issue, but runs for hour but got nothing.
I think the race condition occurs in runtime but not my program, because the code provided shows that, after constructing a os.Command object, i call Start immediately within the same method.

@davecheney
Copy link
Contributor

davecheney commented Apr 11, 2017 via email

@ayanamist
Copy link
Author

ayanamist commented Apr 11, 2017

@davecheney I mentioned above that i got nothing after enabled -race. I suspect race occurs in runtime because i don't use unsafe to manipulate anything and i don't use any third party library, and the exec.Command object does not escape out of the same method.

@davecheney
Copy link
Contributor

davecheney commented Apr 11, 2017 via email

@ayanamist
Copy link
Author

@davecheney So i have no idea why it panics on runtime.SetFinalizer

@davecheney
Copy link
Contributor

Is your source code available? Can you please provide a runnable code sample that will let others try to reproduce and debug the issue.

@ayanamist
Copy link
Author

@davecheney Sorry but source code is not available, and even i attach it, i think it won't reproduce. The same code has been running for months on thousands machines, and even it panics this morning, after restart it runs well until now.

@bradfitz
Copy link
Contributor

For questions about Go, see https://golang.org/wiki/Questions.

@ayanamist
Copy link
Author

@bradfitz Why this issue closed? I'm reporting a panic happening in runtime.SetFinalizer but not consulting for any golang usage.

@odeke-em
Copy link
Member

@ayanamist unfortunately your report as is isn't self contained/reproducible.
@davecheney and I have both asked for a complete program in our comments. Currently with the above content, it looks like more like a question for the Go help sites than a bug report with actionable items for contributors to look at.

Perhaps to help move up its status, here are some points that might be useful:

  1. What is "some-command"?
  2. stdout in your snippet is created but not used, anything related to that?
  3. Here is a sample complete snippet as the template encourages https://play.golang.org/p/JnGVQ8Xyr3 or
package main

import (
	"fmt"
	"io/ioutil"
	"os/exec"
)

func execIt() error {
	cmd := exec.Command("echo")
	cmd.Stderr = ioutil.Discard
	if _, err := cmd.StdoutPipe(); err != nil {
		return err
	}
	if err := cmd.Start(); err != nil {
		if cmd.Process != nil {
			cmd.Process.Kill()
		}
		return err
	}
	return cmd.Wait()
}

func main() {
	fmt.Printf("err: %v\n", execIt())
}

hope that looks similar to what you are doing
4) What is the nature of the code being ran? How heavily is the code used? I see your stack trace points to the use of SetFinalizer which for go1.7.5 is at https://github.com/golang/go/blob/release-branch.go1.7/src/runtime/mfinal.go#L303. Maybe that might help.

@ayanamist
Copy link
Author

  1. some-command is not part of my code, and i think it will not interfere with my program memory.
    1. More codes here
func func3() (boolean, error) {
	foundSomething := false
	cmd := exec.Command("echo", "foobar")
	cmd.Stderr = ioutil.Discard
	stdout, err := cmd.StdoutPipe()
	if err != nil {
		return foundSomething, err
	}
	if err := cmd.Start(); err != nil {  // main.go:186
		if cmd.Process != nil {
			cmd.Process.Kill()
		}
		return foundSomething, err
	}
	go func() {
		time.Sleep(5 * time.Second)
		cmd.Process.Kill()
	}()
	scanner := bufio.NewScanner(stdout)
	for scanner.Scan() {
		line := scanner.Text()
		i := strings.Index(line, ":")
		if i < 0 {
			continue
		}
		if strings.Contains(line, "foo") {
			foundSomething = true
		}
	}
	err = cmd.Wait()
	return foundSomething, err
}
  1. Every 5 minutes run this code in a single goroutine, and runtime.SetFinalizer is used directly by os.exec package but not my code.

@ianlancetaylor
Copy link
Member

I'm sorry, I don't entirely understand what you are saying. Did this crash happen more than once?

The crash reported in the original comment is during the call to clear the finalizer when closing a file descriptor. It's not a race condition. The only thing I can think of that could cause this crash is some sort of memory corruption.

@ayanamist
Copy link
Author

@ianlancetaylor So far, it only happens once. Do you mean other program cause my program memory corruption?

@ianlancetaylor
Copy link
Member

I don't think a different program could cause memory corruption, but it is possible that there was a hardware failure on your machine.

@ayanamist
Copy link
Author

I'll investigate whether there is a hardware problem, thank you.

@golang golang locked and limited conversation to collaborators Apr 12, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants