os: EINTR on darwin from os.Process.Wait #36644

crawshaw · 2020-01-19T20:57:03Z

Background: our Go code is built with -buildmode=c-archive and linked into a macOS app built by xcode. We are not installing any signal handlers, but it is possible some Apple library is doing so behind our backs. We are using the os/exec package to periodically execute netstat -an.

Occasionally, the os/exec.Cmd.Wait method is returning EINTR. This EINTR comes directly from os.Process.Wait. Unfortunately the os/exec package does not expect EINTR, and so it treats it as if the fork process has exited and cleans up its state, so that Cmd.Wait cannot be called again.

The darwin man pages wait(2) and sigaction(2) suggest this should only happen if a signal handler does not have the SA_RESTART flag set. I have not yet determined if Apple is setting another handler for us, or if this is something unusual like Issue #11180.

Even if this is a core Apple library setting a handler (we use no third-party code), it may be worth handling EINTR for darwin in os.Process.Wait, to make the object useful in Go code linked with -buildmode=c-archive.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-01-21T05:16:52Z

Perhaps we do have to change the os package for this, but I'd like to clearly understand the root cause first. We have not historically tried to handle all possible EINTR errors due to a non-Go signal handler that does not use SA_RESTART.

crawshaw · 2020-01-21T15:00:04Z

I did a little investigating. I was unable to replicate this running a normal binary, but it happens reliably when run under lldb:

$ cat eintr.go 
package main

import "os/exec"

import "C"

func main() {}

//export Netstat
func Netstat() {
	cmd := exec.Command("netstat", "-na")
	if _, err := cmd.Output(); err != nil {
		panic(err)
	}
}

$ cat eintr.c 

extern void Netstat();

int main(void) {
	Netstat();
	return 0;
}

$ go build -buildmode=c-archive eintr.go && cc eintr.c eintr.a && lldb -o run ./a.out
(lldb) target create "./a.out"
Current executable set to './a.out' (x86_64).
(lldb) run
panic: wait: interrupted system call

goroutine 17 [running, locked to thread]:
main.Netstat()
	/Users/crawshaw/eintr.go:13 +0xae
main._cgoexpwrap_078bcfb61532_Netstat()
	_cgo_gotypes.go:45 +0x20
Process 41628 exited with status = 2 (0x00000002) 

Process 41628 launched: '/Users/crawshaw/a.out' (x86_64)
(lldb)

This is the typical way xcode runs development binaries, which is why we see it so frequently.

ianlancetaylor · 2020-01-21T15:29:47Z

What happens with gdb?

crawshaw · 2020-01-21T18:54:43Z

(It is surprisingly hard to install gdb on macOS 10.15 with codesigning.)

Results are similar:

~ $ gdb ./a.out 
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin19.2.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...
Loading Go Runtime support.
(gdb) start
Temporary breakpoint 1 at 0x1000843b0: file <autogenerated>, line 1.
Starting program: /Users/crawshaw/a.out 
[New Thread 0x1803 of process 75954]
[New Thread 0x1b03 of process 75954]
[New Thread 0x2603 of process 75954]
warning: unhandled dyld version (16)
[New Thread 0x180f of process 75954]
[New Thread 0x1c03 of process 75954]
[New Thread 0x1d03 of process 75954]
[New Thread 0x2303 of process 75954]
[New Thread 0x2403 of process 75954]
[New Thread 0x2503 of process 75954]
panic: wait: interrupted system call

goroutine 17 [running, locked to thread]:
main.Netstat()
	/Users/crawshaw/eintr.go:13 +0xae
main._cgoexpwrap_078bcfb61532_Netstat()
	_cgo_gotypes.go:45 +0x20
[Inferior 1 (process 75954) exited with code 02]
(gdb)

ianlancetaylor · 2020-01-21T18:58:15Z

Thanks. When I look at that gdb output the first thing that strikes me is: what signal is it? gdb normally reports every signal that a program receives (I don't know about lldb). Why is that not being reported here? When you run info handle, which signals are marked as No under Print?

crawshaw · 2020-01-21T19:11:56Z

Not printed, according to info handle: SIGALRM, SIGURG, SIGCHLD, SIGIO, SIGVTALRM, SIGPROF, SIGWINCH, SIGPOLL, SIGWAITING, SIGLWP, SIGPRIO, SIGCANCEL, SIGLIBRT.

I tried setting handle SIGCHLD print and got:

(gdb) handle SIGCHLD print
Signal        Stop	Print	Pass to program	Description
SIGCHLD       No	Yes	Yes		Child status changed
(gdb) run
Starting program: /Users/crawshaw/a.out 
[New Thread 0x2703 of process 77262]
[New Thread 0x2503 of process 77262]
[New Thread 0x1a03 of process 77262]
warning: unhandled dyld version (16)
[New Thread 0x1b03 of process 77262]
[New Thread 0x1c03 of process 77262]
[New Thread 0x1d03 of process 77262]
[New Thread 0x2303 of process 77262]
[New Thread 0x2403 of process 77262]
[New Thread 0x270f of process 77262]

Thread 3 received signal SIGCHLD, Child status changed.
panic: wait: interrupted system call

goroutine 17 [running, locked to thread]:
main.Netstat()
	/Users/crawshaw/eintr.go:13 +0xae
main._cgoexpwrap_078bcfb61532_Netstat()
	_cgo_gotypes.go:45 +0x20
[Inferior 1 (process 77262) exited with code 02]
(gdb)

(The debugger freezes on run if I attempt to set all signals to print.)

ianlancetaylor · 2020-01-21T19:39:52Z

Thanks. So it sounds like something is installing a SIGCHLD signal handler without setting SA_RESTART. Any idea what that is?

I'm trying to drill down on this because if that is what is happening, and we can't avoid it, then we need to add an EINTR loop to a dozen or more places in the standard library. You are just seeing it in wait because that is where your program is spending its time. We will still see the problem with lower frequency in read, write, sendmsg, etc.

crawshaw · 2020-01-21T20:21:42Z

I only see this behavior under lldb and gdb.

Googling around, here is a reference claiming that gdb installs a SIGCHLD handler on macOS: https://jmmv.dev/2008/10/boostprocess-and-sigchld.html (I notice Julio Merino is a Google employee, maybe jmmv@ will know more?)

Grepping through the GDB sources I don't see any macos-specific code for this, but I am not familiar with the codebase.

ianlancetaylor · 2020-01-21T20:55:56Z

I wonder if this is a side effect of using ptrace.

What happens if your Go c-archive does

    os.Notify(make(chan syscall.Signal, 1), syscall.SIGCHLD)

?

crawshaw · 2020-01-21T21:58:26Z

Calling signal.Notify captures the signal and lets the program exit cleanly.

networkimprov · 2020-01-22T01:18:47Z

cc @jmmv since @crawshaw meant to

ianlancetaylor · 2020-01-22T05:40:33Z

At least there is a workaround.

I don't know what else we can do short of adding EINTR loops throughout.

bcmills · 2020-01-22T15:13:05Z

I don't know what else we can do short of adding EINTR loops throughout.

That is #20400. There, you said (#20400 (comment)):

Unfortunately I agree with you that checking for EINTR in the standard library is the only realistic approach.

ianlancetaylor · 2020-01-22T15:49:46Z

The change can be simultaneously the only realistic approach and yet be undesirable.

gopherbot · 2020-05-08T04:49:25Z

Change https://golang.org/cl/232862 mentions this issue: internal/poll, os: loop on EINTR

networkimprov · 2020-05-12T20:32:02Z

I posted a 1.14 backport request for this: #39026

Historically we've assumed that we can install all signal handlers with the SA_RESTART flag set, and let the system restart slow functions if a signal is received. Therefore, we don't have to worry about EINTR. This is only partially true, and we've added EINTR checks already for connect, and open/read on Darwin, and sendfile on Solaris. Other cases have turned up in golang#36644, golang#38033, and golang#38836. Also, golang#20400 points out that when Go code is included in a C program, the C program may install its own signal handlers without SA_RESTART. In that case, Go code will see EINTR no matter what it does. So, go ahead and check for EINTR. We don't check in the syscall package; people using syscalls directly may want to check for EINTR themselves. But we do check for EINTR in the higher level APIs in os and net, and retry the system call if we see it. This change looks safe, but of course we may be missing some cases where we need to check for EINTR. As such cases turn up, we can add tests to runtime/testdata/testprogcgo/eintr.go, and fix the code. If there are any such cases, their handling after this change will be no worse than it is today. For golang#22838 Fixes golang#20400 Fixes golang#36644 Fixes golang#38033 Fixes golang#38836 Change-Id: I7e46ca8cafed0429c7a2386cc9edc9d9d47a6896 Reviewed-on: https://go-review.googlesource.com/c/go/+/232862 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> (cherry picked from commit 8c1db77)

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 21, 2020

ianlancetaylor added this to the Backlog milestone Jan 21, 2020

bcmills mentioned this issue Mar 23, 2020

os: write should handle EINTR #38033

Closed

gopherbot closed this as completed in 8c1db77 May 12, 2020

liggitt mentioned this issue Jun 25, 2020

consider setting GODEBUG=asyncpreemptoff=1 kubernetes/kubernetes#92521

Closed

golang locked and limited conversation to collaborators May 12, 2021

gopherbot added the FrozenDueToAge label May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os: EINTR on darwin from os.Process.Wait #36644

os: EINTR on darwin from os.Process.Wait #36644

crawshaw commented Jan 19, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020 •

edited

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020 •

edited

networkimprov commented Jan 22, 2020

ianlancetaylor commented Jan 22, 2020

bcmills commented Jan 22, 2020

ianlancetaylor commented Jan 22, 2020

gopherbot commented May 8, 2020

networkimprov commented May 12, 2020

os: EINTR on darwin from os.Process.Wait #36644

os: EINTR on darwin from os.Process.Wait #36644

Comments

crawshaw commented Jan 19, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020 • edited

crawshaw commented Jan 21, 2020

ianlancetaylor commented Jan 21, 2020

crawshaw commented Jan 21, 2020 • edited

networkimprov commented Jan 22, 2020

ianlancetaylor commented Jan 22, 2020

bcmills commented Jan 22, 2020

ianlancetaylor commented Jan 22, 2020

gopherbot commented May 8, 2020

networkimprov commented May 12, 2020

ianlancetaylor commented Jan 21, 2020 •

edited

crawshaw commented Jan 21, 2020 •

edited