Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "fatal error: signal_recv: inconsistent state" crash on ios/arm64 #46466

Closed
stonelgh opened this issue May 31, 2021 · 15 comments
Closed
Labels
arch-arm64 FrozenDueToAge mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@stonelgh
Copy link

stonelgh commented May 31, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/stone/Library/Caches/go-build"
GOENV="/Users/stone/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/stone/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/stone/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/kk/t_y76cl95cz5y1dtl9spqxt00000gn/T/go-build233946771=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Make a library for a iOS App with the latest gomobile. The program crashes a lot of times with the following output:

fatal error: signal_recv: inconsistent state

goroutine 20 [running]:
runtime.throw(0x10686cce1, 0x1f)
        /usr/local/go/src/runtime/panic.go:1117 +0x54 fp=0x13004df90 sp=0x13004df60 pc=0x105c804b4
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:161 +0x1d4 fp=0x13004dfb0 sp=0x13004df90 pc=0x105cb0684
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x20 fp=0x13004dfd0 sp=0x13004dfb0 pc=0x10603ab50
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x13004dfd0 sp=0x13004dfd0 pc=0x105cb3a94
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x44

...

We haven't found a way to consistently reproduce the failure. It usually crashes when users try to resume the app.

What did you expect to see?

The app is resumed and runs normally.

What did you see instead?

The app crashed.

@stonelgh
Copy link
Author

I modified runtime/os_darwin.go to print the return value of read(sigNoteRead, unsafe.Pointer(&b), 1) called in function sigNoteSleep.
It outputs -4 when it crashes. According to runtime/stubs2.go, -4 is a negative errno value. Errno 4 means EINTR(interrupted system call).
That indicates the read system call is interrupted and sigNoteSleep returns prematurely. As a result, sig.state = sigReceiving is detected by runtime/sigqueue.go: func signal_recv() as an inconsistent state and throws.

Shall we check the return value of read and retry when it is interrupted?

@ianlancetaylor
Copy link
Contributor

Yes.

@ianlancetaylor
Copy link
Contributor

At least, yes, try that, and see if it fixes the problem. Thanks.

@stonelgh
Copy link
Author

stonelgh commented Jun 1, 2021

I've made a code change in sigNoteSleep. Will let you know the result.

func sigNoteSleep(*note) {
	for {
		entersyscallblock()
		var b byte
		n := read(sigNoteRead, unsafe.Pointer(&b), 1)
		exitsyscall()
		if n != -_EINTR {
			break
		}
	}
}

Updated at 2021-06-10 to remove the check to sig.state.

@jim3ma
Copy link

jim3ma commented Jun 1, 2021

Same issue in macos/amd64

@dr2chase
Copy link
Contributor

dr2chase commented Jun 1, 2021

You have a reproducer for macos that does not require gomobile?

@stonelgh
Copy link
Author

stonelgh commented Jun 2, 2021

Not yet. Maybe @jim3ma have one?

@dr2chase
Copy link
Contributor

dr2chase commented Jun 2, 2021

Any news on the goodness of the proposed fix?

@stonelgh
Copy link
Author

stonelgh commented Jun 3, 2021

Due to the fact that we haven't found a way to consistently reproduce the issue, we haven't observed that the new logic have been triggered by now. We'll continue testing until we get enough evidences. Will let you know ASAP.

@jim3ma
Copy link

jim3ma commented Jun 3, 2021

Emm, observed many times, but can not consistently reproduce the issue.

@cherrymui
Copy link
Member

@jim3ma It doesn't have to consistently reproduce. As long as it fails somewhat frequently, that would be very helpful. Thanks!

@cherrymui cherrymui added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. mobile Android, iOS, and x/mobile arch-arm64 OS-Darwin labels Jun 7, 2021
@cherrymui cherrymui added this to the Backlog milestone Jun 7, 2021
@stonelgh
Copy link
Author

@dr2chase @cherrymui @ianlancetaylor Confirmed: the proposed patch (with an update to drop the check to sig.state) can fix the mentioned crash.

@kavin808
Copy link

Same issue on our iOS App which causing crashes with SIGPIPE exception.
We are now adding signal(SIGPIPE, SIG_IGN) to ignore this exception type as a temporary solution.Hope this issue can be fixed soon.

@ianlancetaylor
Copy link
Contributor

@stonelgh Thanks for testing and reporting back.

@gopherbot
Copy link

Change https://golang.org/cl/326778 mentions this issue: runtime: loop on EINTR in macOS sigNoteSleep

@golang golang locked and limited conversation to collaborators Jun 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm64 FrozenDueToAge mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests

7 participants