Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: unexpected return pc for runtime.gopark called from 0x1 #19955

Closed
t57root opened this issue Apr 13, 2017 · 8 comments
Closed

runtime: unexpected return pc for runtime.gopark called from 0x1 #19955

t57root opened this issue Apr 13, 2017 · 8 comments
Labels
FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.

Comments

@t57root
Copy link

t57root commented Apr 13, 2017

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/workspace/zagen/src/zagen/../../"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build295860934=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

I use signal.Notify() to get a channel to notify me when SIGTERM/SIGINT occurred. But when I hit ctrl+c(which produce SIGINT), the program sometimes crashed, showing runtime: unexpected return pc for runtime.gopark called from 0x1. This error seems happened conditionally, not inevitably.
It's a large project and I don't know where exactly caused this problem. I kinda think such low-level error should not be caused by my user code. Here's the callstack printed automatically:

runtime: unexpected return pc for runtime.gopark called from 0x1
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x9e8032, 0x11)
    /usr/local/go/src/runtime/panic.go:566 +0x95
runtime.gentraceback(0xffffffffffffffff, 0xc4200236d8, 0x0, 0xc420001380, 0x0, 0x0, 0x7fffffff, 0x7f2d5e7fb8b8, 0x0, 0x0, ...)
    /usr/local/go/src/runtime/traceback.go:317 +0x13f6
runtime.scanstack(0xc420001380, 0xc42001b228)
    /usr/local/go/src/runtime/mgcmark.go:777 +0x259
runtime.scang(0xc420001380, 0xc42001b228)
    /usr/local/go/src/runtime/proc.go:828 +0x14b
runtime.markroot.func1()
    /usr/local/go/src/runtime/mgcmark.go:236 +0x6d
runtime.systemstack(0xc42001a000)
    /usr/local/go/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1079

goroutine 903 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420192698 sp=0xc420192690
runtime.markroot(0xc42001b228, 0xc40000000a)
    /usr/local/go/src/runtime/mgcmark.go:241 +0x2ba fp=0xc420192710 sp=0xc420192698
runtime.gcDrain(0xc42001b228, 0x5)
    /usr/local/go/src/runtime/mgcmark.go:963 +0xab fp=0xc420192748 sp=0xc420192710
runtime.gcBgMarkWorker(0xc42001a000)
    /usr/local/go/src/runtime/mgc.go:1452 +0x1d1 fp=0xc4201927b8 sp=0xc420192748
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4201927c0 sp=0xc4201927b8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [select]:
zagen/core.Schedule()
    /root/workspace/zagen/src/zagen/core/scheduler.go:257 +0x602
main.daemon(0xc4200e5550, 0x13)
    /root/workspace/zagen/src/zagen/daemon.go:65 +0x5b0
main.main()
    /root/workspace/zagen/src/zagen/main.go:110 +0x645

goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 6 [syscall, 1 minutes]:
os/signal.signal_recv(0xc420023fa8)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [chan receive]:
zagen/rpc.(*Logger).Daemon(0xc42013c380)
    /root/workspace/zagen/src/zagen/rpc/log.go:142 +0xa1
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:30 +0xf2

goroutine 10 [select, 1 minutes]:
zagen/rpc.(*JsonRpc).Daemon(0xc4200b81e0)
    /root/workspace/zagen/src/zagen/rpc/json.go:202 +0x152
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:40 +0x1f8

goroutine 11 [select, 1 minutes]:
zagen/rpc.(*Transporter).Work(0xc420016960)
    /root/workspace/zagen/src/zagen/rpc/trans.go:78 +0x18f
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:46 +0x31c

goroutine 12 [select, 1 minutes]:
zagen/rpc.(*Transporter).Work(0xc4200169c0)
    /root/workspace/zagen/src/zagen/rpc/trans.go:78 +0x18f
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:46 +0x31c

goroutine 24 [chan receive]:
zagen/core.MonitorUpdate(0xe404a0, 0xc4200b81e0)
    /root/workspace/zagen/src/zagen/core/scheduler.go:243 +0x80
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:64 +0x5ab

goroutine 25 [select, 1 minutes, locked to thread]:
runtime.gopark(0xa2df80, 0x0, 0x9def92, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc42001f730, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc42001f730)
    /usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 448 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc4200e4420)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 35 [runnable, locked to thread]:
github.com/mattn/go-sqlite3._Cfunc__sqlite3_step(0x22bf508, 0xc42046c860, 0xc42046c868, 0x65)
    github.com/mattn/go-sqlite3/_obj/_cgo_gotypes.go:233 +0x4d
github.com/mattn/go-sqlite3.(*SQLiteStmt).exec(0xc4204b5530, 0x7f2d815061e8, 0xc4200122c8, 0xc4204bee10, 0x6, 0x6, 0x0, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:821 +0x33a
github.com/mattn/go-sqlite3.(*SQLiteConn).exec(0xc42040d810, 0x7f2d815061e8, 0xc4200122c8, 0xc42014f260, 0x6c, 0xc4204bee10, 0x6, 0x6, 0x10, 0x4e2f01, ...)
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:438 +0x2f9
github.com/mattn/go-sqlite3.(*SQLiteConn).Exec(0xc42040d810, 0xc42014f260, 0x6c, 0xc42015ac60, 0x6, 0x6, 0x6, 0x0, 0x0, 0xa2ad9e)
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:418 +0x1f0
database/sql.(*DB).exec(0xc4200e4630, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x4bd37f01, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/database/sql/sql.go:1036 +0x4ca
database/sql.(*DB).Exec(0xc4200e4630, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x10, 0xc420049388, 0xc42046c810, 0xc4200493c8)
    /usr/local/go/src/database/sql/sql.go:1010 +0xa2
github.com/go-xorm/xorm.(*Session).innerExec(0xc42012b180, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x0, 0xc420049358, 0xa2ad7a, 0x10)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/session_raw.go:110 +0x88
github.com/go-xorm/xorm.(*Session).exec.func1(0x10, 0x10, 0xc42046c820, 0x0)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/session_raw.go:130 +0xe8
github.com/go-xorm/xorm.(*Engine).logSQLExecutionTime(0xc420255720, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0xc4200494d0, 0x6c, 0x0, 0x0, ...)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/engine.go:297 +0x54
github.com/go-xorm/xorm.(*Session).exec(0xc42012b180, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x6c, 0x6, 0xc42015aba0, 0x6)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/session_raw.go:133 +0x1f9
github.com/go-xorm/xorm.(*Session).innerInsert(0xc42012b180, 0x9628c0, 0xc42038d860, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/session_insert.go:456 +0xf90
github.com/go-xorm/xorm.(*Session).Insert(0xc42012b180, 0xc420049a68, 0x1, 0x1, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/session_insert.go:49 +0x327
github.com/go-xorm/xorm.(*Engine).Insert(0xc420255720, 0xc420049a68, 0x1, 0x1, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/github.com/go-xorm/xorm/engine.go:1425 +0xa4
zagen/fw.(*DBO).insertKeepOld(0xc4202548c0, 0xe3f360, 0xc42038d860, 0xe3f360, 0xc42038dbc0, 0x0, 0x9a84e0, 0xc4204b4801, 0xc42038dbc0)
    /root/workspace/zagen/src/zagen/fw/dbo.go:307 +0x517
zagen/fw.(*DBO).Insert(0xc4202548c0, 0xe3f360, 0xc42038d860, 0xe3f360, 0xc42038dbc0, 0xc4201fa000, 0x80, 0x100)
    /root/workspace/zagen/src/zagen/fw/dbo.go:254 +0x58
zplug/pkg.Collect(0xe00000000, 0x0, 0x0)
    /root/workspace/zagen/src/zplug/pkg/main.go:118 +0xfa4
zagen/core.(*procRuntime).run(0xc4200e5ad0)
    /root/workspace/zagen/src/zagen/core/proc.go:136 +0x180
created by zagen/core.(*procRuntime).scheduleRoutine
    /root/workspace/zagen/src/zagen/core/proc.go:208 +0x1b8

goroutine 37 [GC assist wait]:
regexp.(*bitState).reset(0xc420184300, 0x5f, 0x2)
    /usr/local/go/src/regexp/backtrack.go:95 +0xff
regexp.(*machine).backtrack(0xc42048b680, 0xe41e80, 0xc42048b758, 0x0, 0x5f, 0x2, 0x7)
    /usr/local/go/src/regexp/backtrack.go:321 +0xa1
regexp.(*Regexp).doExecute(0xc42010cbe0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4201841e0, 0x5f, 0x0, 0x2, ...)
    /usr/local/go/src/regexp/exec.go:437 +0x2e5
regexp.(*Regexp).replaceAll(0xc42010cbe0, 0x0, 0x0, 0x0, 0xc4201841e0, 0x5f, 0x2, 0xc420395850, 0xc4201841e0, 0x5f, ...)
    /usr/local/go/src/regexp/regexp.go:506 +0x10c
regexp.(*Regexp).ReplaceAllString(0xc42010cbe0, 0xc4201841e0, 0x5f, 0x9dbd48, 0x1, 0x5f, 0x6)
    /usr/local/go/src/regexp/regexp.go:467 +0x125
zplug/process.getProcModule(0x3e5, 0xc42046d220, 0x1, 0x1)
    /root/workspace/zagen/src/zplug/process/proc.go:58 +0x1f4
zplug/process.GetProcDetail(0x3e5, 0x3)
    /root/workspace/zagen/src/zplug/process/proc.go:253 +0x5fa
zplug/process.collect(0x0, 0x0)
    /root/workspace/zagen/src/zplug/process/collect.go:47 +0x432
zplug/process.Main(0xe00000030, 0xa2e030, 0xc42019678c)
    /root/workspace/zagen/src/zplug/process/main.go:4 +0x22
zagen/core.(*procRuntime).run(0xc4200e5c30)
    /root/workspace/zagen/src/zagen/core/proc.go:136 +0x180
created by zagen/core.(*procRuntime).scheduleRoutine
    /root/workspace/zagen/src/zagen/core/proc.go:208 +0x1b8

goroutine 149 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc42008fe40)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 92 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc42008e210)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 61 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc42017b340)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 1251 [select]:
github.com/mattn/go-sqlite3.(*SQLiteStmt).exec.func1(0x7f2d815061e8, 0xc4200122c8, 0xc42015b020, 0x21c61a8)
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:813 +0x17b
created by github.com/mattn/go-sqlite3.(*SQLiteStmt).exec
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:818 +0x285

goroutine 452 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc4200e4630)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 154 [chan receive, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc42017a160)
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 1253 [runnable]:
github.com/mattn/go-sqlite3.(*SQLiteStmt).exec.func1(0x7f2d815061e8, 0xc4200122c8, 0xc42015b200, 0x7f2d54014dd8)
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:813 +0x17b
created by github.com/mattn/go-sqlite3.(*SQLiteStmt).exec
    /root/workspace/zagen/src/github.com/mattn/go-sqlite3/sqlite3.go:818 +0x285
@bradfitz
Copy link
Contributor

This might've already been fixed in Go 1.8. Could you try Go 1.8.1?

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 13, 2017
@knightXun
Copy link

@t57root does this appear in Go 1.8?

@t57root
Copy link
Author

t57root commented Apr 16, 2017

Sorry about the late reply. According to my experience so far, seems it doesn't appear in Go 1.8. because it's not inevitable.
if this have been fixed in go 1.8, will the fix be backported to 1.7?

@bradfitz
Copy link
Contributor

We generally only backport security issues. We've occasionally backported runtime crashes, so this might qualify, but it's not clear it's affecting enough people to be worth it. We'd encourage you to use Go 1.8 instead.

/cc @aclements for his opinion.

@bradfitz bradfitz added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Apr 16, 2017
@aclements
Copy link
Member

I agree with @bradfitz that this doesn't seem to qualify for a backport. Lots of things were fixed in 1.8, so we'd much rather users move forward (or tell us if there's an issue in Go preventing them for doing so!). Closing for now, but if you disagree or discover that you can reproduce it with 1.8, please comment and we'll reopen.

@guidovranken
Copy link

Hi, @bradfitz @aclements I am using Go 1.9.2 and I got the same error as OP after several billion runs of a Go program. I happen to have the core dump, which I can send you in private if you wish to analyze it further. Please let me know, thanks.

@ianlancetaylor
Copy link
Contributor

@guidovranken Can you show us the stack trace? I'd like to see if this looks like the same issue or a different one.

Note that one error after several billion runs could easily be a random memory failure. Even if the bug is in Go if it takes several billion runs to repeat the problem we are unlikely to be able to fix it.

@guidovranken
Copy link

@ianlancetaylor due to reasons I only have this screenshot of the original crash: https://i.imgur.com/mvMj24y.png

In case it is relevant, the Go program was compiled as a static library and launched from a C++ program (a fuzzer actually, hence the billions of runs).

@golang golang locked and limited conversation to collaborators Nov 12, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@bradfitz @t57root @aclements @ianlancetaylor @guidovranken @knightXun @gopherbot