runtime: goroutine hung in _Cfunc_.GoString #30098
Labels
FrozenDueToAge
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
WaitingForInfo
Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Comments
odeke-em
changed the title
|
Thank you for the bug report and welcome to the Go project @cliffburdick! I'll kindly page some experts @randall77 @ianlancetaylor @aclements. |
odeke-em
added
the
NeedsInvestigation
|
Thanks for the report. Seeing more of the stack trace will help us debug this. Can you set GOTRACEBACK=crash in the environment, reproduce the issue, and then send a SIGQUIT to the hung process (either using kill or by pressing Ctrl-)? |
|
Hi @aclements, here is the output after a SIGQUIT. I did notice that it's now hanging in a different section that's printing a C.uint16: Here was the dlv bt: EDITED to add the |
|
I suspect that you have a non-preemptible loop. See #10958. In particular, can you use the "created by" line to figure out what code this goroutine is running? |
|
@ianlancetaylor Is the dlv output I posted in the last message what you were looking for? It has a stack trace that was caused by logrus. |
|
Thanks, you're right. I don't know what is going on here. |
|
What is your code doing here: ? |
That's the beginning of the goroutine of the code that handles all the incoming messages using cgo: Line 25 is the first line here: |
|
Hmm, so why then is goroutine 1 stopped here: It's just making a new goroutine. Somewhere inside newproc we're spinning, maybe? casgstatus? |
|
This still seems like a preemption issue, given that goroutine 5 is stuck in the call to Can you use delve to get a traceback of goroutine 1 (the one marked "running" in the traceback that started the goroutine)? It doesn't make sense that it would be stuck on the "go" statement in state "running". Also the output of Also, there are a huge number of GC worker goroutines. What's GOMAXPROCs/how many CPUs are you running with? I'm not sure if this would be relevant.
Yes, that surprised me, too. It's also weird that it didn't traceback goroutine 5. |
|
Hi @aclements, hopefully I've got all the information you need here. Here is the dlv backtrace, but I'm not sure if it's the one you wanted. The only other goroutine when I stop it in dlv is doing: Goroutine 1 - User: .usr/local/go/src/runtime/asm_amd64.s:311 runtime.systemstack_switch (0x4e11b0) The active one is doing this: GOMAXPROCs is the default, so I think that's the number of cores on the system, which is 80. Here is the objdump: |
|
Thanks. Based on the objdump, we're in the obvious place: at the call to newproc at PC 0xa32fd5. The traceback has been truncated since +0x9a is the return PC from that call. @cliffburdick, thanks for the traceback, but I actually wanted to see the Delve traceback of goroutine 1. Though given the inner-most frame you mentioned, which is in Can you attach gdb to it? If so, it would be great to see the output of "thread apply all bt". That might be huge with GOMAXPROCS=80, so if you can reproduce with a smaller GOMAXPROCS I would try that. |
aclements
added
the
WaitingForInfo
|
@aclements I'll get back to this shortly, hopefully. Got pulled onto something else temporarily. |
agnivade
added
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
$ go version
go version go1.11 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I have an application that uses cgo to link to DPDK. After running for about 10 minutes, the main goroutine hangs. When I look in dlv at what it's doing, it's stuck in _Cfunc_GoString with the backtrace:
0 0x00000000004d1230 in runtime.systemstack_switch
at .usr/local/go/src/runtime/asm_amd64.s:311
1 0x0000000000495b4f in runtime.gcStart
at .usr/local/go/src/runtime/mgc.go:1316
2 0x0000000000487dfe in runtime.mallocgc
at .usr/local/go/src/runtime/malloc.go:996
3 0x00000000004c0c9f in runtime.rawstring
at .usr/local/go/src/runtime/string.go:258
4 0x00000000004c1047 in runtime.gostring
at .usr/local/go/src/runtime/string.go:317
5 0x0000000000825afb in github.com/cburdick/myapp/internal/dpdk._Cfunc_GoString
at _cgo_gotypes.go:869
6 0x0000000000827bcc in github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkMain
at .home/cburdick/myapp/internal/dpdk/dpdk.go:178
7 0x000000000083433c in github.com/cburdick/myapp/cmd/myapp/app.Run.func1
at .home/cburdick/myapp/cmd/myapp/app/myapp.go:32
8 0x00000000004d3311 in runtime.goexit
at .usr/local/go/src/runtime/asm_amd64.s:1333
The actual line where it hangs is:
name := C.GoString((*C.char)(unsafe.Pointer(&msg.name[0])))
Where msg.name is a NULL-terminated C character array.
What did you expect to see?
No hanging
The text was updated successfully, but these errors were encountered: