My Linux kernel is "Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux"

	// Some Linux kernels have a bug where futex of
	// FUTEX_WAIT returns an internal error code
	// as an errno. Libpthread ignores the return value
	// here, and so can we: as it says a few lines up,
	// spurious wakeups are allowed.
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT, val, nil, nil, 0)
		return
	}

Could I say that I meet the bug of Linux kernel?

Could I say that I meet the bug of Linux kernel?

Hard to tell w/o a reproduction case.

@cznic Thanks! I am trying to reproduce it. But I am not sure whether I could make it.

I'll also page some other folks to be aware of this as we wait for more information
/cc @randall77 @aclements

I upgrade my gdb to 8.0 and find that there are 74 threads but all of them are waiting here: runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

@davecheney Thanks!
It occurs in the production environment. We could not enable race detection. I will try to build a testing environment to reproduce it.

We just figured out how to print goroutine using gdb. Below is partial stack of all goroutine:

---Type <return> to continue, or q <return> to quit---
  208832207 waiting  runtime.gopark
  208816967 waiting  runtime.gopark
  208824056 waiting  runtime.gopark
  208825674 waiting  runtime.gopark
  208813050 waiting  runtime.gopark
  208821762 waiting  runtime.gopark
  208845379 waiting  runtime.gopark
  208803855 waiting  runtime.gopark
  208819750 waiting  runtime.gopark
  208820173 waiting  runtime.gopark
  208830287 waiting  runtime.gopark
  208835489 waiting  runtime.gopark
  208818500 waiting  runtime.gopark
  208844090 waiting  runtime.gopark
  208803883 waiting  runtime.gopark
  208815260 waiting  runtime.gopark
  208843403 waiting  runtime.gopark
  208817095 waiting  runtime.gopark
  208819772 waiting  runtime.gopark
  208830692 waiting  runtime.gopark
  208808392 waiting  runtime.gopark
  208830801 waiting  runtime.gopark
  208843133 waiting  runtime.gopark
  208839727 waiting  runtime.gopark
  208826235 waiting  runtime.gopark
  208836382 waiting  runtime.gopark
  208828454 waiting  runtime.gopark
  208817075 waiting  runtime.gopark
  208829301 waiting  runtime.gopark

All goroutines are wating on gopark. In. addition, all os threads are waiting on futex.
We came up with two possible reasons: 1. deadlock, 2. g0 is crashed which implies there is no more goroutine can schedule other goroutines.

Deadlock could be caused from kernel bug because our binary in running against linux 3.10 which has a kernel bug causing potential deadlock. For second reason, we are confused why all thread's schedulers are crashed. Per my understanding, all threads have their own g0. The process is still running. Any suggestions about how to debug this will be really appreciated. Of course, tell us what is reason behind this will be also really appreciated.

It's normal for an inactive goroutine to be sitting in gopark. It's normal for an inactive thread to be waiting on a futex.

Your original report, morestack on g0, is a clear sign of a bug or of memory corruption. A deadlock could simply be the result of a program bug.

You suggest that a g0 could crash, causing deadlock, but in general that can't happen. If a g0 crashes, the entire program crashes, as in your original report.

@shenli, it would be useful if you could dump all of the thread stacks from gdb using thread apply all bt.

Unfortunately, I wouldn't expect the info goroutines command to do much useful since the g0 isn't a regular user goroutine and hence isn't printed by info goroutines.

@aclements The output of thread apply all bt is here.

@ianlancetaylor In our scenario, g0 is gone but the whole process is still there.

I don't know what you mean when you say that g0 is gone. Note that g0 is not printed by info goroutines. How are you determining that it is gone?

@ianlancetaylor I meet fatal: morestack on g0 error. So I guess g0 is gone. Please correct me if this is wrong.

That is a fatal error message that should crash the entire program. It is not intended to leave the program running without g0.

Thanks for the stack trace. It does most threads waiting for a lock on runtime.mheap_. The interesting thread stack is this one:

Thread 2 (LWP 12232):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
#1  0x00000000004293f2 in runtime.futexsleep (addr=0x1b13d20 <runtime.mheap_>, val=2, ns=-1)
    at /usr/local/go/src/runtime/os_linux.go:45
#2  0x000000000040f721 in runtime.lock (l=0x1b13d20 <runtime.mheap_>) at /usr/local/go/src/runtime/lock_futex.go:102
#3  0x0000000000414771 in runtime.allocmcache (~r0=0x0) at /usr/local/go/src/runtime/mcache.go:78
#4  0x000000000042c005 in runtime.startpanic_m () at /usr/local/go/src/runtime/panic.go:652
#5  0x000000000045a0cb in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:360
#6  0x000000000042bb8e in runtime.startpanic () at /usr/local/go/src/runtime/panic.go:578
#7  0x000000000044197d in runtime.sighandler (sig=5, info=0xc42006bbf0, ctxt=0xc42006bac0, gp=0xc420000480)
    at /usr/local/go/src/runtime/signal_sighandler.go:81
#8  0x000000000044234d in runtime.sigtrampgo (sig=5, info=0xc42006bbf0, ctx=0xc42006bac0)
    at /usr/local/go/src/runtime/signal_unix.go:334
#9  0x000000000045de03 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:265
#10 0x000000000045def0 in ?? () at /usr/local/go/src/runtime/sys_linux_amd64.s:349
#11 0x0000000000000001 in ?? ()
#12 0x0000000000000000 in ?? ()

This shows that the program got signal 5 (SIGTRAP). Nothing was prepared to handle that signal, so the program started to crash. At the start of the panic code, this condition was true:

	} else if _g_.m.mcache == nil { // can happen if called from signal handler or throw
		_g_.m.mcache = allocmcache()
	}

This makes me wonder whether the signal occurred while holding the lock. I don't know if that can happen while _g_.m.mcache == nil.

I don't understand how to relate this to fatal: morestack on g0. That should not cause a SIGTRAP signal to occur. That only thing that should cause SIGTRAP would be running the program under a debugger (in which case the debugger ought to handle the signal) or using kill to send SIGTRAP to the program.

Unfortunately nothing in the stack trace tells me where the fatal: morestack on g0 is coming from.

@ianlancetaylor We use gdb to get the thread info. So I think it may cause the SIGTRAP signal.

If gdb triggers the SIGTRAP signal, which I think should only happen if you try to use gdb's step or next commands, then it ought to catch the SIGTRAP signal before it is delivered to the program. So while it is certainly possible that gdb is causing it, it's not obvious that it is.

In any case I now see that this is something of a distraction. After the program gets fatal: morestack on g0, there may be some bug that is causing the program to hang instead of crash. The backtrace suggests that this bug has something to do with the error occurring while holding the memory lock. But this is all really just a minor side issue. The question is why the program got fatal: morestack on g0 in the first place. That's the problem that we need to identify and fix.

That brings me back to thinking about memory corruption. Have you been able to run your test program compiled with the race detector?

morestack performs an INT $3 after calling badmorestackg0 to print the "morestack on g0" message. That's probably where the SIGTRAP is coming from. Though, again, that means the SIGTRAP, thread 2's stack, and the deadlock attempting to allocmcache from the signal handler are just side-effects of the real problem.

It does mean that whatever went wrong went wrong on thread 2. I'm surprised gdb couldn't walk over the signal stack switch, but we might be able to get enough from the signal context to unwind the stack ourselves.

@shenli, I assume you're doing this all from a core file? Or do you still have the process around? Either way, could you try running:

thread 2
frame 7
print/x *info
print/x *ctxt
print/x *gp

@aclements The result is here:

(gdb) thread 2
[Switching to thread 2 (LWP 12232)]
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
439	/usr/local/go/src/runtime/sys_linux_amd64.s: No such file or directory.
(gdb) frame 7
#7  0x000000000044197d in runtime.sighandler (sig=5, info=0xc42006bbf0, ctxt=0xc42006bac0, gp=0xc420000480)
    at /usr/local/go/src/runtime/signal_sighandler.go:81
81	/usr/local/go/src/runtime/signal_sighandler.go: No such file or directory.
(gdb) print/x *info
$1 = {si_signo = 0x5, si_errno = 0x0, si_code = 0x80, si_addr = 0x0}
(gdb) print/x *ctxt
Attempt to dereference a generic pointer.
(gdb) print/x *gp
$2 = {stack = {lo = 0xc420070000, hi = 0xc420072000}, stackguard0 = 0xc420070370, stackguard1 = 0xc420070370, _panic = 0x0, 
  _defer = 0x0, m = 0xc420062000, sched = {sp = 0xc420071fc0, pc = 0xffffffffffffffff, g = 0xc420000480, ctxt = 0x0, ret = 0x0, 
    lr = 0x0, bp = 0xc420071fd8}, syscallsp = 0x0, syscallpc = 0x0, stktopsp = 0x0, param = 0x0, atomicstatus = 0x0, 
  stackLock = 0x0, goid = 0x0, waitsince = 0x0, waitreason = "", schedlink = 0x0, preempt = 0x0, paniconfault = 0x0, 
  preemptscan = 0x0, gcscandone = 0x0, gcscanvalid = 0x0, throwsplit = 0x0, raceignore = 0x0, sysblocktraced = 0x0, 
  sysexitticks = 0x0, traceseq = 0x0, tracelastp = 0x0, lockedm = 0x0, sig = 0x0, writebuf =  []uint8, sigcode0 = 0x0, 
  sigcode1 = 0x0, sigpc = 0x0, gopc = 0x0, startpc = 0x0, racectx = 0x0, waiting = 0x0, cgoCtxt =  []uintptr, labels = 0x0, 
  timer = 0x0, gcAssistBytes = 0xfffffffffffff980}
(gdb) 

@ianlancetaylor This happened in a production environment. So we could not enable race detection. I built a test environment and try to produce the same workload. But have not got the error.

We met this issue a few times. We decided to build two binaries: one with Golang 1.7 and another one with Golang 1.10. We will run the two binaries in the production environment and check if the fatal still there.

We build a new binary with Golang 1.10 and put it into the production environment. It has been running for more than two days w/o the fatal problem. I'm keeping my eyes on it.

I took another look at the thread stack dump and there's something I can't figure out. What's holding mheap_.lock? It's unfortunate that the signal handler attempted to acquire it while crashing the system and deadlocked, but what's holding it? Every single other thread is either trying to acquire it, or is happily parked. That leaves only the thread that crashed (thread 2), which we can't see the whole stack of. It's possible it self-deadlocked, but we should only be in allocmcache if the crashing thread doesn't have an mcache, and I don't see how a thread could both hold the mheap_.lock and not have an mcache.

@shenli, thanks for the gdb output. If you still have the crashed process or core dump, could you run the following in gdb?

thread 2
frame 7
x/64a ctxt
set language c
x/1024a *(uint64*)(ctxt+160)
set language go

I will do those operations when the issue occurs.

Thanks. You'll need to tweak the instructions for a different crash instance. To find the right thread number, do thread apply all bt again and look for the stack containing sighandler (also post the new thread stack dumps if you are able to reproduce). Then the frame number I'm interested in is the sighandler frame.

@aclements Thanks! Got it!

Change https://golang.org/cl/88835 mentions this issue: runtime: never allocate during an unrecoverable panic

@aclements The related PR is merged. So I think we will not face the deadlock issue anymore. I will wait for the new release.
Thanks for all of you!

@shenli in the commit message, Austin wrote

This would help with debugging #23360, since the deadlock in allocmcache is currently masking the real failure.

It doesn't say that the CL fixed this issue. So let's not close it (yet).

@ALTree Yes, I will update this issue when we could figure out the real problem. I'm waiting for the new Golang release.

I don't know if this is related or not, but I can reproduce this quite reliably on Mac OS 10.12.6 and go version go1.10.2 darwin/amd64 by doing:

go tool objdump <some go executable> | less

and then quitting less (q) without reaching the end of the assembler listing. In this case this produces this single line on stderr:

fatal: morestack on g0

Should I open a separate ticket?

@CAFxX I believe that that is #25229

@shenli , have you been able to reproduce this with Go 1.10?

@aclements It never happens with Go 1.10. I guess it is fixed. But I will keep watching it. Thanks!

Thanks for checking! I'll go ahead and close this, but let us know if you're able to reproduce it.

I have this occasionally on some Gentoo servers with go 1.10.1, kernel 4.8.17-hardened and prometheus-node-exporter. I cannot reproduce it. And it happens only after some weeks of running.

I also met the issue ,fatal: morestack on g0 , it happens only after some weeks of running. GO VERSION is 1.11.4 .Kernel is 4.9.0-040900-generic

I also meet the same issue,

• go version: 1.10
• kernel version: 3.10.0-862.el7.x86_64

@liuyanhit @ahjdzx i'm sorry to hear you are having issues. This issue has been closed for over six months, please file a new issue including as much detail as possible. Thank you