runtime: process stuck while referencing x.f when x is nil #16069

walktall · 2016-06-15T08:57:21Z

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.5.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

func (dm *driverWithMeta) ReadFile() {
    println("dm", dm)
    println("name", dm.name)
    println("cb", dm.cb)
    // ....
}

What did you expect to see?

dm 0x0
panic: runtime error: invalid memory address or nil pointer dereference

What did you see instead?

dm 0x0

Some info got from delve.

(dlv) goroutine
Thread 1949 at /usr/local/go/src/runtime/sys_linux_amd64.s:289

(dlv) goroutines
[11 goroutines]
  Goroutine 1 - User: /usr/local/go/src/runtime/netpoll.go:157 net.runtime_pollWait (0x43e5cc)
  Goroutine 2 - User: /usr/local/go/src/runtime/proc.go:186 runtime.gopark (0x445003)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:186 runtime.gopark (0x445003)
  Goroutine 4 - User: /usr/local/go/src/runtime/proc.go:186 runtime.gopark (0x445003)
  Goroutine 6 - User: /usr/local/go/src/os/signal/signal_unix.go:22 os/signal.loop (0x4c5443)
  Goroutine 7 - User: /usr/local/go/src/runtime/sys_linux_amd64.s:229 runtime.sigtramp (0x4777e0)
  Goroutine 8 - User: /usr/local/go/src/runtime/proc.go:186 runtime.gopark (0x445003)
  Goroutine 9 - User: ./app/cloud/application.go:81 backend/app/cloud.Start.func1 (0x47920f)
  Goroutine 10 - User: ./xvendor/src/github.com/braintree/manners/server.go:159 github.com/braintree/manners.(*GracefulServer).Serve.func1 (0x4d22e7)
  Goroutine 11 - User: /usr/local/go/src/runtime/proc.go:186 runtime.gopark (0x445003)
  Goroutine 17 - User: /usr/local/go/src/runtime/asm_amd64.s:1722 runtime.goexit (0x4765c1)

(dlv) threads
* Thread 1949 at 0x4778c3 /usr/local/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 1950 at 0x4775fd /usr/local/go/src/runtime/sys_linux_amd64.s:95 runtime.usleep
  Thread 1951 at 0x4778c3 /usr/local/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 1952 at 0x4777e0 /usr/local/go/src/runtime/sys_linux_amd64.s:229 runtime.sigtramp
  Thread 1953 at 0x4778c3 /usr/local/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 1954 at 0x477a49 /usr/local/go/src/runtime/sys_linux_amd64.s:420 runtime.epollwait

(dlv) stack
0  0x00000000004778c3 in runtime.futex
   at /usr/local/go/src/runtime/sys_linux_amd64.s:289
1  0x000000000043fc44 in runtime.futexsleep
   at /usr/local/go/src/runtime/os1_linux.go:39
2  0x000000000042564f in runtime.notesleep
   at /usr/local/go/src/runtime/lock_futex.go:142
3  0x0000000000449179 in runtime.stoplockedm
   at /usr/local/go/src/runtime/proc1.go:1276
4  0x000000000044a1a9 in runtime.schedule
   at /usr/local/go/src/runtime/proc1.go:1598
5  0x000000000044a626 in runtime.park_m
   at /usr/local/go/src/runtime/proc1.go:1706
6  0x0000000000473e79 in runtime.mcall
   at /usr/local/go/src/runtime/asm_amd64.s:204

(dlv) list
> runtime.futex() /usr/local/go/src/runtime/sys_linux_amd64.s:289 (PC: 0x4778c3)
   284:     MOVQ    ts+16(FP), R10
   285:     MOVQ    addr2+24(FP), R8
   286:     MOVL    val3+32(FP), R9
   287:     MOVL    $202, AX
   288:     SYSCALL
=> 289:     MOVL    AX, ret+40(FP)
   290:     RET
   291: 
   292: // int32 clone(int32 flags, void *stack, M *mp, G *gp, void (*fn)(void));
   293: TEXT runtime·clone(SB),NOSPLIT,$0
   294:     MOVL    flags+0(FP), DI

And then cpu usage is 99.9%. And I can not term process without kill -9.
I tried to write a simple piece of code to reproduce this but I can't.
Could you give me some clues of this?

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2016-06-15T14:13:46Z

Does your program use cgo? What is your GOMAXPROCS value? Why are you using runtime.LockOSThread?

Have you tried Go 1.6.2? Go 1.6 has several fixes to signal handling.

The delve information appears to be at the point where the signal occurs. The infinite loop is probably happening later. It might help to see what the goroutine and thread information look like then.

We probably aren't going to be able to help very much without a way to reproduce the problem.

walktall · 2016-06-16T02:40:23Z

@ianlancetaylor

Does your program use cgo?

Yes.

What is your GOMAXPROCS value?

1

Why are you using runtime.LockOSThread?

We are not using runtime.LockOSThread on our code or other import code I think.

Have you tried Go 1.6.2?

Yes.

I change the code to something like this:

func (dm *driverWithMeta) ReadFile() {
    println("dm", dm)
    println("gomaxprocs", runtime.GOMAXPROCS(2))
    println("gomaxprocs now", runtime.GOMAXPROCS(0))
    println("name", dm.name)
    println("cb", dm.cb)
}

And get this:

dm 0x0
gomaxprocs 1
gomaxprocs now 2
unexpected fault address 0x2
fatal error: fault
[signal 0xb code=0x201ca390 addr=0x2 pc=0x531dd8]

goroutine 7 [running]:
runtime.throw(0xc89298, 0x5)
    /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc82004b7b0 sp=0xc82004b798
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ab fp=0xc82004b800 sp=0xc82004b7b0
backend/gateway/cloud/driver.(*driverWithMeta).ReadFile(0x0, 0xc82025d5f0, 0x2a, 0xc82026da00, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /root/go/src/backend/gateway/cloud/driver/driver.go:54 +0x228 fp=0xc82004b840 sp=0xc82004b800
backend/gateway/cloud/driver.(*retryManager).retry.func1(0xc8200faee0, 0x0, 0x0)
    /root/go/src/backend/gateway/cloud/driver/driver.go:219 +0xcba fp=0xc82004bcc0 sp=0xc82004b840
github.com/boltdb/bolt.(*DB).Update(0xc820100380, 0xc82004bd50, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/boltdb/bolt/db.go:573 +0xdb fp=0xc82004bd28 sp=0xc82004bcc0
backend/gateway/cloud/driver.(*retryManager).retry(0xc820252640, 0x0, 0x0)
    /root/go/src/backend/gateway/cloud/driver/driver.go:239 +0x84 fp=0xc82004bd78 sp=0xc82004bd28
backend/gateway/cloud/driver.(*retryManager).run(0xc820252640)
    /root/go/src/backend/gateway/cloud/driver/driver.go:121 +0x30d fp=0xc82004bf98 sp=0xc82004bd78
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82004bfa0 sp=0xc82004bf98
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:324 +0x1a2

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f326e77efe8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82027d260, 0x0, 0x7f326e77f0d0, 0xc82027ab80)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820022228, 0x1, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820022228, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820059a80, 0x7f326e77e068, 0xc820022228, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f326e77e068, 0xc820022228, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd
github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
    /root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
    /root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb24c0, 0xc820020f28, 0xc8cf60, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
    /root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
    /root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:
github.com/braintree/manners.(*GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab00, 0x7f326e77e068, 0xc820022228)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(*GracefulServer).Serve
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

ianlancetaylor · 2016-06-16T03:56:01Z

The last stack trace looks like what you want, right? As I understand it, you expect a crash, and the bug is that the program hangs instead of crashing. So it sounds like it only hangs when GOMAXPROCS is 1. Am I understanding this correctly?

You didn't answer my question about Delve, but let me ask a different question. When the program is hanging, hit ^\ or (equivalently) send the process a SIGQUIT signal. That should dump a stack trace. Tell us that stack trace. Thanks.

walktall · 2016-06-16T04:43:23Z

@ianlancetaylor I do want to answer your question about delve..But I can't because I can not go to the next step, it just frozen there.

Am I understanding this correctly?

Yes.

And this time. It hangs again even with GOMAXPROCS set to 2.

^\SIGQUIT: quit
PC=0x47853d m=2

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0x50a88a2ac901, 0x7f01, 0x2710004426b7, 0x14587672182ed5a7, 0x45d964b800, 0x0, 0x50aa99f0bd9e, 0x0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:95 +0x2d
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3468 +0x9d
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1098 +0xec
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1068 +0x72

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f55fd4fefe8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82029b2c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82029b2c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82029b260, 0x0, 0x7f55fd4ff0d0, 0xc820296ba0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820022230, 0x1, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820022230, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820059a80, 0x7f55fd4fe068, 0xc820022230, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f55fd4fe068, 0xc820022230, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd
github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
    /root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
    /root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [running]:
    goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb24e0, 0xc820021728, 0xc8cf80, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820021728, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820021728)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
    /root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
    /root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:
github.com/braintree/manners.(*GracefulServer).Serve.func1(0xc820056e10, 0xc820296b20, 0x7f55fd4fe068, 0xc820022230)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(*GracefulServer).Serve
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

rax    0xfffffffffffffffc
rbx    0x2d8
rcx    0x47853d
rdx    0x0
rdi    0x0
rsi    0x0
rbp    0x50a887db55d4
rsp    0x7f55fa4b0d48
r8     0x7f55fa4b0d48
r9     0x37b3a8
r10    0x0
r11    0x246
r12    0x0
r13    0x7f55fa4b19c0
r14    0x7f55fa4b1700
r15    0x8
rip    0x47853d
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

This is output when GOMAXPROCS set to 1.

^\SIGQUIT: quit
PC=0x47853d m=2

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0x50d3fd41a701, 0x7f01, 0x2710004426b7, 0x1458769e662a2348, 0x45d964b800, 0x0, 0x50d6e7ec0bce, 0x0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:95 +0x2d
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3468 +0x9d
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1098 +0xec
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1068 +0x72

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f4358feffe8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82027d260, 0x0, 0x7f4358ff00d0, 0xc82027ab80)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820022228, 0x1, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820022228, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820061a80, 0x7f4358fef068, 0xc820022228, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/braintree/manners.(*GracefulServer).Serve(0xc82005ee10, 0x7f4358fef068, 0xc820022228, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd
github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc82005ee10, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
    /root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
    /root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [running]:
    goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2460, 0xc820021728, 0xc8cf20, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820021728, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820021728)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc82005ee10)
    /root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
    /root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:
github.com/braintree/manners.(*GracefulServer).Serve.func1(0xc82005ee10, 0xc82027ab00, 0x7f4358fef068, 0xc820022228)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(*GracefulServer).Serve
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

rax    0xfffffffffffffffc
rbx    0x42c
rcx    0x47853d
rdx    0x0
rdi    0x0
rsi    0x0
rbp    0x50d3fa4a480b
rsp    0x7f4355fa1d48
r8     0x7f4355fa1d48
r9     0x37f5aa
r10    0x0
r11    0x246
r12    0x0
r13    0x7f4355fa29c0
r14    0x7f4355fa2700
r15    0x8
rip    0x47853d
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

ianlancetaylor · 2016-06-16T04:49:07Z

What is this goroutine?

goroutine 7 [running]:
    goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

walktall · 2016-06-16T04:55:54Z

@ianlancetaylor It runs this method

func (rm *retryManager) run() {
    var timer *time.Timer
    var ch <-chan time.Time
    n := time.Duration(1)
    if rm.defaultDriver != nil && rm.backupDriver != nil {
        timer = time.NewTimer(n * time.Second)
        ch = timer.C
    }
    for {
        if timer != nil {
            if rm.retryDone {
                timer.Stop()
            } else {
                timer.Reset(n * time.Second)
            }
        }
        select {
        case re := <-rm.dbchan:
            re.err <- rm.encodeAndWrite(re.retryEntry)
            rm.retryDone = false
        case <-ch:
            err := rm.retry()
            if err != nil {
                n *= 2
                if n > 60 {
                    n = 60
                }
                break
            }
            n = 1
        case q := <-rm.quit:
            q <- struct{}{}
            return
        }
    }
}

And panic I expected should happen inside rm.retry().

davecheney · 2016-06-16T05:45:25Z

ch can be nil if the two conditions for initialising the timer are not
satisfied. A nil channel is never considered ready for selection.

On Thu, 16 Jun 2016, 14:55 Zhiheng Huang notifications@github.com wrote:

@ianlancetaylor https://github.com/ianlancetaylor It runs this method

func (rm *retryManager) run() {
var timer *time.Timer
var ch <-chan time.Time
n := time.Duration(1)
if rm.defaultDriver != nil && rm.backupDriver != nil {
timer = time.NewTimer(n * time.Second)
ch = timer.C
}
for {
if timer != nil {
if rm.retryDone {
timer.Stop()
} else {
timer.Reset(n * time.Second)
}
}
select {
case re := <-rm.dbchan:
re.err <- rm.encodeAndWrite(re.retryEntry)
rm.retryDone = false
case <-ch:
err := rm.retry()
if err != nil {
n *= 2
if n > 60 {
n = 60
}
break
}
n = 1
case q := <-rm.quit:
q <- struct{}{}
return
}
}
}

And panic I expected should happen inside rm.retry().

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA_vL__TPhBiRFKWzwjHNqBEthIdYks5qMNdbgaJpZM4I2Iwt
.

walktall · 2016-06-16T05:49:53Z

@davecheney

ch can be nil if the two conditions for initialising the timer are not
satisfied.

That is what I want.

davecheney · 2016-06-16T05:51:56Z

If ch is nil then the code block that contains rm.retry will not be
reachable.

On Thu, 16 Jun 2016, 15:49 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney

ch can be nil if the two conditions for initialising the timer are not
satisfied.

That is what I want.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA553N5YOt_Bv5-QpuCl6A2r68on_ks5qMOQCgaJpZM4I2Iwt
.

walktall · 2016-06-16T05:55:52Z

@davecheney
This piece of code

func (dm *driverWithMeta) ReadFile() {
    println("dm", dm)
    println("name", dm.name)
    println("cb", dm.cb)
    // ....
}

is called by rm.retry, and it is reachable. output here dm 0x0

davecheney · 2016-06-16T05:58:06Z

OK, then ch isn't nil.

On Thu, 16 Jun 2016, 15:55 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney
This piece of code

func (dm *driverWithMeta) ReadFile() {
println("dm", dm)
println("name", dm.name)
println("cb", dm.cb)
// ....
}

is called by rm.retry, and it is reachable. output here dm 0x0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAyWVn1APFiJghEyYTOAhymO4BDWSks5qMOVpgaJpZM4I2Iwt
.

davecheney · 2016-06-16T05:59:36Z

What happens if you use fmt.Println rather than println?

On Thu, 16 Jun 2016, 15:57 Dave Cheney dave@cheney.net wrote:

OK, then ch isn't nil.

On Thu, 16 Jun 2016, 15:55 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney
This piece of code

func (dm *driverWithMeta) ReadFile() {
println("dm", dm)
println("name", dm.name)
println("cb", dm.cb)
// ....
}

is called by rm.retry, and it is reachable. output here dm 0x0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAyWVn1APFiJghEyYTOAhymO4BDWSks5qMOVpgaJpZM4I2Iwt
.

walktall · 2016-06-16T06:02:47Z

@davecheney
It prints dm <nil>

davecheney · 2016-06-16T06:03:46Z

The issue you reported is that rather than the next line panicking, it just
hung. What do you see now ?

On Thu, 16 Jun 2016, 16:02 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney
It prints dm

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAwftUppcMIixE-H1l89xoplM8iHSks5qMOcIgaJpZM4I2Iwt
.

walktall · 2016-06-16T06:05:41Z

@davecheney
It just hung there.
And cpu 99% used.

davecheney · 2016-06-16T06:06:26Z

Can you please paste a complete stack trace. Thanks.

On Thu, 16 Jun 2016, 16:05 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney
It just hung there.
And cpu 99% used.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA-F_OnNFpgO3kntxSgvmRNS-D1Yqks5qMOe2gaJpZM4I2Iwt
.

walktall · 2016-06-16T06:09:07Z

@davecheney I have pasted above.
And I also found that it doesn't always hung. It may fatal.
Like this:

unexpected fault address 0x10
fatal error: fault
[signal 0xb code=0xf addr=0x10 pc=0x5321e8]

goroutine 7 [running]:
runtime.throw(0xc898f8, 0x5)
    /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820043600 sp=0xc8200435e8
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ab fp=0xc820043650 sp=0xc820043600
backend/gateway/cloud/driver.(*driverWithMeta).ReadFile(0x0, 0xc82025f5f0, 0x2a, 0xc82026da60, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /root/go/src/backend/gateway/cloud/driver/driver.go:52 +0x638 fp=0xc820043840 sp=0xc820043650
backend/gateway/cloud/driver.(*retryManager).retry.func1(0xc8200f8ee0, 0x0, 0x0)
    /root/go/src/backend/gateway/cloud/driver/driver.go:217 +0xcba fp=0xc820043cc0 sp=0xc820043840
github.com/boltdb/bolt.(*DB).Update(0xc8200fe380, 0xc820043d50, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/boltdb/bolt/db.go:573 +0xdb fp=0xc820043d28 sp=0xc820043cc0
backend/gateway/cloud/driver.(*retryManager).retry(0xc820252640, 0x0, 0x0)
    /root/go/src/backend/gateway/cloud/driver/driver.go:237 +0x84 fp=0xc820043d78 sp=0xc820043d28
backend/gateway/cloud/driver.(*retryManager).run(0xc820252640)
    /root/go/src/backend/gateway/cloud/driver/driver.go:119 +0x30d fp=0xc820043f98 sp=0xc820043d78
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820043fa0 sp=0xc820043f98
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f5e6f64afe8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82027d260, 0x0, 0x7f5e6f64b0d0, 0xc82027ac00)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820022248, 0x1, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820022248, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820059a80, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd
github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
    /root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
    /root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
    /root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
    /root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:
github.com/braintree/manners.(*GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab80, 0x7f5e6f64a068, 0xc820022248)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(*GracefulServer).Serve
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

If it hung, it will be like this:

^\SIGQUIT: quit
PC=0x47853d m=2

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0x534b486a3601, 0x7f01, 0x2710004426b7, 0x14587b5e3b9ef151, 0x45d964b800, 0x0, 0x534c6e279b9d, 0x0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:95 +0x2d
runtime.sysmon()
    /usr/local/go/src/runtime/proc.go:3468 +0x9d
runtime.mstart1()
    /usr/local/go/src/runtime/proc.go:1098 +0xec
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1068 +0x72

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f468b452fe8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82027d260, 0x0, 0x7f468b4530d0, 0xc82027abc0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820022238, 0x1, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820022238, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820059a80, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd
github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
    /root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
    /root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 7 [running]:
    goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
    /root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
    /root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
    /root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:
github.com/braintree/manners.(*GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab40, 0x7f468b452068, 0xc820022238)
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(*GracefulServer).Serve
    /root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

rax    0xfffffffffffffffc
rbx    0x18d
rcx    0x47853d
rdx    0x0
rdi    0x0
rsi    0x0
rbp    0x534b45919b6b
rsp    0x7f4688404d48
r8     0x7f4688404d48
r9     0x394d78
r10    0x0
r11    0x246
r12    0x0
r13    0x7f46884059c0
r14    0x7f4688405700
r15    0x8
rip    0x47853d
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

davecheney · 2016-06-16T06:16:36Z

Goroutine 7, your cgo code is the one consuming all the CPU.

On Thu, 16 Jun 2016, 16:09 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney I have pasted above.
And I also found that it doesn't always hung. It may fatal.
Like this:

unexpected fault address 0x10
fatal error: fault
[signal 0xb code=0xf addr=0x10 pc=0x5321e8]

goroutine 7 [running]:
runtime.throw(0xc898f8, 0x5)
/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820043600 sp=0xc8200435e8
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ab fp=0xc820043650 sp=0xc820043600
backend/gateway/cloud/driver.(_driverWithMeta).ReadFile(0x0, 0xc82025f5f0, 0x2a, 0xc82026da60, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/go/src/backend/gateway/cloud/driver/driver.go:52 +0x638 fp=0xc820043840 sp=0xc820043650
backend/gateway/cloud/driver.(_retryManager).retry.func1(0xc8200f8ee0, 0x0, 0x0)
/root/go/src/backend/gateway/cloud/driver/driver.go:217 +0xcba fp=0xc820043cc0 sp=0xc820043840github.com/boltdb/bolt.(_DB).Update(0xc8200fe380, 0xc820043d50, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/boltdb/bolt/db.go:573 +0xdb fp=0xc820043d28 sp=0xc820043cc0
backend/gateway/cloud/driver.(_retryManager).retry(0xc820252640, 0x0, 0x0)
/root/go/src/backend/gateway/cloud/driver/driver.go:237 +0x84 fp=0xc820043d78 sp=0xc820043d28
backend/gateway/cloud/driver.(*retryManager).run(0xc820252640)
/root/go/src/backend/gateway/cloud/driver/driver.go:119 +0x30d fp=0xc820043f98 sp=0xc820043d78
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820043fa0 sp=0xc820043f98
created by backend/gateway/cloud/driver.newRetryManager
/root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f5e6f64afe8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).accept(0xc82027d260, 0x0, 0x7f5e6f64b0d0, 0xc82027ac00)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(_TCPListener).AcceptTCP(0xc820022248, 0x1, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(_TCPListener).Accept(0xc820022248, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(_Server).Serve(0xc820059a80, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd

github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
/root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
/root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/

signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
/root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
/root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:github.com/braintree/manners.(_GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab80, 0x7f5e6f64a068, 0xc820022248)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(_GracefulServer).Serve
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

If it hung, it will be like this:

^\SIGQUIT: quit
PC=0x47853d m=2

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0x534b486a3601, 0x7f01, 0x2710004426b7, 0x14587b5e3b9ef151, 0x45d964b800, 0x0, 0x534c6e279b9d, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:95 +0x2d
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:3468 +0x9d
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1098 +0xec
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1068 +0x72

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f468b452fe8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).accept(0xc82027d260, 0x0, 0x7f468b4530d0, 0xc82027abc0)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(_TCPListener).AcceptTCP(0xc820022238, 0x1, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(_TCPListener).Accept(0xc820022238, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(_Server).Serve(0xc820059a80, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd

github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
/root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
/root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_

amd64.s:1998 +0x1

goroutine 7 [running]:
goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
/root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
/root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
/root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:github.com/braintree/manners.(_GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab40, 0x7f468b452068, 0xc820022238)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(_GracefulServer).Serve
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

rax 0xfffffffffffffffc
rbx 0x18d
rcx 0x47853d
rdx 0x0
rdi 0x0
rsi 0x0
rbp 0x534b45919b6b
rsp 0x7f4688404d48
r8 0x7f4688404d48
r9 0x394d78
r10 0x0
r11 0x246
r12 0x0
r13 0x7f46884059c0
r14 0x7f4688405700
r15 0x8
rip 0x47853d
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAw6TexyQd9JZ4la0_rHU5O2NPGluks5qMOiEgaJpZM4I2Iwt
.

davecheney · 2016-06-16T06:21:11Z

I'm sorry but I don't think there is a bug in go. In the cases where the
panic comes from accessing a nil *deviceManager go panics as expected. If
this nil is ending up in your cgo code then all bets are off.

On Thu, 16 Jun 2016, 16:16 Dave Cheney dave@cheney.net wrote:

Goroutine 7, your cgo code is the one consuming all the CPU.

On Thu, 16 Jun 2016, 16:09 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney I have pasted above.
And I also found that it doesn't always hung. It may fatal.
Like this:

unexpected fault address 0x10
fatal error: fault
[signal 0xb code=0xf addr=0x10 pc=0x5321e8]

goroutine 7 [running]:
runtime.throw(0xc898f8, 0x5)
/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820043600 sp=0xc8200435e8
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ab fp=0xc820043650 sp=0xc820043600
backend/gateway/cloud/driver.(_driverWithMeta).ReadFile(0x0, 0xc82025f5f0, 0x2a, 0xc82026da60, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/go/src/backend/gateway/cloud/driver/driver.go:52 +0x638 fp=0xc820043840 sp=0xc820043650
backend/gateway/cloud/driver.(_retryManager).retry.func1(0xc8200f8ee0, 0x0, 0x0)
/root/go/src/backend/gateway/cloud/driver/driver.go:217 +0xcba fp=0xc820043cc0 sp=0xc820043840github.com/boltdb/bolt.(_DB).Update(0xc8200fe380, 0xc820043d50, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/boltdb/bolt/db.go:573 +0xdb fp=0xc820043d28 sp=0xc820043cc0
backend/gateway/cloud/driver.(_retryManager).retry(0xc820252640, 0x0, 0x0)
/root/go/src/backend/gateway/cloud/driver/driver.go:237 +0x84 fp=0xc820043d78 sp=0xc820043d28
backend/gateway/cloud/driver.(*retryManager).run(0xc820252640)
/root/go/src/backend/gateway/cloud/driver/driver.go:119 +0x30d fp=0xc820043f98 sp=0xc820043d78
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820043fa0 sp=0xc820043f98
created by backend/gateway/cloud/driver.newRetryManager
/root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f5e6f64afe8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).accept(0xc82027d260, 0x0, 0x7f5e6f64b0d0, 0xc82027ac00)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(_TCPListener).AcceptTCP(0xc820022248, 0x1, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(_TCPListener).Accept(0xc820022248, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(_Server).Serve(0xc820059a80, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f5e6f64a068, 0xc820022248, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd

github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
/root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
/root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/

signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
/root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
/root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:github.com/braintree/manners.(_GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab80, 0x7f5e6f64a068, 0xc820022248)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(_GracefulServer).Serve
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

If it hung, it will be like this:

^\SIGQUIT: quit
PC=0x47853d m=2

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0x534b486a3601, 0x7f01, 0x2710004426b7, 0x14587b5e3b9ef151, 0x45d964b800, 0x0, 0x534c6e279b9d, 0x0, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:95 +0x2d
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:3468 +0x9d
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1098 +0xec
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1068 +0x72

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f468b452fe8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(_pollDesc).Wait(0xc82027d2c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82027d2c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).accept(0xc82027d260, 0x0, 0x7f468b4530d0, 0xc82027abc0)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(_TCPListener).AcceptTCP(0xc820022238, 0x1, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(_TCPListener).Accept(0xc820022238, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(_Server).Serve(0xc820059a80, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129github.com/braintree/manners.(*GracefulServer).Serve(0xc820056e10, 0x7f468b452068, 0xc820022238, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:222 +0x2fd

github.com/braintree/manners.(*GracefulServer).ListenAndServe(0xc820056e10, 0x0, 0x0)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:115 +0x124
backend/app/cloud.Start()
/root/go/src/backend/app/cloud/application.go:92 +0xa3d
main.main()
/root/go/src/backend/cmd/putong-cloud-service/main.go:10 +0x38

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_

amd64.s:1998 +0x1

goroutine 7 [running]:
goroutine running on other thread; stack unavailable
created by backend/gateway/cloud/driver.newRetryManager
/root/go/src/backend/gateway/cloud/driver/driver.go:322 +0x1a2

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [select, locked to thread]:
runtime.gopark(0xdb2ba0, 0xc820020f28, 0xc8d5c0, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 9 [chan receive]:
backend/app/cloud.Start.func1(0xc820013500, 0xc820056e10)
/root/go/src/backend/app/cloud/application.go:81 +0x52
created by backend/app/cloud.Start
/root/go/src/backend/app/cloud/application.go:84 +0x73b

goroutine 10 [chan send]:github.com/braintree/manners.(_GracefulServer).Serve.func1(0xc820056e10, 0xc82027ab40, 0x7f468b452068, 0xc820022238)
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:159 +0x4e
created by github.com/braintree/manners.(_GracefulServer).Serve
/root/go/src/backend/xvendor/src/github.com/braintree/manners/server.go:164 +0x14e

rax 0xfffffffffffffffc
rbx 0x18d
rcx 0x47853d
rdx 0x0
rdi 0x0
rsi 0x0
rbp 0x534b45919b6b
rsp 0x7f4688404d48
r8 0x7f4688404d48
r9 0x394d78
r10 0x0
r11 0x246
r12 0x0
r13 0x7f46884059c0
r14 0x7f4688405700
r15 0x8
rip 0x47853d
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcAw6TexyQd9JZ4la0_rHU5O2NPGluks5qMOiEgaJpZM4I2Iwt
.

walktall · 2016-06-16T06:37:03Z

@davecheney @ianlancetaylor
After I remove all cgo code. It panic as I expect.

Thanks for your help!

davecheney · 2016-06-16T06:41:22Z

The new go build -msan flag (go 1.6 or later) may help debug issues with
your cgo code.

https://golang.org/doc/go1.6#go_command

On Thu, 16 Jun 2016, 16:37 Zhiheng Huang notifications@github.com wrote:

@davecheney https://github.com/davecheney @ianlancetaylor
https://github.com/ianlancetaylor
After I remove all cgo code. It panic as I expect.

Thanks for your help!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16069 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8ZOR--sA8UOAl-fsHFOr5x-4G5Qks5qMO8QgaJpZM4I2Iwt
.

walktall · 2016-06-16T06:43:59Z

@davecheney Thanks! Will try it.

ianlancetaylor · 2016-06-16T16:04:54Z

Closing because it sounds like the problem is not one that can be fixed in the Go distribution.

ianlancetaylor changed the title ~~Process stucked while running x.f and x is nil.~~ runtime: process stuck while referencing x.f when x is nil Jun 15, 2016

ianlancetaylor closed this as completed Jun 16, 2016

golang locked and limited conversation to collaborators Jun 16, 2017

gopherbot added the FrozenDueToAge label Jun 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: process stuck while referencing x.f when x is nil #16069

runtime: process stuck while referencing x.f when x is nil #16069

walktall commented Jun 15, 2016

ianlancetaylor commented Jun 15, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016

Navigation Menu

runtime: process stuck while referencing x.f when x is nil #16069

runtime: process stuck while referencing x.f when x is nil #16069

Comments

walktall commented Jun 15, 2016

ianlancetaylor commented Jun 15, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

davecheney commented Jun 16, 2016

walktall commented Jun 16, 2016

ianlancetaylor commented Jun 16, 2016