runtime: infinite loop under FreeBSD #4979
Labels
Milestone
Comments
Please export GOTRACEBACK=2 in the environment before running your server. When it gets into this mode, please try kill -ABRT. Perhaps that will be useful, perhaps not. In the ktrace, when it stops looping, does it start sleeping with nanosleep(2)? This could be caused by the new scheduler but I don't quite know how. Labels changed: added priority-later, removed priority-triage. Status changed to Accepted. |
OK. I'm setting GOTRACEBACK. I'll post a traceback when I get a chance. I can't comment on what happens when the process stops looping, since it doesn't stop (till it's killed). The ktrace dump from a few seconds of this loop is 5.5 million lines exactly like the ones I posted. My wild guess as to the cause would be some subtle difference in behavior between FreeBSD's umtx and Linux's futex that could be hit by the new scheduler. |
Another thing you could do is attach gdb to the thread calling yield and see where it is calling yield from. There are only a few choices: proc.c, lock_futex.c, parfor.c, mgc0.c. Each of those is a very different context, so that information would help narrow down what might be going wrong. Thanks. Russ |
Here's a stack trace (of a _umtx_op call) from GDB:
#0 runtime.sys_umtx_op ()
at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:19
#1 0x000000000043ca62 in runtime.futexsleep (addr=void, val=void, ns=void)
at /usr/local/go/src/pkg/runtime/thread_freebsd.c:56
#2 0x000000000042c089 in runtime.notesleep (n=void)
at /usr/local/go/src/pkg/runtime/lock_futex.c:125
#3 0x0000000000435d7f in stopm () at /usr/local/go/src/pkg/runtime/proc.c:780
#4 0x0000000000436775 in findrunnable ()
at /usr/local/go/src/pkg/runtime/proc.c:1050
#5 0x000000000043685d in schedule ()
at /usr/local/go/src/pkg/runtime/proc.c:1072
#6 0x0000000000436a5c in park0 (gp=void)
at /usr/local/go/src/pkg/runtime/proc.c:1120
#7 0x0000000000441f09 in runtime.mcall (fn=void)
at /usr/local/go/src/pkg/runtime/asm_amd64.s:182
#8 0x000000c203a106e0 in ?? ()
#9 0x0000000000000000 in ?? ()
|
And here's a full stack trace from SIGABRT (attached). Attachments:
|
My latest crash was a little different, but clearly related.
This time it's stalling in the umtx call (0% CPU). GDB gives the following:
(gdb) thread apply all bt
Thread 1 (process 55918):
#0 runtime.sys_umtx_op ()
at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:19
#1 0x000000000043ca62 in runtime.futexsleep (addr=void, val=void, ns=void)
at /usr/local/go/src/pkg/runtime/thread_freebsd.c:56
#2 0x000000000042c089 in runtime.notesleep (n=void)
at /usr/local/go/src/pkg/runtime/lock_futex.c:125
#3 0x00000000004355d0 in runtime.stoptheworld ()
at /usr/local/go/src/pkg/runtime/proc.c:372
#4 0x00000000004307da in gc (args=void)
at /usr/local/go/src/pkg/runtime/mgc0.c:1765
#5 0x0000000000441ff0 in ?? ()
at /usr/local/go/src/pkg/runtime/asm_amd64.s:265
#6 0x0000000811162f8c in ?? ()
#7 0x0000000811162100 in ?? ()
#8 0x0000000811162fb8 in ?? ()
#9 0x0000000811162f60 in ?? ()
#10 0x000000000043072e in runtime.gc (force=void)
at /usr/local/go/src/pkg/runtime/mgc0.c:1736
#11 0x0000000000000000 in ?? ()
ktrace gives a much shorter trace (this time I'm posting the complete trace):
55918 redwood CALL _umtx_op(0xa6c3b8,0x2,0,0,0)
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood CALL _umtx_op(0xa6c3b8,0x2,0,0,0)
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood CALL _umtx_op(0xa6c3b8,0x2,0,0,0)
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood PSIG SIGTERM caught handler=0x4425a0 mask=0x0 code=0x10001
55918 redwood CALL exit(0x2)
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
55918 redwood RET _umtx_op -1 errno 4 Interrupted system call
The interrupted system calls before SIGTERM are times I pressed control-C in gdb. Then I
sent the process SIGTERM to kill it. Do the lines after exit mean that 10 threads were
in calls to _umtx_op when I killed the process? But if so, why did thread apply all bt
only trace one thread?
|
For some reason gdb can only see one thread in the process. If I type 'info threads', it
only lists one. But this time I caught it in runtime.osyield:
(gdb) thread apply all bt
Thread 1 (process 56284):
#0 runtime.osyield () at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:250
#1 0x000000000042be9f in runtime.lock (l=void)
at /usr/local/go/src/pkg/runtime/lock_futex.c:77
#2 0x0000000000426183 in sellock (sel=void) at /usr/local/go/src/pkg/runtime/chan.c:804
#3 0x0000000000426459 in selectgo (selp=void) at
/usr/local/go/src/pkg/runtime/chan.c:918
#4 0x0000000000426272 in runtime.selectgo (sel=void)
at /usr/local/go/src/pkg/runtime/chan.c:841
#5 0x000000000047ab27 in net/http.(*persistConn).roundTrip (pc=0xc20d6bbb00,
req=0xc20e938690, resp=0xc20d6bbb00, err=...)
at /usr/local/go/src/pkg/net/http/transport.go:784
#6 0x00000000004770bc in net/http.(*Transport).RoundTrip (t=0xa34520
<main.transport>,
req=0xc210214b60, resp=0x735708, err=...)
at /usr/local/go/src/pkg/net/http/transport.go:173
#7 0x000000000040bb99 in main.(*retryTransport).RoundTrip (t=0xa34520
<main.transport>,
req=0xc210214b60, resp=0xb, err=...)
at /Users/andy/Programs/Go/src/code.google.com/p/redwood-filter/proxy.go:241
#8 0x000000000040ac47 in main.proxyHandler.ServeHTTP (h=..., w=..., r=0xc210214b60)
at /Users/andy/Programs/Go/src/code.google.com/p/redwood-filter/proxy.go:121
#9 0x000000000041fdf4 in main.(*proxyHandler).ServeHTTP (this=0xc2099bbba0, w=...,
r=0xc210214b60)
#10 0x00000000004716dc in net/http.serverHandler.ServeHTTP (sh=..., rw=...,
req=0xc210214b60)
at /usr/local/go/src/pkg/net/http/server.go:1291
#11 0x000000000046f9a5 in net/http.(*conn).serve (c=0xc20e910700)
at /usr/local/go/src/pkg/net/http/server.go:873
#12 0x0000000000436b40 in ?? () at /usr/local/go/src/pkg/runtime/proc.c:1144
#13 0x000000c20e910700 in ?? ()
#14 0x0000000000000000 in ?? ()
|
About a second before it went into its loop this time, I got the following line in my log file: 2013-03-06 14:35:41,10.1.10.203,ignore,%v(PANIC=runtime error: invalid memory address or nil pointer dereference),GET,200,application/pdf,0,"","","","" The item that was replaced by a panic message was a URL (which I had just used to download a PDF). Is the garbage collector freeing objects that it shouldn't? |
It is certainly possible that the garbage collector is doing bad things. The gdb traces don't have the amount of information I was hoping for. Perhaps the next thing to do is to look for every call to osyield or usleep in the runtime and insert a different print statement before each one (saying the name of the function being called, the length of time for usleep, and something about where the call is). Then the infinite loop becomes an infinite loop that prints a lot, and from the prints we might understand what is looping. One common reason for a loop like this is that usleep is not actually sleeping. However, I don't see anything wrong with the implementation. The prints should tell us more. |
#7 looks like another reincarnation of the bug fixed by https://golang.org/cl/7424054 |
#8 looks legal but if completely hangs there, it suggests memory corruption -- chan mutex state is overwritten and so nobody can lock it there was some GC-related memory corruption fixed some time ago do you run with GOMAXPROCS>1? if so, there can be some data races that lead to memory corruption. Try running the server with under the race detector. |
Memory corruption sounds like a real possibility. This morning I had a crash (program exited with a panic); I'm attaching the tail of the log (including the stack traces) in case it's helpful. So I've been having all three major styles of crashes: freezes at 0% CPU, freezes at 100% or more CPU, and panics. I'll try running it with the race detector today. Attachments:
|
Both crashes are in memmove. Can it allocate more than 2GB? because there is an open bug for it: https://golang.org/cl/7474047/ |
This appears to be a FreeBSD kernel bug (now fixed). I tested Go versions dating back to November, and still got the crashes. But when I updated my FreeBSD base system to the tip of the 9-STABLE branch, the crashes seem to have gone away. I've been crash-free for two days now. From when the crashes started, I suspect the problem was in the ZFS module; they started shortly after I added a ZFS disk to my system. Why a ZFS problem would crash a process that wasn't even accessing a ZFS disk, I don't know; but it looks pretty suspicious. |
Just hit this on linux/amd64. Sounds like the same symptoms:
=== RUN TestInterp-118
Input: coverage.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 6 [running]:
runtime.panic(0x63c600, 0xc200e89e90)
/build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
/build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
/build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
evacuate(0x5d8b00, 0xc2000f7b40, 0x55f)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x939)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef33c9878)
/build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
/build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2000dec40, 0xc2000e6e40,
0xc200cdfac0, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc2000e6e40,
0xc200cdfac0, 0xc200da35a0, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:317 +0x115
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc200100c00,
0xc200cdcdc0, 0xc200da3480, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:312 +0xe3
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc2000e6c00,
0xc200cde780, 0xc200da3480, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:304 +0x3b7
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2000dec40, 0xc2000e6dc0,
0xc200cdfbc0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2381 +0x232c
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2000dec40,
0xc200ce7300, 0x4, 0x4, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2000dec40, 0xc2001650c0,
0xc200ceac30)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2000dec40)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000f20e0,
0xc2000e6d80, 0xc200ceac90)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000f20e0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000f20e0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 1 [chan receive]:
runtime.park(0x40b300, 0xc2000e43b0, 0x80ddea)
/build/go.tip/go/src/pkg/runtime/proc.c:1167 +0x64
runtime.chanrecv(0x5d80e0, 0xc2000e4360, 0x7fbef379ace0, 0x0, 0x0, ...)
/build/go.tip/go/src/pkg/runtime/chan.c:366 +0x566
runtime.chanrecv1()
/build/go.tip/go/src/pkg/runtime/chan.c:458 +0x38
testing.RunTests(0x6d8240, 0x8071d0, 0x1, 0x1, 0xc2000f2901, ...)
/build/go.tip/go/src/pkg/testing/testing.go:427 +0x88e
testing.Main(0x6d8240, 0x8071d0, 0x1, 0x1, 0x810960, ...)
/build/go.tip/go/src/pkg/testing/testing.go:358 +0x8a
main.main()
code.google.com/p/go.exp/ssa/interp/_test/_testmain.go:43 +0x9a
runtime.main()
/build/go.tip/go/src/pkg/runtime/proc.c:182 +0x92
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
goroutine 2 [syscall]:
runtime.entersyscallblock()
/build/go.tip/go/src/pkg/runtime/proc.c:1324 +0x16e
runtime.MHeap_Scavenger()
/build/go.tip/go/src/pkg/runtime/mheap.c:435 +0xee
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by runtime.main
/build/go.tip/go/src/pkg/runtime/proc.c:165
goroutine 3 [semacquire]:
runtime.park(0x40b300, 0x81eb80, 0x806df0)
/build/go.tip/go/src/pkg/runtime/proc.c:1167 +0x64
semacquireimpl(0xc200b3c368, 0x1)
/build/go.tip/go/src/pkg/runtime/zsema_linux_amd64.c:113 +0x116
sync.runtime_Semacquire(0xc200b3c368)
/build/go.tip/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc200d9e1c0)
/build/go.tip/go/src/pkg/sync/waitgroup.go:107 +0xf2
code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages(0xc2000e7380)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2744 +0xf8
code.google.com/p/go.exp/ssa/interp_test.run(0xc2000f5090, 0xc2000bf550, 0x46, 0x6906b0,
0xb, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/interp/interp_test.go:168 +0x7eb
code.google.com/p/go.exp/ssa/interp_test.TestInterp(0xc2000f5090)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/interp/interp_test.go:188 +0x1a7
testing.tRunner(0xc2000f5090, 0x8071d0)
/build/go.tip/go/src/pkg/testing/testing.go:346 +0x8a
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by testing.RunTests
/build/go.tip/go/src/pkg/testing/testing.go:426 +0x86b
goroutine 4 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2620)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 5 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8850)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 7 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8e70)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 8 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2070)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 10 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a8c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 11 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8000)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 12 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2690)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 13 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2ee0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 14 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e87e0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 15 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a3f0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 16 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a460)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 17 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a850)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 11 [running]:
runtime.panic(0x63c600, 0xc2015246e0)
/build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
/build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
/build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
----- stack segment boundary -----
evacuate(0x5d8b00, 0xc2000f7b40, 0x566)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x219)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef33b9220)
/build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
/build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2011ece00, 0xc2000e6e40,
0xc2000e6e80, 0xc2011d9100, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2011ece00,
0xc200c67b80, 0xc201524620, 0xc2000e6e40, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2011ece00,
0xc2000bb860, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2000e6f00,
0xc2000e6ec0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2000e6dc0,
0xc2000e6fc0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2373 +0x2227
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2011ece00,
0xc2000bb8f0, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2001650c0,
0xc2000f7f60)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000e8000,
0xc2000e6d80, 0xc2000f7f90)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2709 +0x3c3
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000e8000)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000e8000)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 4 [running]:
runtime.panic(0x63c600, 0xc201524b70)
/build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
/build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
/build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
evacuate(0x5d8b00, 0xc2000f7b40, 0x56b)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x661)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef35378f0)
/build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
/build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc200420ee0, 0xc2000e6ac0,
0xc2005438a0, 0xc20033ff01, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).addr(0xc2000e7380, 0xc200420ee0, 0xc200107080,
0xc20056fe70, 0xc20033ff00, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:682 +0x5e1
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc200420ee0, 0xc200107080,
0xc20056fe70, 0xc2011d7100, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:936 +0x185a
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc200420ee0,
0xc200c67b80, 0xc201524b30, 0xc200107080, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc200420ee0,
0xc2004aac80, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc200420ee0, 0xc2000e6f00,
0xc200432c40)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc200420ee0,
0xc2003a3c00, 0x12, 0x20, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc200420ee0, 0xc2001650c0,
0xc20056c6c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc200420ee0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000f2620,
0xc2000e6d80, 0xc20056c6f0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000f2620)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000f2620)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 10 [running]:
runtime.panic(0x63c600, 0xc200b46470)
/build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
/build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
/build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
----- stack segment boundary -----
evacuate(0x5d8b00, 0xc2000f7b40, 0x5d5)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0xf28)
/build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef3379268)
/build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
/build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc2000e6ac0,
0xc20059e8a0, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc2000e6c00,
0xc20054e360, 0xc2009352d0, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:837 +0xa3f
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc200165380,
0xc20080b500, 0xc2011e0500, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:861 +0x316e
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cae00,
0xc200c67b80, 0xc200b46420, 0xc200165380, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014cae00,
0xc20094e360, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2000e6f00,
0xc20080b540)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20059e900, 0x2, 0x2, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e390)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2000e6dc0,
0xc20080b580)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2383 +0x23c3
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20094e3d0, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e3c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).forStmt(0xc2000e7380, 0xc2014cae00,
0xc20080b5c0, 0x0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1942 +0x243
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001009c0,
0xc20080b5c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2404 +0x48a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20080b600, 0x4, 0x4, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e3f0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014cae00)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2db0, 0xc2002a0000, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc2000e6ac0,
0xc200976760, 0x0, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014cba80,
0xc200806280, 0xc2011e16f8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014cba80,
0xc200806280, 0xc2011e16f8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc2000e6e40,
0xc200806280, 0xc2011e0500, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:809 +0x12b5
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cba80,
0xc200c67b80, 0xc20026f1e0, 0xc2000e6e40, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).compLit(0xc2000e7380, 0xc2014cba80,
0xc200b40050, 0xc2011e0550, 0xc200806380, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1489 +0x1573
code.google.com/p/go.exp/ssa.(*Builder).addr(0xc2000e7380, 0xc2014cba80, 0xc2000e6cc0,
0xc200806380, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:653 +0x347
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc200100c00,
0xc200976940, 0xc2011e0500, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:816 +0x778
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cba80,
0xc200c67b80, 0xc20151bf70, 0xc200100c00, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014cba80,
0xc20047cfa0, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cba80, 0xc2000e6f00,
0xc2008063c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cba80,
0xc2009d4580, 0x7, 0x8, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cba80, 0xc2001650c0,
0xc2009ff6f0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014cba80)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc200a17330, 0xc2002a0000, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014ca540, 0xc2000e6ac0,
0xc200478fe0, 0xc20151b800, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014ca540,
0xc200431580, 0xc2011e15d8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014ca540,
0xc200431580, 0xc2011e15d8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014ca540, 0xc2000e6e40,
0xc200431580, 0xc2011e1301, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:809 +0x12b5
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014ca540, 0xc200100480,
0xc2005a60c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2307 +0x1b26
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014ca540,
0xc200431540, 0x4, 0x4, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014ca540, 0xc2001650c0,
0xc2005549f0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014ca540)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2c00, 0xc2002a0000, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014c29a0, 0xc2000e6ac0,
0xc20056b140, 0xc200079200, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014c29a0,
0xc200580000, 0xc2011e1278)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014c29a0,
0xc200580000, 0xc2011e1278)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).exprN(0xc2000e7380, 0xc2014c29a0, 0xc2000e6e40,
0xc200580000, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:402 +0x4b2
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014c29a0,
0xc200578fe0, 0x2, 0x2, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1450 +0x702
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6f00,
0xc2005800c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6dc0,
0xc200580340)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2373 +0x2227
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014c29a0,
0xc20040a680, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2001650c0,
0xc200584c30)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6dc0,
0xc200580380)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2383 +0x23c3
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014c29a0,
0xc200a13d80, 0x7, 0x8, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2001650c0,
0xc200576180)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014c29a0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2300, 0xc2002a0000, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc20143b8c0, 0xc2000e6ac0,
0xc200a10f60, 0x40ad00, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc20143b8c0,
0xc200a0fc40, 0xc200da5ed8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc20143b8c0,
0xc200a0fc40, 0xc200da5ed8)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).exprN(0xc2000e7380, 0xc20143b8c0, 0xc2000e6e40,
0xc200a0fc40, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:402 +0x4b2
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc20143b8c0, 0xc200100480,
0xc200a14000)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2300 +0xfc7
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc20143b8c0,
0xc200a059d0, 0x1, 0x1, ...)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc20143b8c0, 0xc2001650c0,
0xc200a12660)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc20143b8c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc200e5a8c0,
0xc2000e6d80, 0xc200a126c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc200e5a8c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc200e5a8c0)
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
/build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
/build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
FAIL code.google.com/p/go.exp/ssa/interp 0.504s
|
It's been a long time since I've had an infinite loop (like in the issue description).
The crashes are basically all panics now.
I'm attaching a stack trace from one I had this morning. It's a nil pointer dereference
where there clearly can't be one if everything is working properly. The crash is
occurring in the line z[0] = x of the following function in math/big/nat.go:
func (z nat) setWord(x Word) nat {
if x == 0 {
return z.make(0)
}
z = z.make(1)
z[0] = x
return z
}
The previous line ensures that z has at least one element.
Attachments:
|
Here's another stack trace. Attachments:
|
and another Attachments:
|
Does the freebsd runtime have any of the mistakes that Dmitry fixed for the big GC bug? There are a couple of places where a byte* is used where a void* would be more appropriate, mostly in the code generated by defs_freebsd.go. I don't know if they can cause a failure like this. Andy, could try going into $GOROOT/src/pkg/runtime/mgc0.c and change the Debug constant to 2, then do a run with the environment variable GOGC=1 ? It will be very slow but should show us something. |
Daniel, This morning I had a crash in one of the zfreebsd files. I don't know if that would have anything to do with the question you have about the runtime, but maybe. I'll try running it with the settings you described. Remy and Mikioh, My code uses unsafe only under Linux. BSD has a simpler way to get the real destination on redirected connections. Attachments:
|
Linux doesn't seem like the best OS to try to replicate the problem. I have the program running on 3 machines: linux/amd64, linux/386, and freebsd/amd64. The linux/386 one isn't crashing, the linux/amd64 one crashes maybe once a week, and the freebsd/amd64 one crashes about twice a day. The linux/amd64 one handles quite a bit more traffic than the FreeBSD one. |
I managed to build and set up the program on a FreeBSD amd64 system. I configured a
browser on my laptop to use that system as a proxy. I do see that some web sites are
denied. I ran several instances of the Go testsuite (cd test; go run run.go) in the
background to push up the load.
I've been poking at this for a while, but I have not yet managed to see a crash of the
redwood-filter program. Oddly, I did see a crash in the Go testsuite.
How long do you usually have to wait until you see a crash?
Go testsuite crash:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4d0fb7]
goroutine 1633 [running]:
regexp/syntax.(*Regexp).MaxCap(0xc2011ba230, 0x1)
/home/gopher/iant/go/src/pkg/regexp/syntax/regexp.go:297 +0x57
regexp/syntax.(*Regexp).MaxCap(0xc2011ba310, 0x1a)
/home/gopher/iant/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp.compile(0xc2011b90e0, 0x1a, 0xd4, 0xc200078380, 0xc2011b90e0, ...)
/home/gopher/iant/go/src/pkg/regexp/regexp.go:148 +0x84
regexp.Compile(0xc2011b90e0, 0x1a, 0xc200257680, 0xc20063da50, 0x5800fc, ...)
/home/gopher/iant/go/src/pkg/regexp/regexp.go:109 +0x3b
regexp.MustCompile(0xc2011b90e0, 0x1a, 0x8107c4768)
/home/gopher/iant/go/src/pkg/regexp/regexp.go:205 +0x38
main.(*test).wantedErrors(0xc2000ece00, 0xc200c9cce0, 0x17, 0x57aef0, 0x8, ...)
/home/gopher/iant/go/test/run.go:785 +0x6c1
main.(*test).errorCheck(0xc2000ece00, 0xc2009ab000, 0x21d8c, 0x81093ab28, 0x2, ...)
/home/gopher/iant/go/test/run.go:676 +0x617
main.(*test).run(0xc2000ece00)
/home/gopher/iant/go/test/run.go:618 +0x2b4d
main.func·001()
/home/gopher/iant/go/test/run.go:243 +0x27
created by main.runTests
/home/gopher/iant/go/test/run.go:245 +0xa2
goroutine 1 [runnable]:
main.main()
/home/gopher/iant/go/test/run.go:116 +0x38d
goroutine 3 [runnable]:
main.runTests()
/home/gopher/iant/go/test/run.go:240 +0x36
created by main.startTest
/home/gopher/iant/go/test/run.go:226 +0xc9
goroutine 4 [runnable]:
exit status 2
|
I was running 4 "GOGC=500 ./test/run" on freebsd/amd64 and get: panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x4d1017] goroutine 1637 [running]: runtime.panic(0x53f100, 0xc201563210) /home/gopher/dvyukov/go/src/pkg/runtime/panic.c:247 +0xb1 runtime.panicstring(0x63e08b) /home/gopher/dvyukov/go/src/pkg/runtime/panic.c:487 +0xa3 runtime.sigpanic() /home/gopher/dvyukov/go/src/pkg/runtime/os_freebsd.c:188 +0x162 regexp/syntax.(*Regexp).MaxCap(0xc201569310, 0x0) /home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:297 +0x57 regexp/syntax.(*Regexp).MaxCap(0xc201569380, 0x0) /home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74 regexp/syntax.(*Regexp).MaxCap(0xc2015691c0, 0x1) /home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74 regexp/syntax.(*Regexp).MaxCap(0xc2015693f0, 0x0) /home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74 regexp/syntax.(*Regexp).MaxCap(0xc201569540, 0x1a) /home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74 regexp.compile(0xc201560460, 0x1a, 0xd4, 0xc200078380, 0xc201560460, ...) /home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:148 +0x84 regexp.Compile(0xc201560460, 0x1a, 0xc200abd340, 0xc2012d6c90, 0x58015c, ...) /home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:109 +0x3b regexp.MustCompile(0xc201560460, 0x1a, 0x8106a4768) /home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:205 +0x38 main.(*test).wantedErrors(0xc20010bd20, 0xc201704cc0, 0x17, 0x57af50, 0x8, ...) /home/gopher/dvyukov/go/test/run.go:785 +0x6c1 main.(*test).errorCheck(0xc20010bd20, 0xc200eaa000, 0x21d8c, 0x8106b5b28, 0x2, ...) /home/gopher/dvyukov/go/test/run.go:676 +0x617 ----- stack segment boundary ----- main.(*test).run(0xc20010bd20) /home/gopher/dvyukov/go/test/run.go:618 +0x2b4d main.func·001() /home/gopher/dvyukov/go/test/run.go:243 +0x27 runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by main.runTests /home/gopher/dvyukov/go/test/run.go:245 +0xa2 goroutine 1 [runnable]: runtime.park(0x415cf0, 0xc20010cad0, 0x63f351) /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64 runtime.chanrecv(0x5034c0, 0xc20010ca80, 0x8106a9b80, 0x0, 0x0, ...) /home/gopher/dvyukov/go/src/pkg/runtime/chan.c:397 +0x234 runtime.chanrecv1() /home/gopher/dvyukov/go/src/pkg/runtime/chan.c:458 +0x38 main.main() /home/gopher/dvyukov/go/test/run.go:116 +0x38d runtime.main() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:182 +0x92 runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 goroutine 2 [syscall]: runtime.entersyscallblock() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1333 +0x16e runtime.MHeap_Scavenger() /home/gopher/dvyukov/go/src/pkg/runtime/mheap.c:435 +0xee runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by runtime.main /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:165 goroutine 3 [chan send]: runtime.park(0x415cf0, 0xc200087410, 0x63f2de) /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64 runtime.chansend(0x5034c0, 0xc2000873c0, 0x8106b7fa0, 0x0, 0x402b36, ...) /home/gopher/dvyukov/go/src/pkg/runtime/chan.c:261 +0x1d7 runtime.chansend1() /home/gopher/dvyukov/go/src/pkg/runtime/chan.c:450 +0x4c main.runTests() /home/gopher/dvyukov/go/test/run.go:240 +0x36 runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by main.startTest /home/gopher/dvyukov/go/test/run.go:226 +0xc9 goroutine 1649 [runnable]: runtime.exitsyscall() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1388 +0x119 syscall.Syscall() /home/gopher/dvyukov/go/src/pkg/syscall/asm_freebsd_amd64.s:34 +0x5b syscall.read(0x5, 0xc2000b8c00, 0x200, 0x200, 0x8, ...) /home/gopher/dvyukov/go/src/pkg/syscall/zsyscall_freebsd_amd64.go:895 +0x70 syscall.Read(0x5, 0xc2000b8c00, 0x200, 0x200, 0x41dd04, ...) /home/gopher/dvyukov/go/src/pkg/syscall/syscall_unix.go:132 +0x5a os.(*File).read(0xc205ee9370, 0xc2000b8c00, 0x200, 0x200, 0x502640, ...) /home/gopher/dvyukov/go/src/pkg/os/file_unix.go:174 +0x60 os.(*File).Read(0xc205ee9370, 0xc2000b8c00, 0x200, 0x200, 0x0, ...) /home/gopher/dvyukov/go/src/pkg/os/file.go:95 +0x96 bytes.(*Buffer).ReadFrom(0xc20130f2a0, 0xc20007fba0, 0xc205ee9370, 0x0, 0x0, ...) /home/gopher/dvyukov/go/src/pkg/bytes/buffer.go:169 +0x203 io.Copy(0xc20007fcf0, 0xc20130f2a0, 0xc20007fba0, 0xc205ee9370, 0x0, ...) /home/gopher/dvyukov/go/src/pkg/io/io.go:331 +0xa3 os/exec.func·003(0x503520, 0xc20130f1c0) /home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:207 +0x60 os/exec.func·004(0xc201704ae0) /home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:276 +0x2a runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by os/exec.(*Cmd).Start /home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:277 +0x732 goroutine 13 [finalizer wait]: runtime.park(0x0, 0x0, 0x6400db) /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64 runfinq() /home/gopher/dvyukov/go/src/pkg/runtime/mgc0.c:2182 +0x6d runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by runtime.gc /home/gopher/dvyukov/go/src/pkg/runtime/mgc0.c:1886 goroutine 1648 [runnable]: runtime.exitsyscall() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1388 +0x119 syscall.Syscall6() /home/gopher/dvyukov/go/src/pkg/syscall/asm_freebsd_amd64.s:57 +0x61 syscall.wait4(0x33e0, 0xc205ee93d0, 0x0, 0xc201705360, 0x100000001, ...) /home/gopher/dvyukov/go/src/pkg/syscall/zsyscall_freebsd_amd64.go:32 +0x7b syscall.Wait4(0x33e0, 0x810ab1f54, 0x0, 0xc201705360, 0xc200ab8480, ...) /home/gopher/dvyukov/go/src/pkg/syscall/syscall_bsd.go:137 +0x6c os.(*Process).wait(0xc201704c60, 0x81092b110, 0x0, 0x0) /home/gopher/dvyukov/go/src/pkg/os/exec_unix.go:22 +0xea ----- stack segment boundary ----- os.(*Process).Wait(0xc201704c60, 0xc200676b20, 0xe, 0xc201704aa0) /home/gopher/dvyukov/go/src/pkg/os/doc.go:43 +0x25 os/exec.(*Cmd).Wait(0xc200abbdc0, 0x0, 0x0) /home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:310 +0x197 os/exec.(*Cmd).Run(0xc200abbdc0, 0xe, 0xc200ab8480) /home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:232 +0x52 main.func·002(0xc20129cd70, 0x5, 0x5, 0x5, 0xffffffffffffff01, ...) /home/gopher/dvyukov/go/test/run.go:438 +0x18e main.(*test).run(0xc20010be70) /home/gopher/dvyukov/go/test/run.go:553 +0xd77 main.func·001() /home/gopher/dvyukov/go/test/run.go:243 +0x27 runtime.goexit() /home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223 created by main.runTests /home/gopher/dvyukov/go/test/run.go:245 +0xa2 |
I was running four copies of while true; do time go run run.go; done and the OOM killer kicked in. My system had run out of swap space (4GB RAM + 1GB swap)! It killed one of the invocations of 6g, but I don't know any way to tell what process it was that was hogging that much memory. Now, with three of the run.go jobs still running, I have 2GB free. So I suspect that it was another one of the transient errors that we're chasing. Besides the OOM, the messages I got so far from the test runs were: run fixedbugs/bug207.go : wait : bad address and runoutput rotate0.go : incorrect output |
I don't think you can run run.go 4 times in parallel without running out of memory. Running index*.go and rotate*.go takes a lot of memory (>300MB) and having 16 of them in parallel doesn't seem possible (assuming you have 4 cores). An OOM may cause the compiler or linker to produce corrupt binaries. |
Andy, here is something else for you to try. Update to tip. Edit src/pkg/runtime/mgc0.c. Change the value of IgnorePreciseGC, near the top of the file, from 0 to 1. Rebuild go. Use the rebuilt go to rebuild your program. See if it crashes. This will help us understand whether the new precise GC in Go 1.1 is related to this crash. Thanks! |
Well, I ran it since yesterday afternoon and only OOMed once. Probably that's when the memory-hungry ones happened to run at the same time. Make that twice�it just OOMed again. As it was running overnight, it had a few more "wait: bad address" errors from files in the fixedbugs directory. |
Andy, it turns out the change Ian referenced might not be sufficient to disable the behavior that is causing the crashes. Can you try patching CL 7569047? Here is some background information: I am able to reproduce "wait: bad address" failures (among others) at ToT. These failures persist with IgnorePreciseGC set to 1. However, with CL 7569047 applied I have yet to observe a failure. |
Well, I got a crash again today, even with the patch. (A caught panic, immediately followed by an uncaught one) Attachments:
|
The issue appears to have been fixed by https://code.google.com/p/go/source/detail?r=0704ea89235b |
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The text was updated successfully, but these errors were encountered: