runtime: runtime panic #5120

nictuku · 2013-03-24T13:34:34Z

$ go version
go version devel +f12b24ea373f Sun Mar 17 19:50:01 2013 +0900 linux/amd64

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x412b6a]

goroutine 5257 [running]:
[fp=0xc20505edd8] runtime.throw(0x9519ea)
        /home/nictuku/go/src/pkg/runtime/panic.c:465 +0x67
[fp=0xc20505edf0] runtime.sigpanic()
        /home/nictuku/go/src/pkg/runtime/thread_linux.c:217 +0xe7
[fp=0xc20505eeb8] flushptrbuf(0x7f0128ff2000, 0xc20505f000, 0xc20505f290, 0xc20505f288,
0xc20505f298, ...)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:388 +0x10a
[fp=0xc20505f288] scanblock(0x7f012351c000, 0x7f012351c018, 0x0, 0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:926 +0xf10
[fp=0xc20505f3b8] gc(0x7f012768df8c)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1863 +0x2c0
----- stack segment boundary -----
[fp=0x7f012768dfa0] runtime.gc(0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1790 +0x11b
[fp=0x7f012768dfb0] forcegchelper(0x7f013a049f60)
        /home/nictuku/go/src/pkg/runtime/mheap.c:355 +0x24
[fp=0x7f012768dfb8] runtime.goexit()
        /home/nictuku/go/src/pkg/runtime/proc.c:1160
created by runtime.MHeap_Scavenger
        /home/nictuku/go/src/pkg/runtime/mheap.c:402


Which compiler are you using (5g, 6g, 8g, gccgo)?
6g

Which operating system are you using?
Debian unstable

Which version are you using?  (run 'go version')
go version devel +f12b24ea373f Sun Mar 17 19:50:01 2013 +0900 linux/amd64

Please provide any additional information below.

This happened in previous dev version too. 
I've initially reported this here:
https://plus.google.com/116078268286389936989/posts/esXoPdHRSfz
while using "go version devel +1ae7ecffb332 Sat Mar 09 14:54:56 2013 -0800
linux/amd64"

Since then I've rebuilt the application with a Mar 17th's repo (seen above), and the
problem still happens once in a while.

I will rebuild the application with today's version and will update this issue if the
problem persists with recent Go repo versions.

DanielMorsing · 2013-03-24T13:43:16Z

Comment 1:

This smells like issue #5074. That was introduced back in february and fixed 3 days ago.

davecheney · 2013-03-25T10:17:17Z

Comment 2:

Hello,
Could you please try updating beyond revision 61fa5c7d741f and confirming if the issue
still occurs.
Cheers
Dave

Status changed to WaitingForReply.

nictuku · 2013-03-27T12:09:48Z

Comment 3:

New crash from mgc0.c, but the stack trace is different so it can be a different issue.
The calling code is here, but I think it's innocent.
https://github.com/nictuku/dht/blob/master/routing_table.go#L264
go version devel +d448734315d3 Mon Mar 25 08:20:22 2013 +0100 linux/amd64
unexpected fault address 0x2ad9
fatal error: fault
[signal 0xb code=0x1 addr=0x2ad9 pc=0x411821]
goroutine 141 [running]:
[fp=0xc202783680] runtime.throw(0xae70f7)
        /home/nictuku/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0xc202783698] runtime.sigpanic()
        /home/nictuku/go/src/pkg/runtime/os_linux.c:239 +0xe7
[fp=0xc2027836e8] hash_gciter_next(0xc202783790, 0xc202783770)
        /home/nictuku/go/src/pkg/runtime/hashmap.c:892 +0x171
[fp=0xc202783a88] scanblock(0x7ffd2fdec000, 0x7ffd2fded500, 0xdf, 0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:912 +0xf2b
[fp=0xc202783bb8] gc(0x7ffd40cd4de4)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1887 +0x2c5
----- stack segment boundary -----
[fp=0x7ffd40cd4df8] runtime.gc(0xc200000000)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1813 +0x11b
[fp=0x7ffd40cd4e50] runtime.mallocgc(0x70, 0x100000000, 0x1)
        /home/nictuku/go/src/pkg/runtime/zmalloc_linux_amd64.c:101 +0x21d
[fp=0x7ffd40cd4e70] runtime.mal(0x70)
        /home/nictuku/go/src/pkg/runtime/zmalloc_linux_amd64.c:611 +0x3f
[fp=0x7ffd40cd4ec0] runtime.makechan_c(0x77f840, 0x1)
        /home/nictuku/go/src/pkg/runtime/chan.c:114 +0xba
[fp=0x7ffd40cd4ed8] runtime.makechan(0x77f840, 0x1, 0x4d7221)
        /home/nictuku/go/src/pkg/runtime/chan.c:141 +0x2f
[fp=0x7ffd40cd4f18] time.NewTimer(0x3611aa85, 0x4d7213)
        /home/nictuku/go/src/pkg/time/sleep.go:63 +0x3b
[fp=0x7ffd40cd4f38] time.After(0x3611aa85, 0xc2000f3600)
        /home/nictuku/go/src/pkg/time/sleep.go:103 +0x33
[fp=0x7ffd40cd4f90] github.com/nictuku/dht.pingSlowly(0xc2000f3600, 0xc2023b7000, 0x39e,
0x500, 0xd18c2e2800, ...)
        /home/nictuku/src/github.com/nictuku/dht/routing_table.go:272 +0x111
[fp=0x7ffd40cd4f98] runtime.goexit()
        /home/nictuku/go/src/pkg/runtime/proc.c:1214
created by github.com/nictuku/dht.(*DHT).DoDHT
        /home/nictuku/src/github.com/nictuku/dht/dht.go:263 +0xc48

ianlancetaylor · 2013-03-27T13:52:25Z

Comment 4:

Labels changed: added priority-soon, go1.1, removed priority-triage.

Status changed to New.

randall77 · 2013-03-27T18:16:31Z

Comment 5:

Any chance you could package your code up into a reproducible test case?  It is ok if it
triggers only occasionally.  I just want to catch it myself under gdb.

DanielMorsing · 2013-03-27T19:42:12Z

Comment 6:

Is the new crash with the updated version that dave pointed out?

randall77 · 2013-03-27T19:50:25Z

Comment 7:

Yes, it is.  But it shouldn't matter, the new map implementation makes that fix obsolete.

nictuku · 2013-03-27T20:15:39Z

Comment 8:

Weird thing is this bug is triggered almost exactly 15 minutes after the program starts.
I'm going to try to reproduce this using the following simplified program:
$ go get github.com/nictuku/dht/examples/find_infohash_and_wait
$ find_infohash_and_wait d1c5676ae7ac98e8b19f63565905105e3c4c37a2
I'm not sure it will work. I'll let you know if it does.

nictuku · 2013-03-27T20:26:21Z

Comment 9:

Stack trace that looks more like the original one:
$ go version
go version devel +d448734315d3 Mon Mar 25 08:20:22 2013 +0100 linux/amd64
unexpected fault address 0x11010100400
fatal error: fault
[signal 0xb code=0x1 addr=0x11010100400 pc=0x411983]
goroutine 104 [running]:
[fp=0xc20201b680] runtime.throw(0xae70f7)
        /home/nictuku/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0xc20201b698] runtime.sigpanic()
        /home/nictuku/go/src/pkg/runtime/os_linux.c:239 +0xe7
[fp=0xc20201b6e8] hash_gciter_next(0xc20201b790, 0xc20201b770)
        /home/nictuku/go/src/pkg/runtime/hashmap.c:938 +0x2d3
[fp=0xc20201ba88] scanblock(0x7f3986afe000, 0x7f3986afe060, 0x3, 0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:912 +0xf2b
[fp=0xc20201bbb8] gc(0x7f3997a53f8c)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1887 +0x2c5
----- stack segment boundary -----
[fp=0x7f3997a53fa0] runtime.gc(0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1813 +0x11b
[fp=0x7f3997a53fb0] forcegchelper(0x7f3997b19f60)
        /home/nictuku/go/src/pkg/runtime/mheap.c:355 +0x24
[fp=0x7f3997a53fb8] runtime.goexit()
        /home/nictuku/go/src/pkg/runtime/proc.c:1214
created by runtime.MHeap_Scavenger
        /home/nictuku/go/src/pkg/runtime/mheap.c:402

nictuku · 2013-03-27T20:26:42Z

Comment 10:

BTW this last stack trace is *not yet* from the simplified example.

nictuku · 2013-03-27T21:37:35Z

Comment 11:

Ok I got a crash using the simplified example program mentioned in #8.
unexpected fault address 0x1ae1
fatal error: fault
[signal 0xb code=0x1 addr=0x1ae1 pc=0x408451]
goroutine 20 [running]:
[fp=0xc2000a3db8] runtime.throw(0x7f5337)
        /home/nictuku/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0xc2000a3dd0] runtime.sigpanic()
        /home/nictuku/go/src/pkg/runtime/os_linux.c:239 +0xe7
[fp=0xc2000a3e20] hash_gciter_next(0xc2000a3ec8, 0xc2000a3ea8)
        /home/nictuku/go/src/pkg/runtime/hashmap.c:892 +0x171
[fp=0xc2000a41c0] scanblock(0x7fc1e0ede000, 0x7fc1e0edea80, 0x6f, 0xc2000a4100)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:912 +0xf2b
[fp=0xc2000a4210] markroot(0xc2000a7000, 0x1000000008)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1198 +0xab
[fp=0xc2000a4288] runtime.parfordo(0xc2000a7000)
        /home/nictuku/go/src/pkg/runtime/parfor.c:105 +0x9b
[fp=0xc2000a43b8] gc(0x7fc1f1f1df8c)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1886 +0x29d
----- stack segment boundary -----
[fp=0x7fc1f1f1dfa0] runtime.gc(0x1)
        /home/nictuku/go/src/pkg/runtime/mgc0.c:1813 +0x11b
[fp=0x7fc1f1f1dfb0] forcegchelper(0x7fc1f1f27f60)
        /home/nictuku/go/src/pkg/runtime/mheap.c:355 +0x24
[fp=0x7fc1f1f1dfb8] runtime.goexit()
        /home/nictuku/go/src/pkg/runtime/proc.c:1214
created by runtime.MHeap_Scavenger
        /home/nictuku/go/src/pkg/runtime/mheap.c:402
(... list of goroutintes ...)
I really don't know how to simplify this further. If you're OK with running this
program, a few notes:
- it uses UDP port 11221 and for better results the program's host:port should be
reachable from the public internet
- you will probably be able to trigger the crasher sooner by searching for a more
popular infohash from the bittorrent network (Google is your friend, or email me
privately)
Let me know if there's anything else I can do on my side to help.

remyoudompheng · 2013-03-27T21:49:59Z

Comment 12:

If the crash is caused by the scavenger, it's easy to make the issue reproduce faster by
reducing the "forcegc" and "limit" constants in src/pkg/runtime/mheap.c.

randall77 · 2013-03-28T20:12:50Z

Comment 13:

This doesn't appear to be a bug in maps, at least directly.  I caught an error in gdb
and it looks like the Hmap structure itself was corrupted on entry to the map code. 
Smells like a GC bug, but not definite yet.  I'll keep investigating.

alberts · 2013-03-29T19:40:20Z

Comment 14:

FWIW, the SSA tests seem to trigger this quite regularly.
https://golang.org/issue/4979?c=25

alberts · 2013-03-29T20:01:26Z

Comment 15:

the net/http tests on linux/386 also seem to be a good way to reproduce something that
looks like this issue.

randall77 · 2013-04-02T02:00:15Z

Comment 16:

This issue was closed by revision 0e7144a.

Status changed to Fixed.

nictuku added fixed labels Apr 2, 2013

rsc added this to the Go1.1 milestone Apr 14, 2015

rsc removed the go1.1 label Apr 14, 2015

golang locked and limited conversation to collaborators Jun 24, 2016

gopherbot added the FrozenDueToAge label Jun 24, 2016

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: runtime panic #5120

runtime: runtime panic #5120

nictuku commented Mar 24, 2013

DanielMorsing commented Mar 24, 2013

davecheney commented Mar 25, 2013

nictuku commented Mar 27, 2013

ianlancetaylor commented Mar 27, 2013

randall77 commented Mar 27, 2013

DanielMorsing commented Mar 27, 2013

randall77 commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

remyoudompheng commented Mar 27, 2013

randall77 commented Mar 28, 2013

alberts commented Mar 29, 2013

alberts commented Mar 29, 2013

randall77 commented Apr 2, 2013

runtime: runtime panic #5120

runtime: runtime panic #5120

Comments

nictuku commented Mar 24, 2013

DanielMorsing commented Mar 24, 2013

davecheney commented Mar 25, 2013

nictuku commented Mar 27, 2013

ianlancetaylor commented Mar 27, 2013

randall77 commented Mar 27, 2013

DanielMorsing commented Mar 27, 2013

randall77 commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

nictuku commented Mar 27, 2013

remyoudompheng commented Mar 27, 2013

randall77 commented Mar 28, 2013

alberts commented Mar 29, 2013

alberts commented Mar 29, 2013

randall77 commented Apr 2, 2013