Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: intermittent checkmarks failures #9309

Closed
minux opened this issue Dec 14, 2014 · 5 comments
Closed

runtime: intermittent checkmarks failures #9309

minux opened this issue Dec 14, 2014 · 5 comments
Labels
FrozenDueToAge
Milestone
Go1.5

Comments

@minux
Copy link
Member

minux commented Dec 14, 2014 

go version devel +2d02ff2 Sat Dec 13 10:18:27 2014 +0000 dragonfly/amd64

I run all.bash three times, and this only happened once:

runtime:greyobject: checkmarks finds unexpected unmarked object obj=0xc2081ef450, mbits->bits=0x1 *mbits->bitp=0x5
runtime:greyobject Span: obj=0xc2081ef450 k=0x61040f7 s.start=0xc2081ee000 s.limit=0xc2081f0000 s.sizeclass=2 s.elemsize=16
 *(obj+0) = 0x36
 *(obj+8) = 0x84
fatal error: checkmark found unmarked object

runtime stack:
runtime.gothrow(0x65a030, 0x1f)
    /home/minux/build/go.git/src/runtime/panic.go:507 +0x98 fp=0xc20815da30 sp=0xc20815da18
runtime.greyobject(0xc2081ef450, 0xc20815db08, 0x800777140, 0x800777140)
    /home/minux/build/go.git/src/runtime/mgc.go:438 +0x23b fp=0xc20815da90 sp=0xc20815da30
runtime.scanobject(0xc20812b200, 0x7d4e00, 0x0, 0x800777140, 0x800777140)
    /home/minux/build/go.git/src/runtime/mgc.go:567 +0x395 fp=0xc20815db28 sp=0xc20815da90
runtime.scanblock(0xc20812b200, 0x58, 0x68a64c)
    /home/minux/build/go.git/src/runtime/mgc.go:629 +0x1e6 fp=0xc20815db68 sp=0xc20815db28
runtime.scanframe(0xc20815dcb8, 0x0, 0x1)
    /home/minux/build/go.git/src/runtime/mgc.go:924 +0x164 fp=0xc20815dc00 sp=0xc20815db68
runtime.gentraceback(0x4254d8, 0xc208203420, 0x0, 0xc208297b00, 0x0, 0x0, 0x7fffffff, 0x698940, 0x0, 0x0, ...)
    /home/minux/build/go.git/src/runtime/traceback.go:313 +0x801 fp=0xc20815dd10 sp=0xc20815dc00
runtime.scanstack(0xc208297b00)
    /home/minux/build/go.git/src/runtime/mgc.go:979 +0x214 fp=0xc20815dd88 sp=0xc20815dd10
runtime.gcphasework(0xc208297b00)
    /home/minux/build/go.git/src/runtime/mgc.go:1093 +0x9c fp=0xc20815dda0 sp=0xc20815dd88
runtime.stopg(0xc208297b00, 0x4)
    /home/minux/build/go.git/src/runtime/proc1.go:444 +0x91 fp=0xc20815ddc0 sp=0xc20815dda0
runtime.markroot(0xc208010000, 0x2e8)
    /home/minux/build/go.git/src/runtime/mgc.go:705 +0x161 fp=0xc20815de30 sp=0xc20815ddc0
runtime.parfordo(0xc208010000)
    /home/minux/build/go.git/src/runtime/parfor.go:84 +0x1af fp=0xc20815dea0 sp=0xc20815de30
runtime.gchelper()
    /home/minux/build/go.git/src/runtime/mgc.go:1467 +0x4b fp=0xc20815ded0 sp=0xc20815dea0
runtime.stopm()
    /home/minux/build/go.git/src/runtime/proc1.go:1057 +0x164 fp=0xc20815def8 sp=0xc20815ded0
runtime.findrunnable(0xc208013200)
    /home/minux/build/go.git/src/runtime/proc1.go:1368 +0x4bd fp=0xc20815df60 sp=0xc20815def8
runtime.schedule()
    /home/minux/build/go.git/src/runtime/proc1.go:1455 +0x1c7 fp=0xc20815df90 sp=0xc20815df60
runtime.goexit0(0xc2080b25a0)
    /home/minux/build/go.git/src/runtime/proc1.go:1576 +0x197 fp=0xc20815dfb8 sp=0xc20815df90
runtime.mcall(0x0)
    /home/minux/build/go.git/src/runtime/asm_amd64.s:187 +0x5a fp=0xc20815dfc8 sp=0xc20815dfb8

goroutine 1 [chan receive]:
runtime.gopark(0x698918, 0xc2080b0478, 0x636c90, 0xc)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc2080f1bc8 sp=0xc2080f1b98
runtime.goparkunlock(0xc2080b0478, 0x636c90, 0xc)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc2080f1bf0 sp=0xc2080f1bc8
runtime.chanrecv(0x5870a0, 0xc2080b0420, 0xc2080f1dc0, 0x46bf01, 0x0)
    /home/minux/build/go.git/src/runtime/chan.go:410 +0x3e0 fp=0xc2080f1c70 sp=0xc2080f1bf0
runtime.chanrecv1(0x5870a0, 0xc2080b0420, 0xc2080f1dc0)
    /home/minux/build/go.git/src/runtime/chan.go:311 +0x2b fp=0xc2080f1ca0 sp=0xc2080f1c70
testing.RunTests(0x698510, 0x752d60, 0x7b, 0x7b, 0xb4fc70aa243c9401)
    /home/minux/build/go.git/src/testing/testing.go:556 +0xad6 fp=0xc2080f1e48 sp=0xc2080f1ca0
testing.(*M).Run(0xc20803e0a0, 0x768060)
    /home/minux/build/go.git/src/testing/testing.go:485 +0x6c fp=0xc2080f1ed0 sp=0xc2080f1e48
main.main()
    runtime/_test/_testmain.go:694 +0x1d5 fp=0xc2080f1f98 sp=0xc2080f1ed0
runtime.main()
    /home/minux/build/go.git/src/runtime/proc.go:81 +0x1cc fp=0xc2080f1fe0 sp=0xc2080f1f98
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc2080f1fe8 sp=0xc2080f1fe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x698918, 0x755420, 0x638190, 0xf)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc20801a798 sp=0xc20801a768
runtime.goparkunlock(0x755420, 0x638190, 0xf)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc20801a7c0 sp=0xc20801a798
runtime.forcegchelper()
    /home/minux/build/go.git/src/runtime/proc.go:115 +0xce fp=0xc20801a7e0 sp=0xc20801a7c0
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc20801a7e8 sp=0xc20801a7e0
created by runtime.init·4
    /home/minux/build/go.git/src/runtime/proc.go:103 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x698918, 0x768400, 0x631db0, 0xd)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc20801d798 sp=0xc20801d768
runtime.goparkunlock(0x768400, 0x631db0, 0xd)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc20801d7c0 sp=0xc20801d798
runtime.bgsweep()
    /home/minux/build/go.git/src/runtime/mgc0.go:96 +0xdd fp=0xc20801d7e0 sp=0xc20801d7c0
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc20801d7e8 sp=0xc20801d7e0
created by runtime.gc
    /home/minux/build/go.git/src/runtime/mgc.go:2042 +0xe5c

goroutine 4 [finalizer wait]:
runtime.gopark(0x698918, 0x7683f8, 0x637a90, 0xe)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc208018f30 sp=0xc208018f00
runtime.goparkunlock(0x7683f8, 0x637a90, 0xe)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc208018f58 sp=0xc208018f30
runtime.runfinq()
    /home/minux/build/go.git/src/runtime/malloc.go:775 +0xba fp=0xc208018fe0 sp=0xc208018f58
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc208018fe8 sp=0xc208018fe0
created by runtime.createfing
    /home/minux/build/go.git/src/runtime/malloc.go:755 +0x5e

goroutine 5 [syscall]:
runtime.notetsleepg(0x755798, 0x1a257a321a, 0x0)
    /home/minux/build/go.git/src/runtime/lock_futex.go:198 +0x59 fp=0xc20801df60 sp=0xc20801df38
runtime.timerproc()
    /home/minux/build/go.git/src/runtime/time.go:207 +0xfa fp=0xc20801dfe0 sp=0xc20801df60
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc20801dfe8 sp=0xc20801dfe0
created by runtime.addtimerLocked
    /home/minux/build/go.git/src/runtime/time.go:113 +0x1ba

goroutine 560 [syscall]:
runtime_test.func·047(0x0, 0x4e94914f0000, 0x64b570, 0x14, 0xc208040060)
    /home/minux/build/go.git/src/runtime/futex_test.go:47 +0x2d fp=0xc20801cfb8 sp=0xc20801cf70
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc20801cfc0 sp=0xc20801cfb8
created by runtime_test.TestFutexsleep
    /home/minux/build/go.git/src/runtime/futex_test.go:51 +0x229

goroutine 65615 [semacquire]:
runtime.gopark(0x698918, 0x75c180, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc208138c50 sp=0xc208138c20
runtime.goparkunlock(0x75c180, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc208138c78 sp=0xc208138c50
runtime.semacquire(0x73e034, 0xc208102400)
    /home/minux/build/go.git/src/runtime/sema.go:89 +0x22f fp=0xc208138cb0 sp=0xc208138c78
runtime.gogc(0x800000000)
    /home/minux/build/go.git/src/runtime/malloc.go:461 +0x9d fp=0xc208138d08 sp=0xc208138cb0
runtime.mallocgc(0x10, 0x590ba0, 0x1, 0xc20825a0f0)
    /home/minux/build/go.git/src/runtime/malloc.go:345 +0x3e5 fp=0xc208138e18 sp=0xc208138d08
runtime.newobject(0x590ba0, 0xc20828aff8)
    /home/minux/build/go.git/src/runtime/malloc.go:387 +0x49 fp=0xc208138e40 sp=0xc208138e18
reflect.Value.MapKeys(0x58f4a0, 0xc20800a540, 0x15, 0x0, 0x0, 0x0)
    /home/minux/build/go.git/src/reflect/value.go:1053 +0x1e6 fp=0xc208138ee8 sp=0xc208138e40
runtime_test.func·058()
    /home/minux/build/go.git/src/runtime/map_test.go:270 +0xab fp=0xc208138fe0 sp=0xc208138ee8
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc208138fe8 sp=0xc208138fe0
created by runtime_test.testConcurrentReadsAfterGrowth
    /home/minux/build/go.git/src/runtime/map_test.go:274 +0x383

goroutine 30829 [semacquire]:
runtime.gopark(0x698918, 0x75bc80, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc208019e38 sp=0xc208019e08
runtime.goparkunlock(0x75bc80, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc208019e60 sp=0xc208019e38
runtime.semacquire(0xc2081f16e0, 0xc2081f1601)
    /home/minux/build/go.git/src/runtime/sema.go:89 +0x22f fp=0xc208019e98 sp=0xc208019e60
runtime.asyncsemacquire(0xc2081f16e0)
    /home/minux/build/go.git/src/runtime/sema.go:43 +0x2d fp=0xc208019eb0 sp=0xc208019e98
sync.(*WaitGroup).Wait(0xc208117d40)
    /home/minux/build/go.git/src/sync/waitgroup.go:132 +0x169 fp=0xc208019ed8 sp=0xc208019eb0
runtime_test.testConcurrentReadsAfterGrowth(0xc20821e090, 0x755801)
    /home/minux/build/go.git/src/runtime/map_test.go:277 +0x3ae fp=0xc208019f60 sp=0xc208019ed8
runtime_test.TestConcurrentReadsAfterGrowthReflect(0xc20821e090)
    /home/minux/build/go.git/src/runtime/map_test.go:287 +0x2d fp=0xc208019f78 sp=0xc208019f60
testing.tRunner(0xc20821e090, 0x753390)
    /home/minux/build/go.git/src/testing/testing.go:447 +0xbf fp=0xc208019fd0 sp=0xc208019f78
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc208019fd8 sp=0xc208019fd0
created by testing.RunTests
    /home/minux/build/go.git/src/testing/testing.go:555 +0xa8b

goroutine 561 [syscall]:
runtime_test.func·047(0x0, 0x1dcd65174876e800, 0x64afb0, 0x13, 0xc2080400c0)
    /home/minux/build/go.git/src/runtime/futex_test.go:47 +0x2d fp=0xc20801c7b8 sp=0xc20801c770
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc20801c7c0 sp=0xc20801c7b8
created by runtime_test.TestFutexsleep
    /home/minux/build/go.git/src/runtime/futex_test.go:51 +0x229

goroutine 65645 [garbage collection]:
runtime.systemstack_switch()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:199 fp=0xc2080d5cb0 sp=0xc2080d5ca8
runtime.gogc(0xc200000000)
    /home/minux/build/go.git/src/runtime/malloc.go:518 +0x28d fp=0xc2080d5d08 sp=0xc2080d5cb0
runtime.mallocgc(0x20, 0x61f0a0, 0xc200000000, 0xc2081e6e30)
    /home/minux/build/go.git/src/runtime/malloc.go:345 +0x3e5 fp=0xc2080d5e18 sp=0xc2080d5d08
runtime.newobject(0x61f0a0, 0xc20828a220)
    /home/minux/build/go.git/src/runtime/malloc.go:387 +0x49 fp=0xc2080d5e40 sp=0xc2080d5e18
reflect.Value.MapIndex(0x58f4a0, 0xc20800a540, 0x15, 0x590ba0, 0xc2081e6868, 0x42, 0x590ba0, 0xc2081e6e30, 0x42)
    /home/minux/build/go.git/src/reflect/value.go:986 +0x32 fp=0xc2080d5ee8 sp=0xc2080d5e40
runtime_test.func·058()
    /home/minux/build/go.git/src/runtime/map_test.go:272 +0x18b fp=0xc2080d5fe0 sp=0xc2080d5ee8
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc2080d5fe8 sp=0xc2080d5fe0
created by runtime_test.testConcurrentReadsAfterGrowth
    /home/minux/build/go.git/src/runtime/map_test.go:274 +0x383

goroutine 65636 [semacquire]:
runtime.gopark(0x698918, 0x75c180, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:146 +0x108 fp=0xc208203450 sp=0xc208203420
runtime.goparkunlock(0x75c180, 0x63c610, 0xa)
    /home/minux/build/go.git/src/runtime/proc.go:152 +0x48 fp=0xc208203478 sp=0xc208203450
runtime.semacquire(0x73e034, 0xc208020c00)
    /home/minux/build/go.git/src/runtime/sema.go:89 +0x22f fp=0xc2082034b0 sp=0xc208203478
runtime.gogc(0x800000000)
    /home/minux/build/go.git/src/runtime/malloc.go:461 +0x9d fp=0xc208203508 sp=0xc2082034b0
runtime.mallocgc(0x10, 0x590ba0, 0x1, 0xc208208000)
    /home/minux/build/go.git/src/runtime/malloc.go:345 +0x3e5 fp=0xc208203618 sp=0xc208203508
runtime.newobject(0x590ba0, 0xc20828a9c0)
    /home/minux/build/go.git/src/runtime/malloc.go:387 +0x49 fp=0xc208203640 sp=0xc208203618
reflect.Value.MapKeys(0x58f4a0, 0xc20800a540, 0x15, 0x0, 0x0, 0x0)
    /home/minux/build/go.git/src/reflect/value.go:1053 +0x1e6 fp=0xc2082036e8 sp=0xc208203640
runtime_test.func·058()
    /home/minux/build/go.git/src/runtime/map_test.go:270 +0xab fp=0xc2082037e0 sp=0xc2082036e8
runtime.goexit()
    /home/minux/build/go.git/src/runtime/asm_amd64.s:2402 +0x1 fp=0xc2082037e8 sp=0xc2082037e0
created by runtime_test.testConcurrentReadsAfterGrowth
    /home/minux/build/go.git/src/runtime/map_test.go:274 +0x383
FAIL    runtime 11.500s
@bradfitz bradfitz assigned bradfitz and RLH and unassigned bradfitz Dec 14, 2014
@bradfitz
Copy link
Contributor

/cc @rsc

@minux minux mentioned this issue Dec 14, 2014
Closed
@RLH
Copy link
Contributor

RLH commented Dec 15, 2014

The GC has a STW mark checker that preforms a second STW mark phase after
the concurrent mark phase is complete. If the STW marker encounters an
object that the Concurrent marker missed it throws. Typically this is due
to some write barrier not being properly called and these reports are of
great value in focusing our attention to where the missing write barriers
might be. Let us know if they are preventing you from making progress and
we will implement a mechanism to turn off the concurrent GC until we can
find and fix the misssing write barriers.

On Sun, Dec 14, 2014 at 8:47 AM, Brad Fitzpatrick notifications@github.com
wrote:

/cc @rsc https://github.com/rsc

—
Reply to this email directly or view it on GitHub
#9309 (comment).

@minux
Copy link
Member Author

minux commented Dec 18, 2014

It also happened on windows/386 with the net package, so it's probably not a dragonfly specific problem. There might be some races in the concurrent mark phase.

http://build.golang.org/log/6a305ee5d22f7d102906d8318325be862f264025

@minux minux changed the title runtime: intermittent checkmarks failure on dragonflybsd/amd64 runtime: intermittent checkmarks failures Dec 18, 2014
@minux minux added this to the Go1.5 milestone Dec 19, 2014
@minux minux mentioned this issue Dec 19, 2014
Closed
@davecheney
Copy link
Contributor

As @minux pointed out, #9388 is dup of this issue, but #9388 occurred on linux, so we can rule out dragonflyBSD as the cause.

@rsc
Copy link
Contributor

rsc commented Jan 6, 2015

Things are unstable enough now that it's probably not worth a bug report for every crash like this. If you are still seeing them in two weeks let us know.

@rsc rsc closed this as completed Jan 6, 2015
This was referenced Jan 10, 2015
Closed
Closed
Closed
@golang golang locked and limited conversation to collaborators Jun 25, 2016
@rsc rsc unassigned RLH Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.5
Development

No branches or pull requests
6 participants
@bradfitz @davecheney @rsc @minux @RLH @gopherbot