Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

Closed
bcmills opened this issue Nov 12, 2019 · 20 comments
Closed

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

bcmills opened this issue Nov 12, 2019 · 20 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.15

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 12, 2019

2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo
2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo

Possibly related to #34835, #32324.

CC @mengzhuo @cherrymui
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 12, 2019
@bcmills bcmills added this to the Go1.14 milestone Nov 12, 2019
@bcmills bcmills mentioned this issue Nov 12, 2019
Closed
@mengzhuo
Copy link
Contributor

I can't reproduce by all.bash nor dist test ( tried serval times)
Bad memory block?

@cherrymui
Copy link
Member

Eyeballing the hex dump, it looks to me that the bad pointer is found in a g struct, in g._defer field.

  • 2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo:
object=0xc000106300 s.base()=0xc000106000 s.limit=0xc000107f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00052a000		g.stack.lo
 *(object+8) = 0xc000532000		g.stack.hi
 *(object+16) = 0xc00052a380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc0005313e0 <==	g._defer
 *(object+48) = 0xc000080380		g.m
 *(object+56) = 0xc0005310b8		g.sched.sp
 *(object+64) = 0x87038			g.sched.pc
 *(object+72) = 0xc000106300		g.sched.g
 ...

The stack bounds matches the stack of goroutine 67, as well as g.sched.pc and g.sched.sp.

What's interesting is that the bad pointer 0xc0005313e0 is also within the stack bound, in particular, within this frame

io/ioutil.readAll(0x3fdc60, 0xc000203a40, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/io/ioutil/ioutil.go:36 +0xd8 fp=0xc000531360 sp=0xc0005312e8 pc=0x170790

Presumably it is a stack-allocated defer. I don't know why spanOf does not find the span of the stack, returns a different span instead...

  • 2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo

This also looks like the g._defer field in a g. The stack bounds match goroutine 1.

What's more interesting is that findObject and badPointer were complaining about pointer 0xc000346ef8, but in the object dump the field is actually containing 0xc00034a688, a different value.

runtime: pointer 0xc000346ef8 to unallocated span span.base()=0xc000346000 span.limit=0xc000355e00 span.state=0
runtime: found in object at *(0xc000000180+0x28)
object=0xc000000180 s.base()=0xc000000000 s.limit=0xc000001f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00033c000		g.stack.lo
 *(object+8) = 0xc00034c000		g.stack.hi
 *(object+16) = 0xc00033c380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc00034a688 <==	g._defer

I think g._defer can only change synchronously, right? But the goroutine is not running...

Also, the span's address range, [0xc000346000, 0xc000355e00], overlaps with the stack bounds [0xc00033c000, 0xc00034c000], assuming my interpretation of the hex dump is right. How could this be possible?

@aclements any idea how g._defer could change this way, or how span tracking could go weirdly?

@cherrymui
Copy link
Member

Another one today https://build.golang.org/log/a810adac990353a9918a366fc13538c03649d206
Also looks like g._defer pointing to the stack.

@mengzhuo
Copy link
Contributor

@cherrymui
The PAGESIZE setting of this builder is 16384 bytes and
0xc000346000 % 16384 = 8192

It looks like some bug of mpagealloc?

@mknyszek
Copy link
Contributor

@mengzhuo Only the scavenger (mgcscavenge.go) operates in terms of physical pages, the runtime page size is always 8 KiB on all platforms, so 8 KiB alignment is exactly what we want.

It could be related to the page allocator, but the physical page size here doesn't matter, I don't think.

@ianlancetaylor
Copy link
Contributor

2019-12-05T17:23:50-50535e6/linux-mips64le-mengzhuo
2019-12-05T01:48:12-fa3a121/linux-mips64le-mengzhuo
2019-12-03T15:26:50-2ac1ca9/linux-mips64le-mengzhuo
2019-11-19T19:59:21-8b1e8a4/linux-mips64le-mengzhuo
2019-11-18T22:08:03-fff4502/linux-mips64le-mengzhuo
2019-11-15T20:56:35-9af8794/freebsd-arm64-dmgk
2019-11-15T02:31:58-498eaee/linux-mips64le-mengzhuo
2019-11-14T00:58:33-e77106c/linux-mips64le-mengzhuo
2019-11-12T22:28:47-13fb147/linux-mips64le-mengzhuo

@bcmills
Copy link
Contributor Author

bcmills commented Jan 22, 2020

Still occurring regularly. Does this need to be addressed for the 1.14 release? Should we mention it as a known issue in the release notes?
(CC @cagedmantis @dmitshur @toothrot)

2020-01-21T23:58:54-fb8cdb7/linux-mips64le-mengzhuo
2020-01-07T18:40:18-7340d15/linux-mips64le-mengzhuo
2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo
2019-12-18T19:09:23-87a5467/linux-mips64le-mengzhuo
2019-12-10T21:11:20-207a0b7/linux-mips64le-mengzhuo

@toothrot
Copy link
Contributor

I think we would need input from the runtime team to make that decision, or anyone else on this thread with sufficient expertise (@cherrymui @ianlancetaylor @aclements @rsc @randall77)

It's interesting that the failure also happened on freebsd-arm64, as @ianlancetaylor linked. Should the title be updated?

@bcmills
Copy link
Contributor Author

bcmills commented Jan 22, 2020

I'm not sure whether the freebsd-arm64-dmgk failure is the same underlying cause.

(I'd prefer to keep this issue focused on the mips64le builder unless we have a specific reason to suspect the same root cause for both.)

@cherrymui
Copy link
Member

I looked at a few new MIPS failures and they all look like the ones before, and are consistent with my previous comment. The "bad pointer" seems from g._defer, which points to the stack (within the stack bounds), but spanOf returns a different span. I don't yet understand how this could happen.

The freebsd-arm64 one is different, and may be fixed by CL https://go-review.googlesource.com/c/go/+/212626 .

@mengzhuo
Copy link
Contributor

mengzhuo commented Jan 22, 2020 via email

@bcmills
Copy link
Contributor Author

bcmills commented Feb 20, 2020

2020-02-20T03:27:46-715e8bb/linux-mips64le-mengzhuo
2020-02-19T21:00:02-1bf9d3c/linux-mips64le-mengzhuo
2020-02-17T10:30:03-88e564e/linux-mips64le-mengzhuo
2020-02-12T17:32:01-7385947/linux-mips64le-mengzhuo
2020-01-31T00:36:45-25d0ffe/linux-mips64le-mengzhuo
2020-01-29T02:43:46-a6701d8/linux-mips64le-mengzhuo
2020-01-24T19:37:58-67539f6/linux-mips64le-mengzhuo

@mengzhuo
Copy link
Contributor

@bcmills It weird that mips-rtrk don't crash not even once by this "unused span".
As I said before the kernel development of Loogson has a lots of patches and out of tree for a long time.

Here is kernel that loongson used.
http://dev.lemote.com:8000/cgit/linux-official.git/log/?h=linux-4.19

@toothrot toothrot modified the milestones: Go1.14, Go1.15 Feb 25, 2020
This was referenced Feb 27, 2020
Open
Open
Closed
@bcmills
Copy link
Contributor Author

bcmills commented Mar 5, 2020 •
edited

This strongly correlates to go/types.(*Checker) (#37602). All of these failures have it on the stack:

2020-03-04T19:06:44-68fea52/linux-mips64le-mengzhuo
2020-03-04T18:09:01-588ee79/linux-mips64le-mengzhuo
2020-03-04T17:33:06-cec0879/linux-mips64le-mengzhuo
2020-03-03T11:36:47-bd6f4cd/linux-mips64le-mengzhuo
2020-03-02T08:32:38-b79acf9/linux-mips64le-mengzhuo
2020-02-28T19:09:53-5a61de3/linux-mips64le-mengzhuo
2020-02-26T18:56:51-e95b858/linux-mips64le-mengzhuo
2020-02-26T17:58:41-a1103dc/linux-mips64le-mengzhuo
2020-02-25T21:08:47-46f9aea/linux-mips64le-mengzhuo
2020-02-24T09:16:18-218f457/linux-mips64le-mengzhuo
2020-02-21T22:00:03-240498d/linux-mips64le-mengzhuo
2020-02-20T03:27:46-715e8bb/linux-mips64le-mengzhuo
2020-02-19T21:00:02-1bf9d3c/linux-mips64le-mengzhuo
2020-02-12T17:32:01-7385947/linux-mips64le-mengzhuo
2020-01-31T00:36:45-25d0ffe/linux-mips64le-mengzhuo
2020-01-29T02:43:46-a6701d8/linux-mips64le-mengzhuo

CC @griesemer

@cherrymui
Copy link
Member

This strongly correlates to go/types.(*Checker)

I noticed that a while ago and looked at that function, but didn't find anything suspicious, especially about defers.

@mengzhuo
Copy link
Contributor

Hi @cherrymui

I've tried turn on debugCachedWork at runtime/mgc.go, now it failed the test every time.

runtime: pointer 0xc0002a9068 to unused region of span span.base()=0xc00016c000 span.limit=0xc0001b9808 span.state=1
runtime: found in object at *(0xc0002ed880+0x28)                                                                    
object=0xc0002ed880 s.base()=0xc0002ec000 s.limit=0xc0002edff0 s.spanclass=16 s.elemsize=112 s.state=mSpanInUse     
 *(object+0) = 0x1000000000020                                                                                      
 *(object+8) = 0xc0002a8dd8                                                                                         
 *(object+16) = 0x2aba30                                                                                            
 *(object+24) = 0x0                                                                                                 
 *(object+32) = 0x0                                                                                                 
 *(object+40) = 0xc0002a9068 <==                                                                                    
 *(object+48) = 0x733f20                                                                                            
 *(object+56) = 0xc0002a8fb0                                                                                        
 *(object+64) = 0x2ab24c                                                                                            
 *(object+72) = 0x0                                                                                                 
 *(object+80) = 0x0                                                                                                 
 *(object+88) = 0x0                                                                                                 
 *(object+96) = 0x0                                                                                                 
 *(object+104) = 0x0                                                                                                
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)                                         
                                                                                                                    
runtime stack:                                                                                                      
runtime.throw(0x72efec, 0x3e)                                                                                       
        /root/godev/src/runtime/panic.go:1114 +0x6c fp=0xc00003be30 sp=0xc00003be08 pc=0x4ef94                      
runtime.badPointer(0xffcded4338, 0xc0002a9068, 0xc0002ed880, 0x28)                                                  
        /root/godev/src/runtime/mbitmap.go:380 +0x2c4 fp=0xc00003be70 sp=0xc00003be30 pc=0x27cac                    
runtime.findObject(0xc0002a9068, 0xc0002ed880, 0x28, 0x0, 0x0, 0x0)                                                 
        /root/godev/src/runtime/mbitmap.go:416 +0xb8 fp=0xc00003bea0 sp=0xc00003be70 pc=0x27d68                     
runtime.scanobject(0xc0002ed880, 0xc000021e98)                                                                      
        /root/godev/src/runtime/mgcmark.go:1274 +0x2a8 fp=0xc00003bf38 sp=0xc00003bea0 pc=0x36060                   
runtime.gcDrain(0xc000021e98, 0x3)                                                                                  
        /root/godev/src/runtime/mgcmark.go:1032 +0x2b0 fp=0xc00003bf98 sp=0xc00003bf38 pc=0x35668                   
runtime.gcBgMarkWorker.func2()                                                                                      
        /root/godev/src/runtime/mgc.go:1932 +0x88 fp=0xc00003bfd0 sp=0xc00003bf98 pc=0x85d20                        
runtime.systemstack(0x4d2)                                                                                          
        /root/godev/src/runtime/asm_mips64x.s:206 +0xa8 fp=0xc00003bfd8 sp=0xc00003bfd0 pc=0x893d8                  
runtime.mstart()                                                                                                    
        /root/godev/src/runtime/proc.go:1042 fp=0xc00003bfd8 sp=0xc00003bfd8 pc=0x55348

@mengzhuo
Copy link
Contributor

mengzhuo commented Apr 26, 2020 •
edited

Hi, @cherrymui

I can reproduce this bug on my box by running runtime/TestDeferHeapAndStack test.
The builder trybot using short test mode which won't trigger this bug.

=== RUN   TestDeferHeapAndStack
runtime: pointer 0xc0004a9ae8 to unallocated span span.base()=0xc0004a8000 span.limit=0xc0004aa000 span.state=0
runtime: found in object at *(0xc00028bc20+0x28)
object=0xc00028bc20 s.base()=0xc00028a000 s.limit=0xc00028bfe0 s.spanclass=12 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0x10000000008
 *(object+8) = 0xc0004a99f0
 *(object+16) = 0x12021178c
 *(object+24) = 0x1202d9340
 *(object+32) = 0x0
 *(object+40) = 0xc0004a9ae8 <==
 *(object+48) = 0x0
 *(object+56) = 0x0
 *(object+64) = 0x0
 *(object+72) = 0xc000687840
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x1202d4b7a, 0x3e)
	/root/godev/src/runtime/panic.go:1116 +0x6c fp=0xffca57e720 sp=0xffca57e6f8 pc=0x120041e2c
runtime.badPointer(0xffc9545718, 0xc0004a9ae8, 0xc00028bc20, 0x28)
	/root/godev/src/runtime/mbitmap.go:380 +0x2c4 fp=0xffca57e760 sp=0xffca57e720 pc=0x12001a644
runtime.findObject(0xc0004a9ae8, 0xc00028bc20, 0x28, 0x0, 0x0, 0x0)
	/root/godev/src/runtime/mbitmap.go:416 +0xb8 fp=0xffca57e790 sp=0xffca57e760 pc=0x12001a700
runtime.scanobject(0xc00028bc20, 0xc000026698)
	/root/godev/src/runtime/mgcmark.go:1383 +0x2f0 fp=0xffca57e828 sp=0xffca57e790 pc=0x120029508
runtime.gcDrainN(0xc000026698, 0x10000, 0x12007e30c)
	/root/godev/src/runtime/mgcmark.go:1235 +0x1a0 fp=0xffca57e850 sp=0xffca57e828 pc=0x120028f08
runtime.gcAssistAlloc1(0xc000001c80, 0x10000)
	/root/godev/src/runtime/mgcmark.go:636 +0x140 fp=0xffca57e898 sp=0xffca57e850 pc=0x120027490
runtime.gcAssistAlloc.func1()
	/root/godev/src/runtime/mgcmark.go:547 +0x44 fp=0xffca57e8b0 sp=0xffca57e898 pc=0x12007ccec
runtime.systemstack(0x0)
	/root/godev/src/runtime/asm_mips64x.s:206 +0xa8 fp=0xffca57e8b8 sp=0xffca57e8b0 pc=0x120084b08
runtime.mstart()
	/root/godev/src/runtime/proc.go:1094 fp=0xffca57e8b8 sp=0xffca57e8b8 pc=0x120048460

goroutine 8 [GC assist marking]:
runtime.systemstack_switch()
	/root/godev/src/runtime/asm_mips64x.s:158 +0x8 fp=0xc0002f7060 sp=0xc0002f7058 pc=0x120084a40
runtime.gcAssistAlloc(0xc000001c80)
	/root/godev/src/runtime/mgcmark.go:546 +0x1b8 fp=0xc0002f70b8 sp=0xc0002f7060 pc=0x120027180
runtime.mallocgc(0x8, 0x1202608e0, 0xffca582701, 0xc0000e8400)
	/root/godev/src/runtime/malloc.go:950 +0x8dc fp=0xc0002f7178 sp=0xc0002f70b8 pc=0x1200110cc
runtime.newobject(0x1202608e0, 0xc0000e8000)
	/root/godev/src/runtime/malloc.go:1187 +0x4c fp=0xc0002f71a0 sp=0xc0002f7178 pc=0x1200113f4
runtime_test.deferHeapAndStack(0x69, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x4c fp=0xc0002f7270 sp=0xc0002f71a0 pc=0x12021171c
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 1 [chan receive, locked to thread]:
runtime.gopark(0x1202d7350, 0xc000064298, 0x1204a170e, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000125aa8 sp=0xc000125a90 pc=0x120045398
runtime.chanrecv(0xc000064240, 0xc000125bb0, 0xc000000101, 0x1201100a0)
	/root/godev/src/runtime/chan.go:567 +0x398 fp=0xc000125b30 sp=0xc000125aa8 pc=0x120008500
runtime.chanrecv1(0xc000064240, 0xc000125bb0)
	/root/godev/src/runtime/chan.go:434 +0x28 fp=0xc000125b58 sp=0xc000125b30 pc=0x120008118
testing.(*T).Run(0xc00011e280, 0x1202c88ec, 0x15, 0x1202d86a0, 0x12010fc01)
	/root/godev/src/testing/testing.go:1074 +0x398 fp=0xc000125c00 sp=0xc000125b58 pc=0x1201100c0
testing.runTests.func1(0xc00011e140)
	/root/godev/src/testing/testing.go:1342 +0x8c fp=0xc000125c48 sp=0xc000125c00 pc=0x120114a9c
testing.tRunner(0xc00011e140, 0xc000125d40)
	/root/godev/src/testing/testing.go:1022 +0xf8 fp=0xc000125c90 sp=0xc000125c48 pc=0x12010fca0
testing.runTests(0xc00000e100, 0x1204ab540, 0x153, 0x153, 0xbfa16e2ad25438c0, 0x8bb2d601f1, 0x1204c75e0, 0xfff2a1d978)
	/root/godev/src/testing/testing.go:1340 +0x328 fp=0xc000125d60 sp=0xc000125c90 pc=0x120111878
testing.(*M).Run(0xc00011a000, 0x0)
	/root/godev/src/testing/testing.go:1250 +0x1e0 fp=0xc000125e88 sp=0xc000125d60 pc=0x120110788
runtime_test.TestMain(0xc00011a000)
	/root/godev/src/runtime/crash_test.go:28 +0x3c fp=0xc000125ed0 sp=0xc000125e88 pc=0x1201ae844
main.main()
	_testmain.go:1209 +0x198 fp=0xc000125f80 sp=0xc000125ed0 pc=0x12023c050
runtime.main()
	/root/godev/src/runtime/proc.go:204 +0x2cc fp=0xc000125fd8 sp=0xc000125f80 pc=0x120044e54
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc000125fd8 sp=0xc000125fd8 pc=0x120086ed4

goroutine 2 [force gc (idle)]:
runtime.gopark(0x1202d7660, 0x1204c6d40, 0x1411, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002efb0 sp=0xc00002ef98 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.forcegchelper()
	/root/godev/src/runtime/proc.go:255 +0x108 fp=0xc00002efd8 sp=0xc00002efb0 pc=0x1200451f8
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002efd8 sp=0xc00002efd8 pc=0x120086ed4
created by runtime.init.5
	/root/godev/src/runtime/proc.go:243 +0x48

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1202d7660, 0x1204c7160, 0x140c, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002f7a8 sp=0xc00002f790 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.bgsweep(0xc000056000)
	/root/godev/src/runtime/mgcsweep.go:90 +0x200 fp=0xc00002f7d0 sp=0xc00002f7a8 pc=0x12002d1d8
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002f7d0 sp=0xc00002f7d0 pc=0x120086ed4
created by runtime.gcenable
	/root/godev/src/runtime/mgc.go:217 +0x74

goroutine 4 [sleep]:
runtime.gopark(0x1202d7660, 0x1204c7120, 0x1313, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002af28 sp=0xc00002af10 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.scavengeSleep(0xebdfb, 0x12b3d0)
	/root/godev/src/runtime/mgcscavenge.go:214 +0x100 fp=0xc00002af78 sp=0xc00002af28 pc=0x12002b038
runtime.bgscavenge(0xc000056000)
	/root/godev/src/runtime/mgcscavenge.go:316 +0x1e8 fp=0xc00002afd0 sp=0xc00002af78 pc=0x12002b248
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002afd0 sp=0xc00002afd0 pc=0x120086ed4
created by runtime.gcenable
	/root/godev/src/runtime/mgc.go:218 +0x9c

goroutine 5 [finalizer wait]:
runtime.gopark(0x1202d7660, 0x1204ef6a0, 0x1410, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000030750 sp=0xc000030738 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.runfinq()
	/root/godev/src/runtime/mfinal.go:175 +0xf0 fp=0xc0000307d8 sp=0xc000030750 pc=0x12001fc78
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000307d8 sp=0xc0000307d8 pc=0x120086ed4
created by runtime.createfing
	/root/godev/src/runtime/mfinal.go:156 +0xb0

goroutine 6 [chan receive]:
runtime.gopark(0x1202d7350, 0xc0000642f8, 0xfff2a1170e, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002e670 sp=0xc00002e658 pc=0x120045398
runtime.chanrecv(0xc0000642a0, 0x0, 0xc000001901, 0x120211604)
	/root/godev/src/runtime/chan.go:567 +0x398 fp=0xc00002e6f8 sp=0xc00002e670 pc=0x120008500
runtime.chanrecv1(0xc0000642a0, 0x0)
	/root/godev/src/runtime/chan.go:434 +0x28 fp=0xc00002e720 sp=0xc00002e6f8 pc=0x120008118
runtime_test.TestDeferHeapAndStack(0xc00011e280)
	/root/godev/src/runtime/stack_test.go:822 +0x1b0 fp=0xc00002e780 sp=0xc00002e720 pc=0x120211640
testing.tRunner(0xc00011e280, 0x1202d86a0)
	/root/godev/src/testing/testing.go:1022 +0xf8 fp=0xc00002e7c8 sp=0xc00002e780 pc=0x12010fca0
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002e7c8 sp=0xc00002e7c8 pc=0x120086ed4
created by testing.(*T).Run
	/root/godev/src/testing/testing.go:1073 +0x378

goroutine 7 [runnable]:
runtime.gcStart(0x0, 0x0, 0xc000000000)
	/root/godev/src/runtime/mgc.go:1232 +0x1c fp=0xc0001e32f8 sp=0xc0001e32f8 pc=0x12002290c
runtime.mallocgc(0x400, 0x120261f20, 0xc000490401, 0xc0005003d8)
	/root/godev/src/runtime/malloc.go:1151 +0x454 fp=0xc0001e33b8 sp=0xc0001e32f8 pc=0x120010c44
runtime.newobject(0x120261f20, 0x38)
	/root/godev/src/runtime/malloc.go:1187 +0x4c fp=0xc0001e33e0 sp=0xc0001e33b8 pc=0x1200113f4
runtime_test.deferHeapAndStack(0x1d, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:848 +0x128 fp=0xc0001e34b0 sp=0xc0001e33e0 pc=0x1202117f8
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 9 [runnable]:
runtime_test.deferHeapAndStack(0x39, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x20 fp=0xc0005a0b70 sp=0xc0005a0b70 pc=0x1202116f0
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 10 [runnable]:
runtime_test.deferHeapAndStack(0x43, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x20 fp=0xc0004a3390 sp=0xc0004a3390 pc=0x1202116f0
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000413600, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002a760 sp=0xc00002a748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000020000)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc00002a7d0 sp=0xc00002a760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002a7d0 sp=0xc00002a7d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500010, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000092760 sp=0xc000092748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000022800)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc0000927d0 sp=0xc000092760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000927d0 sp=0xc0000927d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500020, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000031760 sp=0xc000031748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000025000)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc0000317d0 sp=0xc000031760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000317d0 sp=0xc0000317d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500030, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00008e760 sp=0xc00008e748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000027800)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc00008e7d0 sp=0xc00008e760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00008e7d0 sp=0xc00008e7d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0
exit status 2
FAIL	runtime	0.144s

@mengzhuo
Copy link
Contributor

mengzhuo commented May 8, 2020

I did a git bisect HEAD go1.13.9 and found that this commit is the cause of this issue
a762221

@cherrymui
Copy link
Member

Thanks @mengzhuo . These are all useful information. I'll take a careful look when I get a chance. Thanks.

@gopherbot
Copy link

Change https://golang.org/cl/234478 mentions this issue: runtime: add a barrier in copystack before adjusting pointers

@mengzhuo mengzhuo mentioned this issue Jan 15, 2021
Closed
@golang golang locked and limited conversation to collaborators May 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.15
Development

No branches or pull requests
7 participants
@toothrot @mengzhuo @mknyszek @ianlancetaylor @bcmills @gopherbot @cherrymui