cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

mewmew · 2019-03-14T02:28:49Z

What version of Go are you using (`go version`)?

$ go version
go version devel +04f1b65cc6 Wed Mar 13 15:11:37 2019 +0000 linux/amd64

I also tried to run this on the Go 1.12 release, and the same crash happens there.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/u/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/u/goget:/home/u/Desktop/go"
GOPROXY=""
GORACE=""
GOROOT="/home/u/go1.13"
GOTMPDIR=""
GOTOOLDIR="/home/u/go1.13/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build238606776=/tmp/go-build -gno-record-gcc-switches"

What did you do?

For a minimal reproducible example code see: https://github.com/mewspring/go-cgo-issue

$ git clone https://github.com/mewspring/go-cgo-issue
$ cd go-cgo-issue/foo
$ make -B
$ ./foo
...
fatal error: cgo callback before cgo call

In essence, I have a minimal C program that invokes multiple shared libraries, each one written in C and linked with Go code using -buildmode=c-archive.

The program works great when invoking Go code from C this way when using one library. However, when you invoke Go functions from more than one library, the program crashes with the following error message.

u@x1 ~/D/f/foo> ./foo
function symbol table not sorted by program counter: 0xf7ef1f00 _cgo_topofstack > 0xf7dcbf20 runtime.goexit
    0xf7d80780 __x86.get_pc_thunk.ax
    0xf7d80790 __x86.get_pc_thunk.cx
..
    0xf7d8c840 runtime.evacuate_fast32
    0xf7d8cc30 runtime.mapassign_faststr
a1
       args: ./foo
b1
c1
fatal error: cgo callback before cgo call

Note: if we simply uncomment one line (https://github.com/mewspring/go-cgo-issue/blob/d03e251e54cc0a6821bee5fb15c52d69e90c08d7/c/c.c#L12), the invocation of the Go function FooBar in the c.so library, then the program works as expected.

What did you expect to see?

No crash. In particular, I expected to see the output:

a1
       args: ./foo
b1
c1
       foobar: ./foo

And a status code of 0.

What did you see instead?

The crash "fatal error: cgo callback before cgo call" crash.

The text was updated successfully, but these errors were encountered:

mewmew · 2019-03-16T09:28:24Z

Is there any further information I could provide to help investigate the cause of this issue? Also, is this a previously known issue or limitation of Cgo interaction with Go?

The scenario is as follows.

We have a C program foo and three shared libraries written in C, a.so, b.so and c.so.

The first C library (a.so) links against a go-a.a archive produced with -buildmode=c-archive. The go-a.a archive contains one exported Go function that is invoked from C.
The second C library (b.so) links against a go-b.a archive produced with -buildmode=c-archive. The go-b.a archive contains no exported Go functions.
The third C library (c.so) links against a go-c.a archive produced with -buildmode=c-archive. The go-c.a archive contains one exported Go function that is invoked from C.

Given this setup, the resulting binary foo crashes with fatal error: cgo callback before cgo call.

If we change the scenario to not invoke any Go function in go-c.a, then we can successfully run the foo program, without experiencing any crashes.

So, from what I can tell, it seems that only one shared library may have Cgo code in it (produced with -buildmode=c-archive)? It this a correct understanding of the issue and is it a known limitation?

Cheers,
Robin

mewmew · 2019-03-16T13:00:14Z

A quick update. @acln0 helped me do some initial trouble shooting. We've now reproduced this issue on two machines (and on both 32-bit and 64-bit builds).

mewmew · 2019-03-16T15:46:15Z

We could further conclude that the issue is with the function symbol table not sorted by program counter error (and not the cgo callback before cgo call error).

One way of doing so was to insert a call to sleep before invoking the Go function FooBar in c.c. With the added call to sleep, the ./foo program crashes instead with the error message panic before malloc heap initialized, preceded by function symbol table not sorted by program counter.

crash error message Output

u@x1 ~/D/f/g/foo> ./foo function symbol table not sorted by program counter: 0x7f814ca73920 _cgo_topofstack > 0x7f814c8f0970 runtime.goexit 0x7f814c8a3730 sync/atomic.StoreUint32 0x7f814c8a3740 internal/cpu.Initialize 0x7f814c8a3790 internal/cpu.processOptions 0x7f814c8a3e20 internal/cpu.indexByte 0x7f814c8a3e60 internal/cpu.doinit 0x7f814c8a4270 internal/cpu.cpuid 0x7f814c8a4290 internal/cpu.xgetbv 0x7f814c8a42b0 type..hash.internal/cpu.CacheLinePad 0x7f814c8a42c0 type..eq.internal/cpu.CacheLinePad 0x7f814c8a42d0 type..hash.internal/cpu.arm64 0x7f814c8a4330 type..eq.internal/cpu.arm64 0x7f814c8a43a0 type..hash.internal/cpu.option 0x7f814c8a4420 type..eq.internal/cpu.option 0x7f814c8a44d0 type..hash.internal/cpu.x86 0x7f814c8a4530 type..eq.internal/cpu.x86 0x7f814c8a4590 type..hash.[15]internal/cpu.option 0x7f814c8a4600 type..eq.[15]internal/cpu.option 0x7f814c8a4690 runtime/internal/atomic.Store 0x7f814c8a46a0 internal/bytealg.init.0 0x7f814c8a46d0 internal/bytealg.init 0x7f814c8a4730 runtime.cmpstring 0x7f814c8a4750 cmpbody 0x7f814c8a4990 runtime.memequal 0x7f814c8a49c0 runtime.memequal_varlen 0x7f814c8a49f0 memeqbody 0x7f814c8a4b30 internal/bytealg.IndexByteString 0x7f814c8a4b50 indexbytebody 0x7f814c8a4c70 runtime.memhash0 0x7f814c8a4c80 runtime.memhash8 0x7f814c8a4ce0 runtime.memhash16 0x7f814c8a4d40 runtime.memhash128 0x7f814c8a4da0 runtime.memhash_varlen 0x7f814c8a4df0 runtime.strhash 0x7f814c8a4e50 runtime.f32hash 0x7f814c8a4f70 runtime.f64hash 0x7f814c8a5090 runtime.c64hash 0x7f814c8a5100 runtime.c128hash 0x7f814c8a5170 runtime.interhash 0x7f814c8a52e0 runtime.nilinterhash 0x7f814c8a5450 runtime.memequal0 0x7f814c8a5460 runtime.memequal8 0x7f814c8a5480 runtime.memequal16 0x7f814c8a54a0 runtime.memequal32 0x7f814c8a54c0 runtime.memequal64 0x7f814c8a54e0 runtime.memequal128 0x7f814c8a5510 runtime.f32equal 0x7f814c8a5540 runtime.f64equal 0x7f814c8a5570 runtime.c64equal 0x7f814c8a55b0 runtime.c128equal 0x7f814c8a55f0 runtime.strequal 0x7f814c8a5660 runtime.interequal 0x7f814c8a56d0 runtime.nilinterequal 0x7f814c8a5740 runtime.efaceeq 0x7f814c8a5860 runtime.ifaceeq 0x7f814c8a5980 runtime.alginit 0x7f814c8a5a30 runtime.initAlgAES 0x7f814c8a5b00 runtime.atomicwb 0x7f814c8a5b80 runtime.atomicstorep 0x7f814c8a5bd0 runtime.mmap 0x7f814c8a5d50 runtime.munmap 0x7f814c8a5e00 runtime.sigaction 0x7f814c8a5f60 runtime.cgocall 0x7f814c8a6050 runtime.cgocallbackg 0x7f814c8a61c0 runtime.cgocallbackg1 0x7f814c8a6430 runtime.unwindm 0x7f814c8a64f0 runtime.cgoCheckArg 0x7f814c8a6c40 runtime.cgoCheckUnknownPointer 0x7f814c8a6f90 runtime.cgoIsGoPointer 0x7f814c8a7080 runtime.cgoCheckResult 0x7f814c8a7110 runtime._cgo_panic_internal 0x7f814c8a7190 runtime.cgoCheckWriteBarrier 0x7f814c8a7290 runtime.cgoCheckMemmove 0x7f814c8a7330 runtime.cgoCheckSliceCopy 0x7f814c8a7410 runtime.cgoCheckTypedBlock 0x7f814c8a77a0 runtime.cgoCheckBits 0x7f814c8a7890 runtime.cgoCheckUsingType 0x7f814c8a7af0 runtime.makechan 0x7f814c8a7d30 runtime.chansend1 0x7f814c8a7d70 runtime.chansend 0x7f814c8a83b0 runtime.send 0x7f814c8a84d0 runtime.sendDirect 0x7f814c8a8560 runtime.recvDirect 0x7f814c8a85f0 runtime.closechan 0x7f814c8a8870 runtime.chanrecv1 0x7f814c8a88b0 runtime.chanrecv 0x7f814c8a8f70 runtime.recv 0x7f814c8a9130 runtime.(*waitq).dequeue 0x7f814c8a9230 runtime.init.0 0x7f814c8a9290 runtime.(*cpuProfile).add 0x7f814c8a93b0 runtime.(*cpuProfile).addNonGo 0x7f814c8a94b0 runtime.(*cpuProfile).addExtra 0x7f814c8a96b0 runtime.debugCallCheck 0x7f814c8a9790 runtime.debugCallWrap 0x7f814c8a9840 runtime.gogetenv 0x7f814c8a9990 runtime.(*TypeAssertionError).Error 0x7f814c8a9de0 runtime.errorString.Error 0x7f814c8a9e60 runtime.plainError.Error 0x7f814c8a9e80 runtime.printany 0x7f814c8aa680 runtime.panicwrap 0x7f814c8aaa70 runtime.Callers 0x7f814c8aaaf0 runtime.GOROOT 0x7f814c8aab80 runtime.float64frombits 0x7f814c8aab90 runtime.memhash 0x7f814c8aaea0 runtime.memhash32 0x7f814c8aaf10 runtime.memhash64 0x7f814c8aaf80 runtime.getitab 0x7f814c8ab310 runtime.(*itabTableType).find 0x7f814c8ab370 runtime.itabAdd 0x7f814c8ab4d0 runtime.(*itabTableType).add 0x7f814c8ab530 runtime.(*itab).init 0x7f814c8ab9a0 runtime.itabsinit 0x7f814c8aba90 runtime.panicdottypeI 0x7f814c8abba0 runtime.convT64 0x7f814c8abc20 runtime.convTstring 0x7f814c8abcc0 runtime.assertE2I 0x7f814c8abda0 runtime.assertE2I2 0x7f814c8abe50 runtime.iterate_itabs 0x7f814c8abed0 runtime.(*lfstack).push 0x7f814c8ac030 runtime.(*lfstack).pop 0x7f814c8ac070 runtime.lfnodeValidate 0x7f814c8ac130 runtime.lock 0x7f814c8ac2e0 runtime.unlock 0x7f814c8ac3b0 runtime.notewakeup 0x7f814c8ac480 runtime.notesleep 0x7f814c8ac580 runtime.notetsleep_internal 0x7f814c8ac740 runtime.notetsleep 0x7f814c8ac7d0 runtime.notetsleepg 0x7f814c8ac870 runtime.mallocinit 0x7f814c8acad0 runtime.(*mheap).sysAlloc 0x7f814c8ad2d0 runtime.sysReserveAligned 0x7f814c8ad440 runtime.(*mcache).nextFree 0x7f814c8ad680 runtime.mallocgc 0x7f814c8ae020 runtime.largeAlloc 0x7f814c8ae1c0 runtime.newobject 0x7f814c8ae220 runtime.newarray 0x7f814c8ae2f0 runtime.profilealloc 0x7f814c8ae360 runtime.fastexprand 0x7f814c8ae4e0 runtime.persistentalloc 0x7f814c8ae590 runtime.persistentalloc1 0x7f814c8ae8c0 runtime.inPersistentAlloc 0x7f814c8ae900 runtime.(*linearAlloc).alloc 0x7f814c8ae9e0 runtime.(*hmap).incrnoverflow 0x7f814c8aea70 runtime.(*hmap).newoverflow 0x7f814c8aed50 runtime.makemap_small 0x7f814c8aedf0 runtime.makemap 0x7f814c8aefd0 runtime.makeBucketArray 0x7f814c8af210 runtime.mapaccess2 0x7f814c8af4b0 runtime.mapassign 0x7f814c8afac0 runtime.hashGrow 0x7f814c8afcf0 runtime.growWork 0x7f814c8afda0 runtime.evacuate 0x7f814c8b03e0 runtime.advanceEvacuationMark 0x7f814c8b04d0 runtime.mapaccess1_fast32 0x7f814c8b0690 runtime.mapaccess2_fast32 0x7f814c8b0860 runtime.mapassign_fast32 0x7f814c8b0bf0 runtime.growWork_fast32 0x7f814c8b0ca0 runtime.evacuate_fast32 0x7f814c8b10d0 runtime.mapassign_faststr 0x7f814c8b1530 runtime.growWork_faststr 0x7f814c8b15e0 runtime.evacuate_faststr 0x7f814c8b1a60 runtime.typedmemmove 0x7f814c8b1b30 runtime.reflectcallmove 0x7f814c8b1bd0 runtime.typedslicecopy 0x7f814c8b1d10 runtime.typedmemclr 0x7f814c8b1d80 runtime.memclrHasPointers 0x7f814c8b1de0 runtime.(*mspan).refillAllocCache 0x7f814c8b1e00 runtime.(*mspan).nextFreeIndex 0x7f814c8b1f90 runtime.markBitsForAddr 0x7f814c8b2080 runtime.findObject 0x7f814c8b2450 runtime.heapBits.nextArena 0x7f814c8b24c0 runtime.heapBits.forward 0x7f814c8b2580 runtime.heapBits.forwardOrBoundary 0x7f814c8b2640 runtime.bulkBarrierPreWrite 0x7f814c8b2b00 runtime.bulkBarrierPreWriteSrcOnly 0x7f814c8b2d00 runtime.bulkBarrierBitmap 0x7f814c8b2ea0 runtime.typeBitsBulkBarrier 0x7f814c8b3160 runtime.heapBits.initSpan 0x7f814c8b3370 runtime.heapBits.initCheckmarkSpan 0x7f814c8b34b0 runtime.heapBits.clearCheckmarkSpan 0x7f814c8b3560 runtime.(*mspan).countAlloc 0x7f814c8b35e0 runtime.heapBitsSetType 0x7f814c8b4040 runtime.heapBitsSetTypeGCProg 0x7f814c8b43b0 runtime.progToPointerMask 0x7f814c8b44d0 runtime.runGCProg 0x7f814c8b4b30 runtime.materializeGCProg 0x7f814c8b4be0 runtime.allocmcache 0x7f814c8b4ca0 runtime.freemcache 0x7f814c8b4d00 runtime.(*mcache).refill 0x7f814c8b4e40 runtime.(*mcache).releaseAll 0x7f814c8b4ee0 runtime.(*mcache).prepareForSweep 0x7f814c8b4ff0 runtime.(*mcentral).cacheSpan 0x7f814c8b5450 runtime.(*mcentral).uncacheSpan 0x7f814c8b55b0 runtime.(*mcentral).freeSpan 0x7f814c8b5750 runtime.(*mcentral).grow 0x7f814c8b58d0 runtime.sysAlloc 0x7f814c8b59e0 runtime.sysUnused 0x7f814c8b5bd0 runtime.sysUsed 0x7f814c8b5c50 runtime.sysFree 0x7f814c8b5ca0 runtime.sysMap 0x7f814c8b5d80 runtime.queuefinalizer 0x7f814c8b5fd0 runtime.wakefing 0x7f814c8b6060 runtime.createfing 0x7f814c8b60d0 runtime.runfinq 0x7f814c8b6500 runtime.SetFinalizer 0x7f814c8b6d10 runtime.(*fixalloc).alloc 0x7f814c8b6e70 runtime.gcinit 0x7f814c8b6f50 runtime.readgogc 0x7f814c8b7020 runtime.gcenable 0x7f814c8b70b0 runtime/debug.setGCPercent 0x7f814c8b7180 runtime.(*gcControllerState).startCycle 0x7f814c8b7440 runtime.(*gcControllerState).revise 0x7f814c8b7550 runtime.(*gcControllerState).endCycle 0x7f814c8b7a60 runtime.(*gcControllerState).enlistWorker 0x7f814c8b7bf0 runtime.(*gcControllerState).findRunnableGCWorker 0x7f814c8b7e50 runtime.pollFractionalWorkerExit 0x7f814c8b7f10 runtime.gcSetTriggerRatio 0x7f814c8b82e0 runtime.gcWaitOnMark 0x7f814c8b83b0 runtime.gcStart 0x7f814c8b8a40 runtime.gcMarkDone 0x7f814c8b8d10 runtime.gcMarkTermination 0x7f814c8b9840 runtime.gcBgMarkStartWorkers 0x7f814c8b9910 runtime.gcBgMarkWorker 0x7f814c8b9de0 runtime.gcMark 0x7f814c8ba360 runtime.gcSweep 0x7f814c8ba550 runtime.gcResetMarkState 0x7f814c8ba6a0 sync.runtime_registerPoolCleanup 0x7f814c8ba700 runtime.clearpools 0x7f814c8ba870 runtime.itoaDiv 0x7f814c8ba950 runtime.fmtNSAsMS 0x7f814c8baae0 runtime.(*treapNode).pred 0x7f814c8babf0 runtime.(*treapNode).succ 0x7f814c8bad00 runtime.treapIter.prev 0x7f814c8bad50 runtime.(*mTreap).start 0x7f814c8bad80 runtime.(*mTreap).end 0x7f814c8badb0 runtime.(*mTreap).insert 0x7f814c8bb080 runtime.(*mTreap).removeNode 0x7f814c8bb1a0 runtime.(*mTreap).find 0x7f814c8bb250 runtime.(*mTreap).removeSpan 0x7f814c8bb2e0 runtime.(*mTreap).erase 0x7f814c8bb330 runtime.(*mTreap).rotateLeft 0x7f814c8bb3f0 runtime.(*mTreap).rotateRight 0x7f814c8bb4b0 runtime.gcMarkRootPrepare 0x7f814c8bb600 runtime.gcMarkRootCheck 0x7f814c8bb850 runtime.markroot 0x7f814c8bbbc0 runtime.markrootBlock 0x7f814c8bbc80 runtime.markrootFreeGStacks 0x7f814c8bbdb0 runtime.markrootSpans 0x7f814c8bbfe0 runtime.gcAssistAlloc 0x7f814c8bc2b0 runtime.gcAssistAlloc1 0x7f814c8bc600 runtime.gcWakeAllAssists 0x7f814c8bc680 runtime.gcParkAssist 0x7f814c8bc7d0 runtime.gcFlushBgCredit 0x7f814c8bc960 runtime.scanstack 0x7f814c8bd030 runtime.scanframeworker 0x7f814c8bd210 runtime.gcDrain 0x7f814c8bd650 runtime.gcDrainN 0x7f814c8bd890 runtime.scanblock 0x7f814c8bda10 runtime.scanobject 0x7f814c8bddc0 runtime.shade 0x7f814c8bde70 runtime.greyobject 0x7f814c8be430 runtime.gcDumpObject 0x7f814c8be8d0 runtime.gcmarknewobject 0x7f814c8be960 runtime.gcMarkTinyAllocs 0x7f814c8bea50 runtime.initCheckmarks 0x7f814c8beb90 runtime.clearCheckmarks 0x7f814c8becd0 runtime.init.1 0x7f814c8bece0 runtime.(*stackScanState).putPtr 0x7f814c8bee00 runtime.(*stackScanState).getPtr 0x7f814c8bef10 runtime.(*stackScanState).addObject 0x7f814c8bf080 runtime.binarySearchTree 0x7f814c8bf1e0 runtime.(*stackScanState).findObject 0x7f814c8bf230 runtime.finishsweep_m 0x7f814c8bf280 runtime.bgsweep 0x7f814c8bf3f0 runtime.sweepone 0x7f814c8bf7e0 runtime.(*mspan).ensureSwept 0x7f814c8bf8e0 runtime.(*mspan).sweep 0x7f814c8c0290 runtime.deductSweepCredit 0x7f814c8c03f0 runtime.clobberfree 0x7f814c8c0410 runtime.(*gcSweepBuf).push 0x7f814c8c05d0 runtime.(*gcSweepBuf).block 0x7f814c8c06b0 runtime.init.2 0x7f814c8c06c0 runtime.(*gcWork).init 0x7f814c8c0730 runtime.(*gcWork).put 0x7f814c8c0840 runtime.(*gcWork).putBatch 0x7f814c8c0a10 runtime.(*gcWork).tryGet 0x7f814c8c0b00 runtime.(*gcWork).dispose 0x7f814c8c0bf0 runtime.(*gcWork).balance 0x7f814c8c0cf0 runtime.(*workbuf).checknonempty 0x7f814c8c0d50 runtime.(*workbuf).checkempty 0x7f814c8c0db0 runtime.getempty 0x7f814c8c0fe0 runtime.putempty 0x7f814c8c1040 runtime.putfull 0x7f814c8c10a0 runtime.trygetfull 0x7f814c8c1120 runtime.handoff 0x7f814c8c11e0 runtime.prepareFreeWorkbufs 0x7f814c8c1290 runtime.freeSomeWbufs 0x7f814c8c1370 runtime.(*mheap).coalesce 0x7f814c8c15e0 runtime.(*mspan).scavenge 0x7f814c8c16a0 runtime.recordspan 0x7f814c8c1850 runtime.inHeapOrStack 0x7f814c8c18e0 runtime.spanOfHeap 0x7f814c8c1960 runtime.(*mheap).init 0x7f814c8c1dc0 runtime.(*mheap).reclaim 0x7f814c8c2020 runtime.(*mheap).reclaimChunk 0x7f814c8c2360 runtime.(*mheap).alloc_m 0x7f814c8c2670 runtime.(*mheap).alloc 0x7f814c8c2760 runtime.(*mheap).allocManual 0x7f814c8c2840 runtime.(*mheap).setSpans 0x7f814c8c2910 runtime.(*mheap).pickFreeSpan 0x7f814c8c2a50 runtime.(*mheap).allocSpanLocked 0x7f814c8c2ed0 runtime.(*mheap).grow 0x7f814c8c30e0 runtime.(*mheap).freeSpan 0x7f814c8c3160 runtime.(*mheap).freeManual 0x7f814c8c3210 runtime.(*mheap).freeSpanLocked 0x7f814c8c3500 runtime.(*mheap).scavengeLargest 0x7f814c8c3690 runtime.(*mheap).scavengeAll 0x7f814c8c37e0 runtime.(*mheap).scavenge 0x7f814c8c3a80 runtime.(*mSpanList).remove 0x7f814c8c3c10 runtime.(*mSpanList).insert 0x7f814c8c3d30 runtime.(*mSpanList).insertBack 0x7f814c8c3e50 runtime.(*mSpanList).takeAll 0x7f814c8c3eb0 runtime.addspecial 0x7f814c8c4080 runtime.removespecial 0x7f814c8c4230 runtime.addfinalizer 0x7f814c8c44a0 runtime.removefinalizer 0x7f814c8c4550 runtime.setprofilebucket 0x7f814c8c4610 runtime.freespecial 0x7f814c8c4760 runtime.newMarkBits 0x7f814c8c4aa0 runtime.nextMarkBitArenaEpoch 0x7f814c8c4b60 runtime.newArenaMayUnlock 0x7f814c8c4c50 runtime.newBucket 0x7f814c8c4d20 runtime.(*bucket).mp 0x7f814c8c4d90 runtime.(*bucket).bp 0x7f814c8c4e10 runtime.stkbucket 0x7f814c8c5150 runtime.eqslice 0x7f814c8c51a0 runtime.mProf_NextCycle 0x7f814c8c5220 runtime.mProf_Flush 0x7f814c8c5290 runtime.mProf_FlushLocked 0x7f814c8c5370 runtime.mProf_Malloc 0x7f814c8c5540 runtime.mProf_Free 0x7f814c8c5600 runtime.blockevent 0x7f814c8c5680 runtime.blocksampled 0x7f814c8c5710 runtime.saveblockevent 0x7f814c8c58c0 sync.event 0x7f814c8c59a0 runtime.tracealloc 0x7f814c8c5c20 runtime.tracefree 0x7f814c8c5d80 runtime.tracegc 0x7f814c8c5e70 runtime.init.3 0x7f814c8c5f00 runtime.cachestats 0x7f814c8c5f80 runtime.flushmcache 0x7f814c8c6000 runtime.purgecachedstats 0x7f814c8c60a0 runtime.mSysStatInc 0x7f814c8c6160 runtime.mSysStatDec 0x7f814c8c6220 runtime.(*wbBuf).reset 0x7f814c8c62b0 runtime.wbBufFlush 0x7f814c8c63b0 runtime.wbBufFlush1 0x7f814c8c65f0 internal/poll.runtime_pollServerInit 0x7f814c8c6630 internal/poll.runtime_pollOpen 0x7f814c8c6770 internal/poll.runtime_pollClose 0x7f814c8c6890 runtime.(*pollCache).free 0x7f814c8c68f0 internal/poll.runtime_pollUnblock 0x7f814c8c6ae0 runtime.netpollready 0x7f814c8c6bd0 runtime.netpollunblock 0x7f814c8c6c70 runtime.(*pollCache).alloc 0x7f814c8c6d40 runtime.netpollinit 0x7f814c8c6e20 runtime.netpollopen 0x7f814c8c6eb0 runtime.netpoll 0x7f814c8c70f0 runtime.futexsleep 0x7f814c8c71c0 runtime.futexwakeup 0x7f814c8c7260 runtime.getproccount 0x7f814c8c7360 runtime.newosproc 0x7f814c8c7530 runtime.newosproc0 0x7f814c8c7630 runtime.sysargs 0x7f814c8c78b0 runtime.sysauxv 0x7f814c8c79c0 runtime.osinit 0x7f814c8c7a00 runtime.getRandomData 0x7f814c8c7b60 runtime.libpreinit 0x7f814c8c7b90 runtime.minit 0x7f814c8c7be0 runtime.setsig 0x7f814c8c7ca0 runtime.setsigstack 0x7f814c8c7d30 runtime.sysSigaction 0x7f814c8c7da0 runtime.panicCheckMalloc 0x7f814c8c7e50 runtime.panicindex 0x7f814c8c7f80 runtime.panicslice 0x7f814c8c80b0 runtime.panicdivide 0x7f814c8c8120 runtime.throwinit 0x7f814c8c8170 runtime.deferproc 0x7f814c8c82a0 runtime.testdefersizes 0x7f814c8c84f0 runtime.init.4 0x7f814c8c8570 runtime.newdefer 0x7f814c8c8790 runtime.freedefer 0x7f814c8c89a0 runtime.freedeferpanic 0x7f814c8c89f0 runtime.freedeferfn 0x7f814c8c8a40 runtime.deferreturn 0x7f814c8c8b60 runtime.preprintpanics 0x7f814c8c8d30 runtime.printpanics 0x7f814c8c8e30 runtime.gopanic 0x7f814c8c93a0 runtime.getargp 0x7f814c8c93b0 runtime.gorecover 0x7f814c8c9400 sync.throw 0x7f814c8c9440 runtime.throw 0x7f814c8c94d0 runtime.recovery 0x7f814c8c9630 runtime.fatalthrow 0x7f814c8c96a0 runtime.fatalpanic 0x7f814c8c9750 runtime.startpanic_m 0x7f814c8c9900 runtime.dopanic_m 0x7f814c8c9c90 runtime.shouldPushSigpanic 0x7f814c8c9d50 runtime.recordForPanic 0x7f814c8c9ea0 runtime.printlock 0x7f814c8c9f20 runtime.printunlock 0x7f814c8c9f90 runtime.gwrite 0x7f814c8ca0e0 runtime.printsp 0x7f814c8ca130 runtime.printnl 0x7f814c8ca180 runtime.printbool 0x7f814c8ca1f0 runtime.printfloat 0x7f814c8ca490 runtime.printcomplex 0x7f814c8ca520 runtime.printuint 0x7f814c8ca620 runtime.printint 0x7f814c8ca690 runtime.printhex 0x7f814c8ca7a0 runtime.printpointer 0x7f814c8ca7e0 runtime.printstring 0x7f814c8ca870 runtime.printslice 0x7f814c8ca940 runtime.printeface 0x7f814c8ca9f0 runtime.hexdumpWords 0x7f814c8cac40 runtime.main 0x7f814c8cafd0 runtime.init.5 0x7f814c8cb020 runtime.forcegchelper 0x7f814c8cb150 runtime.gopark 0x7f814c8cb2a0 runtime.goready 0x7f814c8cb310 runtime.acquireSudog 0x7f814c8cb6a0 runtime.releaseSudog 0x7f814c8cba30 runtime.funcPC 0x7f814c8cba40 runtime.badmcall 0x7f814c8cba90 runtime.badmcall2 0x7f814c8cbae0 runtime.badreflectcall 0x7f814c8cbb30 runtime.badmorestackg0 0x7f814c8cbb70 runtime.badmorestackgsignal 0x7f814c8cbbb0 runtime.badctxt 0x7f814c8cbbe0 runtime.allgadd 0x7f814c8cbd30 runtime.cpuinit 0x7f814c8cbec0 runtime.schedinit 0x7f814c8cc120 runtime.checkmcount 0x7f814c8cc1e0 runtime.mcommoninit 0x7f814c8cc410 runtime.ready 0x7f814c8cc6d0 runtime.freezetheworld 0x7f814c8cc780 runtime.casfrom_Gscanstatus 0x7f814c8ccbe0 runtime.castogscanstatus 0x7f814c8cccc0 runtime.casgstatus 0x7f814c8ccec0 runtime.scang 0x7f814c8cd250 runtime.restartg 0x7f814c8cd430 runtime.stopTheWorldWithSema 0x7f814c8cd6f0 runtime.startTheWorldWithSema 0x7f814c8cd970 runtime.mstart 0x7f814c8cda00 runtime.mstart1 0x7f814c8cdb10 runtime.mstartm0 0x7f814c8cdb70 runtime.mexit 0x7f814c8cdde0 runtime.forEachP 0x7f814c8ce170 runtime.runSafePointFn 0x7f814c8ce240 runtime.allocm 0x7f814c8ce4f0 runtime.needm 0x7f814c8ce680 runtime.newextram 0x7f814c8ce710 runtime.oneNewExtraM 0x7f814c8ce8c0 runtime.dropm 0x7f814c8cea00 runtime.lockextra 0x7f814c8ceac0 runtime.newm 0x7f814c8cec10 runtime.newm1 0x7f814c8ced30 runtime.templateThread 0x7f814c8cee50 runtime.stopm 0x7f814c8cefc0 runtime.mspinning 0x7f814c8cefe0 runtime.startm 0x7f814c8cf200 runtime.handoffp 0x7f814c8cf510 runtime.stoplockedm 0x7f814c8cf7d0 runtime.startlockedm 0x7f814c8cf8b0 runtime.gcstopm 0x7f814c8cf9d0 runtime.execute 0x7f814c8cfb40 runtime.findrunnable 0x7f814c8d0740 runtime.pollWork 0x7f814c8d0830 runtime.resetspinning 0x7f814c8d0910 runtime.injectglist 0x7f814c8d0aa0 runtime.schedule 0x7f814c8d0ef0 runtime.parkunlock_c 0x7f814c8d0f40 runtime.park_m 0x7f814c8d1110 runtime.goschedImpl 0x7f814c8d1380 runtime.gosched_m 0x7f814c8d1410 runtime.gopreempt_m 0x7f814c8d14a0 runtime.goexit1 0x7f814c8d1510 runtime.goexit0 0x7f814c8d17e0 runtime.save 0x7f814c8d1830 runtime.reentersyscall 0x7f814c8d1a70 runtime.entersyscall 0x7f814c8d1aa0 runtime.entersyscall_sysmon 0x7f814c8d1b20 runtime.entersyscall_gcwait 0x7f814c8d1c20 runtime.entersyscallblock 0x7f814c8d1dd0 runtime.entersyscallblock_handoff 0x7f814c8d1e60 runtime.exitsyscall 0x7f814c8d20f0 runtime.exitsyscallfast 0x7f814c8d21f0 runtime.exitsyscallfast_reacquired 0x7f814c8d2280 runtime.exitsyscallfast_pidle 0x7f814c8d2370 runtime.exitsyscall0 0x7f814c8d2560 runtime.malg 0x7f814c8d2630 runtime.newproc 0x7f814c8d26b0 runtime.newproc1 0x7f814c8d2bc0 runtime.saveAncestors 0x7f814c8d2f10 runtime.gfput 0x7f814c8d3080 runtime.gfget 0x7f814c8d3210 runtime.gfpurge 0x7f814c8d32e0 runtime.unlockOSThread 0x7f814c8d3370 runtime.badunlockosthread 0x7f814c8d33c0 runtime._System 0x7f814c8d3400 runtime._ExternalCode 0x7f814c8d3440 runtime._LostExternalCode 0x7f814c8d3480 runtime._GC 0x7f814c8d34c0 runtime._VDSO 0x7f814c8d3500 runtime.sigprof 0x7f814c8d39f0 runtime.sigprofNonGo 0x7f814c8d3a80 runtime.sigprofNonGoPC 0x7f814c8d3b00 runtime.setsSP 0x7f814c8d3b90 runtime.procresize 0x7f814c8d4600 runtime.acquirep 0x7f814c8d4660 runtime.wirep 0x7f814c8d47d0 runtime.releasep 0x7f814c8d4a00 runtime.incidlelocked 0x7f814c8d4a70 runtime.checkdead 0x7f814c8d4ef0 runtime.sysmon 0x7f814c8d5410 runtime.retake 0x7f814c8d5650 runtime.preemptall 0x7f814c8d56d0 runtime.schedtrace 0x7f814c8d61b0 runtime.schedEnableUser 0x7f814c8d6300 runtime.schedEnabled 0x7f814c8d6370 runtime.globrunqget 0x7f814c8d64c0 runtime.pidleput 0x7f814c8d6550 runtime.runqempty 0x7f814c8d6590 runtime.runqput 0x7f814c8d6690 runtime.runqputslow 0x7f814c8d68c0 runtime.runqget 0x7f814c8d6970 runtime.runqgrab 0x7f814c8d6ad0 runtime.runqsteal 0x7f814c8d6bd0 sync.runtime_canSpin 0x7f814c8d6c60 sync.runtime_doSpin 0x7f814c8d6c90 runtime.(*randomOrder).reset 0x7f814c8d6db0 runtime.gcd 0x7f814c8d6dd0 runtime.(*profBuf).takeOverflow 0x7f814c8d6e40 runtime.(*profBuf).incrementOverflow 0x7f814c8d6e90 runtime.(*profBuf).canWriteRecord 0x7f814c8d6f40 runtime.(*profBuf).canWriteTwoRecords 0x7f814c8d7040 runtime.(*profBuf).write 0x7f814c8d74c0 runtime.(*profBuf).wakeupExtra 0x7f814c8d7540 syscall.runtime_envs 0x7f814c8d7640 os.runtime_args 0x7f814c8d7740 runtime.args 0x7f814c8d77b0 runtime.goargs 0x7f814c8d7900 runtime.goenvs_unix 0x7f814c8d7a50 runtime.testAtomic64 0x7f814c8d7cb0 runtime.check 0x7f814c8d8050 runtime.parsedebugvars 0x7f814c8d83c0 runtime/debug.SetTraceback 0x7f814c8d8570 runtime.timediv 0x7f814c8d85d0 internal/reflectlite.resolveNameOff 0x7f814c8d8620 internal/reflectlite.resolveTypeOff 0x7f814c8d8680 runtime.extendRandom 0x7f814c8d8770 runtime.waitReason.String 0x7f814c8d87e0 runtime.(*rwmutex).rlock 0x7f814c8d8870 runtime.(*rwmutex).runlock 0x7f814c8d8970 internal/poll.runtime_Semacquire 0x7f814c8d89d0 sync.runtime_Semrelease 0x7f814c8d8a30 sync.runtime_SemacquireMutex 0x7f814c8d8a90 internal/poll.runtime_Semrelease 0x7f814c8d8ae0 runtime.readyWithTime 0x7f814c8d8b50 runtime.semacquire1 0x7f814c8d8e30 runtime.semrelease1 0x7f814c8d9010 runtime.cansemacquire 0x7f814c8d9040 runtime.(*semaRoot).queue 0x7f814c8d9490 runtime.(*semaRoot).dequeue 0x7f814c8d9820 runtime.(*semaRoot).rotateLeft 0x7f814c8d9a30 runtime.(*semaRoot).rotateRight 0x7f814c8d9c20 sync.runtime_notifyListCheck 0x7f814c8d9ce0 sync.runtime_nanotime 0x7f814c8d9d20 runtime.dumpregs 0x7f814c8da410 runtime.(*sigctxt).preparePanic 0x7f814c8da4e0 runtime.sighandler 0x7f814c8dac40 runtime.init.6 0x7f814c8dac50 runtime.initsig 0x7f814c8dae50 runtime.setThreadCPUProfiler 0x7f814c8daf30 runtime.sigtrampgo 0x7f814c8db3d0 runtime.sigpanic 0x7f814c8db890 runtime.dieFromSignal 0x7f814c8db930 runtime.raisebadsignal 0x7f814c8dba50 runtime.crash 0x7f814c8dba80 runtime.noSignalStack 0x7f814c8dbb10 runtime.sigNotOnStack 0x7f814c8dbba0 runtime.signalDuringFork 0x7f814c8dbc30 runtime.badsignal 0x7f814c8dbc90 runtime.sigfwdgo 0x7f814c8dbe50 runtime.unblocksig 0x7f814c8dbee0 runtime.minitSignals 0x7f814c8dbf20 runtime.minitSignalStack 0x7f814c8dc080 runtime.minitSignalMask 0x7f814c8dc190 runtime.unminitSignals 0x7f814c8dc230 runtime.signalstack 0x7f814c8dc290 runtime.sigsend 0x7f814c8dc420 runtime.makeslice 0x7f814c8dc520 runtime.growslice 0x7f814c8dcb60 runtime.stackinit 0x7f814c8dcbc0 runtime.stacklog2 0x7f814c8dcbe0 runtime.stackpoolalloc 0x7f814c8dcdc0 runtime.stackpoolfree 0x7f814c8dcf50 runtime.stackcacherefill 0x7f814c8dd040 runtime.stackcacherelease 0x7f814c8dd160 runtime.stackcache_clear 0x7f814c8dd240 runtime.stackalloc 0x7f814c8dd560 runtime.stackfree 0x7f814c8dd8b0 runtime.adjustpointers 0x7f814c8ddaf0 runtime.adjustframe 0x7f814c8ddf50 runtime.adjustctxt 0x7f814c8ddfa0 runtime.adjustdefers 0x7f814c8de070 runtime.adjustsudogs 0x7f814c8de0b0 runtime.findsghi 0x7f814c8de100 runtime.syncadjustsudogs 0x7f814c8de250 runtime.copystack 0x7f814c8de580 runtime.round2 0x7f814c8de5b0 runtime.newstack 0x7f814c8df2e0 runtime.shrinkstack 0x7f814c8df460 runtime.freeStackSpans 0x7f814c8df600 runtime.getStackMap 0x7f814c8dfeb0 runtime.morestackc 0x7f814c8dfee0 runtime.concatstrings 0x7f814c8e01c0 runtime.concatstring2 0x7f814c8e0230 runtime.concatstring3 0x7f814c8e02b0 runtime.concatstring4 0x7f814c8e0330 runtime.concatstring5 0x7f814c8e03b0 runtime.slicebytetostring 0x7f814c8e04a0 runtime.rawstringtmp 0x7f814c8e0550 runtime.rawstring 0x7f814c8e05e0 runtime.gostring 0x7f814c8e06b0 runtime.index 0x7f814c8e07b0 runtime.atoi 0x7f814c8e08e0 runtime.findnull 0x7f814c8e09a0 runtime.badsystemstack 0x7f814c8e09e0 runtime.fastrand 0x7f814c8e0a30 runtime.modulesinit 0x7f814c8e0cb0 runtime.moduledataverify 0x7f814c8e0d10 runtime.moduledataverify1 0x7f814c8e13a0 runtime.findmoduledatap 0x7f814c8e13e0 runtime.findfunc 0x7f814c8e1560 runtime.pcvalue 0x7f814c8e1b20 runtime.funcname 0x7f814c8e1bd0 runtime.funcnameFromNameoff 0x7f814c8e1c80 runtime.funcline1 0x7f814c8e1e50 runtime.funcline 0x7f814c8e1ed0 runtime.funcspdelta 0x7f814c8e2040 runtime.pcdatavalue 0x7f814c8e20f0 runtime.funcdata 0x7f814c8e21d0 runtime.step 0x7f814c8e2380 runtime.readvarint 0x7f814c8e23e0 runtime.deltimer 0x7f814c8e24b0 runtime.(*timersBucket).deltimerLocked 0x7f814c8e2650 runtime.timejump 0x7f814c8e2720 runtime.timejumpLocked 0x7f814c8e27c0 runtime.siftupTimer 0x7f814c8e28e0 runtime.siftdownTimer 0x7f814c8e2ab0 time.now 0x7f814c8e2b20 runtime.traceReader 0x7f814c8e2c20 runtime.traceProcFree 0x7f814c8e2ce0 runtime.traceEvent 0x7f814c8e2de0 runtime.traceEventLocked 0x7f814c8e3090 runtime.traceStackID 0x7f814c8e31d0 runtime.traceAcquireBuffer 0x7f814c8e3290 runtime.traceReleaseBuffer 0x7f814c8e3310 runtime.traceFlush 0x7f814c8e3500 runtime.(*traceBuf).varint 0x7f814c8e3580 runtime.(*traceStackTable).put 0x7f814c8e3780 runtime.(*traceStackTable).find 0x7f814c8e3830 runtime.(*traceStackTable).newStack 0x7f814c8e3890 runtime.(*traceAlloc).alloc 0x7f814c8e39b0 runtime.traceProcStart 0x7f814c8e3a40 runtime.traceProcStop 0x7f814c8e3b20 runtime.traceGCSweepStart 0x7f814c8e3ba0 runtime.traceGCSweepSpan 0x7f814c8e3c40 runtime.traceGCSweepDone 0x7f814c8e3d20 runtime.traceGoCreate 0x7f814c8e3e20 runtime.traceGoStart 0x7f814c8e3fc0 runtime.traceGoPark 0x7f814c8e4050 runtime.traceGoUnpark 0x7f814c8e4150 runtime.traceGoSysCall 0x7f814c8e41b0 runtime.traceGoSysExit 0x7f814c8e4290 runtime.traceGoSysBlock 0x7f814c8e4370 runtime.traceNextGC 0x7f814c8e4430 runtime.tracebackdefers 0x7f814c8e4660 runtime.gentraceback 0x7f814c8e63a0 runtime.getArgInfo 0x7f814c8e6570 runtime.tracebackCgoContext 0x7f814c8e67e0 runtime.printcreatedby 0x7f814c8e68b0 runtime.printcreatedby1 0x7f814c8e6a70 runtime.traceback 0x7f814c8e6ae0 runtime.tracebacktrap 0x7f814c8e6ba0 runtime.traceback1 0x7f814c8e6ec0 runtime.printAncestorTraceback 0x7f814c8e70c0 runtime.printAncestorTracebackFuncInfo 0x7f814c8e7400 runtime.callers 0x7f814c8e74e0 runtime.gcallers 0x7f814c8e7590 runtime.showframe 0x7f814c8e7660 runtime.showfuncinfo 0x7f814c8e7870 runtime.goroutineheader 0x7f814c8e7b50 runtime.tracebackothers 0x7f814c8e7e00 runtime.tracebackHexdump 0x7f814c8e7ff0 runtime.isSystemGoroutine 0x7f814c8e8100 runtime.printCgoTraceback 0x7f814c8e8240 runtime.printOneCgoTraceback 0x7f814c8e8480 runtime.callCgoSymbolizer 0x7f814c8e8500 runtime.cgoContextPCs 0x7f814c8e85e0 runtime.(*_type).string 0x7f814c8e86a0 runtime.(*_type).uncommon 0x7f814c8e8750 runtime.(*_type).pkgpath 0x7f814c8e8860 runtime.resolveNameOff 0x7f814c8e8b60 runtime.resolveTypeOff 0x7f814c8e8eb0 runtime.(*_type).textOff 0x7f814c8e91f0 runtime.name.tagLen 0x7f814c8e9250 runtime.name.name 0x7f814c8e92a0 runtime.name.tag 0x7f814c8e9330 runtime.name.pkgPath 0x7f814c8e9430 runtime.typelinksinit 0x7f814c8e9ad0 runtime.typesEqual 0x7f814c8eaa10 runtime.vdsoInitFromSysinfoEhdr 0x7f814c8eae60 runtime.vdsoFindVersion 0x7f814c8eafa0 runtime.vdsoParseSymbols 0x7f814c8eb330 runtime.vdsoauxv 0x7f814c8eb410 runtime.inVDSOPage 0x7f814c8eb480 runtime.mmap.func1 0x7f814c8eb500 runtime.munmap.func1 0x7f814c8eb550 runtime.sigaction.func1 0x7f814c8eb5c0 runtime.cgocallbackg1.func1 0x7f814c8eb5d0 runtime.cgoCheckWriteBarrier.func1 0x7f814c8eb690 runtime.cgoCheckTypedBlock.func1 0x7f814c8eb6f0 runtime.chansend.func1 0x7f814c8eb740 runtime.chanrecv.func1 0x7f814c8eb790 runtime.debugCallCheck.func1 0x7f814c8eb990 runtime.debugCallWrap.func1 0x7f814c8eba00 runtime.mallocgc.func1 0x7f814c8eba70 runtime.persistentalloc.func1 0x7f814c8ebae0 runtime.freemcache.func1 0x7f814c8ebba0 runtime.SetFinalizer.func1 0x7f814c8ebbf0 runtime.SetFinalizer.func2 0x7f814c8ebc80 runtime.gcStart.func1 0x7f814c8ebcc0 runtime.gcStart.func2 0x7f814c8ebd30 runtime.gcMarkDone.func1.1 0x7f814c8ebdb0 runtime.gcMarkDone.func1 0x7f814c8ebe50 runtime.gcMarkDone.func2 0x7f814c8ebf30 runtime.gcMarkDone.func3 0x7f814c8ebf80 runtime.gcMarkTermination.func1 0x7f814c8ebfc0 runtime.gcMarkTermination.func2 0x7f814c8ec0e0 runtime.gcMarkTermination.func3 0x7f814c8ec120 runtime.gcMarkTermination.func4.1 0x7f814c8ec170 runtime.gcMarkTermination.func4 0x7f814c8ec1b0 runtime.gcBgMarkWorker.func1 0x7f814c8ec230 runtime.gcBgMarkWorker.func2 0x7f814c8ec3e0 runtime.markroot.func1 0x7f814c8ec4d0 runtime.gcAssistAlloc.func1 0x7f814c8ec520 runtime.scanstack.func1 0x7f814c8ec580 runtime.getempty.func1 0x7f814c8ec5f0 runtime.freeSomeWbufs.func1 0x7f814c8ec6e0 runtime.(*mheap).coalesce.func1 0x7f814c8ec900 runtime.(*mheap).coalesce.func2 0x7f814c8ecae0 runtime.(*mheap).alloc.func1 0x7f814c8ecb50 runtime.(*mheap).freeSpan.func1 0x7f814c8ecc80 runtime.mProf_Malloc.func1 0x7f814c8eccd0 runtime.tracealloc.func1 0x7f814c8ecd30 runtime.tracefree.func1 0x7f814c8ecd90 runtime.wbBufFlush.func1 0x7f814c8ecde0 runtime.futexwakeup.func1 0x7f814c8ece90 runtime.sysSigaction.func1 0x7f814c8ecee0 runtime.newdefer.func1 0x7f814c8ed0b0 runtime.newdefer.func2 0x7f814c8ed200 runtime.freedefer.func1 0x7f814c8ed380 runtime.preprintpanics.func1 0x7f814c8ed3f0 runtime.throw.func1 0x7f814c8ed480 runtime.fatalthrow.func1 0x7f814c8ed510 runtime.fatalpanic.func1 0x7f814c8ed5e0 runtime.fatalpanic.func2 0x7f814c8ed620 runtime.hexdumpWords.func1 0x7f814c8ed6c0 runtime.main.func1 0x7f814c8ed710 runtime.main.func2 0x7f814c8ed750 runtime.goready.func1 0x7f814c8ed7a0 runtime.casgstatus.func1 0x7f814c8ed860 runtime.casgstatus.func2 0x7f814c8ed960 runtime.reentersyscall.func1 0x7f814c8eda60 runtime.entersyscallblock.func1 0x7f814c8edba0 runtime.entersyscallblock.func2 0x7f814c8edce0 runtime.exitsyscallfast.func1 0x7f814c8edd90 runtime.exitsyscallfast_reacquired.func1 0x7f814c8eddf0 runtime.malg.func1 0x7f814c8ede50 runtime.newproc.func1 0x7f814c8edec0 runtime.gfget.func1 0x7f814c8edf20 runtime.(*rwmutex).rlock.func1 0x7f814c8edfe0 runtime.callers.func1 0x7f814c8ee090 runtime.tracebackHexdump.func1 0x7f814c8ee0d0 runtime.vdsoParseSymbols.func1 0x7f814c8ee270 runtime.init.ializers 0x7f814c8ee3f0 runtime.init 0x7f814c8ee480 runtime.skipPleaseUseCallersFrames 0x7f814c8ee580 _rt0_amd64_lib 0x7f814c8ee630 _rt0_amd64_lib_go 0x7f814c8ee650 runtime.rt0_go 0x7f814c8ee7a0 runtime.asminit 0x7f814c8ee7b0 runtime.gogo 0x7f814c8ee810 runtime.mcall 0x7f814c8ee870 runtime.systemstack_switch 0x7f814c8ee880 runtime.systemstack 0x7f814c8ee920 runtime.morestack 0x7f814c8ee9b0 runtime.morestack_noctxt 0x7f814c8ee9c0 runtime.reflectcall 0x7f814c8eeba0 callRet 0x7f814c8eebd0 runtime.call32 0x7f814c8eec50 runtime.call64 0x7f814c8eecd0 runtime.call128 0x7f814c8eed80 runtime.call256 0x7f814c8eee40 runtime.call512 0x7f814c8eef00 runtime.call1024 0x7f814c8eefc0 runtime.call2048 0x7f814c8ef080 runtime.call4096 0x7f814c8ef150 runtime.call8192 0x7f814c8ef220 runtime.call16384 0x7f814c8ef2f0 runtime.call32768 0x7f814c8ef3c0 runtime.call65536 0x7f814c8ef490 runtime.call131072 0x7f814c8ef560 runtime.call262144 0x7f814c8ef630 runtime.call524288 0x7f814c8ef700 runtime.call1048576 0x7f814c8ef7d0 runtime.call2097152 0x7f814c8ef8a0 runtime.call4194304 0x7f814c8ef970 runtime.call8388608 0x7f814c8efa40 runtime.call16777216 0x7f814c8efb10 runtime.call33554432 0x7f814c8efbe0 runtime.call67108864 0x7f814c8efcb0 runtime.call134217728 0x7f814c8efd80 runtime.call268435456 0x7f814c8efe50 runtime.call536870912 0x7f814c8eff20 runtime.call1073741824 0x7f814c8efff0 runtime.procyield 0x7f814c8f0000 runtime.publicationBarrier 0x7f814c8f0010 runtime.jmpdefer 0x7f814c8f0030 gosave 0x7f814c8f0070 runtime.asmcgocall 0x7f814c8f0130 runtime.cgocallback 0x7f814c8f0180 runtime.cgocallback_gofunc 0x7f814c8f0280 runtime.setg 0x7f814c8f02a0 setg_gcc 0x7f814c8f02b0 runtime.abort 0x7f814c8f02c0 runtime.stackcheck 0x7f814c8f02f0 runtime.cputicks 0x7f814c8f0310 runtime.aeshash 0x7f814c8f0330 runtime.aeshashstr 0x7f814c8f0350 runtime.aeshashbody 0x7f814c8f08a0 runtime.aeshash32 0x7f814c8f08e0 runtime.aeshash64 0x7f814c8f0920 runtime.checkASM 0x7f814c8f0940 runtime.return0 0x7f814ca73920 _cgo_topofstack fatal error: invalid runtime symbol table runtime: panic before malloc heap initialized

runtime stack:
runtime.throw(0x7f814c90774d, 0x1c)
/home/u/go1.13/src/runtime/panic.go:627 +0x74 fp=0x7f814c61fd50 sp=0x7f814c61fd20 pc=0x7f814c8c94b4
runtime.moduledataverify1(0x7f814c9a9360)
/home/u/go1.13/src/runtime/symtab.go:448 +0x5a0 fp=0x7f814c61fe58 sp=0x7f814c61fd50 pc=0x7f814c8e12b0
runtime.moduledataverify()
/home/u/go1.13/src/runtime/symtab.go:416 +0x36 fp=0x7f814c61fe78 sp=0x7f814c61fe58 pc=0x7f814c8e0ce6
runtime.schedinit()
/home/u/go1.13/src/runtime/proc.go:537 +0x5c fp=0x7f814c61fed0 sp=0x7f814c61fe78 pc=0x7f814c8cbf1c
runtime.rt0_go(0x7ffe2435c0d8, 0x1, 0x7ffe2435c0d8, 0x7f814c620700, 0x7f814c62aa9d, 0x0, 0x7f814c620700, 0x7f814c620700, 0x9d89969ea2fc8472, 0x7ffe2435beee, ...)
/home/u/go1.13/src/runtime/asm_amd64.s:195 +0x11e fp=0x7f814c61fed8 sp=0x7f814c61fed0 pc=0x7f814c8ee76e

ianlancetaylor · 2019-03-18T00:20:56Z

With the current implementation it's not going to work to use -buildmode=c-archive multiple times and put the results into multiple shared libraries. As you've discovered. The essential problem is that there has to be only one Go runtime, but you have multiple runtimes. When using -buildmode=c-archive there's no way to isolate the different runtimes. I don't see any simple way to make this work.

mewmew · 2019-03-18T02:19:37Z

The essential problem is that there has to be only one Go runtime, but you have multiple runtimes. When using -buildmode=c-archive there's no way to isolate the different runtimes. I don't see any simple way to make this work.

Hi @ianlancetaylor. Thanks for the input. One question, when we generate multiple shared libraries using -buildmode=c-shared, does each one contain the Go runtime? If so, how do shared libraries created with -buildmode=c-archive differ from those produced with -buildmode=c-shared?

I would assume there are cinit functions run in c-shared that are not run in c-archive. If so, are there any theoretical restrictions that make it impossible to implement support for the same initialization in shared libraries produced with c-archived as those produced with c-shared?

If there is a design document, or specific parts of the runtime source code I could attend to to gain further insight into this, I'd be very curious to learn more.

Kindly,
Robin

ianlancetaylor · 2019-03-19T04:14:32Z

The -buildmode=c-shared libraries differ in that they are built with -Bsymbolic which forces all their local references to be local. The effect is that we have multiple Go runtimes, but they don't refer to each other so there is no confusion.

I guess you could do the same with your shared libraries, if your C code doesn't mind being linked with -Bsymbolic.

There is a design doc but I'm not sure it helps with this level of detail: https://golang.org/s/execmodes.

@ianlancetaylor

Thank you @ianlancetaylor for the suggestion and insight! This resolved the issue outlined in golang/go#30822.

mewmew · 2019-03-19T06:23:50Z

Thank you so much @ianlancetaylor! Your insight is invaluable.

Indeed, adding -Wl,-Bsymbolic to the build of each shared library that included Go code in c-archive form resolved the issue.

Thanks once more :)

Cheerful regards,
Robin

katiehockman added the Question label Mar 19, 2019

katiehockman added this to the Unplanned milestone Mar 19, 2019

mewmew added a commit to mewspring/go-cgo-issue that referenced this issue Mar 19, 2019

add -Wl,-Bsymbolic to link c-archive in the same way as c-shared

d50fc66

Thank you @ianlancetaylor for the suggestion and insight! This resolved the issue outlined in golang/go#30822.

mewmew closed this as completed Mar 19, 2019

ncordon mentioned this issue Aug 20, 2019

Solves linking error bblfsh/scala-client#120

Merged

golang locked and limited conversation to collaborators Mar 18, 2020

gopherbot added the FrozenDueToAge label Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

mewmew commented Mar 14, 2019 •

edited

Loading

mewmew commented Mar 16, 2019

mewmew commented Mar 16, 2019

mewmew commented Mar 16, 2019

ianlancetaylor commented Mar 18, 2019

mewmew commented Mar 18, 2019

ianlancetaylor commented Mar 19, 2019

mewmew commented Mar 19, 2019

cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822

Comments

mewmew commented Mar 14, 2019 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

mewmew commented Mar 16, 2019

mewmew commented Mar 16, 2019

mewmew commented Mar 16, 2019

ianlancetaylor commented Mar 18, 2019

mewmew commented Mar 18, 2019

ianlancetaylor commented Mar 19, 2019

mewmew commented Mar 19, 2019

mewmew commented Mar 14, 2019 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?