Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: unexpected return pc for runtime.gcBgMarkWorker called from 0x0 #57550

Closed
xingdl2007 opened this issue Jan 3, 2023 · 4 comments
Closed

runtime: unexpected return pc for runtime.gcBgMarkWorker called from 0x0 #57550

xingdl2007 opened this issue Jan 3, 2023 · 4 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@xingdl2007
Copy link

xingdl2007 commented Jan 3, 2023 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GOOS="linux"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1904677838=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have a C++ program access etcd through its official go client built with -asan and -buildmode=c-shared.
The program runs with GOTRACEBACK=crash GODEBUG=gctrace=1, it crashs frequently.

What did you expect to see?

Not crash.

What did you see instead?

gc 58748 @2292.843s 0%: 0.058+0.62+0.034 ms clock, 3.7+0.13/6.6/1.8+2.2 ms cpu, 6->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58749 @2292.889s 0%: 0.055+0.84+0.039 ms clock, 3.5+0.070/7.7/1.2+2.5 ms cpu, 6->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58750 @2292.922s 0%: 0.087+0.62+0.049 ms clock, 5.6+0.21/6.4/0.96+3.1 ms cpu, 5->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58751 @2292.978s 0%: 0.069+0.68+0.026 ms clock, 4.4+0.21/6.9/1.4+1.7 ms cpu, 6->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58752 @2293.023s 0%: 0.075+0.65+0.056 ms clock, 4.8+0.13/6.6/0.48+3.5 ms cpu, 6->6->3 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58753 @2293.066s 0%: 0.070+0.70+0.080 ms clock, 4.4+0.16/6.0/0.21+5.1 ms cpu, 5->6->3 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58754 @2293.113s 0%: 0.068+0.86+0.061 ms clock, 4.3+0.12/7.4/1.7+3.9 ms cpu, 6->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58755 @2293.163s 0%: 0.052+0.66+0.048 ms clock, 3.3+0.26/7.4/2.6+3.1 ms cpu, 5->6->2 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
gc 58756 @2293.194s 0%: 0.045+0.88+0.074 ms clock, 2.8+0.11/8.2/3.1+4.7 ms cpu, 5->6->3 MB, 6 MB goal, 1 MB stacks, 0 MB globals, 64 P
runtime: g 90: unexpected return pc for runtime.gcBgMarkWorker called from 0x0
stack: frame={sp:0x10c000589f50, fp:0x10c000589fe0} stack=[0x10c000589800,0x10c00058a000)
0x000010c000589e50: 0x0000000000000000 0x0000000000000000
0x000010c000589e60: 0x0000000000000000 0x0000000000000000
0x000010c000589e70: 0x0000000000000000 0x0000000000000000
0x000010c000589e80: 0x0000000000000000 0x0000000000000000
0x000010c000589e90: 0x0000000000000000 0x0000000000000000
0x000010c000589ea0: 0x0000000000000000 0x0000000000000000
0x000010c000589eb0: 0x0000000000000000 0x0000000000000000
0x000010c000589ec0: 0x0000000000000000 0x0000000000000000
0x000010c000589ed0: 0x0000000000000000 0x0000000000000000
0x000010c000589ee0: 0x0000000000000000 0x0000000000000000
0x000010c000589ef0: 0x0000000000000000 0x0000000000000000
0x000010c000589f00: 0x0000000000000000 0x000010c000589f30
0x000010c000589f10: 0x000010c000589f30 0x0000000000000032
0x000010c000589f20: 0x000010c000589f40 0x00007f344eaf5eb6 <runtime.gopark+0x00000000000000d6>
0x000010c000589f30: 0x0000000000000000 0x0000000000000000
0x000010c000589f40: 0x000010c000589fd0 0x00007f344ead71f1 <runtime.gcBgMarkWorker+0x00000000000000f1>
0x000010c000589f50: <0x00ed467121182bfc 0x0000000000000003
0x000010c000589f60: 0x0000000000026b76 0x0000000000000000
0x000010c000589f70: 0x0000000000000000 0x0000000000000000
0x000010c000589f80: 0x00ed467121182bfc 0x00ed46712117a9c0
0x000010c000589f90: 0x000000000000823c 0x0000000000000000
0x000010c000589fa0: 0x000010c0005804b0 0x000010c00058eb40
0x000010c000589fb0: 0x000010c000078500 0x00007f344ead75c0 <runtime.gcBgMarkWorker.func2+0x0000000000000000>
0x000010c000589fc0: 0x0000000000000000 0x0000000000000000
0x000010c000589fd0: 0x0000000000000000 !0x0000000000000000
0x000010c000589fe0: >0x0000000000000000 0x0000000000000000
0x000010c000589ff0: 0x0000000000000000 0x0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x7f344f692bb0?, 0x7f3450022a00?})
/usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x7f341db85a78 sp=0x7f341db85a48 pc=0x7f344eaf329f
runtime.gentraceback(0x20?, 0x1?, 0x0?, 0x7f341db85e68?, 0x0, 0x0, 0x7fffffff, 0x7f341db85fc8, 0x0?, 0x0)
/usr/local/go/src/runtime/traceback.go:269 +0x1cf7 fp=0x7f341db85de8 sp=0x7f341db85a78 pc=0x7f344eb1a8d7
runtime.scanstack(0x10c00058eb40, 0x10c000051c40)
/usr/local/go/src/runtime/mgcmark.go:804 +0x1da fp=0x7f341db85ff0 sp=0x7f341db85de8 pc=0x7f344eada59a
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:240 +0xc5 fp=0x7f341db86040 sp=0x7f341db85ff0 pc=0x7f344ead93c5
runtime.markroot(0x10c000051c40, 0x5a, 0x1)
/usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f341db860e0 sp=0x7f341db86040 pc=0x7f344ead9065
runtime.gcDrain(0x10c000051c40, 0x3)
/usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f341db86140 sp=0x7f341db860e0 pc=0x7f344eadb13f
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1308 +0xa5 fp=0x7f341db86190 sp=0x7f341db86140 pc=0x7f344ead7665
runtime.systemstack()
/usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x7f341db86198 sp=0x7f341db86190 pc=0x7f344eb24c06

goroutine 78 [GC worker (idle)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:459 fp=0x10c00058bf50 sp=0x10c00058bf48 pc=0x7f344eb24ba0
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1295 +0x209 fp=0x10c00058bfe0 sp=0x10c00058bf50 pc=0x7f344ead7309
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058bfe8 sp=0x10c00058bfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 4 [IO wait]:
runtime.gopark(0x7f344eabe350?, 0x7f345000fdf0?, 0x8?, 0x0?, 0x10c0000f8b18?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000f8ae0 sp=0x10c0000f8ac0 pc=0x7f344eaf5eb6
runtime.netpollblock(0x10c0000f8b38?, 0x4ec31aa5?, 0x34?)
/usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0x10c0000f8b18 sp=0x10c0000f8ae0 pc=0x7f344eaee597
internal/poll.runtime_pollWait(0x7f34486d3a98, 0x72)
/usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0x10c0000f8b38 sp=0x10c0000f8b18 pc=0x7f344eb21509
internal/poll.(*pollDesc).wait(0x10c00049e0d8, 0x7f344eac8b1b?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xb7 fp=0x10c0000f8b70 sp=0x10c0000f8b38 pc=0x7f344ec31ab7
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x10c00049e0c0)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x367 fp=0x10c0000f8c20 sp=0x10c0000f8b70 pc=0x7f344ec38a87
net.(*netFD).accept(0x10c00049e0c0)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0x10c0000f8d10 sp=0x10c0000f8c20 pc=0x7f344edc49d5
net.(*TCPListener).accept(0x10c0005a0000)
/usr/local/go/src/net/tcpsock_posix.go:142 +0x36 fp=0x10c0000f8d50 sp=0x10c0000f8d10 pc=0x7f344edeb676
net.(*TCPListener).Accept(0x10c0005a0000)
/usr/local/go/src/net/tcpsock.go:288 +0x58 fp=0x10c0000f8db0 sp=0x10c0000f8d50 pc=0x7f344ede9bd8
net/http.(*onceCloseListener).Accept(0x10c000598080)
:1 +0x42 fp=0x10c0000f8dd0 sp=0x10c0000f8db0 pc=0x7f344efffc22
net/http.(*Server).Serve(0x10c0002ea000, {0x7f344fb83b60, 0x10c0005a0000})
/usr/local/go/src/net/http/server.go:3070 +0x593 fp=0x10c0000f8f58 sp=0x10c0000f8dd0 pc=0x7f344efa7213
net/http.(*Server).ListenAndServe(0x10c0002ea000)
/usr/local/go/src/net/http/server.go:2999 +0x8f fp=0x10c0000f8f88 sp=0x10c0000f8f58 pc=0x7f344efa6c2f
net/http.ListenAndServe(...)
/usr/local/go/src/net/http/server.go:3255
main.NewEtcdClientV3.func1()
/home/xingdongliang/data/ktable2/thirdparty/etcdclient/etcdclient.go:225 +0x10a fp=0x10c0000f8fe0 sp=0x10c0000f8f88 pc=0x7f344f67566a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000f8fe8 sp=0x10c0000f8fe0 pc=0x7f344eb26e41
created by main.NewEtcdClientV3
/home/xingdongliang/data/ktable2/thirdparty/etcdclient/etcdclient.go:224 +0x4a

goroutine 17 [select, 38 minutes, locked to thread]:
runtime.gopark(0x10c0005ffdc0?, 0x2?, 0xaf?, 0x88?, 0x10c0005ffd84?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000fdbf8 sp=0x10c0000fdbd8 pc=0x7f344eaf5eb6
runtime.selectgo(0x10c0000fddc0, 0x10c0005ffd80, 0x7f344fb84608?, 0x0, 0x10c0002c4020?, 0x1)
/usr/local/go/src/runtime/select.go:328 +0x7bc fp=0x10c0000fdd40 sp=0x10c0000fdbf8 pc=0x7f344eb05cdc
main.EtcdLeaderObserve(0x616000000000?, 0x10c0005ffe20?, 0x4eac1578?)
/home/xingdongliang/data/ktable2/thirdparty/etcdclient/etcdclient.go:642 +0x146 fp=0x10c0000fddf0 sp=0x10c0000fdd40 pc=0x7f344f67ada6
_cgoexp_0430734a83f2_EtcdLeaderObserve(0x7f33bb2d29d0)
_cgo_gotypes.go:393 +0x7f fp=0x10c0000fde30 sp=0x10c0000fddf0 pc=0x7f344f67e5df
runtime.cgocallbackg1(0x7f344f67e560, 0x10c0005fffe0?, 0x0)
/usr/local/go/src/runtime/cgocall.go:316 +0x2c2 fp=0x10c0000fdf00 sp=0x10c0000fde30 pc=0x7f344eabef02
runtime.cgocallbackg(0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/cgocall.go:235 +0x109 fp=0x10c0000fdf90 sp=0x10c0000fdf00 pc=0x7f344eabeb89
runtime.cgocallbackg(0x7f344f67e560, 0x7f33bb2d29d0, 0x0)
:1 +0x31 fp=0x10c0000fdfb8 sp=0x10c0000fdf90 pc=0x7f344eb29211
runtime.cgocallback(0x0, 0x0, 0x0)
/usr/local/go/src/runtime/asm_amd64.s:994 +0xb3 fp=0x10c0000fdfe0 sp=0x10c0000fdfb8 pc=0x7f344eb26bf3
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000fdfe8 sp=0x10c0000fdfe0 pc=0x7f344eb26e41

goroutine 2 [force gc (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000ecfb0 sp=0x10c0000ecf90 pc=0x7f344eaf5eb6
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:302 +0xad fp=0x10c0000ecfe0 sp=0x10c0000ecfb0 pc=0x7f344eaf5d4d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000ecfe8 sp=0x10c0000ecfe0 pc=0x7f344eb26e41
created by runtime.init.6
/usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 18 [GC sweep wait]:
runtime.gopark(0x7f3450079601?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000e8790 sp=0x10c0000e8770 pc=0x7f344eaf5eb6
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0x10c0000e87c8 sp=0x10c0000e8790 pc=0x7f344eae0437
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0x10c0000e87e0 sp=0x10c0000e87c8 pc=0x7f344ead5086
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000e87e8 sp=0x10c0000e87e0 pc=0x7f344eb26e41
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 19 [GC scavenge wait]:
runtime.gopark(0xed4671214fded2?, 0x940e2?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000e8f70 sp=0x10c0000e8f50 pc=0x7f344eaf5eb6
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x7f345008b1a0)
/usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0x10c0000e8fa0 sp=0x10c0000e8f70 pc=0x7f344eade493
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0x10c0000e8fc8 sp=0x10c0000e8fa0 pc=0x7f344eadea85
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0x10c0000e8fe0 sp=0x10c0000e8fc8 pc=0x7f344ead5026
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000e8fe8 sp=0x10c0000e8fe0 pc=0x7f344eb26e41
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 3 [finalizer wait, 38 minutes]:
runtime.gopark(0x7f345008b9e0?, 0x10c000009c20?, 0x0?, 0x0?, 0x10c0000ec770?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000ec628 sp=0x10c0000ec608 pc=0x7f344eaf5eb6
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0x10c0000ec7e0 sp=0x10c0000ec628 pc=0x7f344ead418f
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000ec7e8 sp=0x10c0000ec7e0 pc=0x7f344eb26e41
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 34 [GC assist wait, locked to thread]:
runtime.gopark(0x0?, 0x243000?, 0xc0?, 0x3c?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000c83b78 sp=0x10c000c83b58 pc=0x7f344eaf5eb6
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:369
runtime.gcParkAssist()
/usr/local/go/src/runtime/mgcmark.go:651 +0xeb fp=0x10c000c83bb0 sp=0x10c000c83b78 pc=0x7f344eada1cb
runtime.gcAssistAlloc(0x10c0003545a0)
/usr/local/go/src/runtime/mgcmark.go:507 +0x23e fp=0x10c000c83c10 sp=0x10c000c83bb0 pc=0x7f344ead9bde
runtime.mallocgc(0x22, 0x0, 0x0)
/usr/local/go/src/runtime/malloc.go:906 +0x1bc fp=0x10c000c83ca0 sp=0x10c000c83c10 pc=0x7f344eac84bc
runtime.rawstring(0x22)
/usr/local/go/src/runtime/string.go:273 +0x37 fp=0x10c000c83cf0 sp=0x10c000c83ca0 pc=0x7f344eb0f3b7
runtime.gostringn(0x10c000c83d50?, 0x22)
/usr/local/go/src/runtime/string.go:344 +0x2b fp=0x10c000c83d30 sp=0x10c000c83cf0 pc=0x7f344eb0f7cb
main._Cfunc_GoStringN(...)
_cgo_gotypes.go:128
main.EtcdClientGetPrefix(0x3e8, 0x7f344f67ea4d?, 0xc83e20?)
/home/xingdongliang/data/ktable2/thirdparty/etcdclient/etcdclient.go:360 +0x6a fp=0x10c000c83de8 sp=0x10c000c83d30 pc=0x7f344f67706a
_cgoexp_0430734a83f2_EtcdClientGetPrefix(0x7f3379303e10)
_cgo_gotypes.go:263 +0x7d fp=0x10c000c83e30 sp=0x10c000c83de8 pc=0x7f344f67d97d
runtime.cgocallbackg1(0x7f344f67d900, 0x10c000c83fe0?, 0x0)
/usr/local/go/src/runtime/cgocall.go:316 +0x2c2 fp=0x10c000c83f00 sp=0x10c000c83e30 pc=0x7f344eabef02
runtime.cgocallbackg(0x0?, 0x0?, 0x101000000000000?)
/usr/local/go/src/runtime/cgocall.go:235 +0x109 fp=0x10c000c83f90 sp=0x10c000c83f00 pc=0x7f344eabeb89
runtime.cgocallbackg(0x7f344f67d900, 0x7f3379303e10, 0x0)
:1 +0x31 fp=0x10c000c83fb8 sp=0x10c000c83f90 pc=0x7f344eb29211
runtime.cgocallback(0x0, 0x0, 0x0)
/usr/local/go/src/runtime/asm_amd64.s:994 +0xb3 fp=0x10c000c83fe0 sp=0x10c000c83fb8 pc=0x7f344eb26bf3
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000c83fe8 sp=0x10c000c83fe0 pc=0x7f344eb26e41

goroutine 37 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b6750 sp=0x10c0004b6730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b67e0 sp=0x10c0004b6750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b67e8 sp=0x10c0004b67e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 38 [GC worker (idle), 38 minutes]:
runtime.gopark(0x9?, 0x10c0002f4200?, 0xf0?, 0x83?, 0x2?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000ed750 sp=0x10c0000ed730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000ed7e0 sp=0x10c0000ed750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000ed7e8 sp=0x10c0000ed7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 39 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b6f50 sp=0x10c0004b6f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b6fe0 sp=0x10c0004b6f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b6fe8 sp=0x10c0004b6fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 40 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b7750 sp=0x10c0004b7730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b77e0 sp=0x10c0004b7750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b77e8 sp=0x10c0004b77e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 41 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b7f50 sp=0x10c0004b7f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b7fe0 sp=0x10c0004b7f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b7fe8 sp=0x10c0004b7fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 42 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b8750 sp=0x10c0004b8730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b87e0 sp=0x10c0004b8750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b87e8 sp=0x10c0004b87e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 43 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b8f50 sp=0x10c0004b8f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b8fe0 sp=0x10c0004b8f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b8fe8 sp=0x10c0004b8fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 5 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000edf50 sp=0x10c0000edf30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000edfe0 sp=0x10c0000edf50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000edfe8 sp=0x10c0000edfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 6 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000ee750 sp=0x10c0000ee730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000ee7e0 sp=0x10c0000ee750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000ee7e8 sp=0x10c0000ee7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 44 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b9750 sp=0x10c0004b9730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b97e0 sp=0x10c0004b9750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b97e8 sp=0x10c0004b97e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 45 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b9f50 sp=0x10c0004b9f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b9fe0 sp=0x10c0004b9f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b9fe8 sp=0x10c0004b9fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 46 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b2750 sp=0x10c0004b2730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b27e0 sp=0x10c0004b2750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b27e8 sp=0x10c0004b27e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 47 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b2f50 sp=0x10c0004b2f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b2fe0 sp=0x10c0004b2f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b2fe8 sp=0x10c0004b2fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 48 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b3750 sp=0x10c0004b3730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b37e0 sp=0x10c0004b3750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b37e8 sp=0x10c0004b37e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 49 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b3f50 sp=0x10c0004b3f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b3fe0 sp=0x10c0004b3f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b3fe8 sp=0x10c0004b3fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 50 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b4750 sp=0x10c0004b4730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b47e0 sp=0x10c0004b4750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b47e8 sp=0x10c0004b47e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 51 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b4f50 sp=0x10c0004b4f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b4fe0 sp=0x10c0004b4f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b4fe8 sp=0x10c0004b4fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 52 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b5750 sp=0x10c0004b5730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b57e0 sp=0x10c0004b5750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b57e8 sp=0x10c0004b57e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 53 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004b5f50 sp=0x10c0004b5f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004b5fe0 sp=0x10c0004b5f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004b5fe8 sp=0x10c0004b5fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 54 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004c0750 sp=0x10c0004c0730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004c07e0 sp=0x10c0004c0750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004c07e8 sp=0x10c0004c07e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 55 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004c0f50 sp=0x10c0004c0f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004c0fe0 sp=0x10c0004c0f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004c0fe8 sp=0x10c0004c0fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000e9750 sp=0x10c0000e9730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000e97e0 sp=0x10c0000e9750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000e97e8 sp=0x10c0000e97e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 7 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000eef50 sp=0x10c0000eef30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000eefe0 sp=0x10c0000eef50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000eefe8 sp=0x10c0000eefe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 21 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000e9f50 sp=0x10c0000e9f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000e9fe0 sp=0x10c0000e9f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000e9fe8 sp=0x10c0000e9fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 67 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bc750 sp=0x10c0004bc730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bc7e0 sp=0x10c0004bc750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bc7e8 sp=0x10c0004bc7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 68 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bcf50 sp=0x10c0004bcf30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bcfe0 sp=0x10c0004bcf50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bcfe8 sp=0x10c0004bcfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 69 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bd750 sp=0x10c0004bd730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bd7e0 sp=0x10c0004bd750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bd7e8 sp=0x10c0004bd7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 70 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bdf50 sp=0x10c0004bdf30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bdfe0 sp=0x10c0004bdf50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bdfe8 sp=0x10c0004bdfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000ef750 sp=0x10c0000ef730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000ef7e0 sp=0x10c0000ef750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000ef7e8 sp=0x10c0000ef7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 9 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0000eff50 sp=0x10c0000eff30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0000effe0 sp=0x10c0000eff50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0000effe8 sp=0x10c0000effe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 71 [GC worker (idle), 38 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004be750 sp=0x10c0004be730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004be7e0 sp=0x10c0004be750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004be7e8 sp=0x10c0004be7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 72 [GC worker (idle)]:
runtime.gopark(0xed467123699d85?, 0x1?, 0x90?, 0x1a?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bef50 sp=0x10c0004bef30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004befe0 sp=0x10c0004bef50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004befe8 sp=0x10c0004befe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 73 [GC worker (idle)]:
runtime.gopark(0xed4671211799e0?, 0x1?, 0x53?, 0x4b?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bf750 sp=0x10c0004bf730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bf7e0 sp=0x10c0004bf750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bf7e8 sp=0x10c0004bf7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 74 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x1?, 0x3c?, 0xa5?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0004bff50 sp=0x10c0004bff30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0004bffe0 sp=0x10c0004bff50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0004bffe8 sp=0x10c0004bffe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 75 [GC worker (idle)]:
runtime.gopark(0xed46712118342f?, 0x3?, 0xe?, 0x29?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058a750 sp=0x10c00058a730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058a7e0 sp=0x10c00058a750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058a7e8 sp=0x10c00058a7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 76 [GC worker (idle)]:
runtime.gopark(0xed4671211795ee?, 0x3?, 0x25?, 0xc4?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058af50 sp=0x10c00058af30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058afe0 sp=0x10c00058af50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058afe8 sp=0x10c00058afe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 77 [GC worker (idle)]:
runtime.gopark(0xed467123699cfb?, 0x1?, 0xe1?, 0xa3?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058b750 sp=0x10c00058b730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058b7e0 sp=0x10c00058b750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058b7e8 sp=0x10c00058b7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 79 [GC worker (idle)]:
runtime.gopark(0xed46712369a19a?, 0x1?, 0x63?, 0x49?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058c750 sp=0x10c00058c730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058c7e0 sp=0x10c00058c750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058c7e8 sp=0x10c00058c7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 80 [GC worker (idle)]:
runtime.gopark(0xed4671211795bf?, 0x1?, 0x4e?, 0xa5?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058cf50 sp=0x10c00058cf30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058cfe0 sp=0x10c00058cf50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058cfe8 sp=0x10c00058cfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 81 [GC worker (idle)]:
runtime.gopark(0xed46712116f0aa?, 0x3?, 0xb6?, 0x3a?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058d750 sp=0x10c00058d730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058d7e0 sp=0x10c00058d750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058d7e8 sp=0x10c00058d7e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 82 [GC worker (idle)]:
runtime.gopark(0xed467123699d55?, 0x1?, 0x69?, 0x7d?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c00058df50 sp=0x10c00058df30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c00058dfe0 sp=0x10c00058df50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c00058dfe8 sp=0x10c00058dfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 83 [GC worker (idle)]:
runtime.gopark(0xed467123699d8e?, 0x3?, 0xf5?, 0x59?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000586750 sp=0x10c000586730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005867e0 sp=0x10c000586750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005867e8 sp=0x10c0005867e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 84 [GC worker (idle), 7 minutes]:
runtime.gopark(0xed4602fcc25f03?, 0x1?, 0x2b?, 0xf0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c0005cdf50 sp=0x10c0005cdf30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005cdfe0 sp=0x10c0005cdf50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005cdfe8 sp=0x10c0005cdfe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 85 [GC worker (idle)]:
runtime.gopark(0xed467123699cde?, 0x1?, 0x2c?, 0xbc?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000587750 sp=0x10c000587730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005877e0 sp=0x10c000587750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005877e8 sp=0x10c0005877e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 86 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x1?, 0x36?, 0x18?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000587f50 sp=0x10c000587f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000587fe0 sp=0x10c000587f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000587fe8 sp=0x10c000587fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 87 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x1?, 0x7b?, 0x74?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000588750 sp=0x10c000588730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005887e0 sp=0x10c000588750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005887e8 sp=0x10c0005887e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 10 [GC worker (idle)]:
runtime.gopark(0xed467123699e9b?, 0x1?, 0x1d?, 0x25?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000506750 sp=0x10c000506730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005067e0 sp=0x10c000506750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005067e8 sp=0x10c0005067e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 11 [GC worker (idle)]:
runtime.gopark(0xed46712369a2d3?, 0x1?, 0xb4?, 0x51?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000506f50 sp=0x10c000506f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000506fe0 sp=0x10c000506f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000506fe8 sp=0x10c000506fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 12 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x3?, 0x85?, 0x5c?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000507750 sp=0x10c000507730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005077e0 sp=0x10c000507750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005077e8 sp=0x10c0005077e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 13 [GC worker (idle)]:
runtime.gopark(0xed467123699eed?, 0x1?, 0x4c?, 0xe5?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000507f50 sp=0x10c000507f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000507fe0 sp=0x10c000507f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000507fe8 sp=0x10c000507fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 88 [GC worker (idle)]:
runtime.gopark(0xed467121179929?, 0x3?, 0x83?, 0x5f?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000588f50 sp=0x10c000588f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000588fe0 sp=0x10c000588f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000588fe8 sp=0x10c000588fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 14 [GC worker (idle)]:
runtime.gopark(0xed467121171a7e?, 0x3?, 0x77?, 0x2c?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000508750 sp=0x10c000508730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005087e0 sp=0x10c000508750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005087e8 sp=0x10c0005087e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 15 [GC worker (idle)]:
runtime.gopark(0xed46712369a470?, 0x1?, 0xde?, 0xce?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000508f50 sp=0x10c000508f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000508fe0 sp=0x10c000508f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000508fe8 sp=0x10c000508fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 16 [GC worker (idle)]:
runtime.gopark(0xed46712368ec6f?, 0x1?, 0x15?, 0x91?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000509750 sp=0x10c000509730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005097e0 sp=0x10c000509750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005097e8 sp=0x10c0005097e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 99 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x3?, 0xf7?, 0x20?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000509f50 sp=0x10c000509f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000509fe0 sp=0x10c000509f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000509fe8 sp=0x10c000509fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 100 [GC worker (idle)]:
runtime.gopark(0xed46712116f729?, 0x3?, 0xf9?, 0x11?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000502750 sp=0x10c000502730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005027e0 sp=0x10c000502750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005027e8 sp=0x10c0005027e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 101 [GC worker (idle)]:
runtime.gopark(0x7f34500c9280?, 0x1?, 0xe2?, 0x56?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000502f50 sp=0x10c000502f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000502fe0 sp=0x10c000502f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000502fe8 sp=0x10c000502fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 102 [GC worker (idle), 7 minutes]:
runtime.gopark(0x7f34500c9280?, 0x1?, 0xea?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000503750 sp=0x10c000503730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005037e0 sp=0x10c000503750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005037e8 sp=0x10c0005037e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 103 [GC worker (idle)]:
runtime.gopark(0xed46712369a86d?, 0x1?, 0xb4?, 0x5d?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000503f50 sp=0x10c000503f30 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c000503fe0 sp=0x10c000503f50 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c000503fe8 sp=0x10c000503fe0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 104 [GC worker (idle)]:
runtime.gopark(0xed4671211810f3?, 0x1?, 0xb9?, 0x9e?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000504750 sp=0x10c000504730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005047e0 sp=0x10c000504750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005047e8 sp=0x10c0005047e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 89 [GC worker (idle)]:
runtime.gopark(0xed467123699d91?, 0x3?, 0x9a?, 0xa?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000589750 sp=0x10c000589730 pc=0x7f344eaf5eb6
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0x10c0005897e0 sp=0x10c000589750 pc=0x7f344ead71f1
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0x10c0005897e8 sp=0x10c0005897e0 pc=0x7f344eb26e41
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 90 [GC worker (idle) (scan)]:
runtime.gopark(0xed467121182bfc?, 0x3?, 0x76?, 0x6b?, 0x0?)
/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0x10c000589f50 sp=0x10c000589f30 pc=0x7f344eaf5eb6
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x7f344eb19496]

runtime stack:
runtime.throw({0x7f344f6b2363?, 0x7f344eb123b4?})
/usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x7f341db852f8 sp=0x7f341db852c8 pc=0x7f344eaf329f
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7f341db85348 sp=0x7f341db852f8 pc=0x7f344eb09a89
runtime.gentraceback(0x7f344fe7d818?, 0x7f3450022a00?, 0x7f341db85718?, 0x7f344eb1b41e?, 0x0, 0x0, 0x64, 0x0, 0x7f341db85738?, 0x0)
/usr/local/go/src/runtime/traceback.go:257 +0x8b6 fp=0x7f341db856b8 sp=0x7f341db85348 pc=0x7f344eb19496
runtime.traceback1(0x10c00058eb40?, 0x7f344eaf4c00?, 0x3?, 0x10c00058eb40, 0x7f344eb1c5cf?)
/usr/local/go/src/runtime/traceback.go:857 +0x1b1 fp=0x7f341db85878 sp=0x7f341db856b8 pc=0x7f344eb1b8b1
runtime.traceback(...)
/usr/local/go/src/runtime/traceback.go:804
runtime.tracebackothers.func1(0x10c00058eb40)
/usr/local/go/src/runtime/traceback.go:1073 +0xe5 fp=0x7f341db858b8 sp=0x7f341db85878 pc=0x7f344eb1c805
runtime.forEachGRace(0x7f341db85920)
/usr/local/go/src/runtime/proc.go:592 +0x4d fp=0x7f341db858e8 sp=0x7f341db858b8 pc=0x7f344eaf6a0d
runtime.tracebackothers(0x10c000009860?)
/usr/local/go/src/runtime/traceback.go:1059 +0xe5 fp=0x7f341db85950 sp=0x7f341db858e8 pc=0x7f344eb1c6e5
runtime.dopanic_m(0x10c000009860, 0x7f344f7ed963?, 0x1?)
/usr/local/go/src/runtime/panic.go:1270 +0x285 fp=0x7f341db859c8 sp=0x7f341db85950 pc=0x7f344eaf3ca5
runtime.fatalthrow.func1()
/usr/local/go/src/runtime/panic.go:1125 +0x48 fp=0x7f341db85a08 sp=0x7f341db859c8 pc=0x7f344eaf3668
runtime.fatalthrow(0x1db85a50?)
/usr/local/go/src/runtime/panic.go:1122 +0x6c fp=0x7f341db85a48 sp=0x7f341db85a08 pc=0x7f344eaf35ec
runtime.throw({0x7f344f692bb0?, 0x7f3450022a00?})
/usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x7f341db85a78 sp=0x7f341db85a48 pc=0x7f344eaf329f
runtime.gentraceback(0x20?, 0x1?, 0x0?, 0x7f341db85e68?, 0x0, 0x0, 0x7fffffff, 0x7f341db85fc8, 0x0?, 0x0)
/usr/local/go/src/runtime/traceback.go:269 +0x1cf7 fp=0x7f341db85de8 sp=0x7f341db85a78 pc=0x7f344eb1a8d7
runtime.scanstack(0x10c00058eb40, 0x10c000051c40)
/usr/local/go/src/runtime/mgcmark.go:804 +0x1da fp=0x7f341db85ff0 sp=0x7f341db85de8 pc=0x7f344eada59a
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:240 +0xc5 fp=0x7f341db86040 sp=0x7f341db85ff0 pc=0x7f344ead93c5
runtime.markroot(0x10c000051c40, 0x5a, 0x1)
/usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7f341db860e0 sp=0x7f341db86040 pc=0x7f344ead9065
runtime.gcDrain(0x10c000051c40, 0x3)
/usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7f341db86140 sp=0x7f341db860e0 pc=0x7f344eadb13f
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1308 +0xa5 fp=0x7f341db86190 sp=0x7f341db86140 pc=0x7f344ead7665
runtime.systemstack()
/usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x7f341db86198 sp=0x7f341db86190 pc=0x7f344eb24c06
@xingdl2007 xingdl2007 changed the title affected/package: runtime: unexpected return pc for runtime.gcBgMarkWorker called from 0x0 runtime: unexpected return pc for runtime.gcBgMarkWorker called from 0x0 Jan 3, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 3, 2023
@heschi
Copy link
Contributor

heschi commented Jan 3, 2023

cc @golang/runtime

@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 3, 2023
@heschi heschi added this to the Backlog milestone Jan 3, 2023
@cherrymui
Copy link
Member 

0x000010c000589fc0: 0x0000000000000000 0x0000000000000000
0x000010c000589fd0: 0x0000000000000000 !0x0000000000000000
0x000010c000589fe0: >0x0000000000000000 0x0000000000000000
0x000010c000589ff0: 0x0000000000000000 0x0000000000000000

Looks like a memory corruption. At the top (high address) of the stack there should be a runtime.goexit frame, but it's not there. Instead, it is all 0.

We have a C++ program access etcd through its official go client built with -asan and -buildmode=c-shared.

Could you check if the C++ code is memory safe? Could you test the Go code under the race detector?
Are you building the Go code to a shared object with -asan? Does it crash if -asan is not used?
Thanks.

@xingdl2007
Copy link
Author

@cherrymui

Could you check if the C++ code is memory safe?

C++ code is very suspicious, so it's also built with asan. Unfortunately, there are no related output when crash occurs.
This may be related to the lightweight threads we use in c++ code. Go code call a callback registered by c++ code where lightweight thread get involved, I will try to decouple this and see whether it will help.

Could you test the Go code under the race detector?

OK, I will try soon.

Are you building the Go code to a shared object with -asan?

Yes.

Does it crash if -asan is not used?

Yes.

Thanks for reply.

@mdempsky mdempsky added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 4, 2023
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Feb 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@mdempsky @xingdl2007 @gopherbot @cherrymui @heschi