Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: panic: unexpected return pc for runtime.heapBitsSetType #38851

Closed
fasaxc opened this issue May 4, 2020 · 8 comments
Closed

runtime: panic: unexpected return pc for runtime.heapBitsSetType #38851

fasaxc opened this issue May 4, 2020 · 8 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.16

Comments

@fasaxc
Copy link

fasaxc commented May 4, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.7 linux/amd64

Does this issue reproduce with the latest release?

I don't know yet, it's intermittent

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build203255452=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran a large test FV suite for our application (approximately 900 tests, each of which starts and stops multiple copies of our application).

What did you expect to see?

A clean run.

What did you see instead?

One of the tests panicked in the go runtime very soon after our app started; full stack trace from the panic is in the details below. I don't really understand the message of the panic so it's hard to guess what other diagnostics might be useful.

Looking at the chunk of code that generates the panic there's a comment

// This happens if you get a profiling interrupt at just the wrong time.
/// In that context it is okay to stop early.
/// But if callback is set, we're doing a garbage collection and must
/// get everything, so crash loudly.

Does that mean that this might be caused by a profiling interrupt during a GC? Does that refer to go's built-in profiling mechanisms; I think we run with memory profiling turned on by default since it's useful and low overhead but if it causes random panics then we may be forced to turn it off.

runtime: unexpected return pc for runtime.heapBitsSetType called from 0x7273752f3d485441
stack: frame={sp:0xc000694388, fp:0xc000694390} stack=[0xc000690000,0xc000698000)
000000c000694288:  0000000000000000  0000009b00000008 
000000c000694298:  0000000000000000  000000c000458800 
000000c0006942a8:  0000000000000001  0000000000000010 
000000c0006942b8:  0000009c00000008  0000000000000000 
000000c0006942c8:  000000c000458804  0000000000000001 
000000c0006942d8:  000000000000000f  0000009d00000008 
000000c0006942e8:  0000000000000000  000000c000458808 
000000c0006942f8:  0000000000000001  000000000000000e 
000000c000694308:  0000009e00000008  0000000000000000 
000000c000694318:  000000c00045880c  0000000000000001 
000000c000694328:  000000000000000d  0000009f00000008 
000000c000694338:  0000000000000000  000000c000458810 
000000c000694348:  0000000000000001  000000000000000c 
000000c000694358:  000000a000000008  0000000000000000 
000000c000694368:  000000c000458814  0000000000000001 
000000c000694378:  000000000000000b  000000a100000008 
000000c000694388: <7273752f3d485441 >732f6c61636f6c2f 
000000c000694398:  0000000000000000  0000000000000000 
000000c0006943a8:  0000000001a00ac0  0000000000000000 
000000c0006943b8:  0000000000000000  0000000000000000 
000000c0006943c8:  0000000000000008  000000c000010990 
000000c0006943d8:  0000000000000000  0000000000000000 
000000c0006943e8:  000000c000568000  00007f7b481d0d98 
000000c0006943f8:  0000000000000000  0000000000000000 
000000c000694408:  0000000000000000  0000000000000000 
000000c000694418:  0000000000000000  000000c000826450 
000000c000694428:  000000000040ef88   0000000000000008 
000000c000694438:  0000000001a00ac0  0000000000000001 
000000c000694448:  0000000000000000  000000c000826530 
000000c000694458:  000000000044143c   0000000001a00ac0 
000000c000694468:  0000000000000000  0000000000000000 
000000c000694478:  0000000000000000  0000000000000000 
000000c000694488:  0000000000000000 
fatal error: unknown caller pc

runtime stack:

runtime.throw(0x1d4253f, 0x11)

/usr/local/go-cgo/src/runtime/panic.go:774 +0x72

runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc00054d200, 0x0, 0x0, 0x7fffffff, 0x1e06f00, 0x7f7b3effcbc0, 0x0, ...)

/usr/local/go-cgo/src/runtime/traceback.go:273 +0x19e5

runtime.copystack(0xc00054d200, 0x8000, 0x7f7b00000001)

/usr/local/go-cgo/src/runtime/stack.go:886 +0x25a

runtime.newstack()

/usr/local/go-cgo/src/runtime/stack.go:1055 +0x2fb

runtime.morestack()

/usr/local/go-cgo/src/runtime/asm_amd64.s:449 +0x8f


goroutine 143 [copystack]:

runtime: unexpected return pc for runtime.heapBitsSetType called from 0x7273752f3d485441

stack: frame={sp:0xc000694388, fp:0xc000694390} stack=[0xc000690000,0xc000698000)

000000c000694288:  0000000000000000  0000009b00000008

000000c000694298:  0000000000000000  000000c000458800

000000c0006942a8:  0000000000000001  0000000000000010

000000c0006942b8:  0000009c00000008  0000000000000000

000000c0006942c8:  000000c000458804  0000000000000001

000000c0006942d8:  000000000000000f  0000009d00000008

000000c0006942e8:  0000000000000000  000000c000458808

000000c0006942f8:  0000000000000001  000000000000000e

000000c000694308:  0000009e00000008  0000000000000000

000000c000694318:  000000c00045880c  0000000000000001

000000c000694328:  000000000000000d  0000009f00000008

000000c000694338:  0000000000000000  000000c000458810

000000c000694348:  0000000000000001  000000000000000c

000000c000694358:  000000a000000008  0000000000000000

000000c000694368:  000000c000458814  0000000000000001

000000c000694378:  000000000000000b  000000a100000008

000000c000694388: <7273752f3d485441 >732f6c61636f6c2f

000000c000694398:  0000000000000000  0000000000000000

000000c0006943a8:  0000000001a00ac0  0000000000000000

000000c0006943b8:  0000000000000000  0000000000000000

000000c0006943c8:  0000000000000008  000000c000010990

000000c0006943d8:  0000000000000000  0000000000000000

000000c0006943e8:  000000c000568000  00007f7b481d0d98

000000c0006943f8:  0000000000000000  0000000000000000

000000c000694408:  0000000000000000  0000000000000000

000000c000694418:  0000000000000000  000000c000826450

000000c000694428:  000000000040ef88 <runtime.newobject+56>  0000000000000008

000000c000694438:  0000000001a00ac0  0000000000000001

000000c000694448:  0000000000000000  000000c000826530

000000c000694458:  000000000044143c <reflect.typelinks+108>  0000000001a00ac0

000000c000694468:  0000000000000000  0000000000000000

000000c000694478:  0000000000000000  0000000000000000

000000c000694488:  0000000000000000

runtime.heapBitsSetType(0x732f6c61636f6c2f, 0x0, 0x0, 0x1a00ac0)

/usr/local/go-cgo/src/runtime/mbitmap.go:938 +0xa4e fp=0xc000694390 sp=0xc000694388 pc=0x418bbe

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 1 [select]:

github.com/projectcalico/felix/daemon.monitorAndManageShutdown(0xc000470060, 0x0, 0x0, 0x0, 0x0)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:670 +0x281

github.com/projectcalico/felix/daemon.Run(0xc00018615b, 0x15, 0x20722a0, 0x20, 0x20a5be0, 0x28, 0x206bf50, 0x18)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:615 +0x2cbd

main.main()

/go/src/github.com/projectcalico/felix/cmd/calico-felix/calico-felix.go:51 +0x1ca


goroutine 19 [syscall]:

os/signal.signal_recv(0x0)

/usr/local/go-cgo/src/runtime/sigqueue.go:147 +0x9c

os/signal.loop()

/usr/local/go-cgo/src/os/signal/signal_unix.go:23 +0x22

created by os/signal.init.0

/usr/local/go-cgo/src/os/signal/signal_unix.go:29 +0x41


goroutine 35 [chan receive]:

k8s.io/klog.(*loggingT).flushDaemon(0x2f5b2e0)

/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:1018 +0x8b

created by k8s.io/klog.init.0

/go/pkg/mod/k8s.io/klog@v0.4.0/klog.go:404 +0x6c


goroutine 71 [IO wait]:

internal/poll.runtime_pollWait(0x7f7b44760e38, 0x72, 0xffffffffffffffff)

/usr/local/go-cgo/src/runtime/netpoll.go:184 +0x55

internal/poll.(*pollDesc).wait(0xc00028d418, 0x72, 0xa00, 0xa91, 0xffffffffffffffff)

/usr/local/go-cgo/src/internal/poll/fd_poll_runtime.go:87 +0x45

internal/poll.(*pollDesc).waitRead(...)

/usr/local/go-cgo/src/internal/poll/fd_poll_runtime.go:92

internal/poll.(*FD).Read(0xc00028d400, 0xc000546400, 0xa91, 0xa91, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/internal/poll/fd_unix.go:169 +0x1cf

net.(*netFD).Read(0xc00028d400, 0xc000546400, 0xa91, 0xa91, 0x203000, 0x0, 0xa57)

/usr/local/go-cgo/src/net/fd_unix.go:202 +0x4f

net.(*conn).Read(0xc0000d00f0, 0xc000546400, 0xa91, 0xa91, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/net/net.go:184 +0x68

crypto/tls.(*atLeastReader).Read(0xc0000c42a0, 0xc000546400, 0xa91, 0xa91, 0x50, 0xc0000e5d60, 0xc000423970)

/usr/local/go-cgo/src/crypto/tls/conn.go:780 +0x60

bytes.(*Buffer).ReadFrom(0xc00009fe58, 0x206c840, 0xc0000c42a0, 0x40c6f5, 0x1ab8c40, 0x1cacac0)

/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4

crypto/tls.(*Conn).readFromUntil(0xc00009fc00, 0x206efc0, 0xc0000d00f0, 0x5, 0xc0000d00f0, 0x203000)

/usr/local/go-cgo/src/crypto/tls/conn.go:802 +0xec

crypto/tls.(*Conn).readRecordOrCCS(0xc00009fc00, 0x0, 0x0, 0xc0000c4240)

/usr/local/go-cgo/src/crypto/tls/conn.go:609 +0x124

crypto/tls.(*Conn).readRecord(...)

/usr/local/go-cgo/src/crypto/tls/conn.go:577

crypto/tls.(*Conn).Read(0xc00009fc00, 0xc0005d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/crypto/tls/conn.go:1255 +0x161

bufio.(*Reader).Read(0xc0000945a0, 0xc0001f42d8, 0x9, 0x9, 0xc000423d10, 0x0, 0x924d55)

/usr/local/go-cgo/src/bufio/bufio.go:226 +0x26a

io.ReadAtLeast(0x206c6a0, 0xc0000945a0, 0xc0001f42d8, 0x9, 0x9, 0x9, 0xc0000b6050, 0x0, 0x206ca20)

/usr/local/go-cgo/src/io/io.go:310 +0x87

io.ReadFull(...)

/usr/local/go-cgo/src/io/io.go:329

golang.org/x/net/http2.readFrameHeader(0xc0001f42d8, 0x9, 0x9, 0x206c6a0, 0xc0000945a0, 0x0, 0x0, 0xc000618e40, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:237 +0x87

golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001f42a0, 0xc000618e40, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/frame.go:492 +0xa1

golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000423fb8, 0x407295, 0xc000094720)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1746 +0x8c

golang.org/x/net/http2.(*ClientConn).readLoop(0xc0000a1500)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1674 +0xa3

created by golang.org/x/net/http2.(*Transport).newClientConn

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:674 +0x62f


goroutine 73 [chan receive]:

github.com/projectcalico/libcalico-go/lib/logutils.(*Destination).LoopWritingLogs(0xc0002b1440)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/logutils/logutils.go:323 +0x83

created by github.com/projectcalico/libcalico-go/lib/logutils.(*BackgroundHook).Start

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/logutils/logutils.go:468 +0x5e


goroutine 114 [sleep]:

runtime.goparkunlock(...)

/usr/local/go-cgo/src/runtime/proc.go:310

time.Sleep(0x62f5b97)

/usr/local/go-cgo/src/runtime/time.go:105 +0x157

github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518420, 0xc0000d2de0)

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e

created by github.com/projectcalico/felix/jitter.NewTicker

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb


goroutine 53 [runnable]:

syscall.readlen(0x10, 0xc00066f878, 0x8, 0x8, 0xc0003605b0, 0x19)

/usr/local/go-cgo/src/syscall/zsyscall_linux_amd64.go:1025 +0xef

syscall.forkExec(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066f998, 0x18, 0x3e98d35b00000200, 0xc00012ea80)

/usr/local/go-cgo/src/syscall/exec_unix.go:210 +0x3c6

syscall.StartProcess(...)

/usr/local/go-cgo/src/syscall/exec_unix.go:248

os.startProcess(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066fb30, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/os/exec_posix.go:51 +0x2b0

os.StartProcess(0xc0005072e0, 0x10, 0xc0003aa770, 0x7, 0x7, 0xc00066fb30, 0x18, 0x0, 0x1ed)

/usr/local/go-cgo/src/os/exec.go:102 +0x7c

os/exec.(*Cmd).Start(0xc0004ef760, 0x1, 0xc0006f6db0)

/usr/local/go-cgo/src/os/exec/exec.go:416 +0x50c

os/exec.(*Cmd).Run(0xc0004ef760, 0xc0006f6db0, 0xc0004ef760)

/usr/local/go-cgo/src/os/exec/exec.go:338 +0x2b

os/exec.(*Cmd).CombinedOutput(0xc0004ef760, 0x7, 0xc0003aa710, 0x6, 0x6, 0xc0004ef760)

/usr/local/go-cgo/src/os/exec/exec.go:561 +0x91

github.com/projectcalico/felix/bpf.(*PinnedMap).Iter(0xc000630cb0, 0xc0006f2d60, 0x0, 0x0)

/go/src/github.com/projectcalico/felix/bpf/maps.go:179 +0x11e

github.com/projectcalico/felix/bpf/nat.LoadBackendMap(0x20e03a0, 0xc000630cb0, 0xc000802ab0, 0x0, 0x0)

/go/src/github.com/projectcalico/felix/bpf/nat/maps.go:282 +0x44

github.com/projectcalico/felix/bpf/proxy.(*Syncer).loadOrigs(0xc000814000, 0xc000814000, 0xc00048ccc0)

/go/src/github.com/projectcalico/felix/bpf/proxy/syncer.go:190 +0x73

github.com/projectcalico/felix/bpf/proxy.NewSyncer(0xc000802a20, 0x2, 0x2, 0x20e03a0, 0xc000630c40, 0x20e03a0, 0xc000630cb0, 0x20e03a0, 0xc000630d90, 0x20810a0, ...)

/go/src/github.com/projectcalico/felix/bpf/proxy/syncer.go:176 +0x1a2

github.com/projectcalico/felix/bpf/proxy.(*KubeProxy).run(0xc000626000, 0xc0007f4fc0, 0x1, 0x1, 0x0, 0x0)

/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:98 +0x1e9

github.com/projectcalico/felix/bpf/proxy.(*KubeProxy).start(0xc000626000, 0xc0000d3320, 0xc0005667b0)

/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:120 +0xb2

github.com/projectcalico/felix/bpf/proxy.StartKubeProxy.func1(0xc000626000)

/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:67 +0x2b

created by github.com/projectcalico/felix/bpf/proxy.StartKubeProxy

/go/src/github.com/projectcalico/felix/bpf/proxy/kube-proxy.go:66 +0x25c


goroutine 111 [select]:

github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).loopUpdatingDataplane(0xc000183600)

/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:1128 +0x3d5

created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start

/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:773 +0x51


goroutine 112 [sleep]:

runtime.goparkunlock(...)

/usr/local/go-cgo/src/runtime/proc.go:310

time.Sleep(0x2540be400)

/usr/local/go-cgo/src/runtime/time.go:105 +0x157

github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).loopReportingStatus(0xc000183600)

/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:1469 +0xa4

created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start

/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:774 +0x73


goroutine 131 [chan receive]:

github.com/projectcalico/felix/calc.(*SyncerCallbacksDecoupler).SendTo(0xc0005c0038, 0x20811e0, 0xc000347f10)

/go/src/github.com/projectcalico/felix/calc/async_decoupler.go:40 +0x50

created by github.com/projectcalico/felix/daemon.Run

/go/src/github.com/projectcalico/felix/daemon/daemon.go:560 +0x2b44


goroutine 81 [sleep]:

runtime.goparkunlock(...)

/usr/local/go-cgo/src/runtime/proc.go:310

time.Sleep(0x162ba721e4)

/usr/local/go-cgo/src/runtime/time.go:105 +0x157

github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518320, 0xc0000d2d80)

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e

created by github.com/projectcalico/felix/jitter.NewTicker

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb


goroutine 80 [sleep]:

runtime.goparkunlock(...)

/usr/local/go-cgo/src/runtime/proc.go:310

time.Sleep(0x2649aa928)

/usr/local/go-cgo/src/runtime/time.go:105 +0x157

github.com/projectcalico/felix/jitter.(*Ticker).loop(0xc000518240, 0xc0000d2d20)

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:53 +0x4e

created by github.com/projectcalico/felix/jitter.NewTicker

/go/src/github.com/projectcalico/felix/jitter/jittered_ticker.go:46 +0xeb


goroutine 113 [select]:

github.com/projectcalico/felix/ifacemonitor.(*InterfaceMonitor).MonitorInterfaces(0xc0000d2a80)

/go/src/github.com/projectcalico/felix/ifacemonitor/iface_monitor.go:111 +0x597

created by github.com/projectcalico/felix/dataplane/linux.(*InternalDataplane).Start

/go/src/github.com/projectcalico/felix/dataplane/linux/int_dataplane.go:775 +0x9c


goroutine 130 [chan receive]:

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run(0xc0000953e0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:149 +0x2f2

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).Start.func1(0xc0000953e0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:102 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).Start

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:101 +0x139


goroutine 56 [chan receive]:

github.com/vishvananda/netlink.linkSubscribeAt.func1(0xc000186600, 0xc00058c510)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1642 +0x34

created by github.com/vishvananda/netlink.linkSubscribeAt

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1641 +0x36b


goroutine 57 [syscall]:

syscall.Syscall6(0x2d, 0xb, 0xc0006b0000, 0x10000, 0x0, 0xc00017e0e0, 0xc0005f05f4, 0x0, 0xc0005f0590, 0x40ef88)

/usr/local/go-cgo/src/syscall/asm_linux_amd64.s:44 +0x5

golang.org/x/sys/unix.recvfrom(0xb, 0xc0006b0000, 0x10000, 0x10000, 0x0, 0xc00017e0e0, 0xc0005f05f4, 0xc0005f0608, 0x4479ec, 0x10000)

/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/zsyscall_linux_amd64.go:2414 +0xa3

golang.org/x/sys/unix.Recvfrom(0xb, 0xc0006b0000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4302fa)

/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/syscall_unix.go:307 +0x94

github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc00058c510, 0x0, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/nl/nl_linux.go:626 +0x93

github.com/vishvananda/netlink.linkSubscribeAt.func2(0xc000466000, 0xc00058c510, 0x0)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1658 +0xf3

created by github.com/vishvananda/netlink.linkSubscribeAt

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/link_linux.go:1655 +0x204


goroutine 58 [chan receive]:

github.com/vishvananda/netlink.addrSubscribeAt.func1(0xc000186600, 0xc00058c570)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:308 +0x34

created by github.com/vishvananda/netlink.addrSubscribeAt

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:307 +0x36e


goroutine 59 [syscall]:

syscall.Syscall6(0x2d, 0xc, 0xc0006f8000, 0x10000, 0x0, 0xc0001e4310, 0xc0005f1d24, 0xc0005f1ef0, 0xc0005f1cc0, 0x40ef88)

/usr/local/go-cgo/src/syscall/asm_linux_amd64.s:44 +0x5

golang.org/x/sys/unix.recvfrom(0xc, 0xc0006f8000, 0x10000, 0x10000, 0x0, 0xc0001e4310, 0xc0005f1d24, 0xc0005f1d38, 0x4479ec, 0x10000)

/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/zsyscall_linux_amd64.go:2414 +0xa3

golang.org/x/sys/unix.Recvfrom(0xc, 0xc0006f8000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4302fa)

/go/pkg/mod/golang.org/x/sys@v0.0.0-20200202164722-d101bd2416d5/unix/syscall_unix.go:307 +0x94

github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc00058c570, 0x0, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/nl/nl_linux.go:626 +0x93

github.com/vishvananda/netlink.addrSubscribeAt.func2(0xc000466060, 0xc00058c570, 0x0)

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:324 +0xf9

created by github.com/vishvananda/netlink.addrSubscribeAt

/go/pkg/mod/github.com/vishvananda/netlink@v0.0.0-20181108222139-023a6dafdcdf/addr_linux.go:321 +0x207


goroutine 132 [select]:

github.com/projectcalico/felix/calc.(*AsyncCalcGraph).loop(0xc000095620)

/go/src/github.com/projectcalico/felix/calc/async_calc_graph.go:139 +0x192

created by github.com/projectcalico/felix/calc.(*AsyncCalcGraph).Start

/go/src/github.com/projectcalico/felix/calc/async_calc_graph.go:237 +0xb3


goroutine 133 [chan receive]:

github.com/projectcalico/felix/daemon.(*DataplaneConnector).sendMessagesToDataplaneDriver(0xc00062c780)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:1103 +0xb0

created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start

/go/src/github.com/projectcalico/felix/daemon/daemon.go:1185 +0x3f


goroutine 134 [semacquire]:

github.com/sirupsen/logrus.NewEntry(...)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:52

github.com/sirupsen/logrus.(*Logger).newEntry(0xc0000d2120, 0xc0005c0018)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:83 +0x6f

github.com/sirupsen/logrus.(*Logger).Info(0xc0000d2120, 0xc00007bf98, 0x1, 0x1)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:188 +0x41

github.com/sirupsen/logrus.Info(...)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/exported.go:85

github.com/projectcalico/felix/daemon.(*DataplaneConnector).readMessagesFromDataplane(0xc00062c780)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:939 +0xbe

created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start

/go/src/github.com/projectcalico/felix/daemon/daemon.go:1188 +0x61


goroutine 135 [select]:

github.com/projectcalico/felix/daemon.(*DataplaneConnector).handleWireguardStatUpdateFromDataplane(0xc00062c780)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:1070 +0xc8

created by github.com/projectcalico/felix/daemon.(*DataplaneConnector).Start

/go/src/github.com/projectcalico/felix/daemon/daemon.go:1191 +0x83


goroutine 136 [runnable]:

runtime.goparkunlock(...)

/usr/local/go-cgo/src/runtime/proc.go:310

sync.runtime_notifyListWait(0xc0004ef0c0, 0x0)

/usr/local/go-cgo/src/runtime/sema.go:510 +0xf8

sync.(*Cond).Wait(0xc0004ef0b0)

/usr/local/go-cgo/src/sync/cond.go:56 +0x9d

golang.org/x/net/http2.(*pipe).Read(0xc0004ef0a8, 0xc000658200, 0x200, 0x200, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/pipe.go:65 +0xa6

golang.org/x/net/http2.transportResponseBody.Read(0xc0004ef080, 0xc000658200, 0x200, 0x200, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2019 +0xac

bytes.(*Buffer).ReadFrom(0xc0007c63f8, 0x7f7b44507518, 0xc0004ef080, 0x40b5ab, 0xc00003c000, 0x1ab8c40)

/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4

io/ioutil.readAll(0x7f7b44507518, 0xc0004ef080, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/io/ioutil/ioutil.go:36 +0x100

io/ioutil.ReadAll(...)

/usr/local/go-cgo/src/io/ioutil/ioutil.go:45

k8s.io/client-go/rest.(*Request).transformResponse(0xc00067e000, 0xc00018cab0, 0xc000382c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:861 +0x6a7

k8s.io/client-go/rest.(*Request).Do.func1(0xc000382c00, 0xc00018cab0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:829 +0x5f

k8s.io/client-go/rest.(*Request).request.func2(0xc00018cab0, 0xc0007c6820, 0xa, 0xc00067e000, 0xc0007c6948, 0xc000382c00, 0xc0004b0730, 0x49, 0xc000382b00)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:805 +0xd1

k8s.io/client-go/rest.(*Request).request(0xc00067e000, 0xc0007c6948, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:807 +0x6ba

k8s.io/client-go/rest.(*Request).Do(0xc00067e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004889a0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014ef80, 0x0, 0x0, 0x30, 0x30, 0x1c5ca00)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014ef80, 0x0, 0x0, 0x46, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc0001165a0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001165a0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001165a0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 137 [runnable]:

golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616d00, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0

golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616d00, 0xc00078e000, 0x72db95, 0x0, 0xc000207200)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172

golang.org/x/net/http2.(*Transport).RoundTrip(...)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409

golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616d00, 0xc0001255e0, 0x5, 0xc000207288)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e

net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616d00, 0x4119fe, 0x2039510, 0x0)

/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1

net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616d00, 0xc0002b43c0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35

net/http.send(0xc000616d00, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc0000105e8, 0xc0005cfa80, 0x1, 0x0)

/usr/local/go-cgo/src/net/http/client.go:250 +0x443

net/http.(*Client).send(0xc0006637d0, 0xc000616d00, 0x0, 0x0, 0x0, 0xc0000105e8, 0x0, 0x1, 0xc000804810)

/usr/local/go-cgo/src/net/http/client.go:174 +0xfa

net/http.(*Client).do(0xc0006637d0, 0xc000616d00, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce

net/http.(*Client).Do(0xc0006637d0, 0xc000616d00, 0x0, 0x3, 0xc0001255e0)

/usr/local/go-cgo/src/net/http/client.go:509 +0x35

k8s.io/client-go/rest.(*Request).request(0xc0005e8a80, 0xc00078e948, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367

k8s.io/client-go/rest.(*Request).Do(0xc0005e8a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004888f0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014efc0, 0x0, 0x0, 0x30, 0x30, 0x1c5ca00)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014efc0, 0x0, 0x0, 0x46, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116630, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116630, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116630)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 138 [runnable]:

sync.runtime_SemacquireMutex(0xc0000a1554, 0xc00078a100, 0x1)

/usr/local/go-cgo/src/runtime/sema.go:71 +0x47

sync.(*Mutex).lockSlow(0xc0000a1550)

/usr/local/go-cgo/src/sync/mutex.go:138 +0xfc

sync.(*Mutex).Lock(...)

/usr/local/go-cgo/src/sync/mutex.go:81

golang.org/x/net/http2.transportResponseBody.Read(0xc000280580, 0xc000658400, 0x200, 0x200, 0xcc, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2042 +0x584

bytes.(*Buffer).ReadFrom(0xc00078a3f8, 0x7f7b44507518, 0xc000280580, 0x40b5ab, 0xc00003c000, 0x1ab8c40)

/usr/local/go-cgo/src/bytes/buffer.go:204 +0xb4

io/ioutil.readAll(0x7f7b44507518, 0xc000280580, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/io/ioutil/ioutil.go:36 +0x100

io/ioutil.ReadAll(...)

/usr/local/go-cgo/src/io/ioutil/ioutil.go:45

k8s.io/client-go/rest.(*Request).transformResponse(0xc0005e8480, 0xc00018ca20, 0xc000616800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:861 +0x6a7

k8s.io/client-go/rest.(*Request).Do.func1(0xc000616800, 0xc00018ca20)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:829 +0x5f

k8s.io/client-go/rest.(*Request).request.func2(0xc00018ca20, 0xc00078a820, 0xa, 0xc0005e8480, 0xc00078a948, 0xc000616800, 0xc0001246e0, 0x4b, 0xc000616700)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:805 +0xd1

k8s.io/client-go/rest.(*Request).request(0xc0005e8480, 0xc00078a948, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:807 +0x6ba

k8s.io/client-go/rest.(*Request).Do(0xc0005e8480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004882c0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f000, 0x0, 0x0, 0xbfa4231b92b10143, 0x71e74b0, 0x2f5ae40)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f000, 0x0, 0x0, 0x48, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc0001166c0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001166c0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001166c0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 139 [select]:

golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000382600, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0

golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000382600, 0xc0006f0000, 0x72db95, 0x0, 0xc000207200)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172

golang.org/x/net/http2.(*Transport).RoundTrip(...)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409

golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000382600, 0xc0004b1cc0, 0x5, 0xc000207288)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e

net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000382600, 0x0, 0xc0006f03a8, 0x10000000040ef88)

/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1

net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000382600, 0xc0002b43c0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35

net/http.send(0xc000382600, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc0005c0010, 0xc0007ced60, 0x1, 0x0)

/usr/local/go-cgo/src/net/http/client.go:250 +0x443

net/http.(*Client).send(0xc0006637d0, 0xc000382600, 0x0, 0x0, 0x0, 0xc0005c0010, 0x0, 0x1, 0xc000802db0)

/usr/local/go-cgo/src/net/http/client.go:174 +0xfa

net/http.(*Client).do(0xc0006637d0, 0xc000382600, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce

net/http.(*Client).Do(0xc0006637d0, 0xc000382600, 0x0, 0x3, 0xc0004b1cc0)

/usr/local/go-cgo/src/net/http/client.go:509 +0x35

k8s.io/client-go/rest.(*Request).request(0xc00067e600, 0xc0006f0948, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367

k8s.io/client-go/rest.(*Request).Do(0xc00067e600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc000488580, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f040, 0x0, 0x0, 0x30, 0x30, 0xc000802c60)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f040, 0x0, 0x0, 0x44, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116750, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116750, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116750)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 140 [runnable]:

github.com/sirupsen/logrus.Entry.log(0xc0000d2120, 0xc00057ff20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5, ...)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:91 +0x3f

github.com/sirupsen/logrus.(*Entry).Debug(0xc0004b3400, 0xc00007a920, 0x1, 0x1)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:143 +0x10f

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc0001167e0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:80 +0x8e

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc0001167e0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 141 [runnable]:

github.com/json-iterator/go.describeStruct(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x1c20e80)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:393 +0x61c

github.com/json-iterator/go.encoderOfStruct(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_encoder.go:18 +0x5a

github.com/json-iterator/go._createEncoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0xc000194658, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:275 +0x441

github.com/json-iterator/go.createEncoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:245 +0x10b

github.com/json-iterator/go.encoderOfType(0xc0006f6ba0, 0x20f8c00, 0xc000662b40, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:227 +0x78

github.com/json-iterator/go.describeStruct(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x7f7b0002d330)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:384 +0x8da

github.com/json-iterator/go.decoderOfStruct(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc0007c1e10, 0x20f8c00, 0xc0002e8a20, 0xc0004e0208, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc0007c1e10, 0x20f8c00, 0xc0002e8a20, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc00084de10, 0x20f8c00, 0xc0002e8a20, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.decoderOfSlice(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105

github.com/json-iterator/go._createDecoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0xc0004e01f8, 0xc0001c2c60)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff

github.com/json-iterator/go.createDecoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc000619fb0, 0x20f8b60, 0xc0000d9100, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.describeStruct(0xc00084e550, 0x20f8c00, 0xc000619200, 0x2d330)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f

github.com/json-iterator/go.decoderOfStruct(0xc00084e550, 0x20f8c00, 0xc000619200, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc0007c2550, 0x20f8c00, 0xc000619200, 0xc0004e01a8, 0xc0001c2c60)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc0007c2550, 0x20f8c00, 0xc000619200, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc00084e550, 0x20f8c00, 0xc000619200, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.(*frozenConfig).DecoderOf(0xc000175ea0, 0x20f8ac0, 0xc000618f90, 0x20f8ac0, 0xc000618f90)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:114 +0x111

github.com/json-iterator/go.(*Iterator).ReadVal(0xc00018cd80, 0x1cd8f40, 0xc0003d2000)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:72 +0x1ce

github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000175ea0, 0xc00054e000, 0x52f, 0xe00, 0x1cd8f40, 0xc0003d2000, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/config.go:348 +0xd2

k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc00014ea40, 0xc00054e000, 0x52f, 0xe00, 0x0, 0x2085460, 0xc0003d2000, 0x1, 0x0, 0x206f080, ...)

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/serializer/json/json.go:275 +0x7d5

k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode(0x206e680, 0xc00014ea40, 0xc00054e000, 0x52f, 0xe00, 0x0, 0x2085460, 0xc0003d2000, 0x0, 0xc000616e00, ...)

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/helper.go:252 +0x97

k8s.io/client-go/rest.Result.Into(0xc00054e000, 0x52f, 0xe00, 0xc00014a250, 0x10, 0x0, 0x0, 0xc8, 0x2070ec0, 0xc0002aeef0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:1133 +0xba

k8s.io/client-go/kubernetes/typed/core/v1.(*nodes).List(0xc00081e1d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/kubernetes/typed/core/v1/node.go:88 +0x1d2

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*nodeClient).List(0xc0002af810, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f180, 0x0, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/node.go:156 +0x4d3

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f180, 0x0, 0x0, 0x38, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116870, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116870, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116870)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 142 [select]:

golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616500, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0

golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616500, 0xc000678100, 0x72db95, 0x0, 0xc000207200)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172

golang.org/x/net/http2.(*Transport).RoundTrip(...)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409

golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616500, 0xc0004fe300, 0x5, 0xc000207288)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e

net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616500, 0x1a005e0, 0x1, 0xc0000f8f20)

/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1

net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616500, 0x1d36edf, 0xa, 0xc0000f8fc0)

/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35

k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0001f3b20, 0xc000616400, 0xc0001f3b20, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/transport/round_trippers.go:159 +0x1c0

net/http.send(0xc000616400, 0x206eea0, 0xc0001f3b20, 0x0, 0x0, 0x0, 0xc000010108, 0xc0003d4439, 0x1, 0x0)

/usr/local/go-cgo/src/net/http/client.go:250 +0x443

net/http.(*Client).send(0xc000523fb0, 0xc000616400, 0x0, 0x0, 0x0, 0xc000010108, 0x0, 0x1, 0xc000618540)

/usr/local/go-cgo/src/net/http/client.go:174 +0xfa

net/http.(*Client).do(0xc000523fb0, 0xc000616400, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce

net/http.(*Client).Do(0xc000523fb0, 0xc000616400, 0x0, 0x3, 0xc0004fe300)

/usr/local/go-cgo/src/net/http/client.go:509 +0x35

k8s.io/client-go/rest.(*Request).request(0xc0005e9200, 0xc0006789c0, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367

k8s.io/client-go/rest.(*Request).Do(0xc0005e9200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

k8s.io/client-go/kubernetes/typed/core/v1.(*namespaces).List(0xc0004d07d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/kubernetes/typed/core/v1/namespace.go:86 +0x179

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*profileClient).List(0xc00003aba0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f1c0, 0x0, 0x0, 0xbfa4231b93345c5b, 0x7a1cfc3, 0x2f5ae40)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/profile.go:192 +0x85b

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f1c0, 0x0, 0x0, 0x3b, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116900, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116900, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116900)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 144 [runnable]:

github.com/json-iterator/go.createEncoderOfType(0xc0006f7890, 0x20f8ac0, 0xc000618ae0, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:244 +0xc8

github.com/json-iterator/go.encoderOfType(0xc0006f7890, 0x20f8ac0, 0xc000618ae0, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:227 +0x78

github.com/json-iterator/go.describeStruct(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x2d330)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:384 +0x8da

github.com/json-iterator/go.decoderOfStruct(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0xc00080c628, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc00082ea68, 0x20f8c00, 0xc000618780, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.decoderOfSlice(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105

github.com/json-iterator/go._createDecoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0xc0001e33e8, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff

github.com/json-iterator/go.createDecoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc0006f7650, 0x20f8b60, 0xc0002b1b80, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.describeStruct(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x2d330)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f

github.com/json-iterator/go.decoderOfStruct(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0xc00080c338, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc000803a40, 0x20f8c00, 0xc0004971a0, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.describeStruct(0xc00082f720, 0x20f8c00, 0xc000803800, 0x7f7b00009e51)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f

github.com/json-iterator/go.decoderOfStruct(0xc00082f720, 0x20f8c00, 0xc000803800, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc000823720, 0x20f8c00, 0xc000803800, 0xc00080c328, 0x1bfa120)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc000823720, 0x20f8c00, 0xc000803800, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc00082f720, 0x20f8c00, 0xc000803800, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.decoderOfSlice(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_slice.go:12 +0x105

github.com/json-iterator/go._createDecoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0xc00080c318, 0xc0001c2c60)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:180 +0x4ff

github.com/json-iterator/go.createDecoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc000803710, 0x20f8b60, 0xc00014ecc0, 0x1d2df08, 0x1)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.describeStruct(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x1ae42)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_extension.go:380 +0xa4f

github.com/json-iterator/go.decoderOfStruct(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect_struct_decoder.go:14 +0xb6

github.com/json-iterator/go._createDecoderOfType(0xc000823e60, 0x20f8c00, 0xc000802f60, 0xc00080c2c8, 0xc0001c2c60)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:176 +0x4aa

github.com/json-iterator/go.createDecoderOfType(0xc000823e60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:142 +0x10b

github.com/json-iterator/go.decoderOfType(0xc00082fe60, 0x20f8c00, 0xc000802f60, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:124 +0x78

github.com/json-iterator/go.(*frozenConfig).DecoderOf(0xc000175ea0, 0x20f8ac0, 0xc000802ea0, 0x20f8ac0, 0xc000802ea0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:114 +0x111

github.com/json-iterator/go.(*Iterator).ReadVal(0xc00018c990, 0x1c55340, 0xc0003c0460)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/reflect.go:72 +0x1ce

github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000175ea0, 0xc00067a600, 0xc0, 0x600, 0x1c55340, 0xc0003c0460, 0x0, 0x0)

/go/pkg/mod/github.com/json-iterator/go@v1.1.8/config.go:348 +0xd2

k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc00014ea40, 0xc00067a600, 0xc0, 0x600, 0x0, 0x2081f20, 0xc0003c0460, 0x1, 0x0, 0x206f080, ...)

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/serializer/json/json.go:275 +0x7d5

k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode(0x206e680, 0xc00014ea40, 0xc00067a600, 0xc0, 0x600, 0x0, 0x2081f20, 0xc0003c0460, 0xc000617100, 0xc000617000, ...)

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/runtime/helper.go:252 +0x97

k8s.io/client-go/rest.Result.Into(0xc00067a600, 0xc0, 0x600, 0xc00014a250, 0x10, 0x0, 0x0, 0xc8, 0x2070ec0, 0xc0002af7f0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:1133 +0xba

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc000488630, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00083a000, 0x0, 0x0, 0xffffffffffffffff, 0xc000830e70, 0x15cc0f5)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0xa5a

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*networkPolicyClient).List(0xc0006639b0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f240, 0x0, 0x0, 0xbfa4231b9311f897, 0x77f6c0d, 0x2f5ae40)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/networkpolicy.go:243 +0x4dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f240, 0x0, 0x0, 0x42, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116a20, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116a20, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116a20)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 145 [select]:

golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0000a1500, 0xc000616300, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1096 +0x9a0

golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0002b9500, 0xc000616300, 0xc000674000, 0x72db95, 0x0, 0xc000207200)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:448 +0x172

golang.org/x/net/http2.(*Transport).RoundTrip(...)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:409

golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc0002b9500, 0xc000616300, 0xc000140550, 0x5, 0xc000207288)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:2603 +0x3e

net/http.(*Transport).roundTrip(0xc0002b43c0, 0xc000616300, 0xc0004d0f10, 0xc0006743a8, 0x40000000040ef88)

/usr/local/go-cgo/src/net/http/transport.go:485 +0xdc1

net/http.(*Transport).RoundTrip(0xc0002b43c0, 0xc000616300, 0xc0002b43c0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/roundtrip.go:17 +0x35

net/http.send(0xc000616300, 0x206f100, 0xc0002b43c0, 0x0, 0x0, 0x0, 0xc000010098, 0xc0000502e0, 0x1, 0x0)

/usr/local/go-cgo/src/net/http/client.go:250 +0x443

net/http.(*Client).send(0xc0006637d0, 0xc000616300, 0x0, 0x0, 0x0, 0xc000010098, 0x0, 0x1, 0xc0004c5e60)

/usr/local/go-cgo/src/net/http/client.go:174 +0xfa

net/http.(*Client).do(0xc0006637d0, 0xc000616300, 0x0, 0x0, 0x0)

/usr/local/go-cgo/src/net/http/client.go:641 +0x3ce

net/http.(*Client).Do(0xc0006637d0, 0xc000616300, 0x0, 0x3, 0xc000140550)

/usr/local/go-cgo/src/net/http/client.go:509 +0x35

k8s.io/client-go/rest.(*Request).request(0xc0005e9380, 0xc000674948, 0x0, 0x0)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:754 +0x367

k8s.io/client-go/rest.(*Request).Do(0xc0005e9380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

/go/pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/rest/request.go:828 +0xd8

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*customK8sResourceClient).List(0xc0004886e0, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f280, 0x0, 0x0, 0xbfa4231b9336d048, 0x7a443d8, 0x2f5ae40)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/customresource.go:287 +0x9dc

github.com/projectcalico/libcalico-go/lib/backend/k8s.(*KubeClient).List(0xc000668b80, 0x20c15e0, 0xc00014f300, 0x208caa0, 0xc00014f280, 0x0, 0x0, 0x3e, 0x0, 0x0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/k8s.go:592 +0x2f9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).resyncAndCreateWatcher(0xc000116ab0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:203 +0x7fb

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116ab0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:81 +0xb9

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116ab0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 146 [runnable]:

fmt.glob..func1(0x2ed2a40, 0x3)

/usr/local/go-cgo/src/fmt/print.go:132 +0x2d

sync.(*Pool).Get(0x2ed2a40, 0x0, 0x0)

/usr/local/go-cgo/src/sync/pool.go:148 +0xa6

fmt.newPrinter(0x0)

/usr/local/go-cgo/src/fmt/print.go:137 +0x31

fmt.Sprint(0xc00077e920, 0x1, 0x1, 0x0, 0x0)

/usr/local/go-cgo/src/fmt/print.go:248 +0x26

github.com/sirupsen/logrus.(*Entry).Debug(0xc0004b35e0, 0xc00077e920, 0x1, 0x1)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:143 +0x7d

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116b40, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:80 +0x8e

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116b40)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 147 [select]:

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherCache).run(0xc000116bd0, 0x20c15e0, 0xc00014f300)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchercache.go:91 +0x211

github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run.func1(0x20c15e0, 0xc00014f300, 0xc0000953e0, 0xc000116bd0)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:141 +0x43

created by github.com/projectcalico/libcalico-go/lib/backend/watchersyncer.(*watcherSyncer).run

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/watchersyncer/watchersyncer.go:140 +0x14c


goroutine 149 [runnable]:

github.com/projectcalico/felix/daemon.servePrometheusMetrics(0xc0004bb800)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:622 +0x1c8

created by github.com/projectcalico/felix/daemon.Run

/go/src/github.com/projectcalico/felix/daemon/daemon.go:607 +0x2f68


goroutine 151 [chan receive]:

github.com/projectcalico/felix/logutils.RegisterProfilingSignalHandlers.func2(0xc000095aa0, 0xc0004bb800)

/go/src/github.com/projectcalico/felix/logutils/dump.go:106 +0x34

created by github.com/projectcalico/felix/logutils.RegisterProfilingSignalHandlers

/go/src/github.com/projectcalico/felix/logutils/dump.go:104 +0xdc


goroutine 60 [runnable]:

github.com/sirupsen/logrus.NewEntry(...)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/entry.go:52

github.com/sirupsen/logrus.(*Logger).newEntry(0xc0000d2120, 0xc0005f9cd8)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:83 +0x56

github.com/sirupsen/logrus.(*Logger).Info(0xc0000d2120, 0xc0005eefb0, 0x1, 0x1)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/logger.go:188 +0x41

github.com/sirupsen/logrus.Info(...)

/go/pkg/mod/github.com/projectcalico/logrus@v0.0.0-20180701205716-fc9bbf2f5799/exported.go:85

github.com/projectcalico/felix/daemon.monitorAndManageShutdown.func1(0x0, 0xc0001868a0)

/go/src/github.com/projectcalico/felix/daemon/daemon.go:658 +0x137

created by github.com/projectcalico/felix/daemon.monitorAndManageShutdown

/go/src/github.com/projectcalico/felix/daemon/daemon.go:656 +0x1a2


goroutine 63 [runnable]:

golang.org/x/net/http2.awaitRequestCancel(0xc000382c00, 0xc00048c5a0, 0x1, 0x0)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:288 +0x120

golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc0004ef080, 0xc000382c00)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:314 +0x40

created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0


goroutine 115 [runnable]:

golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc000280dc0, 0xc000616900)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313

created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0


goroutine 116 [runnable]:

k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000669240)

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/watch/streamwatcher.go:99

created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher

/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/watch/streamwatcher.go:71 +0xbe


goroutine 117 [select]:

github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.(*k8sWatcherConverter).processK8sEvents(0xc000154a00)

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/watcher.go:103 +0x5fb

created by github.com/projectcalico/libcalico-go/lib/backend/k8s/resources.newK8sWatcherConverterOneToMany

/go/pkg/mod/github.com/projectcalico/libcalico-go@v1.7.2-0.20200427180741-f197f7370140/lib/backend/k8s/resources/watcher.go:60 +0x181


goroutine 162 [runnable]:

golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc0002811e0, 0xc000616d00)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313

created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0


goroutine 163 [runnable]:

golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc000281e40, 0xc000617100)

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:313

created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse

/go/pkg/mod/golang.org/x/net@v0.0.0-20200202094626-16171245cfb2/http2/transport.go:1965 +0x6b0
@ianlancetaylor ianlancetaylor changed the title Panic: runtime: unexpected return pc for runtime.heapBitsSetType runtime: panic: unexpected return pc for runtime.heapBitsSetType May 4, 2020
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 4, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.15 milestone May 4, 2020
@ianlancetaylor
Copy link
Member

Is it feasible for you to test the current release (1.14.2)? Thanks.

@fasaxc
Copy link
Author

fasaxc commented May 5, 2020

Pretty easy to try but we've only seen this crash once so far; I don't have a reliable repro scenario. Any hints as to what might cause it?

@fasaxc
Copy link
Author

fasaxc commented May 7, 2020

Have done 5 or so runs on go v1.14 without repro yet. I did get another repro on 1.13.7 with the same panic but a different named function.

I checked how we use profiling and it looks like we don't enable by default; it's only enabled when sending a signal.

The area of code that seems to trigger the panic is deep in a library that's a transient dependency. Looks like we call through the kubernetes client to json-iter, which uses reflect2, which has a scary name that makes me wonder if it's doing something unsafe and trashing the stack somehow.

@ianlancetaylor
Copy link
Member

CC @aclements @mknyszek

@ianlancetaylor ianlancetaylor modified the milestones: Go1.15, Go1.16 Jun 26, 2020
@fasaxc
Copy link
Author

fasaxc commented Jun 26, 2020

I have been chipping away at this for a few weeks; I made a CI job that runs 20 hours of our FV tests, I get a failure in one in 2 runs. Roughly half the failures are like this one, the other half are nil pointer panics from some nearby code. I set up the job to dump a core file if it fails. The stack of the failing goroutine seems to be corrupted somehow. The leaf function's stack frame looks sane but one or two levels up dlv can't seem to read; then further up it can read again.

@ianlancetaylor
Copy link
Member

Since this looks like memory corruption, I just want to mention the usual comments: make sure to try the race detector (https://golang.org/doc/articles/race_detector.html) and take a close look at any of your code that uses unsafe or cgo to make sure that there aren't any memory errors.

I see that nobody answered your question from earlier:

Does that mean that this might be caused by a profiling interrupt during a GC? Does that refer to go's built-in profiling mechanisms; I think we run with memory profiling turned on by default since it's useful and low overhead but if it causes random panics then we may be forced to turn it off.

Memory profiling does not cause random panics. What that comment means is that there are two different cases to consider when we trace the stack. 1) We might be doing a backtrace because of a profiling interrupt. In that case we might fail to identify the function because the signal was delivered while executing C code or in some other unusual location. We just return with whatever stack backtrace we were able to collect. 2) We might be doing a backtrace in order to collect stack pointers during a garbage collection. In that case we know for sure that we are executing normal Go code, so a failure to identify a function indicates a serious error. So we crash.

You are seeing the latter case. It doesn't have anything to do with memory profiling. The problem is that your stack memory has been corrupted somehow.

@fasaxc
Copy link
Author

fasaxc commented Nov 23, 2020

I haven't seen a repro of this for some time (we moved to Go v1.15 a while back). Possible that it's been fixed or that the code has shuffled so as not to expose the issue.

@ianlancetaylor
Copy link
Member

Thanks for following up. Closing. Please comment if you see another occurrence.

@golang golang locked and limited conversation to collaborators Nov 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.16
Development

No branches or pull requests
3 participants
@fasaxc @ianlancetaylor @gopherbot