Go version

go1.16.15

Output of go env in your module/workspace:

we used official etcd version 3.5.3
https://github.com/etcd-io/etcd/releases/download/v3.5.3/etcd-v3.5.3-linux-amd64.tar.gz

Do not find the go env info.

What did you do?

One of the ETCD server(let's say pod-1) ran into crash loopback for almost 3 days. Then we watched pod-0 fires panic.
I did not want to say there is relationship between the crash loopback and panic. But that was what i observed.

What did you see happen?

2024-03-09T01:22:42.174324508Z
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc005b31d38, fp:0xc005b31d58} stack=[0xc005b2a000,0xc005b32000)
000000c005b31c38: 0000000000000000 0000000000000000
000000c005b31c48: 0000000000000000 0000000000000000
000000c005b31c58: 0000000000000000 0000000000000000
000000c005b31c68: 0000000000000000 0000000000000000
000000c005b31c78: 0000000000000000 0000000000000000
000000c005b31c88: 0000000000000000 0000000000000000
000000c005b31c98: 0000000000000000 0000000000000000
000000c005b31ca8: 0000000000000000 0000000000000000
000000c005b31cb8: 0000000000000000 0000000000000000
000000c005b31cc8: 0000000000000000 0000000000000000
000000c005b31cd8: 0000000000000000 0000000000000000
000000c005b31ce8: 0000000000000000 0000000000000000
000000c005b31cf8: 0000000000000000 0000000000000000
000000c005b31d08: 0000000000000000 0000000000000000
000000c005b31d18: 0000000000000000 0000000000000000
000000c005b31d28: 0000000000000000 0000000000000000
000000c005b31d38: \u003c4040404040404040 4040404040404040
000000c005b31d48: 0000000000000000 !0000000000000000
000000c005b31d58: \u003e0000000000000000 2525252525252525
000000c005b31d68: 2525252525252525 0000000000000000
000000c005b31d78: 0000000000000000 0000000000000000
000000c005b31d88: 0000000000000000 0000000000000000
000000c005b31d98: 0000000000000000 ffffff0000000000
000000c005b31da8: 0000000000000000 0000000000000000
000000c005b31db8: 0000000000000000 0000000000000001
000000c005b31dc8: 0000000000000001 0000000000000002
000000c005b31dd8: 0000000000000002 0000000000000011
000000c005b31de8: 000000c002fd7e48 000000c004d22040
000000c005b31df8: 000000c003d82a80 000000c0024d8000
000000c005b31e08: 000000c0024ea120 000000c0024ea120
000000c005b31e18: 000000c0024ea120 0000000000000000
000000c005b31e28: 000000c003d82b08 000000c002fd7ed4
000000c005b31e38: 000000c003d82bb8 000000c002fd7ed4
000000c005b31e48: 0000000000000002 0000000000000002
fatal error: unknown caller pc

2024-03-09T01:22:42.178142044Z
runtime stack:
runtime.throw(0x1213bd8, 0x11)
\u0009/go/gos/go1.16.15/src/runtime/panic.go:1117 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc003d82a80, 0x0, 0x0, 0x7fffffff, 0x12695e0, 0xc0009afaf8, 0x0, ...)
\u0009/go/gos/go1.16.15/src/runtime/traceback.go:261 +0x1a56
runtime.copystack(0xc003d82a80, 0x8000)
\u0009/go/gos/go1.16.15/src/runtime/stack.go:908 +0x2f1
runtime.shrinkstack(0xc003d82a80)
\u0009/go/gos/go1.16.15/src/runtime/stack.go:1180 +0x13d
runtime.scanstack(0xc003d82a80, 0xc00006fe98)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:720 +0x58e
runtime.markroot.func1()
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:233 +0xc6
runtime.markroot(0xc00006fe98, 0x1ca)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:206 +0x33e
runtime.gcDrain(0xc00006fe98, 0x7)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:1014 +0x118
runtime.gcBgMarkWorker.func2()
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:2003 +0x17e
runtime.systemstack(0xc000644000)
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
\u0009/go/gos/go1.16.15/src/runtime/proc.go:1246

2024-03-09T01:22:42.178333233Z
goroutine 74 [GC worker (idle)]:
runtime.systemstack_switch()
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:339 fp=0xc000612f60 sp=0xc000612f58 pc=0x4705a0
runtime.gcBgMarkWorker()
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:1967 +0x1c7 fp=0xc000612fe0 sp=0xc000612f60 pc=0x41dde7
runtime.goexit()
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000612fe8 sp=0xc000612fe0 pc=0x4722a1
created by runtime.gcBgMarkStartWorkers
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:1835 +0x37

2024-03-09T01:22:42.178404246Z
goroutine 1 [select, 10822 minutes]:
go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2(0xc000226000, 0x15, 0x16)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:216 +0x1472
go.etcd.io/etcd/server/v3/etcdmain.Main(0xc000226000, 0x15, 0x16)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40 +0x13f
main.main()
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32 +0x45

2024-03-09T01:22:42.178600358Z
goroutine 601 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0003a6460, 0x1, 0x0, 0x0, 0x0, 0x0)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/controlbuf.go:407 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00289a0c0, 0x0, 0x0)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/controlbuf.go:527 +0x1dd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000521380)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/http2_server.go:292 +0xd7
created by google.golang.org/grpc/internal/transport.newHTTP2Server
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/http2_server.go:289 +0x1110

2024-03-09T01:22:42.178649053Z
goroutine 370 [chan receive, 10822 minutes]:
go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamHandler).ServeHTTP(0xc000670bc0, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/api/rafthttp/http.go:451 +0x1a4e
net/http.(*ServeMux).ServeHTTP(0xc000670c40, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2429 +0x1ad
net/http.(*ServeMux).ServeHTTP(0xc000670c80, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2429 +0x1ad
go.etcd.io/etcd/server/v3/embed.grpcHandlerFunc.func2(0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/embed/serve.go:210 +0x5c
net/http.HandlerFunc.ServeHTTP(0xc000c81d80, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2050 +0x44
net/http.serverHandler.ServeHTTP(0xc0001541c0, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2868 +0xa3
net/http.(*conn).serve(0xc0002fa000, 0x13a1cb8, 0xc0015f27c0)
\u0009/go/gos/go1.16.15/src/net/http/server.go:1933 +0x8cd
created by net/http.(*Server).Serve
\u0009/go/gos/go1.16.15/src/net/http/server.go:2994 +0x39b

What did you expect to see?

Can anybody help explain what is happening here, and do we already fixed this kind of problem？ i did a search and found that there is several reports similar to this one, but not exactly the same.