Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: Panics and segmentation violation (Go 1.19.8) #61552

Closed
yznima opened this issue Jul 24, 2023 · 3 comments
Closed

runtime: Panics and segmentation violation (Go 1.19.8) #61552

yznima opened this issue Jul 24, 2023 · 3 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@yznima
Copy link

yznima commented Jul 24, 2023 •
edited
Loading

What version of Go are you using (go version)?

$ go version

go version go1.19.8 linux/amd64

Does this issue reproduce with the latest release?

Not sure, we’re not able to reproduce reliably yet but working on t

What operating system and processor architecture are you using (go env)?

Build Envs:

GOOS=linux
GOARCH=amd64
CGO_ENABLED=0

Build flags:

-gcflags -extldflags -ldflags -static -X OtherApplicationFlags=FOOBAR

Go env:

GO111MODULE=""
GOARCH="amd64"
GOBIN="/go/bin"
GOCACHE="/home/****/.cache/go-build"
GOENV="/home/****/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.8"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1501434825=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have recently upgraded go from 1.18.6 to 1.19.8. We run Go based services that are long running and don't restart.

What did you expect to see?

We didn’t expect to see a panic.

What did you see instead?

We saw system panics from the go standard library without any good reason. We have noticed periodical segfaults within the standard library and runtime. These panics are not frequent and only happen about once a day on a running service. We aren't able to pinpoint what is the root cause. Here a few stack traces indicate that there is a segfault.

[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4064de]

goroutine 4454508 [running]:
runtime.throw({0x1104295?, 0x7fcb0967e740?})
    /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc0021c1ba0 sp=0xc0021c1b70 pc=0x439f1d
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0xc0021c1bf0 sp=0xc0021c1ba0 pc=0x44fce9
runtime.(*waitq).dequeue(...)
    /usr/local/go/src/runtime/chan.go:776
runtime.closechan(0xc003b94780)
    /usr/local/go/src/runtime/chan.go:380 +0x15e fp=0xc0021c1c30 sp=0xc0021c1bf0 pc=0x4064de
net/http.(*persistConn).closeLocked(0xc004852d80, {0x126bba0, 0xc00010aca0})
    /usr/local/go/src/net/http/transport.go:2716 +0x145 fp=0xc0021c1cb8 sp=0xc0021c1c30 pc=0x7308c5
net/http.(*persistConn).close(0xc0041cc2f8?, {0x126bba0?, 0xc00010aca0?})
    /usr/local/go/src/net/http/transport.go:2699 +0xb0 fp=0xc0021c1d10 sp=0xc0021c1cb8 pc=0x7306b0
net/http.(*persistConn).readLoop.func1()
    /usr/local/go/src/net/http/transport.go:2061 +0x2d fp=0xc0021c1d40 sp=0xc0021c1d10 pc=0x72e4ad
net/http.(*persistConn).readLoop(0xc004852d80)
    /usr/local/go/src/net/http/transport.go:2236 +0xab6 fp=0xc0021c1fc8 sp=0xc0021c1d40 pc=0x72dc76
net/http.(*Transport).dialConn.func5()
    /usr/local/go/src/net/http/transport.go:1751 +0x26 fp=0xc0021c1fe0 sp=0xc0021c1fc8 pc=0x72bb46
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0021c1fe8 sp=0xc0021c1fe0 pc=0x46aea1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1751 +0x173e

unexpected fault address 0x39e0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x39e0 pc=0x46c649]

goroutine 995 [running]:
runtime.throw({0x105b607?, 0xc001d6a800?})
    /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000f2c7a0 sp=0xc000f2c770 pc=0x439fbd
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:846 +0x2c5 fp=0xc000f2c7f0 sp=0xc000f2c7a0 pc=0x4500e5
runtime.memmove()
    /usr/local/go/src/runtime/memmove_amd64.s:339 +0x3c9 fp=0xc000f2c7f8 sp=0xc000f2c7f0 pc=0x46c649
runtime.typedslicecopy(0xf55900?, 0xc001d6a800, 0x40?, 0x2e60, 0x1be91b9bd6a5e?)
    /usr/local/go/src/runtime/mbarrier.go:294 +0xa6 fp=0xc000f2c838 sp=0xc000f2c7f8 pc=0x414386
github.com/uber/jaeger-client-go.(*Span).FinishWithOptions(0xc000f5ad80, {{0x2e60, 0x2e60, 0x40}, {0x2e60, 0x40, 0x2e60}, {0x40, 0x2e60, 0x40}})

This is the code that is causing the second issue in the FinishWithOptions function

			// Note: bulk logs are not subject to maxLogsPerSpan limit
			if options.LogRecords != nil {
			>>> 	s.logs = append(s.logs, options.LogRecords...)
			}
@bcmills bcmills changed the title affected/package: runtime: Panics and segmentation violation (Go 1.19.8) runtime: Panics and segmentation violation (Go 1.19.8) Jul 24, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 24, 2023
@ianlancetaylor
Copy link
Member

At first glance this looks like memory corruption. Have you tried running your code under the race detector?

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 25, 2023
@yznima
Copy link
Author

yznima commented Jul 25, 2023 •
edited
Loading

@ianlancetaylor We don't run our production code with the race detector but I can give it a try. I'll get back to you with any new information. Thank you

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jul 25, 2023
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 25, 2023
@yznima yznima mentioned this issue Nov 8, 2023
Open
@golang golang locked and limited conversation to collaborators Aug 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ianlancetaylor @gopherbot @seankhliao @yznima