Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: spurious crashes related to timers in go 1.23.0 #69625

Closed
fiber opened this issue Sep 25, 2024 · 7 comments
Closed

runtime: spurious crashes related to timers in go 1.23.0 #69625

fiber opened this issue Sep 25, 2024 · 7 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@fiber
Copy link

fiber commented Sep 25, 2024 •
edited by prattmic
Loading

Go version

go version go1.23.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/gocode/src/go.gtt.net/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3222024183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

recompiled go code that has been running fine for a long time.

What did you see happen?

we are experiencing frequent crashes that follow this pattern:

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad

What did you expect to see?

no crash.
@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@seankhliao
Copy link
Member

are you copying timer values and not pointers?

we need a reproducer for this to be actionable

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 25, 2024
@seankhliao seankhliao changed the title runtime/time.go: spurious crashes related to timers in go 1.23.0 time: spurious crashes related to timers in go 1.23.0 Sep 25, 2024
@fiber
Copy link
Author

fiber commented Sep 25, 2024 •
edited by prattmic
Loading

This is a sizeable piece of code. I would not exclude bugs, however it's unlikely that we I am copying timer values, since we're using *time.Timer/*time.Ticker everywhere.

the below crashes may be related, although I don't understand how.
I am rolling back to 1.22.7 to check

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad
...


unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 72153 gp=0xc004e9bc00 m=518 mp=0xc013509508 [running]:
runtime.throw({0x1d105d2?, 0xc0514c19c9?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc0076ca970 sp=0xc0076ca940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc0076ca9d0 sp=0xc0076ca970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc0514c19c8?}, 0x9b?, {0x80?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc0076ca9d8 sp=0xc0076ca9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262150, {0x1d659d2?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076caa30 sp=0xc0076ca9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076caa68 sp=0xc0076caa30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc004e9bc00?}, 0xc00c9195d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc0076caab8 sp=0xc0076caa68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262298, {0xc0514c1950?}, {0x89?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076cab10 sp=0xc0076caab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x92?, {0x92?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076cab48 sp=0xc0076cab10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x5d7?, {{}, {0x22d52e0?, 0xc049ec4000?}, 0x84?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc0076cab70 sp=0xc0076cab48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22d52e0?, 0xc049ec4000?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc0076caba0 sp=0xc0076cab70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xbc?, 0xe9?}, {0x0, 0x0, 0x0}, {0x22d52e0, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc0076cac70 sp=0xc0076caba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x22b1580, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc0076cacc0 sp=0xc0076cac70 pc=0x8a7c90



unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 63842 gp=0xc0025eae00 m=312 mp=0xc012afdc08 [running]:
runtime.throw({0x1d00572?, 0xc01c07046f?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc000e1b970 sp=0xc000e1b940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc000e1b9d0 sp=0xc000e1b970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc01c07046e?}, 0x14b?, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc000e1b9d8 sp=0xc000e1b9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0150, {0x1d55878?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1ba30 sp=0xc000e1b9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1ba68 sp=0xc000e1ba30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0025eae00?}, 0xc0058300d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc000e1bab8 sp=0xc000e1ba68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0298, {0xc01c070300?}, {0x7f?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1bb10 sp=0xc000e1bab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x188?, {0x188?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1bb48 sp=0xc000e1bb10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x400?, {{}, {0x22c3760?, 0xc018dc5930?}, 0xc?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc000e1bb70 sp=0xc000e1bb48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22c3760?, 0xc018dc5930?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc000e1bba0 sp=0xc000e1bb70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xae?, 0x5e?}, {0x0, 0x0, 0x0}, {0x22c3760, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc000e1bc70 sp=0xc000e1bba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x229fb40, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc000e1bcc0 sp=0xc000e1bc70 pc=0x8a7c90

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 25, 2024
@ianlancetaylor ianlancetaylor changed the title time: spurious crashes related to timers in go 1.23.0 runtime: spurious crashes related to timers in go 1.23.0 Sep 25, 2024
@ianlancetaylor
Copy link
Member

CC @golang/runtime

A fault address like 0x30000000009 looks like memory corruption. It's not necessarily in code related to timers; that may just be where it is showing up. Look closely at any uses of unsafe or cgo.

@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 25, 2024
@ftl
Copy link

ftl commented Feb 23, 2025

I think I also encountered this bug with Go version 1.23.6 on linux/amd when generating audio playback. I built https://github.com/ftl/faultdemo as demonstrator. The panic occurs after a random time, usually under one minute. Please be patient (and turn down your volume).

Here is the top of the stacktrace, you can find the full trace with the demonstrator.

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x456652]

goroutine 102 gp=0xc000203500 m=10 mp=0xc000181c08 [running]:
runtime.throw({0x532f15?, 0x0?})
	/usr/lib/go/src/runtime/panic.go:1067 +0x48 fp=0xc0000ebc30 sp=0xc0000ebc00 pc=0x46b4c8
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:931 +0x26c fp=0xc0000ebc90 sp=0xc0000ebc30 pc=0x46ccec
runtime.(*timer).maybeRunChan(0xc0001af620?)
	/usr/lib/go/src/runtime/time.go:1300 +0x12 fp=0xc0000ebcc0 sp=0xc0000ebc90 pc=0x456652
runtime.selectgo(0xc0000ebe20, 0xc0000ebe18, 0x0?, 0x0, 0x0?, 0x0)
	/usr/lib/go/src/runtime/select.go:177 +0x13f fp=0xc0000ebde8 sp=0xc0000ebcc0 pc=0x449cff
github.com/ftl/digimodes/cw.(*Modulator).nextAction(0x53e558, 0x4025f0b17e4b78dd)
	/home/florian/go/pkg/mod/github.com/ftl/digimodes@v0.1.2/cw/mod.go:191 +0x6b fp=0xc0000ebe70 sp=0xc0000ebde8 pc=0x4ba04b
github.com/ftl/digimodes/cw.(*Modulator).Modulate(0x53e558, 0x4025f0b17e4b78dd, 0xc00003a008?, 0xc0001af728?, 0x0)
	/home/florian/go/pkg/mod/github.com/ftl/digimodes@v0.1.2/cw/mod.go:178 +0x76 fp=0xc0000ebe98 sp=0xc0000ebe70 pc=0x4b9f36
github.com/ftl/patrix/osc.(*Oscillator).Synth32(0xc0001aa000, {0xc0002d6000, 0x800, 0x0?})
	/home/florian/go/pkg/mod/github.com/ftl/patrix@v0.2.0/osc/oscillator.go:51 +0xfb fp=0xc0000ebed8 sp=0xc0000ebe98 pc=0x4f04fb
github.com/ftl/patrix/osc.(*Oscillator).Synth32-fm({0xc0002d6000?, 0xc000203500?, 0x4e89bd?})
	<autogenerated>:1 +0x31 fp=0xc0000ebf08 sp=0xc0000ebed8 pc=0x4f0c51
github.com/jfreymuth/pulse.Float32Reader.Read(0xc0002185b0?, {0xc0002d6000?, 0xc000306000?, 0x2000?})
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/format.go:91 +0x5b fp=0xc0000ebf60 sp=0xc0000ebf08 pc=0x4ef47b
github.com/jfreymuth/pulse.(*PlaybackStream).run(0xc000203180)
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:88 +0x12b fp=0xc0000ebfc8 sp=0xc0000ebf60 pc=0x4efbcb
github.com/jfreymuth/pulse.(*Client).NewPlayback.gowrap1()
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:77 +0x25 fp=0xc0000ebfe0 sp=0xc0000ebfc8 pc=0x4efa65
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000ebfe8 sp=0xc0000ebfe0 pc=0x4728e1
created by github.com/jfreymuth/pulse.(*Client).NewPlayback in goroutine 1
	/home/florian/go/pkg/mod/github.com/jfreymuth/pulse@v0.1.1/playback.go:77 +0x436

@ianlancetaylor
Copy link
Member

@ftl Your program has data races. Build it with go build -race and run it. You will see race detector reports. Fix those, and then see if the problem still occurs. See https://go.dev/doc/articles/race_detector.

@gabyhelp gabyhelp mentioned this issue Feb 28, 2025
Closed
@gabyhelp gabyhelp mentioned this issue Mar 17, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fiber @ftl @ianlancetaylor @gopherbot @seankhliao @gabyhelp