Skip to content

runtime: stack split at bad time for programs built with go1.23rc2 with optimizations disabled on linux/386 #68525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aarzilli opened this issue Jul 19, 2024 · 6 comments · Fixed by ferrmin/go#771
Closed

runtime: stack split at bad time for programs built with go1.23rc2 with optimizations disabled on linux/386 #68525

aarzilli opened this issue Jul 19, 2024 · 6 comments · Fixed by ferrmin/go#771
Assignees
@cuonglm
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.23

Comments

@aarzilli
Copy link
Contributor 

$ go version
go version go1.23rc2 linux/386

How to reproduce:

  • build anything that calls fmt.Println (for example buildtest) on linux/386 with go 1.23rc2 passing -gcflags='all=-N -l'
  • run the executable
  • example crash:
runtime: newstack at runtime.lockWithRankMayAcquire+0x18 sp=0x9c64bf4 stack=[0x9c64000, 0x9c65000]
        morebuf={pc:0x8087426 sp:0x9c64bf8 lr:0x0}
        sched={pc:0x8054688 sp:0x9c64bf4 lr:0x0 ctxt:0x0}
runtime.casgstatus(0x9c06128, 0x2, 0x3)
        /home/d/go-tip/src/runtime/proc.go:1185 +0x56 fp=0x9c64c30 sp=0x9c64bf8 pc=0x8087426
runtime.reentersyscall(0x80e5c70, 0x9c64c74, 0x0)
        /home/d/go-tip/src/runtime/proc.go:4410 +0x89 fp=0x9c64c64 sp=0x9c64c30 pc=0x808da19
runtime.entersyscall()
        /home/d/go-tip/src/runtime/proc.go:4479 +0x23 fp=0x9c64c74 sp=0x9c64c64 pc=0x80b9693
syscall.Syscall(0x4, 0x1, 0x9c120d0, 0xd)
        /home/d/go-tip/src/syscall/syscall_linux.go:73 +0x20 fp=0x9c64cb8 sp=0x9c64c74 pc=0x80e5c70
syscall.write(0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/syscall/zsyscall_linux_386.go:964 +0x96 fp=0x9c64d04 sp=0x9c64cb8 pc=0x80e57d6
syscall.Write(0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/syscall/syscall_unix.go:211 +0x65 fp=0x9c64d3c sp=0x9c64d04 pc=0x80e5395
internal/poll.ignoringEINTRIO(0x8117ae8, 0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/internal/poll/fd_unix.go:745 +0x6c fp=0x9c64d88 sp=0x9c64d3c pc=0x80e8a6c
internal/poll.(*FD).Write(0x9c460c0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/internal/poll/fd_unix.go:381 +0x299 fp=0x9c64e80 sp=0x9c64d88 pc=0x80e8689
os.(*File).write(0x9c142f0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/os/file_posix.go:46 +0x6f fp=0x9c64ec0 sp=0x9c64e80 pc=0x80e9e5f
os.(*File).Write(0x9c142f0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/os/file.go:195 +0xad fp=0x9c64f08 sp=0x9c64ec0 pc=0x80e9a5d
fmt.Fprintln({0x811d6d4, 0x9c142f0}, {0x9c64fb4, 0x1, 0x1})
        /home/d/go-tip/src/fmt/print.go:305 +0xa5 fp=0x9c64f50 sp=0x9c64f08 pc=0x80ee995
fmt.Println({0x9c64fb4, 0x1, 0x1})
        /home/d/go-tip/src/fmt/print.go:314 +0x71 fp=0x9c64f8c sp=0x9c64f50 pc=0x80eea61
main.main()
        /home/d/delve/_fixtures/buildtest/main.go:6 +0x8a fp=0x9c64fc0 sp=0x9c64f8c pc=0x80f399a
runtime.main()
        /home/d/go-tip/src/runtime/proc.go:272 +0x265 fp=0x9c64ff0 sp=0x9c64fc0 pc=0x80851a5
runtime.goexit({})
        /home/d/go-tip/src/runtime/asm_386.s:1393 +0x1 fp=0x9c64ff4 sp=0x9c64ff0 pc=0x80be5b1
fatal error: runtime: stack split at bad time
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 19, 2024
@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@cuonglm cuonglm self-assigned this Jul 19, 2024
@cuonglm cuonglm added the NeedsFix The path to resolution is known, but the work has not been done. label Jul 19, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/599675 mentions this issue: runtime: mark lockWithRankMayAcquire as nosplit

@cuonglm
Copy link
Member

cuonglm commented Jul 19, 2024 •
edited
Loading

git bisect points to https://go-review.googlesource.com/c/go/+/571056

The CL call lockWithRankMayAcquire during casgstatus, which is marked as nosplit. Normally, it's not problem because this function is empty when lockrank is off. However, when inlining is disable, the function is not inlined anymore, and requiring newstack when calling it -> crashing the runtime.

@rhysh
Copy link
Contributor

rhysh commented Jul 19, 2024

Thanks for the report and the fix. I agree with the fix: a nosplit annotation is required there.

It's a big strange to me that GOARCH=386 behaves differently here than GOARCH=amd64. Building with -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' to see if there are other problems to shake out, I see that GOARCH=amd64 and GOARCH=arm64 get trivial implementations of runtime.lockWithRankMayAcquire (nothing more than RET) but that GOARCH=386 has the usual stack growth function preamble.

Lack of nosplit annotation was a bug, the fix is right, what remains is to see if there's anything to be learned from why the bug was hidden except on GOARCH=386.

$ ./bin/go version
go version go1.23rc2 linux/amd64

$ cat /tmp/hello.go 
// run -gcflags='all=-N -l'

// Copyright 2024 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import "os"

func main() {
	os.OpenFile(os.DevNull, os.O_WRONLY, 0)
}

$ GOOS=linux GOARCH=arm64 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:46	0x1c680			d65f03c0		RET			
  lockrank_off.go:46	0x1c684			00000000		?			
  lockrank_off.go:46	0x1c688			00000000		?			
  lockrank_off.go:46	0x1c68c			00000000		?

$ GOOS=linux GOARCH=amd64 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:46	0x40cca0		c3			RET		

$ GOOS=linux GOARCH=386 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:45	0x8053d90		e87b1e0600		CALL runtime.mayMoreStackPreempt(SB)	
  lockrank_off.go:45	0x8053d95		658b0d00000000		MOVL GS:0, CX				
  lockrank_off.go:45	0x8053d9c		8b89fcffffff		MOVL 0xfffffffc(CX), CX			
  lockrank_off.go:45	0x8053da2		3b6108			CMPL SP, 0x8(CX)			
  lockrank_off.go:45	0x8053da5		7601			JBE 0x8053da8				
  lockrank_off.go:46	0x8053da7		c3			RET					
  lockrank_off.go:45	0x8053da8		e8039f0600		CALL runtime.morestack_noctxt(SB)	
  lockrank_off.go:45	0x8053dad		ebe6			JMP 0x8053d95

@ferrmin ferrmin mentioned this issue Jul 20, 2024
Merged
@cuonglm
Copy link
Member

cuonglm commented Jul 20, 2024

Lack of nosplit annotation was a bug, the fix is right, what remains is to see if there's anything to be learned from why the bug was hidden except on GOARCH=386.

Not only 386, this also happens for other platforms where reg ABI is not supported. On these platforms, we need to setup stack for function arguments:

func lockWithRankMayAcquire(l *mutex, rank lockRank) {}

@dmitshur dmitshur added this to the Go1.23 milestone Jul 22, 2024
@rhysh
Copy link
Contributor

rhysh commented Jul 22, 2024

So that's part of how regabi (or lack of it) looks, got it. Thanks @cuonglm .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cuonglm cuonglm

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.23
Development

Successfully merging a pull request may close this issue.

runtime: mark lockWithRankMayAcquire as nosplit golang/go
6 participants
@rhysh @aarzilli @dmitshur @cuonglm @gopherbot @gabyhelp