runtime: systemstack call caused SIGSEGV under race mode #45768

PureWhiteWu · 2021-04-26T07:41:41Z

Hi all,

I'm trying to get time using CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE for performance reasons, and need to use vdso call by hand-written assembly code. That is, I want to reimplement time.Now using CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE.

I referenced the code in runtime and found that there's an issue #20427 indicates that I need to switch to g0 for vdso calls.

I tried to link the runtime.systemstack and use it to do vdso calls:

//go:linkname systemstack runtime.systemstack
//go:noescape
func systemstack(fn func())

My code is something like this:

// now calls vdso and is implemented in asm
func now() (sec int64, nsec int32, mono int64)

func Now() {
  var sec, mono int64
  var nsec int32
  systemstack(func() {
    sec, nsec, mono = now()
  })
  ... // logic copied from time.Now()
}

The code runs well without -race, but I encountered fatal error during go test -race:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x418975]

runtime stack:
runtime.throw(0x6de30d, 0x2a)
	/usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:718 +0x2e5

goroutine 20 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:339 fp=0xc000040540 sp=0xc000040538 pc=0x4a3920
xxx/xxx.Now(0xc019b3c31d55505d, 0xca7404, 0x85d4a0)
	/xxx/time_linux_amd64.go:27 +0xb0 fp=0xc0000405e8 sp=0xc000040540 pc=0x6745b0
xxx/xxx.TestNowStd(0xc000082f00)
	/xxx/time_test.go:27 +0x85 fp=0xc0000406d0 sp=0xc0000405e8 pc=0x674dc5
testing.tRunner(0xc000082f00, 0x6e6040)
	/usr/local/go/src/testing/testing.go:1193 +0x203 fp=0xc0000407d0 sp=0xc0000406d0 pc=0x5518e3
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000407d8 sp=0xc0000407d0 pc=0x4a5761
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x5d8

I debuged and found that this error exists after the systemstack call and before calling my closure(which means this sigsegv is not caused by my code), so I wonder if this is a bug of the race detector or this is my misuse of systemstack?

gdb result Output

(gdb) b xxx/xxx.Now
Breakpoint 1 at 0x674500: file xxx/time_linux_amd64.go, line 24.
(gdb) r
Starting program: xxx.test 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffd09ee700 (LWP 18203)]
[New Thread 0x7fffcbfff700 (LWP 18204)]
[New Thread 0x7fffcb7fe700 (LWP 18205)]
[New Thread 0x7fffcaffd700 (LWP 18206)]
[New Thread 0x7fffca7fc700 (LWP 18207)]
Thread 1 "xxx.test" hit Breakpoint 1,xxx.Now (~r0=...)

at xxx/time_linux_amd64.go:24

24	func Now() time.Time {

(gdb) s

runtime.racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:179

179		MOVQ	callpc+0(FP), R11

(gdb) s

180		JMP	racefuncenter<>(SB)

(gdb) s

racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:185

185		MOVQ	DX, R15		// save function entry context (for closures)

(gdb) s

187		MOVQ	g(R12), R14

(gdb) s

188		MOVQ	g_racectx(R14), RARG0	// goroutine context

(gdb) s

189		MOVQ	R11, RARG1

(gdb) s

191		MOVQ	$__tsan_func_enter(SB), AX

(gdb) p __tsan_func_enter

$3 = {<text variable, no debug info>} 0x418960 <__tsan_func_enter>

(gdb) s

193		CALL	racecall<>(SB)

(gdb) p $rax

$4 = 4295008

(gdb) s

racecall () at /usr/local/go/src/runtime/race_amd64.s:403

403		MOVQ	g(R12), R14

(gdb) s

404		MOVQ	g_m(R14), R13

(gdb) s

406		MOVQ	SP, R12		// callee-saved, preserved across the CALL

(gdb) s

407		MOVQ	m_g0(R13), R10

(gdb) s

408		CMPQ	R10, R14

(gdb) s

409		JE	call	// already on g0

(gdb) s

410		MOVQ	(g_sched+gobuf_sp)(R10), SP

(gdb) s

racecall () at /usr/local/go/src/runtime/race_amd64.s:412

412		ANDQ	$~15, SP	// alignment for gcc ABI

(gdb) s

racecall () at /usr/local/go/src/runtime/race_amd64.s:413

413		CALL	AX

(gdb) s

414		MOVQ	R12, SP

(gdb) s

racecall () at /usr/local/go/src/runtime/race_amd64.s:415

415		RET

(gdb) s

racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:194

194		MOVQ	R15, DX	// restore function entry context

(gdb) s

racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:195

195		RET

(gdb) s

xxx.Now (~r0=...) at xxx/time_linux_amd64.go:25

25		var sec, mono int64

(gdb) s

26		var nsec int32

(gdb) s

27		systemstack(func() {

(gdb) s

runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:343

343		MOVQ	fn+0(FP), DI	// DI = fn

(gdb) s

345		MOVQ	g(CX), AX	// AX = g

(gdb) s

346		MOVQ	g_m(AX), BX	// BX = m

(gdb) s

348		CMPQ	AX, m_gsignal(BX)

(gdb) s

349		JEQ	noswitch

(gdb) s

351		MOVQ	m_g0(BX), DX	// DX = g0

(gdb) s

352		CMPQ	AX, DX

(gdb) s

353		JEQ	noswitch

(gdb) s

355		CMPQ	AX, m_curg(BX)

(gdb) s

356		JNE	bad

(gdb) s

361		MOVQ	$runtime·systemstack_switch(SB), SI

(gdb) s

362		MOVQ	SI, (g_sched+gobuf_pc)(AX)

(gdb) s

363		MOVQ	SP, (g_sched+gobuf_sp)(AX)

(gdb) s

364		MOVQ	AX, (g_sched+gobuf_g)(AX)

(gdb) s

365		MOVQ	BP, (g_sched+gobuf_bp)(AX)

(gdb) s

368		MOVQ	DX, g(CX)

(gdb) s

369		MOVQ	(g_sched+gobuf_sp)(DX), BX

(gdb) s

371		SUBQ	$8, BX

(gdb) s

372		MOVQ	$runtime·mstart(SB), DX

(gdb) s

373		MOVQ	DX, 0(BX)

(gdb) s

374		MOVQ	BX, SP

(gdb) s

runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:377

377		MOVQ	DI, DX

(gdb) s

378		MOVQ	0(DI), DI

(gdb) s

379		CALL	DI

(gdb) s

xxx.Now.func1 () at xxx/time_linux_amd64.go:27

27		systemstack(func() {

(gdb) s

runtime.racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:179

179		MOVQ	callpc+0(FP), R11

(gdb) s

180		JMP	racefuncenter<>(SB)

(gdb) s

racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:185

185		MOVQ	DX, R15		// save function entry context (for closures)

(gdb) s

187		MOVQ	g(R12), R14

(gdb) s

188		MOVQ	g_racectx(R14), RARG0	// goroutine context

(gdb) s

189		MOVQ	R11, RARG1

(gdb) s

191		MOVQ	$__tsan_func_enter(SB), AX

(gdb) p __tsan_func_enter

$7 = {<text variable, no debug info>} 0x418960 <__tsan_func_enter>

(gdb) s

193		CALL	racecall<>(SB)

(gdb) p $rax

$8 = 4295008

(gdb) s

racecall () at /usr/local/go/src/runtime/race_amd64.s:403

403		MOVQ	g(R12), R14

(gdb) s

404		MOVQ	g_m(R14), R13

(gdb) s

406		MOVQ	SP, R12		// callee-saved, preserved across the CALL

(gdb) s

407		MOVQ	m_g0(R13), R10

(gdb) s

408		CMPQ	R10, R14

(gdb) s

409		JE	call	// already on g0

(gdb) s

412		ANDQ	$~15, SP	// alignment for gcc ABI

(gdb) s

413		CALL	AX

(gdb) p __tsan_func_enter

$9 = {<text variable, no debug info>} 0x418960 <__tsan_func_enter>

(gdb) p $rax

$10 = 4295008

(gdb) s
Thread 1 "timex.test" received signal SIGSEGV, Segmentation fault.

0x0000000000418975 in __tsan_func_enter ()

(gdb) bt

#0  0x0000000000418975 in __tsan_func_enter ()

#1  0x000000c000082a80 in ?? ()

#2  0x000000000085d760 in runtime.sched ()

#3  0x00007fffffffde00 in ?? ()

#4  0x00007fffffffddf0 in ?? ()

#5  0x00007fffffffdd90 in ?? ()

#6  0x00000000004a6f03 in racecall () at /usr/local/go/src/runtime/race_amd64.s:413

#7  0x00000000004a6e22 in racefuncenter () at /usr/local/go/src/runtime/race_amd64.s:193

#8  0x0000000000675554 in xxx/xxx.Now.func1 () at xxx/time_linux_amd64.go:27

#9  0x00000000004a39a6 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:379

#10 0x00000000004707c0 in ?? () at /usr/local/go/src/runtime/signal_unix.go:1035

#11 0x00000000004a37f4 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:226

#12 0x0000000000000000 in ?? ()

What version of Go are you using (`go version`)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/purewhite/.cache/go-build"
GOENV="/home/purewhite/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/purewhite/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/purewhite/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/xxx/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1995173489=/tmp/go-build -gno-record-gcc-switches"

The text was updated successfully, but these errors were encountered:

seankhliao · 2021-04-26T07:46:25Z

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

PureWhiteWu · 2021-04-26T08:37:25Z

@seankhliao hello, I've changed to title and I think this is a (maybe) bug report now. Could you please reopen the issue?

PureWhiteWu · 2021-04-26T08:42:39Z

@davecheney Thanks very much!

cherrymui · 2021-04-26T16:24:36Z

I tried to link the runtime.systemstack

runtime.systemstack is a runtime internal function. It is not supposed to be called outside of the runtime. Your use case is not supported. Thanks.

PureWhiteWu · 2021-04-27T03:27:00Z

@cherrymui
Hello, thanks for your reply!
I know this is not supposed to be called outside of the runtime, but I just want to know if this is a bug of the race detector or a by-design behaviour.
If this is a by-design behaviour, than I will go and find another way to get over this.
Thanks again!

ianlancetaylor · 2021-04-27T04:02:26Z

Calling systemstack outside of the runtime package is not supported. I don't know the exact reason why it fails, and finding out that exact reason is of purely academic interest. If that reason is fixed today, some new reason will appear tomorrow. We are never going to support calling systemstack outside of the runtime package, and that is by design.

Sorry this is not helpful, but I believe that it is accurate.

seankhliao closed this as completed Apr 26, 2021

PureWhiteWu changed the title ~~question: How to do vdso calls in my own code?~~ runtime: systemstack call caused SIGSEGV under race mode Apr 26, 2021

davecheney reopened this Apr 26, 2021

cherrymui closed this as completed Apr 26, 2021

golang locked and limited conversation to collaborators Apr 27, 2022

gopherbot added the FrozenDueToAge label Apr 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: systemstack call caused SIGSEGV under race mode #45768

runtime: systemstack call caused SIGSEGV under race mode #45768

PureWhiteWu commented Apr 26, 2021 •

edited

seankhliao commented Apr 26, 2021

PureWhiteWu commented Apr 26, 2021

PureWhiteWu commented Apr 26, 2021

cherrymui commented Apr 26, 2021

PureWhiteWu commented Apr 27, 2021

ianlancetaylor commented Apr 27, 2021

runtime: systemstack call caused SIGSEGV under race mode #45768

runtime: systemstack call caused SIGSEGV under race mode #45768

Comments

PureWhiteWu commented Apr 26, 2021 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

seankhliao commented Apr 26, 2021

PureWhiteWu commented Apr 26, 2021

PureWhiteWu commented Apr 26, 2021

cherrymui commented Apr 26, 2021

PureWhiteWu commented Apr 27, 2021

ianlancetaylor commented Apr 27, 2021

PureWhiteWu commented Apr 26, 2021 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?