Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/trace: using cgo introduces goroutine with empty stack #61037

Open
dominikh opened this issue Jun 27, 2023 · 4 comments
Open

runtime/trace: using cgo introduces goroutine with empty stack #61037

dominikh opened this issue Jun 27, 2023 · 4 comments
Assignees
@mknyszek
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone
Backlog

Comments

@dominikh
Copy link
Member

What version of Go are you using (go version)?

$ go version
go version devel go1.21-a031f4ef83 Sat Jun 24 05:24:25 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dominikh/.cache/go-build'
GOENV='/home/dominikh/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dominikh/prj/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dominikh/prj'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/dominikh/prj/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/dominikh/prj/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.21-a031f4ef83 Sat Jun 24 05:24:25 2023 +0000'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/dominikh/prj/src/example.com/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1830765016=/tmp/go-build -gno-record-gcc-switches'

What did you do?

package main

import "C"

import (
	"os"
	"runtime/trace"
)

func main() {
	trace.Start(os.Stdout)
	trace.Stop()
}

What did you see?

The produced trace contains the following two events:

420 GoCreate p=0 g=0 off=30 g=17 stack=3
456 GoInSyscall p=0 g=17 off=36 g=17

The EvStack for stack 3 contains a single frame, with all values set to zero.

The goroutine seems to be so special that its stack doesn't get printed when sending SIGQUIT to the process.

I've only observed this when using cgo.

/cc @mknyszek
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 27, 2023
@nsrip-dd
Copy link
Contributor

I have noticed this as well. From my investigation, this appears to be an extra goroutine created for cgo programs in anticipation of C-to-Go calls. It's created here in oneNewExtraM with no starting PC. The goroutine will get used in programs where C-created threads call Go code. The creation and InSyscall events get emitted here at the beginning of tracing with a stack with just PC 0 (added in CL 429858) and we see more events for that goroutine if a C-created thread calls back into Go. The goroutine is missing when printing all stacks because its status starts out as _Gdead, so it gets skipped.

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 28, 2023
@cagedmantis cagedmantis added this to the Backlog milestone Jun 28, 2023
@cagedmantis
Copy link
Contributor

/cc @golang/runtime

@prattmic
Copy link
Member

I believe that @nsrip-dd is correct, this is the G from oneNewExtraM, which will be used when executing cgo callbacks that come from a non-Go thread.

This is perhaps an unfortunate internal detail to expose, but I'm not sure we can/should change the trace output? In theory I suppose it would make more sense to delay the GoCreate until a cgo callback actually occurs, but should we do that on every callback that reuses the same G or only the first call?

@mknyszek
Copy link
Contributor

@nsrip-dd is exactly right. I think it needs to operate this way because of the existing requirements around goroutine states. The first thing such a goroutine ever does is, IIRC, emit a GoSysExit event, so the trace parser's state needs to be ready for this event which would otherwise come out of nowhere.

It is a little bit weird that this is treated as "the goroutine has been in a syscall the whole time." That aligns with how the runtime views the goroutine for sure, but not necessarily with how these goroutines behave from a user's perspective. We could, for instance, just pretend the goroutine is created and destroyed on each C->Go call (and return). The GoCreate event could then have the PC of the Go entrypoint for the C->Go call as its starting stack.

I imagine the reason this wasn't modeled this way originally is because it's a bit more complex. The GoSysExit event already has a special case for being emitted upon exiting the scheduler, and this becomes yet-another special case. Not saying this is sufficient reason to not model it this way, though.

@mknyszek mknyszek self-assigned this Jun 28, 2023
@mknyszek mknyszek added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mknyszek mknyszek

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
6 participants
@dominikh @cagedmantis @mknyszek @prattmic @gopherbot @nsrip-dd