Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

process crash in go runtime with "fatal error: invalid g status" #52560

Closed
dproc opened this issue Apr 26, 2022 · 2 comments
Closed

process crash in go runtime with "fatal error: invalid g status" #52560

dproc opened this issue Apr 26, 2022 · 2 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@dproc
Copy link

dproc commented Apr 26, 2022

What version of Go are you using (go version)?

$ go version
go version go1.17.2 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY="*.byted.org,*.everphoto.cn,git.smartisan.com"
GONOSUMDB="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOOS="linux"
GOPATH="/go"
GOPRIVATE="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOPROXY="https://go-mod-proxy.byted.org,https://goproxy.cn,https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.2"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1869258117=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I did nothing, the process(gobgpd) run as a daemon. When it crashed, there was no activity at application level

What did you expect to see?

What did you see instead?

I run gobgpd for about two month until it suddenly crash in go runtime with following error:

runtime: gp: gp=0xc0006836c0, goid=0, gp->atomicstatus=0
runtime:  g:  g=0xc0002f41a0, goid=0,  g->atomicstatus=0
fatal error: invalid g status
runtime: gp: gp=0xc000683860, goid=0, gp->atomicstatus=0
runtime:  g:  g=0xc000a7c1a0, goid=0,  g->atomicstatus=0
fatal error: invalid g status
runtime: unknown pc 0x6
stack: frame={sp:0xc000922b30, fp:0x0} stack=[0xc000922000,0xc000923000)
0x000000c000922a30:  0x0000000000000000  0x0000000000203000 
0x000000c000922a40:  0x0000000000000000  0x000000c000922bb0 
0x000000c000922a50:  0x0000000000419351 <runtime.heapBitsSetType+0x00000000000006f1>  0x000000c0004c2300 
0x000000c000922a60:  0x0000000000000027  0x00000000006e0060 <crypto/tls.(*Conn).writeRecordLocked.func1

It seems to crash inside runtime/preempt.go:suspendG. The switch-case falls in the impossible default case.

func suspendG(gp *g) suspendGState {
    if mp := getg().m; mp.curg != nil && readgstatus(mp.curg) == _Grunning {
        // Since we're on the system stack of this M, the user
        // G is stuck at an unsafe point. If another goroutine
        // were to try to preempt m.curg, it could deadlock.
        throw("suspendG from non-preemptible goroutine")
    }

    // See https://golang.org/cl/21503 for justification of the yield delay.
    const yieldDelay = 10 * 1000
    var nextYield int64

    // Drive the goroutine to a preemption point.
    stopped := false
    var asyncM *m
    var asyncGen uint32
    var nextPreemptM int64
    for i := 0; ; i++ {
        switch s := readgstatus(gp); s {
        default:
            if s&_Gscan != 0 {
                // Someone else is suspending it. Wait
                // for them to finish.
                //
                // TODO: It would be nicer if we could
                // coalesce suspends.
                break
            }

            dumpgstatus(gp)
            throw("invalid g status")

        case _Gdead:
            // Nothing to suspend.
            //
            // preemptStop may need to be cleared, but
            // doing that here could race with goroutine
            // reuse. Instead, goexit0 clears it.
            return suspendGState{dead: true}

        case _Gcopystack:
            // The stack is being copied. We need to wait
            // until this is done.

        case _Gpreempted:
            // We (or someone else) suspended the G. Claim
            // ownership of it by transitioning it to
            // _Gwaiting.
            if !casGFromPreempted(gp, _Gpreempted, _Gwaiting) {
                break
            }

            // We stopped the G, so we have to ready it later.
            stopped = true

            s = _Gwaiting
            fallthrough

        case _Grunnable, _Gsyscall, _Gwaiting:

Please does anyone see this kind of error ever before? How could it happen. Thanks a lot!

@dproc dproc changed the title affected/package: process crash in go runtime with "fatal error: invalid g status" Apr 26, 2022
@ianlancetaylor
Copy link
Contributor

If this only happened once, then it looks like memory got corrupted somehow.

@ianlancetaylor ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 26, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators May 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

3 participants