Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: SIGSEGV at high GOMAXPROCS in init #38474

Closed
prattmic opened this issue Apr 15, 2020 · 6 comments
Closed

runtime: SIGSEGV at high GOMAXPROCS in init #38474

prattmic opened this issue Apr 15, 2020 · 6 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.15

Comments

@prattmic
Copy link
Member

prattmic commented Apr 15, 2020 •
edited
Loading

What version of Go are you using (go version)?

$ ../bin/go version        
go version devel +435b9dd1a1 Wed Apr 15 15:30:46 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

This does not reproduce on go 1.14.2.

What operating system and processor architecture are you using (go env)?

go env Output
$ ../bin/go env    
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/mpratt/.cache/go-build"
GOENV="/usr/local/google/home/mpratt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/google/home/mpratt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/mpratt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/google/home/mpratt/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/google/home/mpratt/src/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/google/home/mpratt/src/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build240555977=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ./make.bash
$ GOMAXPROCS=1024 ../bin/go help

What did you expect to see?

Help output

What did you see instead?

$ GOMAXPROCS=1024 ../bin/go help
zsh: segmentation fault  GOMAXPROCS=1024 ../bin/go help

Full backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x000000000040f093 in runtime.profilealloc (mp=0xeb33a0 <runtime.m0>, x=0xc000932000, size=10240) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1229
1229            return uintptr(fastexprand(MemProfileRate))
(gdb) bt
#0  0x000000000040f093 in runtime.profilealloc (mp=0xeb33a0 <runtime.m0>, x=0xc000932000, size=10240) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1229
#1  0x000000000040e810 in runtime.mallocgc (size=10240, typ=0xa1cb80, needzero=true, ~r3=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1130
#2  0x000000000040ef58 in runtime.newobject (typ=0xa1cb80, ~r1=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/malloc.go:1179
#3  0x000000000043fbf2 in runtime.procresize (nprocs=1024, ~r1=<optimized out>) at /usr/local/google/home/mpratt/src/go/src/runtime/proc.go:4279
#4  0x00000000004372db in runtime.schedinit () at /usr/local/google/home/mpratt/src/go/src/runtime/proc.go:584
#5  0x0000000000463e45 in runtime.rt0_go () at /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:214
#6  0x0000000000000002 in ?? ()
#7  0x00007fffffffdbf8 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0x00007fffffffdbf8 in ?? ()
#10 0x0000000000000000 in ?? ()
@ianlancetaylor
Copy link
Member

Yeah, don't start doing profiling allocations before the scheduler is initialized. The crash is because mp.p is 0 in profilealloc, and that is so because it has not yet been set by the call from procresize to acquirep, which happens after the point of the crash above.

@ianlancetaylor ianlancetaylor added NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Apr 15, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.15 milestone Apr 15, 2020
@prattmic
Copy link
Member Author

For completeness:

$ git bisect bad                          
3093959ee10f5c28211094e784c954f6a304b9c9 is the first bad commit                                                                              
commit 3093959ee10f5c28211094e784c954f6a304b9c9                  
Author: Ian Lance Taylor <iant@golang.org>                                                                                                    
Date:   Mon Nov 4 14:25:22 2019 -0800                          
                                                                       
    runtime: remove mcache field from m                                
                                                                                                                                              
    Having an mcache field in both m and p is confusing, so remove it from m.
    Always use mcache field from p. Use new variable mcache0 during bootstrap.                                                                
                                                                                                                                              
    Change-Id: If2cba9f8bb131d911d512b61fd883a86cf62cc98 
    Reviewed-on: https://go-review.googlesource.com/c/go/+/205239
    Run-TryBot: Ian Lance Taylor <iant@golang.org>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    Reviewed-by: Austin Clements <austin@google.com>            
                                                                       
 src/runtime/malloc.go   | 19 +++++++++++++++----                                                                                             
 src/runtime/mgcsweep.go |  2 +-                                                                                                                                                                                                                                                            
 src/runtime/mheap.go    | 29 ++++++++++++++++++++---------                                                                                   
 src/runtime/proc.go     | 29 ++++++++++++-----------------                                                                                   
 src/runtime/runtime1.go |  5 -----                                                                                                           
 src/runtime/runtime2.go |  1 -                           
 src/runtime/stack.go    | 12 ++++++------           
 7 files changed, 54 insertions(+), 43 deletions(-)

@ianlancetaylor
Copy link
Member

Interesting, it's true that before that change profilealloc updated mp.mcache but after that change it updates mp.p.ptr().mcache. So only after that change did profilealloc require that mp.p be initialized.

@ianlancetaylor
Copy link
Member

I'll send a CL.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/228420 mentions this issue: runtime: use mcache0 if no P in profilealloc

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/228786 mentions this issue: runtime: allow proflock above globalAlloc.mutex

@bcmills bcmills mentioned this issue Apr 20, 2020
Closed
gopherbot pushed a commit that referenced this issue Apr 21, 2020
@prattmic
runtime: allow proflock and mheap.speciallock above globalAlloc.mutex
300ff5d 
During schedinit, these may occur in:

mProf_Malloc
  stkbucket
    newBucket
      persistentalloc
        persistentalloc1

mProf_Malloc
  setprofilebucket
    fixalloc.alloc
      persistentalloc
        persistentalloc1

These seem to be legitimate lock orderings.

Additionally, mheap.speciallock had a defined rank, but it was never
actually used. That is fixed now.

Updates #38474

Change-Id: I0f6e981852eac66dafb72159f426476509620a65
Reviewed-on: https://go-review.googlesource.com/c/go/+/228786
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dan Scales <danscales@google.com>
@golang golang locked and limited conversation to collaborators Apr 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Milestone
Go1.15
Development

No branches or pull requests
3 participants
@prattmic @ianlancetaylor @gopherbot