Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: panic fatal error: malloc deadlock #24059

Closed
dgryski opened this issue Feb 23, 2018 · 11 comments
Closed

runtime: panic fatal error: malloc deadlock #24059

dgryski opened this issue Feb 23, 2018 · 11 comments
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Milestone
Go1.10.1

Comments

@dgryski
Copy link
Contributor

dgryski commented Feb 23, 2018

What version of Go are you using (go version)?

I've bisected the issue to 2edc4d4

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin/amd64

What did you do?

During development of a large internal service, we were running our test suite. The tests triggered a nil pointer dereference. On 1.9.4 the output was the usual

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1300854]

But on 1.10 we instead got

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/Users/dgryski/work/src/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/Users/dgryski/work/src/go/src/runtime/proc.go:1170

goroutine 207 [running]:
runtime.systemstack_switch()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:363 fp=0xc420373a88 sp=0xc420373a80 pc=0x1057fe0
runtime.startpanic()
	/Users/dgryski/work/src/go/src/runtime/panic.go:592 +0x1e fp=0xc420373aa0 sp=0xc420373a88 pc=0x102c8be
runtime.throw(0x1367aa4, 0xf)
	/Users/dgryski/work/src/go/src/runtime/panic.go:618 +0x74 fp=0xc420373ac0 sp=0xc420373aa0 pc=0x102c9e4
runtime.mallocgc(0x2010, 0x0, 0x1303401, 0x12ae3a8)
	/Users/dgryski/work/src/go/src/runtime/malloc.go:621 +0x99d fp=0xc420373b60 sp=0xc420373ac0 pc=0x10126cd
runtime.itabAdd(0x2541840)
	/Users/dgryski/work/src/go/src/runtime/iface.go:122 +0x83 fp=0xc420373bb0 sp=0xc420373b60 pc=0x100f403
runtime.getitab(0x1303480, 0x12e2680, 0xc4202e4301, 0xc4202e43b0)
	/Users/dgryski/work/src/go/src/runtime/iface.go:70 +0x404 fp=0xc420373c28 sp=0xc420373bb0 pc=0x100f284
runtime.assertE2I2(0x1303480, 0x12e2680, 0xc42022ad30, 0x7, 0xc400000007, 0xc420373cb8)
	/Users/dgryski/work/src/go/src/runtime/iface.go:592 +0x43 fp=0xc420373c58 sp=0xc420373c28 pc=0x10107e3
runtime.printany(0x12e2680, 0xc42022ad30)
	/Users/dgryski/work/src/go/src/runtime/error.go:77 +0x69 fp=0xc420373d18 sp=0xc420373c58 pc=0x1007709
runtime.printpanics(0xc420373da0)
	/Users/dgryski/work/src/go/src/runtime/panic.go:420 +0x71 fp=0xc420373d38 sp=0xc420373d18 pc=0x102c101
panic(0x12fd360, 0x1528330)
	/Users/dgryski/work/src/go/src/runtime/panic.go:553 +0x3b4 fp=0xc420373dd8 sp=0xc420373d38 pc=0x102c544
runtime.panicmem()
	/Users/dgryski/work/src/go/src/runtime/panic.go:63 +0x5e fp=0xc420373df8 sp=0xc420373dd8 pc=0x102b24e
runtime.sigpanic()
	/Users/dgryski/work/src/go/src/runtime/signal_unix.go:388 +0x17a fp=0xc420373e48 sp=0xc420373df8 pc=0x1041fca
github.com/fastly/service1/service2.(*GetStats).Add(...)
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:222
github.com/fastly/service1/service2.(*Client).GetAll.func2()
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:276 +0x27d fp=0xc420373fe0 sp=0xc420373e48 pc=0x1292c8d
runtime.goexit()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420373fe8 sp=0xc420373fe0 pc=0x105ab31
created by github.com/fastly/service1/service2.(*Client).GetAll
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:285 +0x232

[ output trimmed of all the other goroutine stack traces]

Some notes from trying to minimize this into a useful test case:

  • There are lots of goroutines running, and the nil dereference happens in one
  • When running with -test.v, the normal panic output is produced
  • I need to run multiple tests from this package in addition to the failing test. Running just the one test that fails produced the normal panic output.
  • Adding -race doesn't make a difference
@mvdan mvdan added this to the Go1.10.1 milestone Feb 23, 2018
@mvdan mvdan added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 23, 2018
@ianlancetaylor
Copy link
Contributor

We crash in the unlikely case that we have to grow the itab table while adding an itab entry for converting string to stringer. I will send a CL.

@gopherbot
Copy link

Change https://golang.org/cl/96755 mentions this issue: runtime: in prepanic, ensure printany doesn't allocate printing a string

@dgryski
Copy link
Contributor Author

dgryski commented Feb 23, 2018

I can confirm this CL fixes my issue.

@fatedier fatedier mentioned this issue Mar 14, 2018
Closed
@andybons
Copy link
Member

Reopening for cherry-pick

@andybons andybons reopened this Mar 26, 2018
@andybons andybons mentioned this issue Mar 27, 2018
Closed
@andybons
Copy link
Member

CL 96755 OK for Go 1.10.1

@andybons andybons added CherryPickApproved Used during the release process for point releases and removed NeedsFix The path to resolution is known, but the work has not been done. labels Mar 27, 2018
@gopherbot
Copy link

Change https://golang.org/cl/102781 mentions this issue: [release-branch.go1.10] runtime: don't check for String/Error methods in printany

gopherbot pushed a commit that referenced this issue Mar 29, 2018
@ianlancetaylor @andybons
[release-branch.go1.10] runtime: don't check for String/Error methods…
e1c0834 
… in printany

They have either already been called by preprintpanics, or they can
not be called safely because of the various conditions checked at the
start of gopanic.

Fixes #24059

Change-Id: I4a6233d12c9f7aaaee72f343257ea108bae79241
Reviewed-on: https://go-review.googlesource.com/96755
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-on: https://go-review.googlesource.com/102781
Run-TryBot: Andrew Bonventre <andybons@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@grobie grobie mentioned this issue Mar 31, 2018
Closed
grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018
@grobie
Use go1.10.1 and go1.9.5
380ada1 
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059
@grobie grobie mentioned this issue Mar 31, 2018
Merged
grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018
@grobie
Use go1.10.1 and go1.9.5
0c07722 
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059
grobie added a commit to prometheus/golang-builder that referenced this issue Mar 31, 2018
@grobie
Use go1.10.1 and go1.9.5 (#39)
f0f4e83 
The latest patch versions include two important fixes for the Prometheus
project:

* golang/go#23995
* golang/go#24059
@SuperQ
Copy link

SuperQ commented Apr 27, 2018

We're still seeing issues that appear to be related on go1.10.1, should we file a separate issue or reopen?

@ianlancetaylor
Copy link
Contributor

@SuperQ Please file a new issue. We're pretty sure this problem is fixed. I would guess that you are seeing a different problem with similar symptoms.

@denismakogon
Copy link

faced with this type of error while was doing something like this:

a function that generates a slice of int64 items:

func makeInt64Range(min, max int64) []int64 {
	a := make([]int64, max-min+1)
	for i := range a {
		a[i] = min + int64(i)
	}
	return a
}

accidentally (was distracted by random convo) it was called with mismatched boundaries:

	int64Range := makeInt64Range(1025, 1000)

and got this:

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/usr/local/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1170

Golang comes from: golang:1.10-alpine3.7 docker image.

Tried that on playground: https://play.golang.org/p/D5j5EzDgNiW
Error is: panic: runtime error: makeslice: len out of range which appear to be different from what i have.

@aclements
Copy link
Member

@denismakogon, please open a new issue, thanks!

@denismakogon
Copy link

@aclements thanks, it appears that fixed was merged somewhere on the road to 1.10.2 since i don't get malloc error on the playground (which is 1.10.2). Will test on golang:1.10.2-alpine3.7 (when that will be available) and if it's still there I will definitely open an issue.

@golang golang locked and limited conversation to collaborators May 1, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Projects
None yet
Milestone
Go1.10.1
Development

No branches or pull requests
8 participants
@andybons @dgryski @SuperQ @aclements @denismakogon @ianlancetaylor @mvdan @gopherbot