Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: large stack makes the program hang #26154

Closed
kjk opened this issue Jun 29, 2018 · 3 comments
Closed

runtime: large stack makes the program hang #26154

kjk opened this issue Jun 29, 2018 · 3 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.12

Comments

@kjk
Copy link

kjk commented Jun 29, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

kjkmacpro:tst kjk$ go version
go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kjk/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kjk/src/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v_/ksw1dqvd59v790zk2wqf_t_80000gn/T/go-build146241323=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

https://play.golang.org/p/bEhVv3kwiNr

package main

import (
	"fmt"
)

func stackOverflow(x *byte) {
	var buf [1024 * 1024 * 11]byte
	stackOverflow(&buf[0])
}

func main() {
	fmt.Printf("Before stack overflow\n")
	stackOverflow(nil)
	fmt.Printf("After stack overflow\n")
}

This program never finishes.

Change the buf size to [1024 * 1024 * 10] (https://play.golang.org/p/s5sf7m0qWcu) and it'll crash with stack overflow error.

It looks like runtime is bouncing system threads:

(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:56
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
* 8    Thread 10720.0xdd0 0x00007ffd1f3649bf in ?? ()
(gdb) c
Continuing.
[Thread 10720.0xdd0 exited with code 0]
[New Thread 10720.0x1538]
[New Thread 10720.0x2ba8]
[New Thread 10720.0x6a4]

Thread 11 received signal SIGINT, Interrupt.
[Switching to Thread 10720.0x6a4]
0x00007ffd1f3649bf in ?? ()
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc 0x00007ffd22d29f34 in ?? ()
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
  9    Thread 10720.0x1538 0x00007ffd22d29f34 in ?? ()
  10   Thread 10720.0x2ba8 runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:51
* 11   Thread 10720.0x6a4 0x00007ffd1f3649bf in ?? ()
(gdb) c
Continuing.
[Thread 10720.0x6a4 exited with code 0]
[New Thread 10720.0x2bf0]

Thread 12 received signal SIGINT, Interrupt.
[Switching to Thread 10720.0x2bf0]
0x00007ffd1f3649bf in ?? ()
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc 0x00007ffd22d29f34 in ?? ()
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
  9    Thread 10720.0x1538 runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
  10   Thread 10720.0x2ba8 0x00007ffd22d29f34 in ?? ()
* 12   Thread 10720.0x2bf0 0x00007ffd1f3649bf in ?? ()

(gdb) thread 9
[Switching to thread 9 (Thread 10720.0x1538)]
#0  runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
47              MOVOU   X0, 16(DI)
(gdb) bt
#0  runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
#1  0x0000000000421351 in runtime.(*mheap).alloc (h=<optimized out>, large=<optimized out>, needzero=true, npage=<optimized out>, spanclass=<optimized out>,
    ~r4=<optimized out>) at C:/Users/kjk/src/go/src/runtime/mheap.go:340
#2  0x000000000040bafe in runtime.largeAlloc (needzero=true, noscan=true, size=11534336, ~r3=<optimized out>) at C:/Users/kjk/src/go/src/runtime/malloc.go:1011
#3  0x000000000044c8ed in runtime.mallocgc.func1 () at C:/Users/kjk/src/go/src/runtime/malloc.go:906
#4  0x000000000044f1bb in runtime.systemstack () at C:/Users/kjk/src/go/src/runtime/asm_amd64.s:351
#5  0x000000000042e720 in runtime.startTheWorldWithSema () at C:/Users/kjk/src/go/src/runtime/proc.go:1146
#6  0x0000000000000000 in ?? ()

This came as part of investigating #21382
@ianlancetaylor ianlancetaylor changed the title Large stack makes the program hang runtime: large stack makes the program hang Jun 29, 2018
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 29, 2018
@ianlancetaylor
Copy link
Contributor

Go 1.6 crashes with a stack overflow error, as does gccgo. Go versions after 1.6 appear to just burn CPU without allocating memory.

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Jun 29, 2018
@randall77
Copy link
Contributor

It looks like you're tripping across the boundary between stack-allocating and heap-allocating buf. We only allow objects to be stack allocated below a certain size, which happens to be:

cmd/compile/internal/gc/go.go:	maxStackVarSize = 10 * 1024 * 1024

When buf is stack allocated (when it is 1024*1024*10 bytes), we blow through stack really quickly, and you get a stack overflow error.
When buf is heap allocated (when it is 1024*1024*11 bytes), the stack frames are really small and it takes lots of them to overflow the stack. For each frame, we need to allocate 11MB in the heap, which is promptly garbage and needs to be collected. That's a lot of work per frame, and thus it takes a really long time to get to an overflow (but it will: 32 bytes/frame, ~1ms/frame, 1GB max stack, should take ~9 hours).

I'm going to close as not-a-bug. Definitely odd, but I'd argue that with the heap-allocated buf we're actually trying harder, and mostly succeeding, in doing what the programmer asked. Stack overflow errors are best effort to help the programmer avoid obvious bugs; they aren't required by the spec or anything like that.

@aclements This is kind of a weird case, where the GC is collecting a single 11MB object each iteration. Probably a larger heap goal size would be useful in situations like these (although this particular situation is contrived enough to be irrelevant).

gc 10415 @12.031s 7%: 0.002+0.72+0.048 ms clock, 0.025+0.69/0.042/0.69+0.58 ms cpu, 11->11->0 MB, 12 MB goal, 12 P

@aclements
Copy link
Member

This is kind of a weird case, where the GC is collecting a single 11MB object each iteration.

Perhaps the proper solution is to solve #23044 (comment). That would happen to space out these GCs, and would mean that when they did run, the cost of marking (0.72 ms in that example) would be amortized against the less frequent GC scheduling.

@golang golang locked and limited conversation to collaborators Jul 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.12
Development

No branches or pull requests
5 participants
@kjk @aclements @ianlancetaylor @randall77 @gopherbot