Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/playground: out of memory crashing #36469

Open
bradfitz opened this issue Jan 9, 2020 · 1 comment
Open

x/playground: out of memory crashing #36469

bradfitz opened this issue Jan 9, 2020 · 1 comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Jan 9, 2020

I was looking at the logs for the playground and see lots of crashes like:

A 2020-01-08T22:59:11Z fatal error: runtime: out of memory
 
A 2020-01-08T22:59:11Z 
 
A 2020-01-08T22:59:11Z runtime stack:
 
A 2020-01-08T22:59:11Z runtime.throw(0xbc60ed, 0x16)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/panic.go:774 +0x72
 
A 2020-01-08T22:59:11Z runtime.sysMap(0xc088000000, 0x24000000, 0x12028f8)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mem_linux.go:169 +0xc5
 
A 2020-01-08T22:59:11Z runtime.(*mheap).sysAlloc(0x11ea160, 0x20040000, 0x7f5eb32eed08, 0x4377c7)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/malloc.go:701 +0x1cd
 
A 2020-01-08T22:59:11Z runtime.(*mheap).grow(0x11ea160, 0x10020, 0xffffffff)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mheap.go:1255 +0xa3
 
A 2020-01-08T22:59:11Z runtime.(*mheap).allocSpanLocked(0x11ea160, 0x10020, 0x1202908, 0x1195e80)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mheap.go:1170 +0x266
 
A 2020-01-08T22:59:11Z runtime.(*mheap).alloc_m(0x11ea160, 0x10020, 0x101, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mheap.go:1022 +0xc2
 
A 2020-01-08T22:59:11Z runtime.(*mheap).alloc.func1()
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mheap.go:1093 +0x4c
 
A 2020-01-08T22:59:11Z runtime.(*mheap).alloc(0x11ea160, 0x10020, 0x7f5eb3000101, 0x429510)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/mheap.go:1092 +0x8a
 
A 2020-01-08T22:59:11Z runtime.largeAlloc(0x20040000, 0x450100, 0xc04be66000)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/malloc.go:1138 +0x97
 
A 2020-01-08T22:59:11Z runtime.mallocgc.func1()
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/malloc.go:1033 +0x46
 
A 2020-01-08T22:59:11Z runtime.systemstack(0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
 
A 2020-01-08T22:59:11Z runtime.mstart()
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/proc.go:1146
 
A 2020-01-08T22:59:11Z 
 
A 2020-01-08T22:59:11Z goroutine 416 [running]:
 
A 2020-01-08T22:59:11Z runtime.systemstack_switch()
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc000157058 sp=0xc000157050 pc=0x45a6b0
 
A 2020-01-08T22:59:11Z runtime.mallocgc(0x20040000, 0x0, 0x0, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/malloc.go:1032 +0x895 fp=0xc0001570f8 sp=0xc000157058 pc=0x40e1b5
 
A 2020-01-08T22:59:11Z runtime.slicebytetostring(0x0, 0xc04be66000, 0x20040000, 0x20800bcc, 0x0, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/string.go:102 +0x9f fp=0xc000157128 sp=0xc0001570f8 pc=0x448dff
 
A 2020-01-08T22:59:11Z main.(*Recorder).Events(0xc00039e8c0, 0x0, 0x0, 0x6, 0x6, 0xc0003022c0)
 
A 2020-01-08T22:59:11Z 	/go/src/playground/play.go:93 +0x36f fp=0xc0001572e0 sp=0xc000157128 pc=0x9fe9ef
 
A 2020-01-08T22:59:11Z main.compileAndRun(0xd35260, 0xc00026ec80, 0xc0003869a0, 0x0, 0x0, 0x0)
 
A 2020-01-08T22:59:11Z 	/go/src/playground/sandbox.go:493 +0x14ed fp=0xc000157940 sp=0xc0001572e0 pc=0xa0220d
 
A 2020-01-08T22:59:11Z main.(*server).commandHandler.func1(0xd336e0, 0xc0000cea80, 0xc0002f2800)
 
A 2020-01-08T22:59:11Z 	/go/src/playground/sandbox.go:115 +0x468 fp=0xc000157b30 sp=0xc000157940 pc=0xa09df8
 
A 2020-01-08T22:59:11Z net/http.HandlerFunc.ServeHTTP(0xc0002734a0, 0xd336e0, 0xc0000cea80, 0xc0002f2800)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2007 +0x44 fp=0xc000157b58 sp=0xc000157b30 pc=0x753d34
 
A 2020-01-08T22:59:11Z net/http.(*ServeMux).ServeHTTP(0xc00026eac0, 0xd336e0, 0xc0000cea80, 0xc0002f2800)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2387 +0x1bd fp=0xc000157bb8 sp=0xc000157b58 pc=0x755d0d
 
A 2020-01-08T22:59:11Z main.(*server).ServeHTTP(0xc0000ebdb0, 0xd336e0, 0xc0000cea80, 0xc0002f2800)
 
A 2020-01-08T22:59:11Z 	/go/src/playground/server.go:92 +0x193 fp=0xc000157c18 sp=0xc000157bb8 pc=0xa05043
 
A 2020-01-08T22:59:11Z net/http.serverHandler.ServeHTTP(0xc000362000, 0xd336e0, 0xc0000cea80, 0xc0002f2800)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2802 +0xa4 fp=0xc000157c48 sp=0xc000157c18 pc=0x757284
 
A 2020-01-08T22:59:11Z net/http.(*conn).serve(0xc00019c140, 0xd35260, 0xc00007a040)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:1890 +0x875 fp=0xc000157fc8 sp=0xc000157c48 pc=0x752b25
 
A 2020-01-08T22:59:11Z runtime.goexit()
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000157fd0 sp=0xc000157fc8 pc=0x45c781
 
A 2020-01-08T22:59:11Z created by net/http.(*Server).Serve
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2928 +0x384
 
A 2020-01-08T22:59:11Z 
 
A 2020-01-08T22:59:11Z goroutine 1 [IO wait]:
 
A 2020-01-08T22:59:11Z internal/poll.runtime_pollWait(0x7f5eb6446fe8, 0x72, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/runtime/netpoll.go:184 +0x55
 
A 2020-01-08T22:59:11Z internal/poll.(*pollDesc).wait(0xc0002e2898, 0x72, 0x0, 0x0, 0xba4397)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
 
A 2020-01-08T22:59:11Z internal/poll.(*pollDesc).waitRead(...)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
 
A 2020-01-08T22:59:11Z internal/poll.(*FD).Accept(0xc0002e2880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
 
A 2020-01-08T22:59:11Z net.(*netFD).accept(0xc0002e2880, 0xc0005c9c80, 0x758214, 0xc0003620a0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/fd_unix.go:238 +0x42
 
A 2020-01-08T22:59:11Z net.(*TCPListener).accept(0xc00000e4c0, 0x5e165eba, 0xc0005c9c80, 0x4c90a6)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
 
A 2020-01-08T22:59:11Z net.(*TCPListener).Accept(0xc00000e4c0, 0xc0005c9cd0, 0x18, 0xc000000180, 0x757744)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/tcpsock.go:261 +0x47
 
A 2020-01-08T22:59:11Z net/http.(*Server).Serve(0xc000362000, 0xd33420, 0xc00000e4c0, 0x0, 0x0)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2896 +0x280
 
A 2020-01-08T22:59:11Z net/http.(*Server).ListenAndServe(0xc000362000, 0xc000362000, 0x1)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:2825 +0xb7
 
A 2020-01-08T22:59:11Z net/http.ListenAndServe(...)
 
A 2020-01-08T22:59:11Z 	/usr/local/go/src/net/http/server.go:3081
 
A 2020-01-08T22:59:11Z main.main()
 
A 2020-01-08T22:59:11Z 	/go/src/playground/main.go:58 +0x2cd
 
A 2020-01-08T22:59:11Z

I haven't looked into it yet but filing before I forget.

I don't think it's related to the new gvisor sandbox code (that was just submitted before I saw these logs) because the new code is disabled right now and I don't see how the other changes could cause this (too much memory being used by the Recorder type?).

Investigate.
@gopherbot gopherbot added this to the Unreleased milestone Jan 9, 2020
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 9, 2020
@dmitshur
Copy link
Contributor

dmitshur commented Mar 3, 2020

/cc @toothrot @cagedmantis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @dmitshur @gopherbot