cmd/trace: runs out of memory processing 30 MB file #21870
Comments
|
So I ran it with the ram/cpu monitor opened and noticed it was eating 100% ram before my linux box froze. Added some swap and looks like it ate 6 GB of ram to open the webpage and then trying to view a trace ate another 1.5 GB - 2 GB and then froze my linux box without showing the graph. Do these numbers sound right? They are from looking at the total ram +swap usage so it's not too accurate. |
|
Yes. Traces can be very large.
… On 14 Sep 2017, at 07:11, Chotepud Teo ***@***.***> wrote:
So I ran it with the ram/cpu monitor opened and noticed it was eating 100% ram before my linux box froze. Added some swap and looks like it ate 6 GB of ram to open the webpage and then trying to view a trace ate another 1.5 GB - 2 GB and then froze my linux box without showing the graph.
Do these numbers sound right? They are from looking at the total ram +swap usage so it's not too accurate.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Any options? Or am I just screwed now cause this is from just 5 seconds of run time. |
|
How much ram does your host have?
…On Thu, Sep 14, 2017 at 7:25 AM, Chotepud Teo ***@***.***> wrote:
Any options? Or am I just screwed now cause this is from just 5 seconds of
run time.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21870 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcAyArgyXPeVnm9xqtz-b5dFInhBejks5siEhIgaJpZM4PWswJ>
.
|
|
8 GB (7.7 according to ram monitor), but even with swap I couldn't get the graphs to load before it freezes. |
|
/cc @francesc
TL;DR you need more ram. Sorry.
…On Thu, Sep 14, 2017 at 9:14 AM, Chotepud Teo ***@***.***> wrote:
8 GB, but even with swap I couldn't get the graphs to load before it
freezes.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21870 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA0rLZcXwOmi1L_7zCGKgTFz6CfTXks5siGHEgaJpZM4PWswJ>
.
|
|
Was expecting that... Do you know of any other similar tools but live so shouldn't eat a ton of ram? |
|
What was the program doing during that trace. Based on what I've seen the resources required to analyse a trace are proportional to the number of goroutines events during that time. Where an event is, a goroutines starting and stopping execution.
… On 14 Sep 2017, at 09:38, Chotepud Teo ***@***.***> wrote:
Was expecting that...
But if this is from 5 seconds of runtime, then there is no amount of RAM that would be enough for longer runs. And turning trace on/off (runtime/trace) wouldn't be a very nice experience if you are trying to catch a event you can't predict.
Do you know of any other similar tools but live so shouldn't eat a ton of ram?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Running I think 4 goroutines and calling a bunch of C funcs in a loop (each goroutine does this)... I guess the worst case for trace resource consumption? |
|
Pretty much.
…On Thu, Sep 14, 2017 at 9:55 AM, Chotepud Teo ***@***.***> wrote:
Running I think 4 goroutines and calling a bunch of C funcs in a loop... I
guess the worst case for trace resource consumption?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21870 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA4DCvtKghJD_PTfl1pKm6poQDv0Nks5siGtUgaJpZM4PWswJ>
.
|
ianlancetaylor
changed the title
ianlancetaylor
added
the
NeedsInvestigation
|
Thanks Ian for fixing the label mistake. It is indeed a pathological case. The trace contains more than 10.5million events, mostly ProcStart/ProcStop or Syscall events. When parsing the trace, each event is mapped to the trace.Event object Then, the 10.5million events are mapped into 17million TraceViewer type objects Memory fragmentation makes this issue worse. |
|
Change https://golang.org/cl/92375 mentions this issue: |
|
Change https://golang.org/cl/92355 mentions this issue: |
|
Change https://golang.org/cl/92376 mentions this issue: |
Enabled when the tool runs with DEBUG_MEMORY_USAGE=1 env var. After reporting the usage, it waits until user enters input (helpful when checking top or other memory monitor) Also adds net/http/pprof to export debug endpoints. From the trace included in #21870 $ DEBUG_MEMORY_USAGE=1 go tool trace trace.out 2018/02/21 16:04:49 Parsing trace... after parsing trace Alloc: 3385747848 Bytes Sys: 3661654648 Bytes HeapReleased: 0 Bytes HeapSys: 3488907264 Bytes HeapInUse: 3426377728 Bytes HeapAlloc: 3385747848 Bytes Enter to continue... 2018/02/21 16:05:09 Serializing trace... after generating trace Alloc: 4908929616 Bytes Sys: 5319063640 Bytes HeapReleased: 0 Bytes HeapSys: 5032411136 Bytes HeapInUse: 4982865920 Bytes HeapAlloc: 4908929616 Bytes Enter to continue... 2018/02/21 16:05:18 Splitting trace... after spliting trace Alloc: 4909026200 Bytes Sys: 5319063640 Bytes HeapReleased: 0 Bytes HeapSys: 5032411136 Bytes HeapInUse: 4983046144 Bytes HeapAlloc: 4909026200 Bytes Enter to continue... 2018/02/21 16:05:39 Opening browser. Trace viewer is listening on http://127.0.0.1:33661 after httpJsonTrace Alloc: 5288336048 Bytes Sys: 7790245896 Bytes HeapReleased: 0 Bytes HeapSys: 7381123072 Bytes HeapInUse: 5324120064 Bytes HeapAlloc: 5288336048 Bytes Enter to continue... Change-Id: I88bb3cb1af3cb62e4643a8cbafd5823672b2e464 Reviewed-on: https://go-review.googlesource.com/92355 Reviewed-by: Peter Weinberger <pjw@google.com>
Update #21870 The Sys went down to 4.25G from 6.2G. $ DEBUG_MEMORY_USAGE=1 go tool trace trace.out 2018/03/07 08:49:01 Parsing trace... after parsing trace Alloc: 3385757184 Bytes Sys: 3661195896 Bytes HeapReleased: 0 Bytes HeapSys: 3488841728 Bytes HeapInUse: 3426516992 Bytes HeapAlloc: 3385757184 Bytes Enter to continue... 2018/03/07 08:49:18 Splitting trace... after spliting trace Alloc: 2352071904 Bytes Sys: 4243825464 Bytes HeapReleased: 0 Bytes HeapSys: 4025712640 Bytes HeapInUse: 2377703424 Bytes HeapAlloc: 2352071904 Bytes Enter to continue... after httpJsonTrace Alloc: 3228697832 Bytes Sys: 4250379064 Bytes HeapReleased: 0 Bytes HeapSys: 4025647104 Bytes HeapInUse: 3260014592 Bytes HeapAlloc: 3228697832 Bytes Change-Id: I546f26bdbc68b1e58f1af1235a0e299dc0ff115e Reviewed-on: https://go-review.googlesource.com/92375 Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> Reviewed-by: Peter Weinberger <pjw@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
to return memory to the OS after completing potentially large operations. Update #21870 Sys went down to 3.7G $ DEBUG_MEMORY_USAGE=1 go tool trace trace.out 2018/03/07 09:35:52 Parsing trace... after parsing trace Alloc: 3385754360 Bytes Sys: 3662047864 Bytes HeapReleased: 0 Bytes HeapSys: 3488907264 Bytes HeapInUse: 3426549760 Bytes HeapAlloc: 3385754360 Bytes Enter to continue... 2018/03/07 09:36:09 Splitting trace... after spliting trace Alloc: 3238309424 Bytes Sys: 3684410168 Bytes HeapReleased: 0 Bytes HeapSys: 3488874496 Bytes HeapInUse: 3266461696 Bytes HeapAlloc: 3238309424 Bytes Enter to continue... 2018/03/07 09:36:39 Opening browser. Trace viewer is listening on http://100.101.224.241:12345 after httpJsonTrace Alloc: 3000633872 Bytes Sys: 3693978424 Bytes HeapReleased: 0 Bytes HeapSys: 3488743424 Bytes HeapInUse: 3030966272 Bytes HeapAlloc: 3000633872 Bytes Enter to continue... Change-Id: I56f64cae66c809cbfbad03fba7bd0d35494c1d04 Reviewed-on: https://go-review.googlesource.com/92376 Reviewed-by: Peter Weinberger <pjw@google.com>
|
According to the cl/92376 description, the memory usage went down to 3.7G (from 6.2G). |
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?go version go1.9 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="
/go"/.local/go"GORACE=""
GOROOT="
GOTOOLDIR="~/.local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build582547481=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
go tool trace trace.out
What did you expect to see?
2017/09/14 03:16:35 Parsing trace...
2017/09/14 03:16:40 Serializing trace...
2017/09/14 03:16:41 Splitting trace...
2017/09/14 03:16:47 Opening browser
What did you see instead?
2017/09/14 04:31:17 Parsing trace...
2017/09/14 04:31:32 Serializing trace...
go tool trace: signal: killed
Something to note, OS froze (keyboard/mouse unresponsive) for a few seconds before trace terminated.
trace.out.tar.gz
The text was updated successfully, but these errors were encountered: