cmd/go: go build takes too long to compile table driven test code #8259
Labels
Milestone
Comments
The table entries in the provided sample are interface values so this long commpilatin times might be a related to issue #8354. |
CL https://golang.org/cl/125720043 mentions this issue. |
|
This issue was closed by revision 7aa3031. Status changed to Fixed. |
I tried taking the example with 500 values in the slice, and I made more copies so there were 8000 values, and I let the compiler at tip run for 20 minutes before killing it. I am not sure what's going on. Maybe it's not really fixed, but that 20 minutes is much longer than the reported 55 seconds. It would be nice to have a test case demonstrating the 55 seconds. I feel fairly confident the fix is correct, but it's still a bit larger than I would like for a point release, and unless I can prove it actually fixes the original case I am inclined not to include it. Not sure about 1.3.1 at this point. |
It may not fix this case, but it does fix 8354. Moving 1.3.1 tag there. nicolas.riesch, can you please try Go at tip and see if your original case is fixed? If not, can you send a test case? The 500-entry case is working better but you mentioned an 8000-entry case, and I haven't been able to reproduce that. Labels changed: added release-go1.4, removed release-go1.3.1. Status changed to WaitingForReply. |
I have just installed Go 1.3.1 and run my test files. These test files are table-driven, and contain about 8000 samples. My configuration is: HP Pavilion dv7 AMD Phenom II P960 Quad-Core Processor 1.80 GHz 4 Gb RAM Windows 7 On top of this, I have Virtualbox, with Linux Mint 17 64bits Xfce. All the work is done inside this Linux Mint virtual machine. I have run the tests with the virtual machine memory setting at 1024 Mb of RAM, or 1914 Mb of RAM. You can see that for Go 1.3.1, with 1914 Mb of RAM available for the virtual machine, the tests go 4 times faster than with 1024 Mb of RAM. Here are the figures. --------------------------------- go 1.2.2 on Linux Mint with 1024 Mb of RAM ---------------------- nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 7.007s real 1m20.271s user 0m58.783s sys 0m5.230s nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 6.899s real 1m17.577s user 0m58.069s sys 0m5.599s --------------------------------- go 1.3 with 1024 Mb of RAM ---------------------- no figure, compilation never ends... --------------------------------- go 1.3.1 with 1024 Mb of RAM ---------------------- At my first attempt, the compiler stopped with a message indicating it cannot allocate more memory. At the second attempt, the OS just sent a SIGKILL to stop the compiler. But at the third attempt, the compilation works all right and performed the test successfully. I don't know what happened, as I just retried the same "go test" command each time. Seems to be some memory pressure somewhere... nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 6.946s real 5m56.815s user 1m5.199s sys 0m16.318s nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 8.619s real 6m32.995s user 1m5.150s sys 0m15.858s nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 8.057s real 6m3.998s user 1m5.329s sys 0m16.053s This is the "top" command output when VIRT and RES were at at their highest during compilation : top - 19:39:07 up 41 min, 3 users, load average: 1.65, 1.33, 1.30 Tasks: 138 total, 2 running, 136 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.3 us, 8.9 sy, 0.0 ni, 0.0 id, 82.5 wa, 2.3 hi, 0.0 si, 0.0 st KiB Mem: 1017912 total, 921784 used, 96128 free, 452 buffers KiB Swap: 1047548 total, 964600 used, 82948 free. 9448 cached Mem scroll coordinates: y = 1/138 (tasks), x = 1/12 (fields) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4449 nico 20 0 1513344 713208 512 D 2.6 70.1 0:44.88 6g 1329 root 20 0 299800 23296 972 S 0.7 2.3 1:08.13 Xorg 4382 nico 20 0 259308 4692 32 S 0.0 0.5 0:00.96 go 2968 nico 20 0 377328 3752 1772 S 6.6 0.4 2:14.72 xfce4-taskmanag 2182 nico 20 0 532564 3448 1596 S 0.0 0.3 0:06.61 xfce4-terminal 1857 nico 20 0 475620 3164 1904 S 0.0 0.3 0:00.95 xfce4-panel 1853 nico 20 0 164176 2952 1460 S 0.0 0.3 0:02.93 xfwm4 2086 nico 20 0 147584 2860 2088 S 0.3 0.3 0:00.46 panel-5-datetim 1861 nico 20 0 570316 1612 852 S 0.0 0.2 0:01.64 xfdesktop --------------------------------- go 1.3.1 with 1914 Mb of RAM ---------------------- nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 6.698s real 1m29.073s user 1m2.295s sys 0m10.791s nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 6.741s real 1m42.629s user 1m2.167s sys 0m10.041s nico@nico-Qiana-Xfce ~/z_go/src/rsql/data $ time go test PASS ok rsql/data 6.620s real 1m29.094s user 1m1.978s sys 0m10.681s This is the "top" command output when VIRT and RES were at at their highest during compilation : top - 20:34:04 up 48 min, 3 users, load average: 0.84, 0.44, 0.21 Tasks: 139 total, 3 running, 136 sleeping, 0 stopped, 0 zombie %Cpu(s): 90.1 us, 9.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st KiB Mem: 1914936 total, 1473636 used, 441300 free, 1260 buffers KiB Swap: 1047548 total, 14356 used, 1033192 free. 89032 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2465 nico 20 0 1128648 1.075g 768 R 99.5 58.8 0:25.57 6g 1329 root 20 0 290700 38920 3960 S 0.0 2.0 0:12.42 Xorg 1932 nico 20 0 561652 18560 684 S 0.0 1.0 0:00.85 mintUpdate 1826 nico 20 0 569412 10220 844 S 0.0 0.5 0:01.18 xfdesktop 1834 nico 20 0 247560 9656 496 S 0.0 0.5 0:00.28 applet.py 2398 nico 20 0 259180 9604 3688 S 0.0 0.5 0:00.96 go 2188 nico 20 0 531980 8160 2640 S 0.3 0.4 0:06.00 xfce4-terminal The compiler seems to allocate about 1 Gb of memory, which explains why the configuration with only 1024 Mb of RAM is crawling and swapping. Is the compiler expected to allocate this amount of memory ? But the compile time on 1.3.1 with 1914 Mb of RAM is similar to 1.2.2 with 1024 Mb of RAM. So, in my case, the problem is fixed. I thank you very much for having fixed this issue so quickly ;-))) @rsc: For your question of August 11th: The test case demonstrating the 55 seconds on Go 1.3 is http://play.golang.org/p/qpxcVGkzuk, which contains ~500 samples. My original test case contains 8000 samples, and Go 1.3 takes forever to compile it. |
adg
added a commit
that referenced
this issue
May 11, 2015
««« CL 125720043 / b92e5df7d3ba cmd/gc: make liveness ~10x faster 1) The arrayindexof lookup function is O(n). Replace with O(1) lookups. 2) The checkptxt function is O(n²) and is purely for debugging. Only run when the debugging flags are turned on. 3) Iterating over sparse bitmaps can be done faster word by word. Introduce and use bvnext for that. Run times before and after, on my 2.5 GHz Core i5 MacBook Pro. x.go 9.48 0.84 issue 8259 x100.go 0.01 0.01 issue 8354 x1000.go 0.10 0.10 x2000.go 0.62 0.19 x3000.go 1.33 0.34 x4000.go 2.29 0.49 x5000.go 3.89 0.67 x6000.go 5.00 0.90 x7000.go 6.70 1.13 x8000.go 9.44 1.38 x9000.go 11.23 1.87 x10000.go 13.78 2.09 Fixes #8259. Fixes #8354. LGTM=iant, r R=golang-codereviews, iant, r CC=golang-codereviews https://golang.org/cl/125720043 »»» TBR=rsc CC=golang-codereviews https://golang.org/cl/121600043
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
1) The arrayindexof lookup function is O(n). Replace with O(1) lookups. 2) The checkptxt function is O(n²) and is purely for debugging. Only run when the debugging flags are turned on. 3) Iterating over sparse bitmaps can be done faster word by word. Introduce and use bvnext for that. Run times before and after, on my 2.5 GHz Core i5 MacBook Pro. x.go 9.48 0.84 issue 8259 x100.go 0.01 0.01 issue 8354 x1000.go 0.10 0.10 x2000.go 0.62 0.19 x3000.go 1.33 0.34 x4000.go 2.29 0.49 x5000.go 3.89 0.67 x6000.go 5.00 0.90 x7000.go 6.70 1.13 x8000.go 9.44 1.38 x9000.go 11.23 1.87 x10000.go 13.78 2.09 Fixes golang#8259. Fixes golang#8354. LGTM=iant, r R=golang-codereviews, iant, r CC=golang-codereviews https://golang.org/cl/125720043
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
1) The arrayindexof lookup function is O(n). Replace with O(1) lookups. 2) The checkptxt function is O(n²) and is purely for debugging. Only run when the debugging flags are turned on. 3) Iterating over sparse bitmaps can be done faster word by word. Introduce and use bvnext for that. Run times before and after, on my 2.5 GHz Core i5 MacBook Pro. x.go 9.48 0.84 issue 8259 x100.go 0.01 0.01 issue 8354 x1000.go 0.10 0.10 x2000.go 0.62 0.19 x3000.go 1.33 0.34 x4000.go 2.29 0.49 x5000.go 3.89 0.67 x6000.go 5.00 0.90 x7000.go 6.70 1.13 x8000.go 9.44 1.38 x9000.go 11.23 1.87 x10000.go 13.78 2.09 Fixes golang#8259. Fixes golang#8354. LGTM=iant, r R=golang-codereviews, iant, r CC=golang-codereviews https://golang.org/cl/125720043
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
by nicolas.riesch:
The text was updated successfully, but these errors were encountered: