Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test/bench/go1: benchmark is noisy on node/wasm #33749

Closed
ghost opened this issue Aug 21, 2019 · 9 comments
Closed

test/bench/go1: benchmark is noisy on node/wasm #33749

ghost opened this issue Aug 21, 2019 · 9 comments
Labels
arch-wasm WebAssembly issues FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Testing An issue that has been verified to require only test changes, not just a test failure. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@ghost
Copy link

ghost commented Aug 21, 2019 •
edited by ghost

Run the go1 benchmark twice, with the same go version.

(with node 8.16.0 and the recent master branch of go)

name                   old time/op    new time/op    delta
BinaryTree17              25.7s ± 0%     25.7s ± 0%  +0.13%  (p=0.000 n=25+29)
Fannkuch11                13.6s ± 0%     13.9s ± 0%  +1.71%  (p=0.000 n=29+30)
FmtFprintfEmpty           273ns ± 0%     272ns ± 0%  -0.37%  (p=0.000 n=25+30)
FmtFprintfString          565ns ± 1%     554ns ± 0%  -1.91%  (p=0.000 n=30+20)
FmtFprintfInt             716ns ± 1%     674ns ± 0%  -5.88%  (p=0.000 n=30+30)
FmtFprintfIntInt         1.24µs ± 0%    1.16µs ± 0%  -6.35%  (p=0.000 n=30+29)
FmtFprintfPrefixedInt    1.19µs ± 1%    1.11µs ± 0%  -6.89%  (p=0.000 n=30+30)
FmtFprintfFloat          2.38µs ± 0%    2.56µs ± 0%  +7.70%  (p=0.000 n=30+30)
FmtManyArgs              5.77µs ± 0%    5.69µs ± 0%  -1.39%  (p=0.000 n=30+29)
GobDecode                65.8ms ± 0%    67.5ms ± 0%  +2.57%  (p=0.000 n=30+29)
GobEncode                33.5ms ± 0%    34.5ms ± 0%  +2.91%  (p=0.000 n=30+30)
Gzip                      1.09s ± 0%     1.08s ± 0%  -0.81%  (p=0.000 n=28+28)
Gunzip                    239ms ± 0%     229ms ± 0%  -4.21%  (p=0.000 n=27+29)
HTTPClientServer          323µs ± 0%     327µs ± 0%  +1.22%  (p=0.000 n=29+28)
JSONEncode               70.0ms ± 0%    71.9ms ± 0%  +2.63%  (p=0.000 n=30+29)
JSONDecode                591ms ± 0%     572ms ± 0%  -3.30%  (p=0.000 n=30+29)
Mandelbrot200            5.79ms ± 0%    5.79ms ± 0%  +0.01%  (p=0.000 n=25+25)
GoParse                  38.8ms ± 0%    39.5ms ± 0%  +1.69%  (p=0.000 n=29+29)
RegexpMatchEasy0_32       654ns ± 0%     670ns ± 0%  +2.52%  (p=0.000 n=29+25)
RegexpMatchEasy0_1K      5.72µs ± 1%    5.41µs ± 0%  -5.55%  (p=0.000 n=28+29)
RegexpMatchEasy1_32       578ns ± 0%     546ns ± 0%  -5.46%  (p=0.000 n=24+28)
RegexpMatchEasy1_1K      5.68µs ± 0%    5.74µs ± 0%  +1.11%  (p=0.000 n=29+29)
RegexpMatchMedium_32     26.8ns ± 0%    26.8ns ± 0%    ~     (all equal)
RegexpMatchMedium_1K      208µs ± 0%     225µs ± 0%  +8.36%  (p=0.000 n=28+25)
RegexpMatchHard_32       11.7µs ± 0%    12.1µs ± 0%  +3.32%  (p=0.000 n=30+30)
RegexpMatchHard_1K        341µs ± 0%     360µs ± 0%  +5.57%  (p=0.000 n=30+29)
Revcomp                   2.32s ± 0%     2.31s ± 0%  -0.65%  (p=0.000 n=23+26)
Template                  723ms ± 0%     724ms ± 0%  +0.16%  (p=0.000 n=30+30)
TimeParse                3.59µs ± 0%    3.66µs ± 0%  +1.96%  (p=0.000 n=29+29)
TimeFormat               3.93µs ± 1%    3.74µs ± 0%  -4.74%  (p=0.000 n=30+30)
[Geo mean]                340µs          339µs       -0.21%
	
name                   old speed      new speed      delta
GobDecode              11.7MB/s ± 0%  11.4MB/s ± 0%  -2.51%  (p=0.000 n=30+29)
GobEncode              22.9MB/s ± 0%  22.2MB/s ± 0%  -2.82%  (p=0.000 n=30+29)
Gzip                   17.8MB/s ± 0%  17.9MB/s ± 0%  +0.83%  (p=0.000 n=24+30)
Gunzip                 81.0MB/s ± 0%  84.6MB/s ± 0%  +4.40%  (p=0.000 n=28+28)
JSONEncode             27.7MB/s ± 0%  27.0MB/s ± 0%  -2.56%  (p=0.000 n=30+29)
JSONDecode             3.28MB/s ± 0%  3.40MB/s ± 0%  +3.45%  (p=0.000 n=30+30)
GoParse                1.49MB/s ± 0%  1.47MB/s ± 0%  -1.34%  (p=0.000 n=30+30)
RegexpMatchEasy0_32    49.0MB/s ± 0%  47.7MB/s ± 0%  -2.51%  (p=0.000 n=28+28)
RegexpMatchEasy0_1K     179MB/s ± 1%   189MB/s ± 0%  +5.87%  (p=0.000 n=28+29)
RegexpMatchEasy1_32    55.4MB/s ± 0%  58.6MB/s ± 0%  +5.78%  (p=0.000 n=29+28)
RegexpMatchEasy1_1K     180MB/s ± 0%   178MB/s ± 0%  -1.09%  (p=0.000 n=29+29)
RegexpMatchMedium_32   37.3MB/s ± 0%  37.3MB/s ± 0%  -0.15%  (p=0.000 n=27+30)
RegexpMatchMedium_1K   4.92MB/s ± 0%  4.54MB/s ± 0%  -7.72%  (p=0.000 n=30+30)
RegexpMatchHard_32     2.73MB/s ± 0%  2.64MB/s ± 0%  -3.41%  (p=0.000 n=24+30)
RegexpMatchHard_1K     3.01MB/s ± 0%  2.85MB/s ± 0%  -5.22%  (p=0.000 n=30+30)
Revcomp                 109MB/s ± 0%   110MB/s ± 0%  +0.65%  (p=0.000 n=23+27)
Template               2.69MB/s ± 0%  2.68MB/s ± 0%  -0.19%  (p=0.000 n=30+30)
[Geo mean]             17.9MB/s       17.8MB/s       -0.57%
@ghost
Copy link
Author

ghost commented Aug 21, 2019

any better way to check if my optimzation on WASM works as expected ?

@bcmills
Copy link
Contributor

bcmills commented Aug 21, 2019

Run the go1 benchmark twice, with the same go version.

On what kind of machine, under what conditions? (Did you use perflock first? What else was running on the machine at the time?)

CC @mvdan (for his related recent talks on benchmarking), @dr2chase (general interest in benchmarking), @neelance (WASM)

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Testing An issue that has been verified to require only test changes, not just a test failure. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 21, 2019
@bcmills bcmills changed the title The go1 benchmark is noisy on node/wasm test/bench/go1: benchmark is noisy on node/wasm Aug 21, 2019
@bcmills bcmills added the arch-wasm WebAssembly issues label Aug 21, 2019
@bcmills bcmills added this to the Unplanned milestone Aug 21, 2019
@mvdan
Copy link
Member

mvdan commented Aug 21, 2019

On top of what @bcmills said: Is the machine idle? Have you tried randomizing the order of the benchmark executions? Is the machine throttling during any of the benchmarks?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 21, 2019
@cherrymui
Copy link
Member

I'm not sure what we can do on the Go side. The WASM bytecode is run by V8 (or any other execution engine), which has a complex JIT system.

@ghost
Copy link
Author

ghost commented Aug 22, 2019 •
edited by ghost

my go1 test condition:

  1. a single desktop machine, with quad-core i5 @ 2.6GHz, ubuntu 18.04 (and its default GUI).
  2. the benchmark test ran from 8:00 PM to 6:00 AM, for 8 rounds, without any other active program running simultaneously, and the GUI was shutdown by the screen saver.
  3. idle or throttling should not happen, during the benchmark's running.
  4. no other special setting.

Actually I use this environment for go1's AMD64/i386 benchmark tests, whose results are OK.

@agnivade
Copy link
Contributor

I agree that there isn't much that can be done on the Go side. The entire execution is run by V8 which can optimize or de-optimize code at will. We can run using perflock on a quiet machine and hope for the best. Perhaps also run with Node 10 to check if things are any better.

@beenshi - Did you want anything else to be done on this issue ?

@agnivade agnivade added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 21, 2019
@ghost
Copy link
Author

ghost commented Sep 25, 2019

Have you tried the way you suggested? Or even the result can be shown? Though we can do less on the go side, I expect to see a solid way to verify WASM backend's performance.

@agnivade
Copy link
Contributor

agnivade commented May 8, 2020

I haven't tried with perflock, but on some optimization CLs, I have indeed noticed performance improvements with the Go1 benchmarks. Although I confess I haven't tried running the test multiple times with the same Go version. But what I was trying to say is that, personally, I don't see an actionable item that can be done in the code to address this.

@neelance
Copy link
Member

This does not seem to be actionable. I'm closing it for now. Maybe in the future some non-V8 runtime will give more stable results.

@golang golang locked and limited conversation to collaborators Oct 28, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-wasm WebAssembly issues FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Testing An issue that has been verified to require only test changes, not just a test failure. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@neelance @agnivade @mvdan @bcmills @gopherbot @cherrymui