wiki: Would like a page to list community benchmarks #21664

dr2chase · 2017-08-28T15:26:49Z

What did you expect to see?

A table listing important benchmarks from the open source community, so that Go compiler and runtime developers can get a better idea of which optimizations matter and which do not, and to avoid performance surprises late in the release cycle. The existing go1 benchmark suite is not that good.

Ideally these benchmarks would have the following properties:

they should matter; does it cost money if the benchmarked code runs slowly?
they should be go-gettable and not require customized steps for building the benchmark
they should run relatively quickly; ideally each individual benchmark iteration runs in less than a second (we'll probably want different sets of benchmarks for runtime and compiler)
ideally their timings are not noisy
they should run cleanly in a restricted environment, such as a Docker container
because only a few benchmarks from each suite can be run, they should not overlap other benchmarks in the list; we don't need a half-dozen benchmarks of the same transcendental functions

Information for each benchmark would include:

a short name for the benchmark
the path to the benchmark directory
a regexp for the benchmark specifying the one or two best benchmarks to run
a contact person for questions about the benchmarks

There may be more than one directory in a project containing benchmarks, and that's okay, these can be listed as separate benchmarks.

For example:

  Name = "gonum_blas_native"
  Repo = "github.com/gonum/blas/native"
  Benchmarks = "Benchmark(DasumMediumUnitaryInc|Dnrm2MediumPosInc)"
  Contact = "SomeoneOnTheGoNumTeam@example.com"

and

  Name = "gonum_matrix_mat64"
  Repo = "github.com/gonum/matrix/mat64"
  Benchmarks = "Benchmark(MulWorkspaceDense1000Hundredth|ScaleVec10000Inc20)"
  Contact = "SomeoneOnTheGoNumTeam@example.com"

I've done some work creating a preliminary list of benchmarks I think is likely to be interesting, and it includes code from the following github projects:

   ethereum_bitutil (repo=github.com/ethereum/go-ethereum/common/bitutil)
   ethereum_storage (repo=github.com/ethereum/go-ethereum/swarm/storage)
   ethereum_ethash (repo=github.com/ethereum/go-ethereum/consensus/ethash)
   ethereum_core (repo=github.com/ethereum/go-ethereum/core)
   ethereum_sha3 (repo=github.com/ethereum/go-ethereum/crypto/sha3)
   ethereum_ecies (repo=github.com/ethereum/go-ethereum/crypto/ecies)
   ethereum_corevm (repo=github.com/ethereum/go-ethereum/core/vm)
   ethereum_trie (repo=github.com/ethereum/go-ethereum/trie)
   eolian_dsp (repo=buddin.us/eolian/dsp)
   spexs2 (repo=github.com/egonelbre/spexs2/_benchmark)
   minio (repo=github.com/minio/minio/cmd)
   gonum_blas_native (repo=github.com/gonum/blas/native)
   gonum_lapack_native (repo=github.com/gonum/lapack/native)
   gonum_matrix_mat64 (repo=github.com/gonum/matrix/mat64)
   semver (repo=github.com/Masterminds/semver)
   hugo_helpers (repo=github.com/gohugoio/hugo/helpers)
   hugo_hugolib (repo=github.com/gohugoio/hugo/hugolib)
   k8s_api (repo=k8s.io/kubernetes/pkg/api)
   k8s_schedulercache (repo=k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache)

In the list above, ethereum_whisperv5 does not appear because all the benchmarks were noisy.
I also don't have official contacts for these benchmarks, and I don't know if the benchmarks I selected are representative or not.

What did you see instead?

No such page.

@aclements

The text was updated successfully, but these errors were encountered:

kevinburke · 2017-08-28T16:56:49Z

A good place to start would probably be the list of "Popular Packages" here, which I think measures how many times a given package is imported... if the packages in that list have benchmarks, we could add them.

https://godoc.org/

odeke-em · 2017-08-28T18:42:42Z

@dr2chase awesome, thank you for starting this! It's great to see this up as a follow up of the team discussions at the Go contributor summit.

So in regards to

In the list above, ethereum_whisperv5 does not appear because all the benchmarks were noisy.
I also don't have official contacts for these benchmarks, and I don't know if the benchmarks I selected are representative or not.

For official contacts of the Ethereum benchmarks, kindly pinging @karalabe @fjl @gluk256: might y'all have some input on @dr2chase's questions?

kevinburke · 2017-08-28T19:11:58Z

I'd appreciate at least one benchmark that looks at UUID serialization. There are a few different packages.

dr2chase · 2017-08-28T19:21:15Z

If you can name an example or two of UUID serialization benchmarks, I'll give them a look.

kevinburke · 2017-08-28T19:23:20Z

https://github.com/satori/go.uuid/blob/master/benchmarks_test.go
https://github.com/google/uuid/tree/master/json_test.go (other benchmarks in other files there)

kevinburke · 2017-08-28T19:23:53Z

for logging, uber/zap has a series of logging benchmarks for popular logging libraries

https://github.com/uber-go/zap#performance

helpfully compiled here

https://github.com/uber-go/zap/tree/master/benchmarks

dlsniper · 2017-08-28T20:44:09Z

@dr2chase while I think this is awesome, and I can see the benefit / need for this, I think that this also needs to be tackled with care as people might misinterpret those projects as being reference projects and we already have a problem with the most frequent question being: what's the fastest X? (mainly, what's the fastest framework?).

dr2chase · 2017-08-28T21:19:06Z

We could add a disclaimer. Wanting to be sure that a benchmark "matters" will tend to bias us towards popular projects, however.

karalabe · 2017-08-29T07:59:47Z

I can help with getting benchmarks out of Ethereum if need be, though it will take a bit of looking around to see what's "meaningful" from Go's perspective and which ones aren't (some of our VM benchmarks are there to aid in pricing decisions, not so much in measuring raw performance).

karalabe · 2017-08-29T08:01:04Z

Perhaps another aspect that would be nice to touch on is the performance on different platforms. In our specific use cases, some hashing packages are implemented in assembly for amd64, but not for others. It might be useful to see the discrepancies between architectures too thus.

dlsniper · 2017-08-29T08:08:25Z

@dr2chase yes, disclaimer should be good enough imho (hopefully).

For some recommendations of popular packages:

http stuff: https://github.com/gorilla/mux
logging: https://github.com/sirupsen/logrus
mysql: https://github.com/go-sql-driver/mysql
postgres: https://github.com/lib/pq

Should I also list applications that contain benchmarks rather than just libraries?

flanglet · 2017-08-29T14:42:27Z

Maybe this data compression package would be useful as well:

https://github.com/flanglet/kanzi/tree/master/go/src/kanzi/benchmark

dr2chase · 2017-08-29T18:59:50Z

Note that it has to be go-gettable. That is, there should be a one-line command like this:

go get -d -t -v github.com/flanglet/kanzi/tree/master/go/src/kanzi/benchmark

that succeeds. The intent is that this be input to a program that gets benchmarks, compiles them, and runs them in a container.

kevinburke · 2018-01-18T06:20:31Z

A go-bindata benchmark would be helpful. How long does it take to e.g. compile JS, CSS, templates, images and SQL queries into a Go package.

https://github.com/jteeuwen/go-bindata
I have a fork here: https://github.com/kevinburke/go-bindata

dr2chase · 2018-01-18T18:47:28Z

I went and looked, and there are tests, but no benchmarks. Does the proposed benchmark measure

the time to generate bindata,
the time to compile the generated bindata, or
the time to access the bindata?

The last one is the one that seems relevant to application performance.

dr2chase self-assigned this Aug 28, 2017

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 29, 2018

ianlancetaylor added this to the Unplanned milestone Mar 29, 2018

rsc unassigned dr2chase Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wiki: Would like a page to list community benchmarks #21664

wiki: Would like a page to list community benchmarks #21664

dr2chase commented Aug 28, 2017

kevinburke commented Aug 28, 2017

odeke-em commented Aug 28, 2017

kevinburke commented Aug 28, 2017

dr2chase commented Aug 28, 2017

kevinburke commented Aug 28, 2017

kevinburke commented Aug 28, 2017 •

edited

dlsniper commented Aug 28, 2017

dr2chase commented Aug 28, 2017

karalabe commented Aug 29, 2017

karalabe commented Aug 29, 2017

dlsniper commented Aug 29, 2017

flanglet commented Aug 29, 2017

dr2chase commented Aug 29, 2017

kevinburke commented Jan 18, 2018

dr2chase commented Jan 18, 2018

wiki: Would like a page to list community benchmarks #21664

wiki: Would like a page to list community benchmarks #21664

Comments

dr2chase commented Aug 28, 2017

What did you expect to see?

What did you see instead?

kevinburke commented Aug 28, 2017

odeke-em commented Aug 28, 2017

kevinburke commented Aug 28, 2017

dr2chase commented Aug 28, 2017

kevinburke commented Aug 28, 2017

kevinburke commented Aug 28, 2017 • edited

dlsniper commented Aug 28, 2017

dr2chase commented Aug 28, 2017

karalabe commented Aug 29, 2017

karalabe commented Aug 29, 2017

dlsniper commented Aug 29, 2017

flanglet commented Aug 29, 2017

dr2chase commented Aug 29, 2017

kevinburke commented Jan 18, 2018

dr2chase commented Jan 18, 2018

kevinburke commented Aug 28, 2017 •

edited