Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/benchmarks: monitor binary size #57770

Open
aclements opened this issue Jan 13, 2023 · 7 comments
Open

x/benchmarks: monitor binary size #57770

aclements opened this issue Jan 13, 2023 · 7 comments
Assignees
Labels
binary-size compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@aclements
Copy link
Member

We should monitor binary size in the performance dashboard.

Probably this makes the most sense to report as additional metrics on the GoBuild* benchmarks, which means integrating this into Sweet. As a starting point, we can just report total binary size. It would be nice to break it out into at least text and data, but that would mostly serve to help us understand changes in binary size.

We do have a way to indicate that certain metrics have an "exact" distribution, but I'm not sure if that's wired into the dashboard analysis.

@mknyszek @prattmic @dr2chase

@aclements aclements added this to the Unreleased milestone Jan 13, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 13, 2023
@golang golang deleted a comment Jan 13, 2023
@qmuntal
Copy link
Member

qmuntal commented Jan 13, 2023

Given that the binary size metric is builder-agnostic (cgo aside), could we generate this metric for all first-class ports by cross-compiling from the linux-amd64-perf builder?

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 13, 2023
@prattmic
Copy link
Member

prattmic commented Jan 13, 2023

We do have a way to indicate that certain metrics have an "exact" distribution, but I'm not sure if that's wired into the dashboard analysis.

I'm not sure if it is wired up or not, but if not we already run GoBuild benchmarks N times, so we can just report the same binary size N times which should result in low/zero variance.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462718 mentions this issue: cmd/bench: benchmark binary size and build time via bent

gopherbot pushed a commit to golang/benchmarks that referenced this issue Jan 18, 2023
benchsize adds binary size benchmark results for each package in bent.
bent doesn't log these to stdout by default, we need to add -v for that.

It turns out that bent also has build time benchmarks that it doesn't
log without -v. So now we get those for free too.

Updates golang/go#57770.

Change-Id: I2b714a4f2e73842fe79d5d227a397efeca794d06
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462718
Run-TryBot: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: David Chase <drchase@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462956 mentions this issue: cmd/bent: log toolchain prior to compilation

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462957 mentions this issue: cmd/bent: clean up benchsize

gopherbot pushed a commit to golang/benchmarks that referenced this issue Jan 20, 2023
Compilation may emit compilation benchmark results, so we need to log
the toolchain in use first.

For golang/go#57770.

Change-Id: Ia1fdbbabba9a556ff6be720fcd7455de23d3d147
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462956
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Pratt <mpratt@google.com>
gopherbot pushed a commit to golang/benchmarks that referenced this issue Jan 20, 2023
1. Remove the section from the benchmark names. It is already in the
   unit, so this will provide better grouping.

2. Capitalize the first letter in the name. This makes the benchmark
   name match the name used for the build time benchmark.

3. Change section size parsing to use exact matches.

The existing substring match could match multiple lines. e.g.,
`zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this
happens, $zdebug_loc contains a string like "12345 123", which breaks
the later addition in `expr`.

Fix this by requiring an exact match of the section name field with the
desired section name. Rather than making a more complex regexp, just
include this as a condition in the awk command. Darwin and GNU size has
different formats, so they each require their own format.

For golang/go#57770.

Change-Id: I482bb90b29578179a3b59637c265580f06becaf8
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462957
Auto-Submit: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/463195 mentions this issue: cmd/bent: move toolchain key print to lower layer

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/463196 mentions this issue: cmd/bent: add flag to disable build time reporting

gopherbot pushed a commit to golang/benchmarks that referenced this issue Jan 26, 2023
CL 462956 added logging of the toolchain key to properly annotate build
/ after-build benchmarks. This worked fine for the perf dashboard, but
results also go to a file, which this logging did not do. Additionally,
it logged even in non-verbose mode, when benchmark results are not.

Move this logging down near the benchmarks themselves to fix these
inconsistencies.

For golang/go#57770.

Change-Id: I6502092bd93a2dd6932018989c69fba5b51adff8
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/463195
Reviewed-by: David Chase <drchase@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
gopherbot pushed a commit to golang/benchmarks that referenced this issue Jan 26, 2023
We do not want these benchmarks in cmd/bench because we only run builds
once and thus do not get statistically significant results anyway.

For golang/go#57770.

Change-Id: I75987416458f826ee0a3e71c69f36fd48206eb77
Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/463196
Run-TryBot: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
benchsize adds binary size benchmark results for each package in bent.
bent doesn't log these to stdout by default, we need to add -v for that.

It turns out that bent also has build time benchmarks that it doesn't
log without -v. So now we get those for free too.

Updates golang/go#57770.

Change-Id: I2b714a4f2e73842fe79d5d227a397efeca794d06
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
Compilation may emit compilation benchmark results, so we need to log
the toolchain in use first.

For golang/go#57770.

Change-Id: Ia1fdbbabba9a556ff6be720fcd7455de23d3d147
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
1. Remove the section from the benchmark names. It is already in the
   unit, so this will provide better grouping.

2. Capitalize the first letter in the name. This makes the benchmark
   name match the name used for the build time benchmark.

3. Change section size parsing to use exact matches.

The existing substring match could match multiple lines. e.g.,
`zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this
happens, $zdebug_loc contains a string like "12345 123", which breaks
the later addition in `expr`.

Fix this by requiring an exact match of the section name field with the
desired section name. Rather than making a more complex regexp, just
include this as a condition in the awk command. Darwin and GNU size has
different formats, so they each require their own format.

For golang/go#57770.

Change-Id: I482bb90b29578179a3b59637c265580f06becaf8
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
1. Remove the section from the benchmark names. It is already in the
   unit, so this will provide better grouping.

2. Capitalize the first letter in the name. This makes the benchmark
   name match the name used for the build time benchmark.

3. Change section size parsing to use exact matches.

The existing substring match could match multiple lines. e.g.,
`zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this
happens, $zdebug_loc contains a string like "12345 123", which breaks
the later addition in `expr`.

Fix this by requiring an exact match of the section name field with the
desired section name. Rather than making a more complex regexp, just
include this as a condition in the awk command. Darwin and GNU size has
different formats, so they each require their own format.

For golang/go#57770.

Change-Id: I482bb90b29578179a3b59637c265580f06becaf8
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
CL 462956 added logging of the toolchain key to properly annotate build
/ after-build benchmarks. This worked fine for the perf dashboard, but
results also go to a file, which this logging did not do. Additionally,
it logged even in non-verbose mode, when benchmark results are not.

Move this logging down near the benchmarks themselves to fix these
inconsistencies.

For golang/go#57770.

Change-Id: I6502092bd93a2dd6932018989c69fba5b51adff8
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
We do not want these benchmarks in cmd/bench because we only run builds
once and thus do not get statistically significant results anyway.

For golang/go#57770.

Change-Id: I75987416458f826ee0a3e71c69f36fd48206eb77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binary-size compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Development

No branches or pull requests

6 participants