-
Notifications
You must be signed in to change notification settings - Fork 18k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/benchmarks: monitor binary size #57770
Comments
Given that the binary size metric is builder-agnostic (cgo aside), could we generate this metric for all first-class ports by cross-compiling from the |
I'm not sure if it is wired up or not, but if not we already run GoBuild benchmarks N times, so we can just report the same binary size N times which should result in low/zero variance. |
Change https://go.dev/cl/462718 mentions this issue: |
benchsize adds binary size benchmark results for each package in bent. bent doesn't log these to stdout by default, we need to add -v for that. It turns out that bent also has build time benchmarks that it doesn't log without -v. So now we get those for free too. Updates golang/go#57770. Change-Id: I2b714a4f2e73842fe79d5d227a397efeca794d06 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462718 Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: David Chase <drchase@google.com>
Change https://go.dev/cl/462956 mentions this issue: |
Change https://go.dev/cl/462957 mentions this issue: |
Compilation may emit compilation benchmark results, so we need to log the toolchain in use first. For golang/go#57770. Change-Id: Ia1fdbbabba9a556ff6be720fcd7455de23d3d147 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462956 Reviewed-by: Michael Knyszek <mknyszek@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com>
1. Remove the section from the benchmark names. It is already in the unit, so this will provide better grouping. 2. Capitalize the first letter in the name. This makes the benchmark name match the name used for the build time benchmark. 3. Change section size parsing to use exact matches. The existing substring match could match multiple lines. e.g., `zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this happens, $zdebug_loc contains a string like "12345 123", which breaks the later addition in `expr`. Fix this by requiring an exact match of the section name field with the desired section name. Rather than making a more complex regexp, just include this as a condition in the awk command. Darwin and GNU size has different formats, so they each require their own format. For golang/go#57770. Change-Id: I482bb90b29578179a3b59637c265580f06becaf8 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/462957 Auto-Submit: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
Change https://go.dev/cl/463195 mentions this issue: |
Change https://go.dev/cl/463196 mentions this issue: |
CL 462956 added logging of the toolchain key to properly annotate build / after-build benchmarks. This worked fine for the perf dashboard, but results also go to a file, which this logging did not do. Additionally, it logged even in non-verbose mode, when benchmark results are not. Move this logging down near the benchmarks themselves to fix these inconsistencies. For golang/go#57770. Change-Id: I6502092bd93a2dd6932018989c69fba5b51adff8 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/463195 Reviewed-by: David Chase <drchase@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Michael Pratt <mpratt@google.com>
We do not want these benchmarks in cmd/bench because we only run builds once and thus do not get statistically significant results anyway. For golang/go#57770. Change-Id: I75987416458f826ee0a3e71c69f36fd48206eb77 Reviewed-on: https://go-review.googlesource.com/c/benchmarks/+/463196 Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com>
benchsize adds binary size benchmark results for each package in bent. bent doesn't log these to stdout by default, we need to add -v for that. It turns out that bent also has build time benchmarks that it doesn't log without -v. So now we get those for free too. Updates golang/go#57770. Change-Id: I2b714a4f2e73842fe79d5d227a397efeca794d06
Compilation may emit compilation benchmark results, so we need to log the toolchain in use first. For golang/go#57770. Change-Id: Ia1fdbbabba9a556ff6be720fcd7455de23d3d147
1. Remove the section from the benchmark names. It is already in the unit, so this will provide better grouping. 2. Capitalize the first letter in the name. This makes the benchmark name match the name used for the build time benchmark. 3. Change section size parsing to use exact matches. The existing substring match could match multiple lines. e.g., `zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this happens, $zdebug_loc contains a string like "12345 123", which breaks the later addition in `expr`. Fix this by requiring an exact match of the section name field with the desired section name. Rather than making a more complex regexp, just include this as a condition in the awk command. Darwin and GNU size has different formats, so they each require their own format. For golang/go#57770. Change-Id: I482bb90b29578179a3b59637c265580f06becaf8
1. Remove the section from the benchmark names. It is already in the unit, so this will provide better grouping. 2. Capitalize the first letter in the name. This makes the benchmark name match the name used for the build time benchmark. 3. Change section size parsing to use exact matches. The existing substring match could match multiple lines. e.g., `zdebug_loc` matching `.zdebug_loc` and `.zdebug_loclists`. When this happens, $zdebug_loc contains a string like "12345 123", which breaks the later addition in `expr`. Fix this by requiring an exact match of the section name field with the desired section name. Rather than making a more complex regexp, just include this as a condition in the awk command. Darwin and GNU size has different formats, so they each require their own format. For golang/go#57770. Change-Id: I482bb90b29578179a3b59637c265580f06becaf8
CL 462956 added logging of the toolchain key to properly annotate build / after-build benchmarks. This worked fine for the perf dashboard, but results also go to a file, which this logging did not do. Additionally, it logged even in non-verbose mode, when benchmark results are not. Move this logging down near the benchmarks themselves to fix these inconsistencies. For golang/go#57770. Change-Id: I6502092bd93a2dd6932018989c69fba5b51adff8
We do not want these benchmarks in cmd/bench because we only run builds once and thus do not get statistically significant results anyway. For golang/go#57770. Change-Id: I75987416458f826ee0a3e71c69f36fd48206eb77
We should monitor binary size in the performance dashboard.
Probably this makes the most sense to report as additional metrics on the GoBuild* benchmarks, which means integrating this into Sweet. As a starting point, we can just report total binary size. It would be nice to break it out into at least text and data, but that would mostly serve to help us understand changes in binary size.
We do have a way to indicate that certain metrics have an "exact" distribution, but I'm not sure if that's wired into the dashboard analysis.
@mknyszek @prattmic @dr2chase
The text was updated successfully, but these errors were encountered: