Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: perf.golang.org crashing on influx sync of some results #61362

Closed
prattmic opened this issue Jul 14, 2023 · 4 comments
Closed

x/build: perf.golang.org crashing on influx sync of some results #61362

prattmic opened this issue Jul 14, 2023 · 4 comments
Assignees
@prattmic
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@prattmic
Copy link
Member 

ERROR 2023-07-14T04:00:01.888178068Z [resource.labels.containerName: perf] 2023/07/14 04:00:01 Processing upload 20230714.1...
ERROR 2023-07-14T04:00:05.630862579Z [resource.labels.containerName: perf] 2023/07/14 04:00:05 http2: panic serving 10.128.0.18:20577: runtime error: invalid memory address or nil pointer dereference
ERROR 2023-07-14T04:00:05.630954809Z [resource.labels.containerName: perf] goroutine 8364363 [running]:
ERROR 2023-07-14T04:00:05.630961409Z [resource.labels.containerName: perf] net/http.(*http2serverConn).runHandler.func1()
ERROR 2023-07-14T04:00:05.630965499Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/h2_bundle.go:5899 +0x125
ERROR 2023-07-14T04:00:05.630968479Z [resource.labels.containerName: perf] panic({0xe21fa0, 0x1bff4f0})
ERROR 2023-07-14T04:00:05.630971379Z [resource.labels.containerName: perf] /usr/local/go/src/runtime/panic.go:838 +0x207
ERROR 2023-07-14T04:00:05.630974889Z [resource.labels.containerName: perf] golang.org/x/perf/benchseries.(*Builder).AllComparisonSeries(0xc000a5c3c0, {0x0, 0x0, 0xf8e0c6?}, 0x0)
ERROR 2023-07-14T04:00:05.630977779Z [resource.labels.containerName: perf] /go/pkg/mod/golang.org/x/perf@v0.0.0-20221222170352-3fd27c239283/benchseries/benchseries.go:579 +0x1d60
ERROR 2023-07-14T04:00:05.630999559Z [resource.labels.containerName: perf] golang.org/x/build/perf/app.(*App).compareAndPush(0xc0003d4600?, {0x111dca8, 0xc000723e00}, {0x1122de8, 0xc0003d4300}, 0xc0003d4600, 0x0?, {0xfadd89, 0x23})
ERROR 2023-07-14T04:00:05.631004309Z [resource.labels.containerName: perf] /app/perf/app/influx.go:335 +0x1d5
ERROR 2023-07-14T04:00:05.631009469Z [resource.labels.containerName: perf] golang.org/x/build/perf/app.(*App).pushRunToInflux(0xc00007f020, {0x111dca8, 0xc000723e00}, {0x1122de8, 0xc0003d4300}, {0x0?, {0xc000a937a0?, 0xc000941030?}, 0x0?})
ERROR 2023-07-14T04:00:05.631013069Z [resource.labels.containerName: perf] /app/perf/app/influx.go:304 +0x62e
ERROR 2023-07-14T04:00:05.631017669Z [resource.labels.containerName: perf] golang.org/x/build/perf/app.(*App).syncInflux(0xc00007f020, {0x111d4b0, 0xc0005c4160}, 0x0?)
ERROR 2023-07-14T04:00:05.631024059Z [resource.labels.containerName: perf] /app/perf/app/influx.go:91 +0x8b5
ERROR 2023-07-14T04:00:05.631027589Z [resource.labels.containerName: perf] net/http.HandlerFunc.ServeHTTP(0xc000341300?, {0x111d4b0?, 0xc0005c4160?}, 0xc0004d8c40?)
ERROR 2023-07-14T04:00:05.631030379Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/server.go:2084 +0x2f
ERROR 2023-07-14T04:00:05.631033299Z [resource.labels.containerName: perf] net/http.(*ServeMux).ServeHTTP(0x617202?, {0x111d4b0, 0xc0005c4160}, 0xc00088a700)
ERROR 2023-07-14T04:00:05.631036049Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/server.go:2462 +0x149
ERROR 2023-07-14T04:00:05.631039129Z [resource.labels.containerName: perf] golang.org/x/build/internal/https.ListenAndServeOpts.func1({0x111d4b0?, 0xc0005c4160?}, 0xc00003e030?)
ERROR 2023-07-14T04:00:05.631041899Z [resource.labels.containerName: perf] /app/internal/https/https.go:73 +0x9c
ERROR 2023-07-14T04:00:05.631045129Z [resource.labels.containerName: perf] net/http.HandlerFunc.ServeHTTP(0x0?, {0x111d4b0?, 0xc0005c4160?}, 0xc00082e8e8?)
ERROR 2023-07-14T04:00:05.631047869Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/server.go:2084 +0x2f
ERROR 2023-07-14T04:00:05.631050609Z [resource.labels.containerName: perf] net/http.serverHandler.ServeHTTP({0x6210c0?}, {0x111d4b0, 0xc0005c4160}, 0xc00088a700)
ERROR 2023-07-14T04:00:05.631053809Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/server.go:2916 +0x43b
ERROR 2023-07-14T04:00:05.631056989Z [resource.labels.containerName: perf] net/http.initALPNRequest.ServeHTTP({{0x111dd50?, 0xc000a78de0?}, 0xc000c56e00?, {0xc0003422a0?}}, {0x111d4b0, 0xc0005c4160}, 0xc00088a700)
ERROR 2023-07-14T04:00:05.631061219Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/server.go:3523 +0x245
ERROR 2023-07-14T04:00:05.631064059Z [resource.labels.containerName: perf] net/http.(*http2serverConn).runHandler(0x111c598?, 0x1c648e8?, 0x0?, 0x0?)
ERROR 2023-07-14T04:00:05.631066859Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/h2_bundle.go:5906 +0x78
ERROR 2023-07-14T04:00:05.631069639Z [resource.labels.containerName: perf] created by net/http.(*http2serverConn).processHeaders
ERROR 2023-07-14T04:00:05.631072409Z [resource.labels.containerName: perf] /usr/local/go/src/net/http/h2_bundle.go:5641 +0x59b

https://perf.golang.org/search?q=upload:20230714.1

cc @mknyszek @dr2chase
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 14, 2023
@gopherbot gopherbot added this to the Unreleased milestone Jul 14, 2023
@prattmic prattmic self-assigned this Jul 14, 2023
@prattmic prattmic added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 14, 2023
@gopherbot
Copy link

Change https://go.dev/cl/510475 mentions this issue: benchseries: ignore results with mismatched compare value

gopherbot pushed a commit to golang/perf that referenced this issue Jul 17, 2023
@prattmic @gopherbot
benchseries: ignore results with mismatched compare value
1ba3a21 
Currently, benchseries assumes that results with compare value ==
denominator is in the denominator set, and everything else is in the
numerator.

In addition to being confusing (BuilderOptions.Numerator is completely
ignored), this breaks things when the compare key isn't set at all. In
that case cmpCfg.StringValues() is always "", so everything ends up in
the numerator and nothing in the denominator, which causes a later crash
due to a missing denominator [1].

Be more explicit and only save results that explicitly fall into the
numerator or denominator.

[1] This fix isn't perfect, as benchmark results that contain only
numerator results and no denominator results will still cause a crash.
That should probably be an error or simply result in no comparisons.

For golang/go#61362.

Change-Id: I1444d0fcb101bcea0f1cdf31de7292211dcbc901
Reviewed-on: https://go-review.googlesource.com/c/perf/+/510475
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: David Chase <drchase@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/510495 mentions this issue: DO NOT SUBMIT: command to process a benchfmt file through per-influx

@gopherbot
Copy link

Change https://go.dev/cl/510496 mentions this issue: all: update golang.org/x/perf

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 17, 2023
@findleyr
Copy link
Contributor

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prattmic prattmic

Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@prattmic @dmitshur @gopherbot @findleyr