Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: linux-amd64-perf repeatedly failing on x/tools with "error committing perfdata file: upload failed: 500 Internal Server Error no valid benchmark lines found" #58875

Closed
dmitshur opened this issue Mar 5, 2023 · 7 comments
Closed

x/build: linux-amd64-perf repeatedly failing on x/tools with "error committing perfdata file: upload failed: 500 Internal Server Error no valid benchmark lines found" #58875

dmitshur opened this issue Mar 5, 2023 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Mar 5, 2023 •
edited

Looking at https://build.golang.org/?repo=golang.org%2fx%2ftools, the linux-amd64-perf builder column has some "ok" rows but most are empty.

At https://farmer.golang.org/#completed right now there are many repeated recent failures:

linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 59s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 5m33s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 8m45s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 13m14s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 16m30s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 21m1s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 23m29s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 28m28s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 30m40s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 35m39s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 38m16s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 43m17s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 45m57s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 51m0s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 53m30s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 58m24s ago
linux-amd64-perf rev 0d288f90 (sub-repo tools rev c0742f58); failed, 1h1m14s ago
linux-amd64-perf rev 26eeaec8 (sub-repo tools rev c0742f58); failed, 1h6m11s ago

It fails on the "upload_bench_results" step:

Error: runTests: error committing perfdata file: upload failed: 500 Internal Server Error
no valid benchmark lines found
Full Log 
  builder: linux-amd64-perf
      rev: 0d288f90c4da361044b5876bf15985abb98fae07
 buildlet: http://10.128.0.82 GCE VM: buildlet-linux-amd64-perf-rn7ca4de1
  started: 2023-03-05 07:42:31.988741418 +0000 UTC m=+211383.907853116
    ended: 2023-03-05 07:50:10.21701817 +0000 UTC m=+211842.136129857
  success: false

Events:
  2023-03-05T07:42:32Z get_buildlet 
  2023-03-05T07:43:18Z finish_get_buildlet after 46.5s
  2023-03-05T07:43:18Z using_buildlet 10.128.0.82:80
  2023-03-05T07:43:18Z write_go_bootstrap_tar 
  2023-03-05T07:43:18Z write_version_tar 
  2023-03-05T07:43:18Z get_source go@0d288f90c4da361044b5876bf15985abb98fae07
  2023-03-05T07:43:18Z finish_get_source after 0s; go@0d288f90c4da361044b5876bf15985abb98fae07
  2023-03-05T07:43:18Z write_go_src_tar 
  2023-03-05T07:43:19Z finish_write_go_src_tar after 966.5ms
  2023-03-05T07:43:20Z finish_write_go_bootstrap_tar after 1.46s
  2023-03-05T07:43:20Z make_and_test 
  2023-03-05T07:43:20Z make src/make.bash
  2023-03-05T07:44:35Z finish_make after 1m15.6s; src/make.bash
  2023-03-05T07:44:35Z install_baseline 
  2023-03-05T07:44:35Z list_go_releases 
  2023-03-05T07:44:35Z finish_list_go_releases after 5.2ms
  2023-03-05T07:44:35Z write_version_tar 
  2023-03-05T07:44:35Z get_source go@8656c03fee94ce9cdc4da120b831c2fb9fd68d9d
  2023-03-05T07:44:35Z finish_get_source after 0s; go@8656c03fee94ce9cdc4da120b831c2fb9fd68d9d
  2023-03-05T07:44:35Z write_go_src_tar 
  2023-03-05T07:44:36Z finish_write_go_src_tar after 882.8ms
  2023-03-05T07:44:36Z make src/make.bash
  2023-03-05T07:45:48Z finish_make after 1m11.9s; src/make.bash
  2023-03-05T07:45:48Z finish_install_baseline after 1m13.1s
  2023-03-05T07:45:48Z fetching_benchmarks 
  2023-03-05T07:45:48Z get_source benchmarks@a97ee2e8b498a20e35cbf1e1c03576b513556ba3
  2023-03-05T07:45:49Z finish_get_source after 0s; benchmarks@a97ee2e8b498a20e35cbf1e1c03576b513556ba3
  2023-03-05T07:45:49Z finish_fetching_benchmarks after 110.2ms
  2023-03-05T07:45:49Z fetching_subrepo tools
  2023-03-05T07:45:49Z get_source tools@c0742f584bb22bf982da6377387cc647cafc0ffd
  2023-03-05T07:45:49Z finish_get_source after 0s; tools@c0742f584bb22bf982da6377387cc647cafc0ffd
  2023-03-05T07:45:49Z get_source tools@611cff71b9afeb81ae6eb16c5ce9376c051b1828
  2023-03-05T07:45:49Z finish_get_source after 0s; tools@611cff71b9afeb81ae6eb16c5ce9376c051b1828
  2023-03-05T07:45:49Z running_benchmark_tests tools
  2023-03-05T07:50:09Z upload_bench_results 
  2023-03-05T07:50:10Z finish_upload_bench_results after 177.5ms; err=error committing perfdata file: upload failed: 500 Internal Server Error
no valid benchmark lines found

  2023-03-05T07:50:10Z finish_running_benchmark_tests after 4m20.4s; err=error committing perfdata file: upload failed: 500 Internal Server Error
no valid benchmark lines found
; tools
  2023-03-05T07:50:10Z finish_make_and_test after 6m50s; err=runTests: error committing perfdata file: upload failed: 500 Internal Server Error
no valid benchmark lines found


Build log:
linux-amd64-perf at 0d288f90c4da361044b5876bf15985abb98fae07 building tools at c0742f584bb22bf982da6377387cc647cafc0ffd

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-amd64-perf-rn7ca4de1" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=12s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-amd64-perf" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PWD=/workdir/go/src"] in dir /workdir/go/src

Building Go cmd/dist using /workdir/go1.4. (go1.17.13 linux/amd64)
Building Go toolchain1 using /workdir/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /workdir/go
Installed commands in /workdir/go/bin
Baseline toolchain 8656c03fee94ce9cdc4da120b831c2fb9fd68d9d
:: Running /workdir/go-baseline/src/make.bash with args ["/workdir/go-baseline/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-amd64-perf-rn7ca4de1" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=12s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64-perf" "GOBIN=" "GOROOT_BOOTSTRAP=/workdir/go" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PWD=/workdir/go-baseline/src"] in dir /workdir/go-baseline/src

Building Go cmd/dist using /workdir/go. (go1.19.6 linux/amd64)
Building Go toolchain1 using /workdir/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /workdir/go-baseline
Installed commands in /workdir/go-baseline/bin
Baseline subrepo 611cff71b9afeb81ae6eb16c5ce9376c051b1828
:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "run" "golang.org/x/benchmarks/cmd/bench"] and env ["HOSTNAME=buildlet-linux-amd64-perf-rn7ca4de1" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=12s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-amd64-perf" "BENCH_BASELINE_GOROOT=/workdir/go-baseline" "BENCH_BRANCH=release-branch.go1.19" "BENCH_REPOSITORY=tools" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://gk3-services-nap-jetxd907-d9790f34-bx4g.c.symbolic-datum-552.internal:30157" "BENCH_SUBREPO_PATH=/workdir/subrepo" "BENCH_SUBREPO_BASELINE_PATH=/workdir/subrepo-baseline" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/workdir/benchmarks"] in dir /workdir/benchmarks

go: downloading github.com/BurntSushi/toml v1.0.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
2023/03/05 07:45:50 Load average: 7.30 3.33 1.29 1/277 21007
2023/03/05 07:45:50 Waiting for load average to drop below 0.20...
2023/03/05 07:46:20 Load average: 4.43 3.01 1.25 1/277 21007
2023/03/05 07:46:20 Waiting for load average to drop below 0.20...
2023/03/05 07:46:50 Load average: 2.68 2.72 1.21 1/277 21007
2023/03/05 07:46:50 Waiting for load average to drop below 0.20...
2023/03/05 07:47:20 Load average: 1.62 2.46 1.17 1/277 21007
2023/03/05 07:47:20 Waiting for load average to drop below 0.20...
2023/03/05 07:47:50 Load average: 0.98 2.22 1.13 1/277 21007
2023/03/05 07:47:50 Waiting for load average to drop below 0.20...
2023/03/05 07:48:20 Load average: 0.59 2.01 1.09 1/251 21007
2023/03/05 07:48:20 Waiting for load average to drop below 0.20...
2023/03/05 07:48:50 Load average: 0.36 1.82 1.06 1/249 21007
2023/03/05 07:48:50 Waiting for load average to drop below 0.20...
2023/03/05 07:49:20 Load average: 0.22 1.64 1.02 1/249 21007
2023/03/05 07:49:20 Waiting for load average to drop below 0.20...
2023/03/05 07:49:50 Load average: 0.13 1.48 0.99 1/249 21007
repository: tools
branch: release-branch.go1.19
runstamp: 2023-03-05T07:49:50.220705407Z
2023/03/05 07:49:50 Running sub-repo benchmarks for tools
go: downloading golang.org/x/tools v0.3.1-0.20221213193459-ca17b2c27ca8
go: downloading github.com/sergi/go-diff v1.1.0
go: downloading honnef.co/go/tools v0.3.3
go: downloading mvdan.cc/gofumpt v0.4.0
go: downloading mvdan.cc/xurls/v2 v2.4.0
go: downloading golang.org/x/mod v0.7.0
go: downloading golang.org/x/text v0.4.0
go: downloading github.com/google/go-cmp v0.5.9
go: downloading golang.org/x/sys v0.2.0
go: downloading golang.org/x/sync v0.1.0
go: downloading golang.org/x/vuln v0.0.0-20221109205719-3af8368ee4fe
go: downloading golang.org/x/exp/typeparams v0.0.0-20221031165847-c99f073a8326
go: downloading golang.org/x/exp v0.0.0-20221031165847-c99f073a8326
go: downloading github.com/BurntSushi/toml v1.2.1
go: downloading honnef.co/go/tools v0.4.2
go: downloading golang.org/x/mod v0.9.0
go: downloading golang.org/x/text v0.8.0
go: downloading golang.org/x/sys v0.6.0
go: downloading golang.org/x/vuln v0.0.0-20230110180137-6ad3e3d07815
go: downloading golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e
go: downloading golang.org/x/exp/typeparams v0.0.0-20221212164502-fae10dda9338
toolchain: baseline
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
PASS
ok  	golang.org/x/tools/gopls/internal/regtest/bench	0.036s
toolchain: experiment
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
=== RUN   TestPilosaStress
    stress_test.go:37: -pilosa_path not configured
--- SKIP: TestPilosaStress (0.00s)
PASS
ok  	golang.org/x/tools/gopls/internal/regtest/bench	0.037s


Error: runTests: error committing perfdata file: upload failed: 500 Internal Server Error
no valid benchmark lines found

Also relevant is that this builder is currently considered as one that must be passing for purpose of golang.org/x tagging (CC @heschi):

# findGreen("x/tools", "c0742f584bb22bf982da6377387cc647cafc0ffd")
branch release-branch.go1.19 at 0d288f90c4da361044b5876bf15985abb98fae07: green = false (missing: [linux-amd64-perf])
branch release-branch.go1.20 at 26eeaec89c1e78696bfa8ad000d4c8275e3bdb75: green = false (missing: [linux-amd64-perf])
branch master at b94dc384cabf75e7e8703265cd80f5324f84b642: green = true (missing: [])
rev c0742f584bb22bf982da6377387cc647cafc0ffd green = false
green = "", ok = false, err = nil

CC @golang/release, @prattmic, @mknyszek.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 5, 2023
@dmitshur dmitshur added this to the Unreleased milestone Mar 5, 2023
@dmitshur dmitshur changed the title x/build: linux-amd64-perf continuously failing on x/tools with "error committing perfdata file: upload failed: 500 Internal Server Error no valid benchmark lines found" x/build: linux-amd64-perf repeatedly failing on x/tools with "error committing perfdata file: upload failed: 500 Internal Server Error no valid benchmark lines found" Mar 5, 2023
@prattmic
Copy link
Member

prattmic commented Mar 6, 2023

cc @findleyr

There should be runs of BenchmarkGoToDefinition, those seem to be missing in the log.

FWIW, https://build.golang.org/?repo=golang.org%2fx%2ftools looks clean now, no fails. Is it possible to see more of the history at subrepo commit c0742f5?

@prattmic
Copy link
Member

prattmic commented Mar 6, 2023

Ah, the failures on the farmer are 1.19 and 1.20 release branch commits. It seems that the dashboard can't display history for those at all? :(

https://build.golang.org/?repo=golang.org%2Fx%2Ftools&branch=release-branch.go1.20

@prattmic
Copy link
Member

prattmic commented Mar 6, 2023

This was broken by https://go-review.git.corp.google.com/c/tools/+/468940/10/gopls/internal/regtest/bench/definition_test.go (cc @findleyr), which changed the name of the benchmark from BenchmarkGoToDefinition to BenchmarkDefinition.

x/benchmarks is configured to benchmark only BenchmarkGoToDefinition: https://cs.opensource.google/go/x/benchmarks/+/master:cmd/bench/gotest.go;l=47.

Switching to a new name is painful, because x/benchmarks is going to test tip x/tools vs the previous gopls release, which currently have different names for this benchmark. The simplest way to make things work would be to run all benchmarks in the package rather than filtering them, though the perf dashboard will still have a disconnect where the benchmark changes name.

@heschi
Copy link
Contributor

heschi commented Mar 6, 2023

FYI, this failure is blocking tagging of x/tools. Not saying it makes sense, but that's how I wrote it.

@findleyr
Copy link
Contributor

findleyr commented Mar 6, 2023

On it.

@gopherbot
Copy link

Change https://go.dev/cl/473756 mentions this issue: cmd/bench: update gopls benchmarks to filter to -short

@findleyr
Copy link
Contributor

findleyr commented Mar 7, 2023

The fix also needs https://go.dev/cl/473677, which I just submitted. Builders should now succeed.

@golang golang locked and limited conversation to collaborators Mar 6, 2024
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
@findleyr
cmd/bench: update gopls benchmarks to filter to -short
ff228c7 
This fixes a breakage due to renaming of BenchmarkGoToDefinition, and
provides a means for us to configure the benchmarks run by x/benchmark
in the future.

Also add some documentation, and refactor somewhat. Remove -v for gopls
benchmarks, as that prints noisy results.

Fixes golang/go#58875

Change-Id: I862ace4af029b8042d50467567bf314b2e5521ae
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
@findleyr
cmd/bench: update gopls benchmarks to filter to -short
c93803d 
Also add some documentation, and refactor somewhat.

Fixes golang/go#58875

Change-Id: I862ace4af029b8042d50467567bf314b2e5521ae
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
@findleyr
cmd/bench: update gopls benchmarks to filter to -short
b8820ff 
This fixes a breakage due to renaming of BenchmarkGoToDefinition, and
provides a means for us to configure the benchmarks run by x/benchmark
in the future.

Also add some documentation, and refactor somewhat. Remove -v for gopls
benchmarks, as that prints noisy results.

Fixes golang/go#58875

Change-Id: I862ace4af029b8042d50467567bf314b2e5521ae
DarrylWong pushed a commit to DarrylWong/go-benchmarks that referenced this issue Apr 22, 2024
@findleyr
cmd/bench: update gopls benchmarks to filter to -short
b06c3ca 
This fixes a breakage due to renaming of BenchmarkGoToDefinition, and
provides a means for us to configure the benchmarks run by x/benchmark
in the future.

Also add some documentation, and refactor somewhat.

Fixes golang/go#58875

Change-Id: I862ace4af029b8042d50467567bf314b2e5521ae
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@prattmic @dmitshur @gopherbot @heschi @findleyr