Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/perf: storage test takes too long on linux-riscv64 builder #53745

Closed
mengzhuo opened this issue Jul 8, 2022 · 7 comments
Closed

x/perf: storage test takes too long on linux-riscv64 builder #53745

mengzhuo opened this issue Jul 8, 2022 · 7 comments
Assignees
Labels
arch-riscv Issues solely affecting the riscv64 architecture. Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@mengzhuo
Copy link
Contributor

mengzhuo commented Jul 8, 2022

I can run and complete x/perf less than 5 mins, however the it get failed while testing storage and not trigger the failure of this test.

....
ok  	golang.org/x/perf/internal/stats	0.462s
ok  	golang.org/x/perf/storage	0.101s

Error: runTests: error copying response: unexpected EOF

Full log:

  builder: linux-riscv64-unmatched
      rev: 1ebc983000ed411a1c06f6b8a61770be1392e707
 buildlet: http://unmatched-057.farm.rvlab.org reverse peer unmatched-057.farm.rvlab.org/43.153.1.86:50160 for host type host-linux-riscv64-unmatched
  started: 2022-07-08 07:28:47.096638795 +0000 UTC m=+33527.991837162
   status: still running

Events:
2022-07-08T07:28:47Z checking_for_snapshot
2022-07-08T07:28:47Z finish_checking_for_snapshot after 17.9ms
2022-07-08T07:28:47Z get_buildlet
2022-07-08T07:28:47Z finish_get_buildlet after 183.3ms
2022-07-08T07:28:47Z using_buildlet unmatched-057.farm.rvlab.org
2022-07-08T07:28:47Z write_snapshot_tar
2022-07-08T07:30:24Z finish_write_snapshot_tar after 1m37.4s
2022-07-08T07:30:24Z make_and_test
2022-07-08T07:30:24Z fetching_subrepo perf
2022-07-08T07:30:25Z get_source perf@84e58bfe0a7e5416369e236afa007d5d9c58a0fa
2022-07-08T07:30:25Z finish_get_source after 0s; perf@84e58bfe0a7e5416369e236afa007d5d9c58a0fa
2022-07-08T07:30:40Z listing_subrepo_modules perf
2022-07-08T07:30:40Z finish_listing_subrepo_modules after 528ms; perf
2022-07-08T07:30:40Z running_subrepo_tests perf
+460.0s (now)

Build log:
linux-riscv64-unmatched at 1ebc983000ed411a1c06f6b8a61770be1392e707 building perf at 84e58bfe0a7e5416369e236afa007d5d9c58a0fa

:: Running /tmp/workdir-host-linux-riscv64-unmatched/go/bin/go with args ["/tmp/workdir-host-linux-riscv64-unmatched/go/bin/go" "test" "-short" "golang.org/x/perf/..."] and env ["LANG=C.UTF-8" "PATH=/tmp/workdir-host-linux-riscv64-unmatched/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/sh" "INVOCATION_ID=9a25a87216aa4a67aff3f2c5c3f199cb" "JOURNAL_STREAM=8:14199526" "GO_BUILDER_ENV=host-linux-riscv64-unmatched" "WORKDIR=/tmp/workdir-host-linux-riscv64-unmatched" "HTTPS_PROXY=http://uplink.golangcn.org:8123" "HTTP_PROXY=http://uplink.golangcn.org:8123" "GOPROXY=https://proxy.golang.com.cn,direct" "NO_PROXY=*.cn" "GOSUMDB=off" "GO_STAGE0_NET_DELAY=400ms" "GO_STAGE0_DL_DELAY=3.1s" "GOROOT_BOOTSTRAP=/tmp/workdir-host-linux-riscv64-unmatched/go1.4" "GO_BUILDER_NAME=linux-riscv64-unmatched" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/local/goboot" "GO_TEST_TIMEOUT_SCALE=4" "GOROOT=/tmp/workdir-host-linux-riscv64-unmatched/go" "GOPATH=/tmp/workdir-host-linux-riscv64-unmatched/gopath" "GO111MODULE=on" "TMPDIR=/tmp/workdir-host-linux-riscv64-unmatched/tmp" "GOCACHE=/tmp/workdir-host-linux-riscv64-unmatched/gocache"] in dir /tmp/workdir-host-linux-riscv64-unmatched/gopath/src/golang.org/x/perf

go: downloading google.golang.org/appengine v1.4.0
go: downloading golang.org/x/net v0.0.0-20190620200207-3b0461eec859
go: downloading github.com/aclements/go-gg v0.0.0-20170118225347-6dbb4e4fefb0
go: downloading github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794
go: downloading gonum.org/v1/plot v0.10.0
go: downloading golang.org/x/oauth2 v0.0.0-20170207211851-4464e7848382
go: downloading github.com/go-sql-driver/mysql v1.4.1
go: downloading google.golang.org/api v0.0.0-20170206182103-3d017632ea10
go: downloading github.com/mattn/go-sqlite3 v1.14.5
go: downloading cloud.google.com/go v0.0.0-20170206221025-ce650573d812
go: downloading github.com/golang/protobuf v1.2.0
go: downloading github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9
go: downloading github.com/fogleman/gg v1.3.0
go: downloading golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d
go: downloading github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac
go: downloading github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82
go: downloading github.com/gonum/internal v0.0.0-20181124074243-f884aa714029
go: downloading github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9
go: downloading github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81
go: downloading github.com/googleapis/gax-go v0.0.0-20161107002406-da06d194a00e
go: downloading github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
go: downloading github.com/go-fonts/liberation v0.2.0
go: downloading github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527
go: downloading github.com/go-pdf/fpdf v0.5.0
go: downloading golang.org/x/text v0.3.6
go: downloading google.golang.org/grpc v0.0.0-20170208002647-2a6bf6142e96
? golang.org/x/perf/analysis [no test files]
ok golang.org/x/perf/analysis/app 0.189s
? golang.org/x/perf/analysis/appengine [no test files]
? golang.org/x/perf/analysis/localperf [no test files]
ok golang.org/x/perf/benchfmt 0.070s
? golang.org/x/perf/benchfmt/internal/bytesconv [no test files]
ok golang.org/x/perf/benchmath 0.047s
ok golang.org/x/perf/benchproc 0.098s
ok golang.org/x/perf/benchproc/internal/parse 0.020s
? golang.org/x/perf/benchproc/syntax [no test files]
? golang.org/x/perf/benchseries [no test files]
ok golang.org/x/perf/benchstat 0.174s
ok golang.org/x/perf/benchunit 0.029s
? golang.org/x/perf/cmd/benchsave [no test files]
ok golang.org/x/perf/cmd/benchseries 310.364s
ok golang.org/x/perf/cmd/benchstat 1.515s
? golang.org/x/perf/internal/basedir [no test files]
? golang.org/x/perf/internal/diff [no test files]
ok golang.org/x/perf/internal/stats 0.462s
ok golang.org/x/perf/storage 0.101s

Error: runTests: error copying response: unexpected EOF

I've tried login into the builders while running this test and I found that it failed when it still linking db.test and app.test ld -plugin ...

root     3222984  4.5  0.6 765336 111188 pts/0   Sl+  08:29   0:06 /usr/lib/go-1.16/pkg/tool/linux_riscv64/link -o /tmp/go-build485723735/b349/db.test -importcfg /tmp/go-build485723735/b349/importcfg.link -s -w -buildmode=e[24/987]did=w-XRZ47IGaZIzdRCKlAv/0ehAeeARcjTQjUqWKBwJ/aWsqw_fx2l7hUyrqp8eF/w-XRZ47IGaZIzdRCKlAv -extld=gcc /tmp/go-build485723735/b349/_pkg_.a
                                                                                            root     3222996  6.3  0.8 832892 133872 pts/0   Sl+  08:29   0:09 /usr/lib/go-1.16/pkg/tool/linux_riscv64/link -o /tmp/go-build485723735/b316/app.test -importcfg /tmp/go-build485723735/b316/importcfg.link -s -w -buildmode=exe -buildid=0NfFpeT_w6Naf7XuFb2f/cgyu__5Ymmqv0KSerh5u/vjaJne72VvyLRSG6KIZ5/0NfFpeT_w6Naf7XuFb2f -extld=gcc /tmp/go-build485723735/b316/_pkg_.a
                                                                                           root     3223057  0.0  0.0   6640   616 pts/0    S+   08:29   0:00 gcc -s -o /tmp/go-build485723735/b349/db.test -rdynamic -Wl,--compress-debug-sections=zlib-gnu /tmp/go-link-020566952/go.o /tmp/go-link-020566952/000000.o /tmp/go-link-020566952/000001.o /tmp/go-link-020566952/000002.o /tmp/go-link-020566952/000003.o /tmp/go-link-020566952/000004.o /tmp/go-link-020566952/000005.o /tmp/go-link-020566952/000006.o /tmp/go-link-020566952/000007.o /tmp/go-link-020566952/000008.o /tmp/go-link-020566952/000009.o /tmp/go-link-020566952/000010.o /tmp/go-link-020566952/000011.o /tmp/go-link-020566952/000012.o /tmp/go-link-020566952/000013.o /tmp/go-link-020566952/000014.o /tmp/go-link-020566952/000015.o /tmp/go-link-020566952/000016.o /tmp/go-link-020566952/000017.o /tmp/go-link-020566952/000018.o /tmp/go-link-020566952/000019.o /tmp/go-link-020566952/000020.o /tmp/go-link-020566952/000021.o /tmp/go-link-020566952/000022.o /tmp/go-link-020566952/000023.o /tmp/go-link-020566952/000024.o /tmp/go-link-020566952/000025.o /tmp/go-link-020566952/000026.o /tmp/go-link-020566952/000027.o -g -O2 -g -O2 -ldl -g -O2 -lpthread -no-pie                          root     3223058  0.0  0.0   5300   544 pts/0    S+   08:29   0:00 /usr/lib/gcc/riscv64-linux-gnu/10/collect2 -plugin /usr/lib/gcc/riscv64-linux-gnu/10/liblto_plugin.so -plugin-opt=/usr/lib/gcc/riscv64-linux-gnu/10/lto-wrapper -plugin-opt=-fresolution=/tmp/ccRCil8p.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -hash-style=gnu --as-needed -melf64lriscv -export-dynamic -dynamic-linker /lib/ld-linux-riscv64-lp64d.so.1 -z relro -o /tmp/go-build485723735/b349/db.test -s /lib/riscv64-linux-gnu/crt1.o /usr/lib/gcc/riscv64-linux-gnu/10/crti.o /usr/lib/gcc/riscv64-linux-gnu/10/crtbegin.o -L/usr/lib/gcc/riscv64-linux-gnu/10 -L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu --compress-debug-sections=zlib-gnu /tmp/go-link-020566952/go.o /tmp/go-link-020566952/000000.o /tmp/go-link-020566952/000001.o /tmp/go-link-020566952/000002.o /tmp/go-link-020566952/000003.o /tmp/go-link-020566952/000004.o /tmp/go-link-020566952/000005.o /tmp/go-link-020566952/000006.o /tmp/go-link-020566952/000007.o /tmp/go-link-020566952/000008.o /tmp/go-link-020566952/000009.o /tmp/go-link-020566952/000010.o /tmp/go-link-020566952/000011.o /tmp/go-link-020566952/000012.o /tmp/go-link-020566952/000013.o /tmp/go-link-020566952/000014.o /tmp/go-link-020566952/000015.o /tmp/go-link-020566952/000016.o /tmp/go-link-020566952/000017.o /tmp/go-link-020566952/000018.o /tmp/go-link-020566952/000019.o /tmp/go-link-020566952/000020.o /tmp/go-link-020566952/000021.o /tmp/go-link-020566952/000022.o /tmp/go-link-020566952/000023.o /tmp/go-link-020566952/000024.o /tmp/go-link-020566952/000025.o /tmp/go-link-020566952/000026.o /tmp/go-link-020566952/000027.o -ldl -lpthread -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/riscv64-linux-gnu/10/crtend.o /usr/lib/gcc/riscv64-linux-gnu/10/crtn.o</pre>

Possible solutions:

  1. Increase builder's test time scale from 4 to 6
  2. disable perf test for riscv64

cc @golang/riscv64 @bcmills

@mengzhuo mengzhuo added Builders x/build issues (builders, bots, dashboards) arch-riscv Issues solely affecting the riscv64 architecture. labels Jul 8, 2022
@gopherbot gopherbot added this to the Unreleased milestone Jul 8, 2022
@heschi heschi added the NeedsFix The path to resolution is known, but the work has not been done. label Jul 8, 2022
@bcmills
Copy link
Contributor

bcmills commented Jul 8, 2022

Possible solutions:

The folks on @golang/runtime have been doing the x/perf work lately — perhaps they'll have some insight into the choice?

@mknyszek
Copy link
Contributor

mknyszek commented Jul 8, 2022

It would be nice to understand what's causing the failure long link times, but I'm not opposed to just skipping the test on some platforms with a reference to this issue.

@prattmic
Copy link
Member

prattmic commented Jul 8, 2022

ok  	golang.org/x/perf/cmd/benchseries	310.364s

Slow benchseries tests seem to be the root cause here. Increasing the scale might be a fix, but I wonder if the benchseries tests should be modified to have a shorter -short mode.

@mknyszek @dr2chase

@gopherbot
Copy link

Change https://go.dev/cl/417314 mentions this issue: dashboard: increase linux-riscv64-unmatched test timeout scale

gopherbot pushed a commit to golang/build that referenced this issue Jul 13, 2022
linux-riscv64-unmatched takes too long to complete perf test.
This CL increase builder's test timeout scale.

For golang/go#53745

Change-Id: Ic91abc974f803e36a62215cb870db8ce9993106e
Reviewed-on: https://go-review.googlesource.com/c/build/+/417314
Run-TryBot: Meng Zhuo <mzh@golangcn.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Meng Zhuo <mzh@golangcn.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/417495 mentions this issue: dashboard: skip perf test for linux-riscv64-unmatched

gopherbot pushed a commit to golang/build that referenced this issue Jul 16, 2022
The perf test still timeed out even CL 417314 increased builder
timeout scale up to 6.
Unfortunately we had to skip this test for now and revert timeout
scale back to 4.

For golang/go#53745

Change-Id: I247179b42c575ca05bb0ec6b51d260ab51d912f3
Reviewed-on: https://go-review.googlesource.com/c/build/+/417495
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Meng Zhuo <mzh@golangcn.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Dec 15, 2022
@mknyszek
Copy link
Contributor

benchseries tests still take 300+ seconds on this builder. We should probably fix the tests.

@gopherbot
Copy link

Change https://go.dev/cl/459075 mentions this issue: cmd/benchseries: shorten tests by 2/3, bootstrapping by 1/2, add a skip.

@golang golang locked and limited conversation to collaborators Dec 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-riscv Issues solely affecting the riscv64 architecture. Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

7 participants