Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: linux-amd64-race TryBot straggling #32032

Open
bcmills opened this issue May 14, 2019 · 3 comments
Open

x/build: linux-amd64-race TryBot straggling #32032

bcmills opened this issue May 14, 2019 · 3 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented May 14, 2019

In the TryBot run for https://golang.org/cl/176580 at Patchset 4, most of the TryBots ran relatively quickly (less than 10m), but the linux-amd64-race builder took much longer (over 20m).

I'm not sure exactly why it was so slow, but it leads me to believe that there may be something wrong with the sharding for that builder.

(CC @bradfitz @dmitshur @andybons)
@bcmills bcmills added Performance Testing An issue that has been verified to require only test changes, not just a test failure. labels May 14, 2019
@gopherbot gopherbot added this to the Unreleased milestone May 14, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label May 14, 2019
@bradfitz
Copy link
Contributor

All the timing data is in BigQuery. That almost always reveals the mystery.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 14, 2019
@bradfitz
Copy link
Contributor

Looks like the @rsc's recent suffixarray changes made it slower?

https://console.cloud.google.com/bigquery?sq=872405196845:3b455f3831c54226ba07eda8cea81247

...

SELECT
    Builder, Event, APPROX_QUANTILES(Seconds, 100)[OFFSET(50)] as MedianSec, COUNT(*) as N
FROM
    builds.Spans
WHERE
    Error = ''
    AND Builder = 'linux-amd64-race'
    AND StartTime > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24*7 HOUR)
    AND Repo = "go"
    And IsTry = true
GROUP BY 1, 2
ORDER BY 3 DESC
Row Builder Event MedianSec N  
1 linux-amd64-race make_and_test 625.462161107 285  
2 linux-amd64-race run_test:go_test_bench:index/suffixarray 388.381372503 278  
3 linux-amd64-race make 130.729277047 276  
4 linux-amd64-race run_test:go_test_bench:runtime 117.176056921 278  
5 linux-amd64-race get_helper 58.835877058 1381  
6 linux-amd64-race get_buildlet 57.736557395 285  
7 linux-amd64-race run_test:go_test_bench:math/big 48.183967875 278  
8 linux-amd64-race create_gce_buildlet 47.280073747 1666  
9 linux-amd64-race run_test:go_test:runtime 43.819345904 279  
10 linux-amd64-race run_test:go_test_bench:net 38.822814725 278  
11 linux-amd64-race run_test:go_test_bench:text/tabwriter 35.119872601 278  
12 linux-amd64-race run_test:go_test:net 33.723890843 281  
13 linux-amd64-race run_test:go_test_bench:bytes 30.623900061 278  
14 linux-amd64-race run_test:go_test_bench:strings 30.278805037 278  
15 linux-amd64-race run_test:go_test_bench:compress/flate 27.559467869 278  
16 linux-amd64-race run_test:go_test:os/exec 26.914272922 279  
17 linux-amd64-race create_gce_instance 25.102813238 1669  
18 linux-amd64-race run_test:go_test_bench:crypto/tls 24.369628886 278  
19 linux-amd64-race run_test:go_test_bench:net/http 21.715966586 278  
20 linux-amd64-race wait_buildlet_start 21.17260563 1666  
21 linux-amd64-race run_test:go_test:os/signal 20.97289739 279  
22 linux-amd64-race run_test:go_test:go/internal/gcimporter 19.999651301 281  
23 linux-amd64-race run_test:go_test:runtime/race 19.223213255 277  
24 linux-amd64-race run_test:go_test_bench:sync 17.451266096 277  
25 linux-amd64-race run_test:go_test_bench:strconv 17.317289797 277  
26 linux-amd64-race run_test:go_test:net/http 17.160193173 280  
27 linux-amd64-race install_race_std 16.297853892 276  
28 linux-amd64-race run_test:go_test_bench:math 15.798795113 277  
29 linux-amd64-race run_test:go_test_bench:context 12.820779637 277  
30 linux-amd64-race run_test:go_test:os 12.79174036 278  
31 linux-amd64-race run_test:go_test:runtime/pprof 12.656720241 274  
32 linux-amd64-race run_test:go_test:go/types 12.650777936 280  
33 linux-amd64-race run_test:go_test:compress/flate 12.288297681 281  
34 linux-amd64-race run_test:go_test_bench:encoding/json 11.248818408 273  
35 linux-amd64-race run_test:go_test:internal/cpu 10.493141873999999 280  
36 linux-amd64-race run_test:go_test_bench:time 10.472495406 272  
37 linux-amd64-race run_test:go_test_bench:math/bits 10.459982868 272  
38 linux-amd64-race run_test:go_test:internal/trace 10.349812287 279  
39 linux-amd64-race run_test:go_test:crypto/tls 10.322860367 281  
40 linux-amd64-race run_test:go_test:compress/zlib 10.282296089999999 281  

@bradfitz
Copy link
Contributor

But suffixarray seems fast on linux with -race locally, even before Russ's recent commit.

@bcmills bcmills mentioned this issue Jan 24, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @bcmills @gopherbot