Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/app: dashboard should not hide missing builders #19930

Closed
rsc opened this issue Apr 11, 2017 · 6 comments
Closed

x/build/app: dashboard should not hide missing builders #19930

rsc opened this issue Apr 11, 2017 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@rsc
Copy link
Contributor

rsc commented Apr 11, 2017

build.golang.org should know what builders it expects results from and not hide columns with no results. This makes it look like we don't have builders - or that all builders are OK - when in fact a builder is missing. This made #19929 (linux/arm64 builder not reporting results) look like we don't have one at all.
@rsc rsc added this to the Soon milestone Apr 11, 2017
@bradfitz bradfitz changed the title builders: dashboard should not hide missing builders x/build/app: dashboard should not hide missing builders Apr 11, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Apr 11, 2017
@vstefanovic
Copy link
Member

Speaking of missing builders, mips and mips64 are also missing.
Comparing the mipsle (where all is ok) and mips buildlet logs, seems like mips' make.bash breaks the ten minute limit:

#mipsle:
Running /data/mipsle/go/src/make.bash with args ...
Run = ok, after 5m48.777024107s
Running /data/mipsle/go/bin/go with args ...

#mips:
Running /data/mipsle/go/src/make.bash with args ...
Run = ok, after 16m22.922689703s
Error dialing coordinator: http.Serve on reverse connection complete: revdial: Listener closed; EOF

Then there's this line in coordinator's awaitTryBuild():

timeout := time.NewTimer(10 * time.Minute)

@bradfitz Does this make sense, are mips and mips64 builder timing out?

cc @MIPSbkirby

@bradfitz
Copy link
Contributor

@vstefanovic, the 10 minute timer you found (in awaitTryBuilt) is about trybots (pre-submit builds), which we don't run against MIPS machines, since they're too slow.

I'm not aware of any other time limit in the build coordinator.

I see a mips64 build running now. (temporary URL: https://farmer.golang.org/temporarylogs?name=linux-mips64&rev=9191c0b3fdd7b129007fab66f9034d595abb3876&st=0xc4215b0e00&nostream=1)

The logs at the top say:

  builder: linux-mips64
      rev: 9191c0b3fdd7b129007fab66f9034d595abb3876
 buildlet: http://mipsswbrd041 reverse peer mipsswbrd041/4.16.204.77:31479 for host type host-linux-mips64
  started: 2017-04-12 19:58:32.520845441 +0000 UTC
   status: still running

Events:
  2017-04-12T19:58:32Z checking_for_snapshot 
  2017-04-12T19:58:32Z finish_checking_for_snapshot after 29.4256ms
  2017-04-12T19:58:32Z get_buildlet 
  2017-04-12T19:58:32Z wait_static_builder host-linux-mips64
  2017-04-12T19:58:32Z waiting_machine_in_use 
  2017-04-12T20:16:12Z finish_wait_static_builder after 17m39.348136797s; host-linux-mips64
  2017-04-12T20:16:12Z clean_buildlet http://mipsswbrd041 reverse peer mipsswbrd041/4.16.204.77:31479 for host type host-linux-mips64
  2017-04-12T20:16:13Z finish_clean_buildlet after 627.228276ms; http://mipsswbrd041 reverse peer mipsswbrd041/4.16.204.77:31479 for host type host-linux-mips64
  2017-04-12T20:16:13Z finish_get_buildlet after 17m40.194793237s
  2017-04-12T20:16:13Z using_buildlet mipsswbrd041
  2017-04-12T20:16:13Z write_version_tar 
  2017-04-12T20:16:13Z get_source 
  2017-04-12T20:16:13Z finish_get_source after 49.163µs
  2017-04-12T20:16:13Z write_go_src_tar 
  2017-04-12T20:18:50Z finish_write_go_src_tar after 2m36.890646068s
  2017-04-12T20:18:50Z make_and_test 
  2017-04-12T20:18:50Z make src/make.bash
 +522.5s (now)

Build log:
linux-mips64 at 9191c0b3fdd7b129007fab66f9034d595abb3876

:: Running /data/mips64/go/src/make.bash with args ["/data/mips64/go/src/make.bash"] and env ["LANG=en_US.UTF-8" "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/mips/proj/build-compiler/upstream-testing/go-lang" "LOGNAME=buildbot" "USER=buildbot" "SHELL=/sbin/nologin" "GOARCH=mips64" "GOROOT_BOOTSTRAP=/usr/local/go-bootstrap-mips64" "WORKDIR=/data/mips64" "GO_BUILDER_NAME=linux-mips64" "GOHOSTARCH=mips64" "GO_TEST_TIMEOUT_SCALE=4" "GOBIN="] in dir /data/mips64/go/src

It's setting GO_TEST_TIMEOUT_SCALE=4, which is already cranking up limits.

But if it violates those limits and fails, it should report a failure to the build dashboard.

If it disappears from the build dashboard, that implies the machine is somehow dropping off the network or not reporting anything.

@bradfitz
Copy link
Contributor

Does make.bash really take 13+ minutes and counting on mips64?

Or is other stuff happening on that machine slowing it down? Is it swapping?

@bradfitz bradfitz mentioned this issue Apr 12, 2017
Closed
@bradfitz
Copy link
Contributor

Moved mips64 discussion to #19953

@bradfitz bradfitz removed their assignment Jul 20, 2017
@bradfitz
Copy link
Contributor

/cc @andybons

@bradfitz bradfitz modified the milestones: Unreleased, Soon Jul 20, 2017
@bradfitz bradfitz added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jul 20, 2017
@bradfitz bradfitz mentioned this issue Jul 27, 2017
Closed
6 tasks
@gopherbot
Copy link

Change https://golang.org/cl/170443 mentions this issue: app/appengine: show all builder columns, even if they don't have results

@golang golang locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@bradfitz @rsc @gopherbot @vstefanovic