Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/coordinator: missing timeouts during build (make.bash phase? overall?) #10921

Closed
minux opened this issue May 21, 2015 · 3 comments
Closed

x/build/cmd/coordinator: missing timeouts during build (make.bash phase? overall?) #10921

minux opened this issue May 21, 2015 · 3 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@minux
Copy link
Member

minux commented May 21, 2015

http://farmer.golang.org/logs?name=darwin-arm-a5ios&rev=8fa14ea8b4744576bd28073901154c15813e29de&st=0xc20847ca80

  builder: darwin-arm-a5ios
      rev: 8fa14ea8b4744576bd28073901154c15813e29de
 buildlet: http://none
  started: 2015-05-16 02:02:58.850844883 +0000 UTC
   status: still running

Events:
         2015-05-16T02:02:58Z get_buildlet
   +0.0s 2015-05-16T02:02:58Z waiting_machine_in_use
 +406284.8s 2015-05-20T18:54:23Z got_machine
   +0.5s 2015-05-20T18:54:24Z cleaned_up
   +0.0s 2015-05-20T18:54:24Z got_buildlet
   +0.0s 2015-05-20T18:54:24Z start_write_version_tar
   +0.0s 2015-05-20T18:54:24Z start_fetch_gerrit_tgz
   +0.2s 2015-05-20T18:54:24Z start_write_go14_tar
   +0.0s 2015-05-20T18:54:24Z start_write_go_tar
   +2.5s 2015-05-20T18:54:26Z end_write_go14_tar
   +1.6s 2015-05-20T18:54:28Z end_write_go_tar
   +0.0s 2015-05-20T18:54:28Z pre_exec
   +0.0s 2015-05-20T18:54:28Z running_exec

FAIL    hash/adler32    239.226s
 process continue
Process 493 resuming
go_darwin_arm_exec: timeout running tests
FAIL    hash/crc32  239.234s
 process continue
Process 499 resuming
go_darwin_arm_exec: timeout running tests
FAIL    hash/crc64  239.225s
 process continue
Process 504 resuming
go_darwin_arm_exec: timeout running tests
FAIL    hash/fnv    239.228s
 process continue
Process 509 resuming
go_darwin_arm_exec: timeout running tests
FAIL    html    239.227s
 process continue
Process 515 resuming
go_darwin_arm_exec: timeout running tests
FAIL    html/template   239.280s
 process continue
Process 520 resuming
go_darwin_arm_exec: timeout running tests
FAIL    image   239.266s
 process continue
Process 525 resuming
go_darwin_arm_exec: timeout running tests
FAIL    image/color 239.259s
?       image/color/palette [no test files]
 process continue
Process 531 resuming
go_darwin_arm_exec: timeout running tests
FAIL    image/draw  239.268s
 process continue
Process 536 resuming
go_darwin_arm_exec: timeout running tests
FAIL    image/gif   239.266s
?       image/internal/imageutil    [no test files]
 process continue
Process 541 resuming
go_darwin_arm_exec: timeout running tests
FAIL    image/jpeg  239.274s
....

The build has been running for more than 5 hours, and waited more than 4 days
to get a machine. Clearly the timeout limit logic is missing or doesn't work.

/cc @crawshaw
@minux minux added this to the Unreleased milestone May 21, 2015
@rsc rsc added the Builders x/build issues (builders, bots, dashboards) label Jun 11, 2015
@bradfitz bradfitz changed the title x/build: reverse builtlet doesn't limit all.bash run time? x/build: reverse buildlet doesn't limit all.bash run time? Jul 7, 2015
@josharian
Copy link
Contributor

The dev.ssa 386 and arm builders have been hung in a CPU-burning infinite loop for three or four days.

I'll mail a fix shortly, but a global timeout would be really good.

@bradfitz
Copy link
Contributor

They don't run all.bash. They run make.bash and then go tool dist test --list and then run go tool dist test <name> for each.

But yeah, perhaps a timeout is missing.

@bradfitz bradfitz changed the title x/build: reverse buildlet doesn't limit all.bash run time? x/build/cmd/coordinator: missing timeouts during build (make.bash phase? overall?) Nov 25, 2019
@bradfitz bradfitz mentioned this issue Nov 25, 2019
Closed
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 18, 2022
@dmitshur
Copy link
Contributor

We generally have a global timeout for VM/pod-based builders (see #52929), though not sure about if reverse builders have something equivalent or not. I'll close this in favor of #35364, a more recent coordinator issue also about adding a timeout to make.bash.

@golang golang locked and limited conversation to collaborators May 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
6 participants
@bradfitz @josharian @rsc @minux @dmitshur @gopherbot