Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: disable linux/arm trybot #22592

Closed
rsc opened this issue Nov 5, 2017 · 9 comments
Closed

build: disable linux/arm trybot #22592

rsc opened this issue Nov 5, 2017 · 9 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@rsc
Copy link
Contributor

rsc commented Nov 5, 2017

As I write this, https://farmer.golang.org/try?commit=5728eb6a says the linux-arm buildbot has been running for 26.1 min.

linux-arm rev 5728eb6a (trybot set for I179c431); running; http://scaleway-prod-36 reverse peer scaleway-prod-36/10.240.0.8:51088 for host type host-linux-arm-scaleway, 26m5.027460472s ago
  2017-11-05T21:32:36Z checking_for_snapshot 
  2017-11-05T21:32:36Z finish_checking_for_snapshot after 161.5ms
  2017-11-05T21:32:36Z get_buildlet_cross 
  2017-11-05T21:32:36Z creating_kube_pod buildlet-linux-armhf-cross-rn918aa96
  2017-11-05T21:32:36Z pod_creating 
  2017-11-05T21:32:57Z pod_created 
  2017-11-05T21:32:57Z got_pod_info waiting_for_buildlet...
  2017-11-05T21:32:57Z finish_get_buildlet_cross after 21s
  2017-11-05T21:32:57Z write_version_tar 
  2017-11-05T21:32:57Z get_source 
  2017-11-05T21:32:57Z finish_get_source after 0s
  2017-11-05T21:32:57Z write_go_src_tar 
  2017-11-05T21:33:00Z finish_write_go_src_tar after 3.03s
  2017-11-05T21:33:00Z make_cross_compile_kube 
  2017-11-05T21:35:56Z clean_for_snapshot 
  2017-11-05T21:35:56Z finish_clean_for_snapshot after 4.9ms
  2017-11-05T21:35:56Z write_snapshot_to_gcs 
  2017-11-05T21:35:56Z fetch_snapshot_reader_from_buildlet 
  2017-11-05T21:35:57Z finish_fetch_snapshot_reader_from_buildlet after 377.1ms
  2017-11-05T21:37:08Z finish_write_snapshot_to_gcs after 1m11.1s
  2017-11-05T21:37:08Z finish_make_cross_compile_kube after 4m7.5s
  2017-11-05T21:37:08Z get_buildlet 
  2017-11-05T21:37:08Z wait_static_builder host-linux-arm-scaleway
  2017-11-05T21:37:08Z finish_wait_static_builder after 0s; host-linux-arm-scaleway
  2017-11-05T21:37:08Z clean_buildlet http://scaleway-prod-36 reverse peer scaleway-prod-36/10.240.0.8:51088 for host type host-linux-arm-scaleway
  2017-11-05T21:37:08Z finish_clean_buildlet after 110ms; http://scaleway-prod-36 reverse peer scaleway-prod-36/10.240.0.8:51088 for host type host-linux-arm-scaleway
  2017-11-05T21:37:08Z finish_get_buildlet after 213.8ms
  2017-11-05T21:37:08Z using_buildlet scaleway-prod-36
  2017-11-05T21:37:08Z write_snapshot_tar 
 +1292.5s (now)

It's now T21:59 or so.

I am not sure what's going on or how to find out. It may be related to my cmd/go changes, or not.

/cc @bradfitz
@rsc rsc added this to the Unreleased milestone Nov 5, 2017
@0intro
Copy link
Member

0intro commented Nov 5, 2017 •
edited

The linux-arm builder is currently running for 49 minutes on the trybot for my last CL.

@bradfitz
Copy link
Contributor

bradfitz commented Nov 6, 2017

Sounds like a dup of #21839.

@adams-sarah, did you ever make progress on debugging it?

@s-mang
Copy link
Contributor

s-mang commented Nov 6, 2017 •
edited

Yea, dupe of my other issue.
Nope, did not start debugging.

I assume it is a network issue though. Didn’t the scaleway people say that moving our builders to Tokyo would give us a much slower external network conn? I think you were on that thread.
Anyway, I think it started happening around that time.

It’s the write to GCS that’s taking forever.

@bradfitz
Copy link
Contributor

bradfitz commented Nov 7, 2017

It was the arm64 builders (packet.net) that moved us to Tokyo, not Scaleway (still in Paris).

@s-mang
Copy link
Contributor

s-mang commented Nov 7, 2017

My apologies. You’re totally right.

@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Nov 7, 2017
@bradfitz bradfitz self-assigned this Nov 7, 2017
@rsc
Copy link
Contributor Author

rsc commented Nov 8, 2017

Another, this one at 122 min (>2 hours!!).

linux-arm	(running)	
linux-arm rev b24e5dba (trybot set for I6716bed); running; http://scaleway-prod-49 reverse peer scaleway-prod-49/10.240.0.8:54220 for host type host-linux-arm-scaleway, 2h2m30.153086268s ago
  2017-11-08T16:35:59Z checking_for_snapshot 
  2017-11-08T16:35:59Z finish_checking_for_snapshot after 92.9ms
  2017-11-08T16:35:59Z get_buildlet_cross 
  2017-11-08T16:35:59Z creating_kube_pod buildlet-linux-armhf-cross-rncf5b44a
  2017-11-08T16:35:59Z pod_creating 
  2017-11-08T16:36:04Z pod_created 
  2017-11-08T16:36:04Z got_pod_info waiting_for_buildlet...
  2017-11-08T16:36:04Z finish_get_buildlet_cross after 5.42s
  2017-11-08T16:36:04Z write_version_tar 
  2017-11-08T16:36:04Z get_source 
  2017-11-08T16:36:11Z finish_get_source after 6.21s
  2017-11-08T16:36:11Z write_go_src_tar 
  2017-11-08T16:36:12Z finish_write_go_src_tar after 1.23s
  2017-11-08T16:36:12Z make_cross_compile_kube 
  2017-11-08T16:37:52Z clean_for_snapshot 
  2017-11-08T16:37:52Z finish_clean_for_snapshot after 3.2ms
  2017-11-08T16:37:52Z write_snapshot_to_gcs 
  2017-11-08T16:37:52Z fetch_snapshot_reader_from_buildlet 
  2017-11-08T16:37:52Z finish_fetch_snapshot_reader_from_buildlet after 125.8ms
  2017-11-08T16:37:57Z finish_write_snapshot_to_gcs after 4.39s
  2017-11-08T16:37:57Z finish_make_cross_compile_kube after 1m44.6s
  2017-11-08T16:37:57Z get_buildlet 
  2017-11-08T16:37:57Z wait_static_builder host-linux-arm-scaleway
  2017-11-08T16:37:57Z finish_wait_static_builder after 0s; host-linux-arm-scaleway
  2017-11-08T16:37:57Z clean_buildlet http://scaleway-prod-49 reverse peer scaleway-prod-49/10.240.0.8:54220 for host type host-linux-arm-scaleway
  2017-11-08T16:37:57Z finish_clean_buildlet after 106.4ms; http://scaleway-prod-49 reverse peer scaleway-prod-49/10.240.0.8:54220 for host type host-linux-arm-scaleway
  2017-11-08T16:37:57Z finish_get_buildlet after 223.6ms
  2017-11-08T16:37:57Z using_buildlet scaleway-prod-49
  2017-11-08T16:37:57Z write_snapshot_tar 
 +7231.8s (now)

This is making Run-TryBot essentially useless. It seems like we should turn off the linux-arm trybot until this is fixed.

@rsc rsc changed the title build: linux/arm trybots taking too long build: disable linux/arm trybot Nov 8, 2017
@gopherbot
Copy link

Change https://golang.org/cl/77370 mentions this issue: cmd/buildlet: use tmpfs workdir if flag value is unspecified

gopherbot pushed a commit to golang/build that referenced this issue Nov 13, 2017
@bradfitz
cmd/buildlet: use tmpfs workdir if flag value is unspecified
3da79c2 
Fixes golang/go#21839
Updates golang/go#22592

Change-Id: Ia96236c61847193f930564b4dd5e0d7965e89945
Reviewed-on: https://go-review.googlesource.com/77370
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@bradfitz
Copy link
Contributor

Filed #22749 and #22748

@gopherbot
Copy link

Change https://golang.org/cl/78032 mentions this issue: dashboard: remove linux-arm as a trybot

@golang golang locked and limited conversation to collaborators Nov 15, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @rsc @s-mang @0intro @gopherbot