Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gitmirror: build repo is broken due to updateDashboard failing to postCommit on a commit with 2 parents #35828

Closed
dmitshur opened this issue Nov 25, 2019 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@dmitshur
Copy link
Contributor

From https://farmer.golang.org/#health right now:

This has happened a few times in the last few days. I've fixed previous instances by restarting gitmirror, but since it keeps recurring, this will need to be investigated and fixed more.

/cc @toothrot @cagedmantis

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 25, 2019
@dmitshur dmitshur added this to the Unreleased milestone Nov 25, 2019
@gopherbot
Copy link

Change https://golang.org/cl/208663 mentions this issue: cmd/gitmirror: log error details when Repo.Loop operations fail

@bradfitz
Copy link
Contributor

If it helps:

$ for IP in $(kubectl get pods  -l app=gitmirror -o=jsonpath={.items[*].status.podIP}); do curl http://$IP:8585/debug/goroutines; done

.. to look at the goroutine stacks of the active pods.

gopherbot pushed a commit to golang/build that referenced this issue Nov 25, 2019
The error value is not exposed anywhere. Start by logging it,
so that it's possible to look up the error details from logs.

Updates golang/go#35828

Change-Id: I0cfc89685269a3511c8aee767459be7a198860b1
Reviewed-on: https://go-review.googlesource.com/c/build/+/208663
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur dmitshur self-assigned this Nov 25, 2019
@bradfitz
Copy link
Contributor

If it turns out this is hanging due to updating the dashboard, note that all the code's gone as of pending https://go-review.googlesource.com/c/build/+/208697

@dmitshur
Copy link
Contributor Author

dmitshur commented Nov 25, 2019

It is. It happened again, and here are the logs:

[...]
2019/11/25 22:18:37 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:37 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:18:49 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:49 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:19:01 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:19:01 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
[...]

The "parent commit not found" error is likely related to commit golang/build@63986c1 having 2 parents instead of the usual 1.

It sounds like we can let this wait for CL 208697 before trying to make progress here, does that sound good to you @bradfitz?

@dmitshur dmitshur removed their assignment Nov 25, 2019
@dmitshur dmitshur changed the title x/build/cmd/gitmirror: build repo keeps breaking/hanging x/build/cmd/gitmirror: build repo is broken due to updateDashboard failing to postCommit on a commit with 2 parents Nov 25, 2019
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 25, 2019
@bradfitz
Copy link
Contributor

Yup. Let's wait.

That code is all scary and it's not worth debugging. Easier to delete it all.

@gopherbot
Copy link

Change https://golang.org/cl/208697 mentions this issue: app, gitmirror, maintner: use maintner for dashboard, not datastore

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

3 participants