x/build/cmd/gitmirror: not mirroring some subrepos at this time #32931

dmitshur · 2019-07-03T22:13:20Z

The gitmirror service was redeployed recently and it's currently mirroring the following repositories:

arch - ok
benchmarks - ok
blog - ok
crypto - ok
debug - ok
exp - ok
go - ok
image - ok
mobile - ok
net - ok
oauth2 - ok
perf - ok
review - ok
sync - ok
sys - ok
talks - ok
term - ok
text - ok
time - ok
tools - ok
tour - ok

(I got that list by port forwarding its HTTP port to my machine via kubectl port-forward <gitmirror pod name> 8585:8585.)

There are a few smaller subrepos missing, for example, mod, xerrors, website, build, playground.

The issue seems to be that the build dashboard is serving a subset of subrepos at its https://build.golang.org/packages?kind=subrepo endpoint, and gitmirror uses that endpoint to find a list of all subrepos to consider mirroring (see here). So we need to resolve the issue in the build dashboard, then redeploy gitmirror so it picks up the change.

/cc @toothrot @andybons

The text was updated successfully, but these errors were encountered:

cherrymui · 2019-07-03T22:49:33Z

Seems gofrontend repo mirroring also stopped.

dmitshur · 2019-07-03T23:11:03Z

Yes, that's one of the affected subrepos as well.

The earlier analysis overlooked that there's another code path to mirror repos that talks to Gerrit API directly via instead of the build dashboard:

https://github.com/golang/build/blob/54405f243e45b89d4b1e4e1f0716387c58890f6f/cmd/gitmirror/gitmirror.go#L228

The problem seems to be there. I added a log statement to see the value of err:

        meta, err := gerritClient.GetProjects(ctx, "master")
        if err != nil {
+               log.Println("gerritClient.GetProjects:", err)
                return nil
        }

And got an 404 error. That's unexpected and I suspect is the source of the problem.

2019/07/03 19:08:57 gerritClient.GetProjects: HTTP status 404 Not Found; <!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><title>Repository is not found - Git at Google</title><link rel="stylesheet" type="text/css" href="/+static/base.vsKBklzePi_Td7VvkjGVKw.cache.css"><!-- default customHeadTagPart --></head><body class="Site"><header class="Site-header"><div class="Header"><!-- default customHeader --><div class="Header-title">Git at Google</div><div class="Header-menu"> <a class="Header-menuItem" href="https://go-review.googlesource.com/">Code Review</a> <a class="Header-menuItem" href="https://accounts.google.com/o/oauth2/auth?response_type=code&amp;access_type=offline&amp;approval_prompt=force&amp;client_id=413937457453.apps.googleusercontent.com&amp;scope=https://www.googleapis.com/auth/gerritcodereview&amp;redirect_uri=https://www.googlesource.com/new-password&amp;state=go">Generate Password</a> <a class="Header-menuItem" href="https://security.google.com/settings/security/permissions">Revoke Passwords</a> <a class="Header-menuItem" href="https://accounts.google.com/AccountChooser?service=gerritcodereview&amp;continue=https://go.googlesource.com/login/projects/?b%3Dmaster%26format%3DJSON">Sign in</a> </div></div></header><div class="Site-content"><div class="Container "><h1>Repository is not found</h1></div> <!-- Container --></div> <!-- Site-content --><!-- default customFooter --><footer class="Site-footer"><div class="Footer"><span class="Footer-poweredBy">Powered by <a href="https://gerrit.googlesource.com/gitiles/">Gitiles</a></span><span class="Footer-formats"><a class="u-monospace Footer-formatsItem" href="?format=TEXT">txt</a> <a class="u-monospace Footer-formatsItem" href="?format=JSON">json</a></span></div></footer></body></html>

I think there's been some change with Gerrit and/or Gitilies and/or the frontend that routes traffic in between them. While https://go.googlesource.com/projects/?b=master&format=JSON worked before, it no longer does. But https://go-review.googlesource.com/projects/?b=master&format=JSON does work now.

dmitshur · 2019-07-03T23:15:25Z

I'll update the Gerrit client in gitmirror to use https://go-review.googlesource.com/ as the URL, which should fix the immediate problem.

We can investigate and decide on follow up actions after.

gopherbot · 2019-07-03T23:32:37Z

Change https://golang.org/cl/184922 mentions this issue: cmd/gitmirror: update Gerrit API URL to include "-review"

dmitshur · 2019-07-03T23:41:19Z

I've deployed CL 184922 to test it and it fixed this issue. All other subrepos are being mirrored now:

Repo List

arch - ok
benchmarks - ok
blog - ok
build - ok
crypto - ok
debug - ok
dl - ok
example - ok
exp - ok
gddo - ok
go - ok
gofrontend - ok
gollvm - ok
grpc-review - ok
image - ok
lint - ok
mobile - ok
mod - ok
net - ok
oauth2 - ok
perf - ok
playground - ok
proposal - ok
protobuf - ok
review - ok
scratch - ok
sublime-build - ok
sublime-config - ok
sync - ok
sys - ok
talks - ok
term - ok
text - ok
time - ok
tools - ok
tour - ok
vgo - ok
website - ok
xerrors - ok

As another followup item here, we should check if any other services are depending on a Gerrit API URL without the "-review" part, as they're likely to be affected too. I'm not aware of any at the moment.

cherrymui · 2019-07-04T00:11:59Z

Thanks, @dmitshur

dmitshur · 2019-07-04T00:32:41Z

Thanks to @toothrot as well for helping investigate this issue.

dmitshur · 2019-07-04T02:16:56Z

As another followup item here, we should check if any other services are depending on a Gerrit API URL without the "-review" part, as they're likely to be affected too. I'm not aware of any at the moment.

I've checked the importers of golang.org/x/build/gerrit package, and all of the relevant ones are contained in golang.org/x/build. Other than gitmirror, all others already use https://go-review.googlesource.com as the Gerrit API URL.

I also checked the importers of github.com/andygrunwald/go-gerrit and didn't spot any issues in any of the relevant packages there either.

Using "https://go.googlesource.com" without "-review" as the Gerrit API URL no longer works. Use "https://go-review.googlesource.com" instead. This is the canonical Gerrit API URL for the Go project, and it's already being used in many other places in x/build. Return and handle errors from gerritMetaMap. Make it so gitmirror treats an error from gerritMetaMap on startup as fatal, otherwise it would have started in an incorrect state. While running, skip transient errors from gerritMetaMap as before, but also log them when they happen. Remove unneeded trailing slashes from the Gerrit API URLs in packages maintner/maintnerd and maintner/maintnerd/maintapi for consistency. This is a no-op as the gerrit client library used to trim them anyway. Fixes golang/go#32931 Change-Id: Icc20b798946d6317ee89533691f92b1f4d1564f0 Reviewed-on: https://go-review.googlesource.com/c/build/+/184922 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 3, 2019

gopherbot added this to the Unreleased milestone Jul 3, 2019

dmitshur assigned toothrot and dmitshur Jul 3, 2019

cherrymui mentioned this issue Jul 3, 2019

gccgo: bug in type descriptor equality #32901

Closed

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 3, 2019

gopherbot closed this as completed in golang/build@25c4be9 Jul 4, 2019

golang locked and limited conversation to collaborators Jul 3, 2020

gopherbot added the FrozenDueToAge label Jul 3, 2020

rsc unassigned toothrot and dmitshur Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build/cmd/gitmirror: not mirroring some subrepos at this time #32931

x/build/cmd/gitmirror: not mirroring some subrepos at this time #32931

dmitshur commented Jul 3, 2019 •

edited

cherrymui commented Jul 3, 2019

dmitshur commented Jul 3, 2019

dmitshur commented Jul 3, 2019

gopherbot commented Jul 3, 2019

dmitshur commented Jul 3, 2019 •

edited

cherrymui commented Jul 4, 2019

dmitshur commented Jul 4, 2019

dmitshur commented Jul 4, 2019

x/build/cmd/gitmirror: not mirroring some subrepos at this time #32931

x/build/cmd/gitmirror: not mirroring some subrepos at this time #32931

Comments

dmitshur commented Jul 3, 2019 • edited

cherrymui commented Jul 3, 2019

dmitshur commented Jul 3, 2019

dmitshur commented Jul 3, 2019

gopherbot commented Jul 3, 2019

dmitshur commented Jul 3, 2019 • edited

cherrymui commented Jul 4, 2019

dmitshur commented Jul 4, 2019

dmitshur commented Jul 4, 2019

dmitshur commented Jul 3, 2019 •

edited

dmitshur commented Jul 3, 2019 •

edited