Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: productionize maintner #19866

Closed
bradfitz opened this issue Apr 6, 2017 · 20 comments
Closed

x/build: productionize maintner #19866

bradfitz opened this issue Apr 6, 2017 · 20 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Apr 6, 2017

Move maintner to GKE & give it a hostname.

@bradfitz bradfitz self-assigned this Apr 6, 2017
@gopherbot gopherbot added this to the Unreleased milestone Apr 6, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Apr 6, 2017
@jessfraz
Copy link
Contributor

does this: https://github.com/golang/build/blob/master/maintner/maintnerd/maintnerd.go need a Dockerfile then? 😇

@bradfitz
Copy link
Contributor Author

Yup!

It'll likely look similar to https://github.com/golang/build/blob/master/cmd/gitmirror/Dockerfile which you'll probably gag looking at.

But so far that's the best mechanism I've found to pin versions of Go deps in container images.

@jessfraz
Copy link
Contributor

I saw it cringed and then realized it was actually an inventive way to do a versioning system lol

@bradfitz
Copy link
Contributor Author

Really there should be one Dockerfile to build the binary, and then a Makefile (cringe?) or other driver tool to copy the binary out of Dockerfile 1 into super-minimal Dockerfile 2 that lacks the go compiler or any go source code.

@jessfraz
Copy link
Contributor

eventually we can use this new feature for nested builds: moby/moby#32063

BUT I have a makefile technique for other repos with the one dockerfile to build, one to actually use song and dance

@gopherbot
Copy link

CL https://golang.org/cl/40478 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 12, 2017
Adds a Dockerfile and Makefile so maintnerd can be deployed to GKE.

Updates golang/go#19866

Change-Id: I83d8d409c2acab3022c2b74516157d32515fe28b
Reviewed-on: https://go-review.googlesource.com/40478
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@jessfraz
Copy link
Contributor

jessfraz commented Apr 12, 2017

can you assign this one to me so i remember to close it when i make them all minimal :)

@gopherbot
Copy link

CL https://golang.org/cl/40572 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 17, 2017
It's starting to pollute the package, and seems separable. I also plan
to use it more from elsewhere in upcoming CLs.

Updates golang/go#19866

Change-Id: I7b2add37f74ed42c2f78939924f19d8322179823
Reviewed-on: https://go-review.googlesource.com/40868
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/40868 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2017
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     786MB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     8.85MB

Updates golang/go#18817
Updates golang/go#19866

Change-Id: Ibfdadbaf043e44b26d5807c1da45746e934a7dce
Reviewed-on: https://go-review.googlesource.com/40572
Reviewed-by: Kevin Burke <kev@inburke.com>
@bradfitz
Copy link
Contributor Author

Thanks for the Dockerfile. I'll try to productize this soon. Still fighting a Github sync issue that's easier to debug locally for now.

@gopherbot
Copy link

CL https://golang.org/cl/41824 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/42146 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017
…package

This will also be used by maintner.golang.org shortly.

Updates golang/go#19866

Change-Id: Id952065831920a206e3cb97bd1f451ceaea34927
Reviewed-on: https://go-review.googlesource.com/42146
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017
And Kubernetes config.

Updates golang/go#19866

Change-Id: I67b9c93c9c931c19d5c4275a55753ee146e13f21
Reviewed-on: https://go-review.googlesource.com/41824
Reviewed-by: Kevin Burke <kev@inburke.com>
@gopherbot
Copy link

CL https://golang.org/cl/42148 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2017
Also, the maintner.Corpus constructor is now gone. An API is added for
maintnerd to become the leader and specify the cache dir.

Updates golang/go#19866

Change-Id: Ia726aa00ca1337b6c130cfee040ff9a1f935d0c2
Reviewed-on: https://go-review.googlesource.com/42148
Reviewed-by: Kevin Burke <kev@inburke.com>
@gopherbot
Copy link

CL https://golang.org/cl/42184 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 1, 2017
When resuming from a point, the reclog reading code would double check
the record headers offsets but because netsource MutationSource would
seek on the *os.File, what reclog saw and expected didn't match what
it read.

Also move temporary 5 second sleep to 2 seconds and adjust some
logging.

Updates golang/go#19866

Change-Id: I66d1f9df8bb36cf028b715ddd284cb10bc74b45b
Reviewed-on: https://go-review.googlesource.com/42184
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
kevinburke pushed a commit to kevinburke/build that referenced this issue May 2, 2017
When resuming from a point, the reclog reading code would double check
the record headers offsets but because netsource MutationSource would
seek on the *os.File, what reclog saw and expected didn't match what
it read.

Also move temporary 5 second sleep to 2 seconds and adjust some
logging.

Updates golang/go#19866

Change-Id: I66d1f9df8bb36cf028b715ddd284cb10bc74b45b
Reviewed-on: https://go-review.googlesource.com/42184
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@AlekSi
Copy link
Contributor

AlekSi commented May 4, 2017

There are data races that should be fixed. Noticed with

go get -u -v -race golang.org/x/build/maintner/...
maintnerd -config=go

@bradfitz
Copy link
Contributor Author

bradfitz commented May 4, 2017

@AlekSi, thanks! I sent out https://go-review.googlesource.com/c/42615/

gopherbot pushed a commit to golang/build that referenced this issue May 4, 2017
The lastUpdated field can only be used by the sync goroutine, but we
did labels and milestones concurrently in their own goroutines.

Remove the update because it was redundant with the caller's update
anyway.

Noticed in:
golang/go#19866 (comment)

Change-Id: I4b0270a00eaf2994ed096631e24a7bd75d0210ab
Reviewed-on: https://go-review.googlesource.com/42615
Reviewed-by: Kevin Burke <kev@inburke.com>
@AlekSi
Copy link
Contributor

AlekSi commented May 4, 2017

Another thing: there is no handling of termination signals in maintnerd. Ctrl+C instantly kills the processes without any graceful context cancelations. If I understand correctly, this greatly increases the chances of data corruption (cache becoming out-of-sync), especially during initial sync.

I'm interested in helping with this if dust is settled enough in this place.

@bradfitz
Copy link
Contributor Author

bradfitz commented May 4, 2017

Lack of graceful shutdown is not a problem because all robust programs must be able to also survive un-graceful shutdowns, which maintnerd should. It has a pretty paranoid on-disk record format.

@gopherbot
Copy link

CL https://golang.org/cl/43557 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 17, 2017
For upcoming API development, permit running maintnerd in non-syncing
(no new mutations logged) mode, using only the data from the godata
package.

Updates golang/go#19866
Updates golang/go#20222

Change-Id: I8c66c7f5e29b5702b481da0fec8f270f93f484a9
Reviewed-on: https://go-review.googlesource.com/43557
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/43560 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 18, 2017
Also, update docs on how to re-generate.
Switches to using go4.org/grpc and go4.org/grpc-codegen/protoc-gen-go4grpc
for now.

Updates golang/go#19866
Updates golang/go#20222

Change-Id: Ifa8a123fca2a30f17270c3c558b7395a02064eae
Reviewed-on: https://go-review.googlesource.com/43560
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@golang golang locked and limited conversation to collaborators May 18, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Development

No branches or pull requests

4 participants