-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: cleanup failed reverse buildlets on coordinator #10682
Labels
Comments
What does health mean? TCP keep-alives would say the conn is alive at least. But a hung reverse buildlet mid-build is also bad. |
Stack from such a a dead builder:
|
TCP keep-alive should already be enabled on this incoming TLS reverse connections: func listenAndServeTLS() {
addr := ":443"
if *mode == "dev" {
addr = ":8119"
}
ln, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalf("net.Listen(%s): %v", addr, err)
}
serveTLS(ln)
}
func serveTLS(ln net.Listener) {
certPEM, err := readGCSFile("farmer-cert.pem")
if err != nil {
log.Printf("cannot load TLS cert, skipping https: %v", err)
return
}
keyPEM, err := readGCSFile("farmer-key.pem")
if err != nil {
log.Printf("cannot load TLS key, skipping https: %v", err)
return
}
cert, err := tls.X509KeyPair(certPEM, keyPEM)
if err != nil {
log.Printf("bad TLS cert: %v", err)
return
}
server := &http.Server{Addr: ln.Addr().String()}
config := &tls.Config{
NextProtos: []string{"http/1.1"},
Certificates: []tls.Certificate{cert},
}
tlsLn := tls.NewListener(tcpKeepAliveListener{ln.(*net.TCPListener)}, config)
log.Printf("Coordinator serving on: %v", tlsLn.Addr())
if err := server.Serve(tlsLn); err != nil {
log.Fatalf("serve https: %v", err)
}
}
type tcpKeepAliveListener struct {
*net.TCPListener
}
func (ln tcpKeepAliveListener) Accept() (c net.Conn, err error) {
tc, err := ln.AcceptTCP()
if err != nil {
return
}
tc.SetKeepAlive(true)
tc.SetKeepAlivePeriod(3 * time.Minute)
return tc, nil
} |
CL https://golang.org/cl/9851 mentions this issue. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
If a reverse buildlet dies in the middle of the build, the coordinator health check isn't sent and stays on the dashboard in a stuck state. Set a Conn timeout?
The text was updated successfully, but these errors were encountered: