Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: renewing MTLS Certificates in TLS.Conn Connections before remote error: tls: bad certificate error is thrown #48817

Closed
ShivanshVij opened this issue Oct 6, 2021 · 3 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@ShivanshVij
Copy link

ShivanshVij commented Oct 6, 2021

What version of Go are you using (go version)?

$ go version
1.17.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/shivanshvij/Library/Caches/go-build"
GOENV="/Users/shivanshvij/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/shivanshvij/go/pkg/mod"
GONOPROXY="github.com/loophole-labs"
GONOSUMDB="github.com/loophole-labs"
GOOS="darwin"
GOPATH="/Users/shivanshvij/go"
GOPRIVATE="github.com/loophole-labs"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/shivanshvij/go/go1.17.1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/shivanshvij/go/go1.17.1/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4h/yyr7_ggd2td0dsymd6750q740000gn/T/go-build2007742489=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Creating tls.Conn connections using short-lived certificates with Client Authentication.

Client Config:

t.tlsConfig = &tls.Config{
		Certificates: []tls.Certificate{{
			Certificate: [][]byte{mtls.Raw},
			PrivateKey:  privateKey,
		}},
		RootCAs: caCerts,
	}

Server Config:

tlsConfig := &tls.Config{
		Certificates: []tls.Certificate{{
			Certificate: [][]byte{cert.Raw},
			PrivateKey:  privateKey,
		}},
		RootCAs:    caCerts,
		ClientCAs:  caCerts,
		ClientAuth: tls.RequireAndVerifyClientCert,
	}

What did you expect to see?

No errors thrown once the certificate expires, and the existing connections allowed to continue functioning.

What did you see instead?

For a few minutes, everything works fine and it's possible to read and write data exactly as expected. Then, as soon as the short-lived x509 certificate expires, the client TLS.Conn connections throw a remote error: tls: bad certificate error.

If that's expected behaviour, is it possible to replace the certificates in the connection somehow? For our use case, we would rather not have to re-create the connections if possible.

@mknyszek mknyszek changed the title Renewing MTLS Certificates in TLS.Conn Connections before remote error: tls: bad certificate error is thrown crypto/tls: renewing MTLS Certificates in TLS.Conn Connections before remote error: tls: bad certificate error is thrown Oct 6, 2021
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 6, 2021
@mknyszek mknyszek added this to the Backlog milestone Oct 6, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Oct 6, 2021

@ShivanshVij
Copy link
Author

So it looks like during any renegotiation, the GetClientCertificate gets called. This should (in theory) allow us to swap certificates on the fly for any new connections that get created as well as for any renegotiation.

I'm still concerned about what happens to connections that have already completed their handshake - do they get killed once the certificate expires?

I'm pretty sure that's what happened in our case, but that seems like unexpected behaviour after reading this: https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/

@ShivanshVij
Copy link
Author

Looks like this was a false alarm. We're not able to reproduce this issue on our end anymore, and swapping the certificates is handled thanks to the GetClientCertificate function that's provided for us.

I'm closing this issue.

@golang golang locked and limited conversation to collaborators Oct 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants