Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/crypto/acme/autocert: new certificate rejected by Chrome client for 1h if client's clock is behind. #39638

Open
h8liu opened this issue Jun 17, 2020 · 3 comments
Open

x/crypto/acme/autocert: new certificate rejected by Chrome client for 1h if client's clock is behind. #39638

h8liu opened this issue Jun 17, 2020 · 3 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@h8liu
Copy link

h8liu commented Jun 17, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/h8liu/Library/Caches/go-build"
GOENV="/Users/h8liu/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/h8liu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4l/dcqzjhqn65gggmw9qrlnksx00000gn/T/go-build462787337=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. use autocert to host a new site
  2. change client's system clock to 3 second behind UTC. (1 second would also work).
  3. visit the site with Google Chrome, the site will be rejected due to SCT timestamp was in the past.
  4. in fact, even 5 seconds later, when the certificate is valid again, the site will still get rejected.

Chrome version: 83.0.4103.97

What did you expect to see?

Certificate won't get rejected, at least not for one hour.

I think the rejection can also happen when the visit is on a new renewal, but renewals often happen in the background, so it is slightly harder to hit the particular time window in a debug session.

What did you see instead?

Certificate got rejected for 1 hour.

to be fair, I think the root cause is on chrome browser side. I also filed the issue to chrome, and chrome marked it as won't fix:
https://bugs.chromium.org/p/chromium/issues/detail?id=1093183

so one mitigation autocert can have, is to have an config to optionally insert a sleeping delay after a certificate is issued and before the certificate can be returned to a GetCertificate() call.
@gopherbot gopherbot added this to the Unreleased milestone Jun 17, 2020
@davecheney
Copy link
Contributor

Thank you for raising this issue. Given that Chrome have decided they will not address the certificate caching in the browser, I'm not sure what can or should be fixed on the Go side. The server running autocert should try to keep its clock in sync -- although I appreciate how hard accurate time keeping is these days.

Sleeping for an amount of time greater than the system's clock drift is an answer to the problem, but I don't think it is a good answer to the problem because this is fundamentally a caching problem on the browser. Any delaying returning from the certificate generation call is always a guess against unknown clock skew on the client.

@davecheney davecheney added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 17, 2020
@h8liu
Copy link
Author

h8liu commented Jun 18, 2020

Given the update on side of the chrome bug, it seems that the invalid result caching issue (invalid for 30+ minutes) will go away eventually.

The client clock skew/drift issue will still be there though: 1st time visit has a chance that the cert will be rejected for a short time (which also can happen in background refreshing scenarios when the client hits the exact time window), and you are right that it is a guess on the client side's system time error.

Based on my understanding, letsencrypt already had a guess (in the old days), which is 1 hour. That is a certificate issued at say 3pm will have NotValidBefore set to 2pm.

Assuming that nowadays most clients (that check SCTs) sync system time from the Internet, I think adding even just a 1-second sleep would greatly mitigate the issue in practice. I was hoping that this can be an config option where default is 0, but the server hosting user can set to a small time duration.

I am not sure what will be a better solution either.. maybe the client should compare the SCT timestamp with a queried time from the SCT's signing server (rather than local system time)?

@ALTree ALTree added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 18, 2021
@mitar
Copy link
Contributor

mitar commented Oct 29, 2022

Related: https://community.letsencrypt.org/t/possible-race-condition-observed-with-autocert/137271

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@davecheney @mitar @h8liu @ALTree @gopherbot