Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

time: Location interprets wrong timezone (DST) with slim zoneinfo [1.15 backport] #42138

Closed
hlubek opened this issue Oct 22, 2020 · 36 comments
Closed
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Milestone

Comments

@hlubek
Copy link

hlubek commented Oct 22, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

Yes, but it seems to be fixed in master with d83168e.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build587886067=/tmp/go-build -gno-record-gcc-switches"

What did you do?

How to reproduce: https://gist.github.com/hlubek/f46a73bc9d150cf1f2af585b0849e3d9

  • Added a compiled Go binary to a very simple Alpine Docker image with installed tzdata package
  • Timezones (e.g. Europe/Berlin) uses wrong daylight saving information (e.g. CEST instead of CET for dates after October, 25th in 2020)
  • This seems to be caused by changes in the tzdata package in Alpine
  • It does not occur if the tzdata Alpine package is not installed and the embedded zoneinfo time/tzdata is used

What did you expect to see?

  • An error when loading the location with a non compatible zoneinfo

What did you see instead?

  • A wrongly interpreted date

Notes

This is already fixed with d83168e but I wonder why there is no info about that issue (took me some time to figure it out) or a test for the code added to zoneinfo_read.go. This is quite a standard setup for packaging Go programs so I could think many people are affected by this. The fix turns out to only work if the cached zone for now() is the same as the last transition. This broke after the last clock change on October, 25th for CET/CEST.

@MattBrittan
Copy link

MattBrittan commented Oct 22, 2020

I believe this may be a broader issue. This commit to zic changed the default format from fat to slim and that change was pushed out with the 2020b release. This means that the issue is likely to occur wherever 2020b or later timezone files are used (unless the package maintainer overrides the defaults when running zic). With daylight saving ending in many countries this weekend this issue may have a significant impact.

Edit: For those looking for a workaround here are a few options until a fix is released:

  • Use the ZONEINFO environmental variable to select a different zone file (e.g. export ZONEINFO=/usr/local/go/lib/time/zoneinfo.zip github ).
  • Include the tzdata package in your app (and do not install tzdata in the container - the package is only used if the time package cannot find tzdata files on the system).
  • Use a container built from scratch (in combination with one of the above options)
  • Recompile the relevant tz files in 'fat' mode (instructions)
  • Pin an earlier alpine version (i.e. alpine:3.8) that uses 2020a or earlier (note that version 3.8 is past its End of Support date).

@tklauser
Copy link
Member

/cc @ianlancetaylor

@tklauser tklauser added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 22, 2020
@tklauser
Copy link
Member

tklauser commented Oct 22, 2020

When working on https://golang.org/cl/261363 (commit 5b509d9) I noticed that some of the tests in package time failed due to the zic change mentioned in #42138 (comment), see e.g. https://storage.googleapis.com/go-build-log/29cd3aeb/linux-amd64_58c05150.log.

I investigated this further and figured out that there is an issue with the way we cache location data, which lead to https://golang.org/cl/261877 (commit d83168e). In addition, this should probably haven been reported as an issue to make it easier to discover, which I didn't do. Sorry about that.

I guess what we could do is backport the changes in src/time/zoneinfo_read.go from https://golang.org/cl/261877 to 1.15 but I'll defer to @ianlancetaylor to decide on that.

@ianlancetaylor
Copy link
Contributor

Yes, I think this is likely due to the very recent change in the default tzdata install. We didn't notice until now because the change was so recent. Since fortunately @tklauser has already found and fixed the problem on tip, and since more and more people are going to be using slim tzdata installations, I agree that we should backport the fix.

This should be a straightforward backport for 1.15. Unfortunately it is more work for 1.14, which doesn't have the tzset function introduced in https://golang.org/cl/215539. So perhaps we will have to backport that also to 1.14.

I'll repurpose this issue for 1.15 and open a backport issue for 1.14.

@ianlancetaylor
Copy link
Contributor

@gopherbot Please open backport for 1.14.

Without a backport people using Go 1.14 with a new tzdata installation will not get the correct timezone information.

@gopherbot
Copy link

Backport issue(s) opened: #42155 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot
Copy link

Change https://golang.org/cl/264301 mentions this issue: [release-branch.go1.15] time: support slim tzdata format

@ianlancetaylor
Copy link
Contributor

@hlubek It would be helpful if you could check that https://golang.org/cl/264301 fixes the problem for you. It is a patch against 1.15. Thanks.

@ianlancetaylor ianlancetaylor added the CherryPickCandidate Used during the release process for point releases label Oct 22, 2020
@hlubek
Copy link
Author

hlubek commented Oct 23, 2020

@hlubek It would be helpful if you could check that https://golang.org/cl/264301 fixes the problem for you. It is a patch against 1.15. Thanks.

Sure, will do!

@dsantos
Copy link

dsantos commented Oct 26, 2020

@hlubek I've tried using https://golang.org/cl/264301 and it didn't work as expected, unless I am missing something

@hlubek
Copy link
Author

hlubek commented Oct 26, 2020

@dsantos I could reproduce that the patch fixes the issue reproduced in https://gist.github.com/hlubek/f46a73bc9d150cf1f2af585b0849e3d9 (Alpine with tzdata package) by using a patched Go toolchain to compile a binary for the image.

Could you provider more details on your issue?

@dsantos
Copy link

dsantos commented Oct 26, 2020

Picking up from your example, this is how I've tested it:

FROM golang:1.15 as builder

RUN git clone --branch release-branch.go1.15 https://go.googlesource.com/go /root/goroot &&\
	cd /root/goroot/src &&\
	git fetch https://go.googlesource.com/go refs/changes/01/264301/1 && git checkout FETCH_HEAD &&\
	./make.bash

WORKDIR /app

COPY . .

RUN CGO_ENABLED=0 /root/goroot/bin/go build -o bin/test

# ---

FROM alpine:3.12

RUN apk update &&\
    apk --no-cache add tzdata &&\
    rm -rf /var/cache/apk/*

COPY --from=builder /app/bin/test /app/test

CMD ["/app/test"]

@hlubek
Copy link
Author

hlubek commented Oct 26, 2020

@dsantos Thanks, I can see it's failing and I digged a little deeper here.

I took the test from https://go-review.googlesource.com/c/go/+/264301 and applied it to the golang image together with the patch and it failed. Before it was working for me. I think it's possible that the extend handling is not correct depending on the now where the cache zone is built (because we had the clock change to DST since early Sunday). And if I update the now seconds to the day before (sec -= 200000) the test will succeed.

@gopherbot
Copy link

Change https://golang.org/cl/264939 mentions this issue: time: fix LoadLocationFromTZData with slim tzdata

@hlubek
Copy link
Author

hlubek commented Oct 26, 2020

I implemented a fix for tip that could be applied to the backport: https://go-review.googlesource.com/c/go/+/264939

@hlubek hlubek changed the title time: Location interprets wrong timezone (DST) with slim zoneinfo time: Location interprets wrong timezone (DST) with slim zoneinfo [1.15] Oct 26, 2020
@e3matheus
Copy link

For anyone facing this issue, I switched to time zone data en Go using this: https://medium.com/@mhcbinder/using-local-time-in-a-golang-docker-container-built-from-scratch-2900af02fbaf.

This caused a lot of problems in our site since last week, using America/Mexico_City. A new deploy with docker, would not calculate the correct tz data and we displayed different dates on our site.

@rob-lowcock
Copy link

This caused us a few problems as we were using alpine:latest as our base image, which currently uses 2020c-r0 for tzdata. For anyone looking for a workaround, we pinned our version to alpine:3.8 and that seemed to do the trick (as it uses 2020a-r0).

@butuzov
Copy link

butuzov commented Oct 27, 2020 via email

bogosj added a commit to bogosj/go-tesla-go that referenced this issue Oct 28, 2020
Pin to alpine:3.8.5 until golang/go#42138 is resolved, hopefully in go 1.15.4.
@magiconair
Copy link
Contributor

I am able to reproduce this with Alpine 3.12 but not Alpine 3.11. So you might not have to go back as far as 3.8. Can someone else confirm this?

@MattBrittan
Copy link

MattBrittan commented Oct 28, 2020

@magiconair I based the versions in my comment on the Alpine packages page which indicates that tzdata 2020c is current for all versions >3.8. I have just tested this (under docker) and can replicate the problem on versions 3.9-3.12 but 3.8 is OK.

@dmitshur
Copy link
Contributor

Approving per discussion in a release meeting. This is a serious issue without a workaround. This backport applies to both 1.15 (this issue) and 1.14 (#42155).

@dmitshur dmitshur added CherryPickApproved Used during the release process for point releases and removed CherryPickCandidate Used during the release process for point releases labels Oct 29, 2020
@gopherbot
Copy link

Closed by merging 414668c to release-branch.go1.15.

@gopherbot
Copy link

Change https://golang.org/cl/266299 mentions this issue: [release-branch.go1.15] time: fix LoadLocationFromTZData with slim tzdata

@ianlancetaylor
Copy link
Contributor

gopherbot why are you closing this issue?

gopherbot pushed a commit that referenced this issue Oct 29, 2020
Backport of part of https://golang.org/cl/261877 to support the slim
tzdata format. As of tzdata 2020b, the default is to use the slim format.
We need to support that format so that Go installations continue to
work when tzdata is updated.

Relevant part of the CL description:

    The reason for the failed tests was that when caching location data, the
    extended time format past the end of zone transitions was not
    considered. The respective change was introduced in (*Location).lookup
    by CL 215539.

For #42138

Change-Id: I37f52a0917b2c6e3957e6b4612c8ef104c736e65
Reviewed-on: https://go-review.googlesource.com/c/go/+/264301
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
@ianlancetaylor
Copy link
Contributor

Filed #42277 about gopherbot.

@mangas
Copy link

mangas commented Oct 29, 2020

@dmitshur @ianlancetaylor any idea when we can expect a release with this fix on 1.15?

@ianlancetaylor
Copy link
Contributor

We usually do minor releases at the start of each month, so, if all goes well, next week.

@mangas
Copy link

mangas commented Oct 29, 2020

This tz issue has a considerable impact and the workaround (finding a compatible tzdata version) is not great, don't suppose there is any chance to have it earlier?

@dmitshur dmitshur reopened this Oct 29, 2020
@ianlancetaylor
Copy link
Contributor

Earlier than next week? It's already Thursday, and we don't want to make a release on Friday, so I don't see how we can do earlier than next week.

There is an easy temporary workaround for Go 1.15: go build -tags=timetzdata. See https://golang.org/pkg/time/tzdata.

gopherbot pushed a commit that referenced this issue Oct 29, 2020
…data

The extend information of a time zone file with last transition < now
could result in a wrong cached zone because it used the zone of the
last transition.

This could lead to wrong zones in systems with slim zoneinfo.

For #42216
Fixes #42138

Change-Id: I7c57c35b5cfa58482ac7925b5d86618c52f5444d
Reviewed-on: https://go-review.googlesource.com/c/go/+/264939
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 70e022e)
Reviewed-on: https://go-review.googlesource.com/c/go/+/266299
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
@ianlancetaylor
Copy link
Contributor

This should now be fixed on the 1.15 release branch.

@choonkeat
Copy link
Contributor

choonkeat commented Oct 30, 2020

For those looking for a workaround here are a few options until a fix is released:
Use the ZONEINFO environmental variable to select a different zone file (e.g. export ZONEINFO=/usr/local/go/lib/time/zoneinfo.zip github ).

curious: what's the downside of using /usr/local/go/lib/time/zoneinfo.zip all the time (not as a "workaround", regardless of whether 1.15 etc is patched) instead of depending on another packaging system (alpine tzdata) ?

UPDATE: relying on Go toolchain update schedule for time zone data updates isn't the best setup. e.g. there could be projects that for whatever reason need to be pegged at certain Go versions, and having that to mean the time zone info is also outdated isn't right 🙇‍♂️ Thanks Matt

@gonzaloserrano
Copy link

Sorry to bother but I'm able to understand if there is any workaround while the new version is released (which is not using the release branch one I mean).

@tklauser
Copy link
Member

tklauser commented Nov 5, 2020

Sorry to bother but I'm able to understand if there is any workaround while the new version is released (which is not using the release branch one I mean).

A temporary workaround is described in #42138 (comment)

@toothrot toothrot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 5, 2020
wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this issue Nov 28, 2020
BREAKING CHANGES:

* agent: The enable_central_service_config option now defaults to
  true. [GH-8746]
* connect: Switch the default gateway port from 443 to 8443 to avoid
  assumption of Envoy running as root. [GH-9113]
* connect: Update Envoy metrics names and labels for proxy listeners so
  that attributes like datacenter and namespace can be extracted. [GH-9207]
* connect: intention destinations can no longer be reassigned [GH-8834]
* raft: Raft protocol v2 is no longer supported. If currently using
  protocol v2 then an intermediate upgrade to a version supporting both v2
  and v3 protocols will be necessary (1.0.0 - 1.8.x). Note that the Raft
  protocol configured with the raft_protocol setting and the Consul RPC
  protocol configured with the protocol setting and output by the consul
  version command are distinct and supported Consul RPC protocol versions are
  not altered. [GH-9103]
* sentinel: (Consul Enterprise only) update to v0.16.0, which replaces
  whitelist and blacklist with allowlist and denylist
* server: (Enterprise only) Pre-existing intentions defined with
  non-existent destination namespaces were non-functional and are erased
  during the upgrade process. This should not matter as these intentions had
  nothing to enforce. [GH-9186]
* server: (OSS only) Pre-existing intentions defined with either a source
  or destination namespace value that is not "default" are rewritten or
  deleted during the upgrade process. Wildcards first attempt to downgrade to
  "default" unless an intention already exists, otherwise these
  non-functional intentions are deleted. [GH-9186]
* xds: Drop support for Envoy versions 1.12.0, 1.12.1, 1.12.2, and 1.13.0,
  due to a lack of support for url_path in RBAC. [GH-8839]

SECURITY:

* Fix Consul Enterprise Namespace Config Entry Replication DoS. Previously
  an operator with service:write ACL permissions in a Consul Enterprise
  cluster could write a malicious config entry that caused infinite raft
  writes due to issues with the namespace replication logic. [CVE-2020-25201]
  [GH-9024]
* Increase the permissions to read from the /connect/ca/configuration
  endpoint to operator:write. Previously Connect CA configuration, including
  the private key, set via this endpoint could be read back by an operator
  with operator:read privileges. CVE-2020-28053 [GH-9240]

FEATURES:

* agent: Add a new RPC endpoint for streaming cluster state change events
  to clients.
* agent: Allow client agents to be configured with an advertised reconnect
  timeout to control how long until the nodes are reaped by others in the
  cluster. [GH-8781]
* agent: moved ui config options to a new ui_config stanza in agent
  configuration and added new options to display service metrics in the
  UI. [GH-8694]
* agent: return the default ACL policy to callers as a header [GH-9101]
* autopilot: A new /v1/operator/autopilot/state HTTP API was created to
  give greater visibility into what autopilot is doing and how it has
  classified all the servers it is tracking. [GH-9103]
* autopilot: Added a new consul operator autopilot state command to
  retrieve and view the Autopilot state from consul. [GH-9142]
* cli: update snapshot inspect command to provide more detailed snapshot
  data [GH-8787]
* connect: support defining intentions using layer 7 criteria [GH-8839]
* telemetry: add initialization and definition for non-expiring key metrics
  in Prometheus [GH-9088]
* telemetry: track node and service counts and emit them as metrics
  [GH-8603]
* ui: If Prometheus is being used for monitoring the sidecars, the topology
  view can be configured to display overview metrics for the
  services. [GH-8858]
* ui: Services using Connect with Envoy sidecars have a topology tab in the
  UI showing their upstream and downstream services. [GH-8788]
* xds: use envoy's rbac filter to handle intentions entirely within envoy
  [GH-8569]

IMPROVEMENTS:

* agent: Return HTTP 429 when connections per clients
  limit (limits.http_max_conns_per_client) has been reached. [GH-8221]
* agent: add path_allowlist config option to restrict metrics proxy queries
  [GH-9059]
* agent: allow the /v1/connect/intentions/match endpoint to use the agent
  cache [GH-8875]
* agent: protect the metrics proxy behind ACLs [GH-9099]
* api: The v1/connect/ca/roots endpoint now accepts a pem=true query
  parameter and will return a PEM encoded certificate chain of all the
  certificates that would normally be in the JSON version of the
  response. [GH-8774]
* api: support GetMeta() and GetNamespace() on all config entry kinds
  [GH-8764]
* autopilot: (Enterprise Only) Autopilot now supports using both Redundancy
  Zones and Automated Upgrades together. [GH-9103]
* checks: add health status to the failure message when gRPC healthchecks
  fail. [GH-8726]
* chore: Update to Go 1.15 with mitigation for golang/go#42138 [GH-9036]
* command: remove conditional envoy bootstrap generation for versions
  <=1.10.0 since those are not supported [GH-8855]
* connect: The Vault provider will now automatically renew the lease of the
  token used, if supported. [GH-8560]
* connect: add support for specifying load balancing policy in
  service-resolver [GH-8585]
* connect: intentions are now managed as a new config entry kind
  "service-intentions" [GH-8834]
* raft: Update raft to v1.2.0 to prevent non-voters from becoming eligible
  for leader elections and adding peer id as metric label to reduce
  cardinality in metric names [GH-8822]
* server: (Consul Enterprise only) ensure that we also shutdown network
  segment serf instances on server shutdown [GH-8786]
* server: break up Intention.Apply monolithic method [GH-9007]
* server: create new memdb table for storing system metadata [GH-8703]
* server: make sure that the various replication loggers use consistent
  logging [GH-8745]
* server: remove config entry CAS in legacy intention API bridge code
  [GH-9151]
* snapshot agent: Deregister critical snapshotting TTL check if leadership
  is transferred.
* telemetry: All metrics should be present and available to prometheus
  scrapers when Consul starts. If any non-deprecated metrics are missing
  please submit an issue with its name. [GH-9198]
* telemetry: add config flag telemetry { disable_compat_1.9 = (true|false)
  } to disable deprecated metrics in 1.9 [GH-8877]
* telemetry: add counter consul.api.http with labels for each HTTP path and
  method. This is intended to replace consul.http... [GH-8877]
* ui: Add the Upstreams and Exposed Paths tabs for services in mesh
  [GH-9141]
* ui: Moves the Proxy health checks to be displayed with the Service health
  check under the Health Checks tab [GH-9141]
* ui: Upstream and downstream services in the topology tab will show a
  visual indication if a deny intention or intention with L7 policies is
  configured. [GH-8846]
* ui: add dashboard_url_template config option for external dashboard links
  [GH-9002]

DEPRECATIONS:

* Go 1.15 has dropped support for 32-bit binaries for Darwin, so darwin_386
  builds will not be available for any 1.9.x+ releases. [GH-9036]
* agent: ui, ui_dir and ui_content_path are now deprecated for use in agent
  configuration files. Use ui_config.{enable, dir, content_path} instead. The
  command arguments -ui, -ui-dir, and -ui-content-path remain
  supported. [GH-8694]
* telemetry: The measurements in all of the consul.http... prefixed metrics
  have been migrated to consul.api.http. consul.http... prefixed metrics will
  be removed in a future version of Consul. [GH-8877]
* telemetry: the disable_compat_1.9 config will cover more metrics
  deprecations in future 1.9 point releases. These metrics will be emitted
  twice for backwards compatibility - if the flag is true, only the new
  metric name will be written. [GH-9181]

BUG FIXES:

* agent: make the json/hcl decoding of ConnectProxyConfig fully work with
  CamelCase and snake_case [GH-8741]
* agent: when enable_central_service_config is enabled ensure agent reload
  doesn't revert check state to critical [GH-8747]
* api: Fixed a bug where the Check.GRPCUseTLS field could not be set using
  snake case. [GH-8771]
* autopilot: (Enterprise Only) Previously servers in other zones would not
  be promoted when all servers in a second zone had failed. Now the actual
  behavior matches the docs and autopilot will promote a healthy non-voter
  from any zone to replace failure of an entire zone. [GH-9103]
* autopilot: Prevent panic when requesting the autopilot health immediately
  after a leader is elected. [GH-9204]
* command: when generating envoy bootstrap configs use the datacenter
  returned from the agent services endpoint [GH-9229]
* connect: Fixed an issue where the Vault intermediate was not renewed in
  the primary datacenter. [GH-8784]
* connect: fix Vault provider not respecting IntermediateCertTTL [GH-8646]
* connect: fix connect sidecars registered via the API not being
  automatically deregistered with their parent service after an agent restart
  by persisting the LocallyRegisteredAsSidecar property. [GH-8924]
* connect: use stronger validation that ingress gateways have compatible
protocols defined for their upstreams [GH-8470]
* license: (Enterprise only) Fixed an issue where the UI would see
  Namespaces and SSO as licensed when they were not.
* license: (Enterprise only) Fixed an issue where warnings about Namespaces
  being unlicensed would be emitted erroneously.
* namespace: (Enterprise Only) Fixed a bug that could case snapshot
  restoration to fail when it contained a namespace marked for deletion while
  still containing other resources in that namespace. [GH-9156]
* namespace: (Enterprise Only) Fixed an issue where namespaced services and
  checks were not being deleted when the containing namespace was deleted.
* raft: (Enterprise only) properly update consul server meta non_voter for
  non-voting Enterprise Consul servers [GH-8731]
* server: skip deleted and deleting namespaces when migrating intentions to
  config entries [GH-9186]
* telemetry: fixed a bug that caused logs to be flooded with [WARN]
  agent.router: Non-server in server-only area [GH-8685]
* ui: show correct datacenter for gateways [GH-8704]
@golang golang locked and limited conversation to collaborators Nov 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Projects
None yet
Development

No branches or pull requests