Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

time: time.Parse unable to parse timestamps with unusual UTC offsets #26032

Open
ALTree opened this issue Jun 24, 2018 · 9 comments
Open

time: time.Parse unable to parse timestamps with unusual UTC offsets #26032

ALTree opened this issue Jun 24, 2018 · 9 comments
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@ALTree
Copy link
Member

ALTree commented Jun 24, 2018

$ gotip version
go version devel +d6a27e8edc Sat Jun 23 00:16:14 2018 +0000 linux/amd64

I dumped the time for each of the 425 timezones in my system's tz database in a file and tried to parse the timestamps. I got 17 errors, all caused by time.Parse inability to parse unusual offsets like +0545 (used in Asia/Kathmandu) or +13 (used in Pacific/Fakaofo).

Complete failures log:

Timezone:	 America/Scoresbysund
zdump time:	 Sun Jun 24 10:43:46 2018 +00
time.Parse:	 parsing time "Sun Jun 24 10:43:46 2018 +00": extra text: +00 

Timezone:	 Asia/Colombo
zdump time:	 Sun Jun 24 16:13:46 2018 +0530
time.Parse:	 parsing time "Sun Jun 24 16:13:46 2018 +0530": extra text: +0530 

Timezone:	 Asia/Kabul
zdump time:	 Sun Jun 24 15:13:46 2018 +0430
time.Parse:	 parsing time "Sun Jun 24 15:13:46 2018 +0430": extra text: +0430 

Timezone:	 Asia/Kathmandu
zdump time:	 Sun Jun 24 16:28:46 2018 +0545
time.Parse:	 parsing time "Sun Jun 24 16:28:46 2018 +0545": extra text: +0545 

Timezone:	 Asia/Tehran
zdump time:	 Sun Jun 24 15:13:46 2018 +0430
time.Parse:	 parsing time "Sun Jun 24 15:13:46 2018 +0430": extra text: +0430 

Timezone:	 Asia/Yangon
zdump time:	 Sun Jun 24 17:13:46 2018 +0630
time.Parse:	 parsing time "Sun Jun 24 17:13:46 2018 +0630": extra text: +0630 

Timezone:	 Atlantic/Azores
zdump time:	 Sun Jun 24 10:43:46 2018 +00
time.Parse:	 parsing time "Sun Jun 24 10:43:46 2018 +00": extra text: +00 

Timezone:	 Australia/Eucla
zdump time:	 Sun Jun 24 19:28:46 2018 +0845
time.Parse:	 parsing time "Sun Jun 24 19:28:46 2018 +0845": extra text: +0845 

Timezone:	 Australia/Lord_Howe
zdump time:	 Sun Jun 24 21:13:46 2018 +1030
time.Parse:	 parsing time "Sun Jun 24 21:13:46 2018 +1030": extra text: +1030 

Timezone:	 Indian/Cocos
zdump time:	 Sun Jun 24 17:13:46 2018 +0630
time.Parse:	 parsing time "Sun Jun 24 17:13:46 2018 +0630": extra text: +0630 

Timezone:	 Pacific/Apia
zdump time:	 Sun Jun 24 23:43:46 2018 +13
time.Parse:	 parsing time "Sun Jun 24 23:43:46 2018 +13": extra text: +13 

Timezone:	 Pacific/Chatham
zdump time:	 Sun Jun 24 23:28:46 2018 +1245
time.Parse:	 parsing time "Sun Jun 24 23:28:46 2018 +1245": extra text: +1245 

Timezone:	 Pacific/Enderbury
zdump time:	 Sun Jun 24 23:43:46 2018 +13
time.Parse:	 parsing time "Sun Jun 24 23:43:46 2018 +13": extra text: +13 

Timezone:	 Pacific/Fakaofo
zdump time:	 Sun Jun 24 23:43:46 2018 +13
time.Parse:	 parsing time "Sun Jun 24 23:43:46 2018 +13": extra text: +13 

Timezone:	 Pacific/Kiritimati
zdump time:	 Mon Jun 25 00:43:46 2018 +14
time.Parse:	 parsing time "Mon Jun 25 00:43:46 2018 +14": extra text: +14 

Timezone:	 Pacific/Marquesas
zdump time:	 Sun Jun 24 01:13:46 2018 -0930
time.Parse:	 parsing time "Sun Jun 24 01:13:46 2018 -0930": extra text: -0930 

Timezone:	 Pacific/Tongatapu
zdump time:	 Sun Jun 24 23:43:46 2018 +13
time.Parse:	 parsing time "Sun Jun 24 23:43:46 2018 +13": extra text: +13 

https://gist.github.com/ALTree/de33561c1cc00ac46e2e9e6d6cca52fe

@ALTree ALTree added the NeedsFix The path to resolution is known, but the work has not been done. label Jun 24, 2018
@ALTree ALTree added this to the Go1.12 milestone Jun 24, 2018
@ALTree ALTree self-assigned this Jun 24, 2018
@gopherbot
Copy link

Change https://golang.org/cl/120558 mentions this issue: time: accept UTC offsets +13,+14 in Parse, reject -13,-14

gopherbot pushed a commit that referenced this issue Aug 21, 2018
time.Parse currently rejects numeric timezones names with UTC offsets
bigger than +12, but this is incorrect: there's a +13 timezone and a
+14 timezone:

  $ zdump Pacific/Kiritimati
  Pacific/Kiritimati  Mon Jun 25 02:15:03 2018 +14

For convenience, this cl changes the ranges of accepted offsets from
-14..+12 to -23..+23 (zero still excluded), i.e. every possible offset
that makes sense. We don't validate three-letter abbreviations for the
timezones names, so there's no need to be too strict on numeric names.

This change also fixes a bug in the parseTimeZone, that is currently
unconditionally returning true (i.e. valid timezone), without checking
the value returned by parseSignedOffset.

This fixes 5 of 17 time.Parse() failures listed in Issue #26032.

Updates #26032

Change-Id: I2f08ca9aa41ea4c6149ed35ed2dd8f23eeb42bff
Reviewed-on: https://go-review.googlesource.com/120558
Reviewed-by: Rob Pike <r@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/130696 mentions this issue: time: allow +00 as numeric timezone name and GMT offset

gopherbot pushed a commit that referenced this issue Aug 24, 2018
A timezone with a zero offset from UTC and without a three-letter
abbreviation will have a numeric name in timestamps: "+00".

There are currently two of them:

  $ zdump Atlantic/Azores America/Scoresbysund
  Atlantic/Azores       Wed Aug 22 09:01:05 2018 +00
  America/Scoresbysund  Wed Aug 22 09:01:05 2018 +00

These two timestamp are rejected by Parse, since it doesn't allow for
zero offsets:

  parsing time "Wed Aug 22 09:01:05 2018 +00": extra text: +00

This change modifies Parse to accept a +00 offset in numeric timezone
names.

As side effect of this change, Parse also now accepts "GMT+00". It was
explicitely disallowed (with a unit test ensuring it got rejected),
but the restriction seems incorrect.

DATE(1), for example, allows it:

  $ date --debug --date="2009-01-02 03:04:05 GMT+00"

  date: parsed date part: (Y-M-D) 2009-01-02
  date: parsed time part: 03:04:05
  date: parsed zone part: UTC+00
  date: input timezone: parsed date/time string (+00)
  date: using specified time as starting value: '03:04:05'
  date: starting date/time: '(Y-M-D) 2009-01-02 03:04:05 TZ=+00'
  date: '(Y-M-D) 2009-01-02 03:04:05 TZ=+00' = 1230865445 epoch-seconds
  date: timezone: system default
  date: final: 1230865445.000000000 (epoch-seconds)
  date: final: (Y-M-D) 2009-01-02 03:04:05 (UTC)
  date: final: (Y-M-D) 2009-01-02 04:04:05 (UTC+01)
  Fri  2 Jan 04:04:05 CET 2009

This fixes 2 of 17 time.Parse() failures listed in Issue #26032.

Updates #26032

Change-Id: I01cd067044371322b7bb1dae452fb3c758ed3cc2
Reviewed-on: https://go-review.googlesource.com/130696
Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ALTree
Copy link
Member Author

ALTree commented Aug 26, 2018

+14 and +00 are now parsed correctly, down to 10 failures, all caused by non-integral UTC offsets:

$ gotip run parse_timezones.go 
Timezone:    Indian/Cocos
zdump time:  Sun Aug 5 18:25:10 2018 +0630
time.Parse:  parsing time "Sun Aug 5 18:25:10 2018 +0630" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0630" as "MST" 

Timezone:    Asia/Kabul
zdump time:  Sun Aug 5 16:25:10 2018 +0430
time.Parse:  parsing time "Sun Aug 5 16:25:10 2018 +0430" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0430" as "MST" 

Timezone:    Australia/Lord_Howe
zdump time:  Sun Aug 5 22:25:10 2018 +1030
time.Parse:  parsing time "Sun Aug 5 22:25:10 2018 +1030" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+1030" as "MST" 

Timezone:    Australia/Eucla
zdump time:  Sun Aug 5 20:40:10 2018 +0845
time.Parse:  parsing time "Sun Aug 5 20:40:10 2018 +0845" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0845" as "MST" 

Timezone:    Asia/Kathmandu
zdump time:  Sun Aug 5 17:40:10 2018 +0545
time.Parse:  parsing time "Sun Aug 5 17:40:10 2018 +0545" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0545" as "MST" 

Timezone:    Pacific/Chatham
zdump time:  Mon Aug 6 00:40:10 2018 +1245
time.Parse:  parsing time "Mon Aug 6 00:40:10 2018 +1245" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+1245" as "MST" 

Timezone:    Pacific/Marquesas
zdump time:  Sun Aug 5 02:25:10 2018 -0930
time.Parse:  parsing time "Sun Aug 5 02:25:10 2018 -0930" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "-0930" as "MST" 

Timezone:    Asia/Colombo
zdump time:  Sun Aug 5 17:25:10 2018 +0530
time.Parse:  parsing time "Sun Aug 5 17:25:10 2018 +0530" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0530" as "MST" 

Timezone:    Asia/Yangon
zdump time:  Sun Aug 5 18:25:10 2018 +0630
time.Parse:  parsing time "Sun Aug 5 18:25:10 2018 +0630" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0630" as "MST" 

Timezone:    Asia/Tehran
zdump time:  Sun Aug 5 16:25:10 2018 +0430
time.Parse:  parsing time "Sun Aug 5 16:25:10 2018 +0430" as "Mon Jan 2 15:04:05 2006 MST": cannot parse "+0430" as "MST" 

Failures:    10

@cstockton
Copy link

cstockton commented Aug 22, 2019

I just ran into some timestamps formatted as 2019-08-22T10:41:59.006928+0000 - I know time parsing is complex but I believe the above is ISO8601 compliant. I believe the below should return the same times as date command, and definitely not have a parse failure:

Format Cmd Result
2019-08-22T10:41:59.006928+0000 date Thu Aug 22 03:41:59 MST 2019
2019-08-22T10:41:59.006928+0000 go parsing time ""2019-08-22T10:41:59.006928+0000"" as ""2006-01-02T15:04:05Z07:00"": cannot parse "+0000"" as "Z07:00"
2019-08-22T10:41:59.006928+00:00 date Thu Aug 22 03:41:59 MST 2019
2019-08-22T10:41:59.006928+00:00 go 2019-08-22 10:41:59.006928 +0000 +0000
2019-08-22T10:41:59.006928Z date Thu Aug 22 03:41:59 MST 2019
2019-08-22T10:41:59.006928Z go 2019-08-22 10:41:59.006928 +0000 UTC

Go 1.8-1.12.9 all have same result.

edit: sorry, I left out the fact it's parsing JSON. I could work around it easily if it was just a regular string. But there is never a clean way to fix JSON time parsing in without wrapping it in a custom type which makes working with the data structure unfortunate since you have to unwrap the type to do anything with it, or have a multiple step json.Unmarshal in the parent type which doesn't scale very well for nested occurrences. That said I think it may be worth fixing in json.Unmarshal even if it's a simple special case that accepts another set of 00 in place of a :.

@ALTree
Copy link
Member Author

ALTree commented Aug 22, 2019

@cstockton It seems to be working (if you use the right parsing format):

package main

import (
	"fmt"
	"time"
)

func main() {
	tstamp := "2019-08-22T10:41:59.006928+0000"
	t, err := time.Parse("2006-01-02T15:04:05-0700", tstamp)
	if err != nil {
		panic(err)
	}
	fmt.Println(t.Format(time.UnixDate))
}

https://play.golang.org/p/RjhP0xB6AtX

what am I missing?

@cstockton
Copy link

I am using JSON Unmarshal.

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@goku321

This comment has been minimized.

@goku321

This comment has been minimized.

AlexanderYastrebov added a commit to AlexanderYastrebov/go that referenced this issue Oct 20, 2021
With this CL Parse accepts all of `timedatectl list-timezones | xargs zdump`

Fixes golang#26032
@gopherbot
Copy link

Change https://golang.org/cl/357450 mentions this issue: time: allow +0430,-0930 UTC offsets in Parse

AlexanderYastrebov added a commit to AlexanderYastrebov/go that referenced this issue Oct 22, 2021
With this CL Parse accepts all of `timedatectl list-timezones | xargs zdump`

Fixes golang#26032
AlexanderYastrebov added a commit to AlexanderYastrebov/go that referenced this issue Oct 22, 2021
With this CL Parse accepts all of `timedatectl list-timezones | xargs zdump`

Fixes golang#26032
@ALTree ALTree removed their assignment Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants