Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: time: support Colon for fractional seconds #36145

Closed
trbroyles1 opened this issue Dec 15, 2019 · 18 comments
Closed

proposal: time: support Colon for fractional seconds #36145

trbroyles1 opened this issue Dec 15, 2019 · 18 comments

Comments

@trbroyles1
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?

Yes, I can reproduce it on the playground

What operating system and processor architecture are you using (go env)?

GOOS=windows
GOARCH=amd64
GOHOSTARCH=amd64

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\N555074\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Development\go
set GOPROXY=
set GORACE=
set GOROOT=D:\Go
set GOTMPDIR=
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\N555074\AppData\Local\Temp\go-build289214998=/tmp/go-build -gno-record-gcc-switches

What did you do?

Attempt to parse a date/time string containing milliseconds preceded by a colon. Example:
https://play.golang.org/p/eY-B7yU7cbT

What did you expect to see?

Time is 02/12/2019 15:45:48:746

What did you see instead?

parsing time "02/12/2019 15:45:48:746" as "02/01/2006 15:04:05:000": cannot parse "746" as ":000"
@tannerryan
Copy link

This is not an issue. Please review the documentation for package time:

A fractional second is represented by adding a period and zeros to the end of the seconds section of layout string, as in "15:04:05.000" to format a time stamp with millisecond precision.

For your example, the format string must be 02/01/2006 15:04:05.000, not 02/01/2006 15:04:05:000. The input string must also be corrected.

Here is your example corrected: https://play.golang.org/p/E46bFklY7L3

@robpike robpike closed this as completed Dec 16, 2019
@trbroyles1
Copy link
Author

I'm sorry but I have to disagree. I reviewed the documentation thoroughly before writing the original issue, and I am well aware that the problem does not occur if a period is used in place of a colon. That was the whole point of the report I filed. The generated file I am processing uses a colon instead of a period. Which is non-standard, I completely agree. But, that's kind of my point. Non-standard things happen "in the wild", and the library needs to be able to handle them.

Yes, I can work around it by doing a search and replace on the colon in the string. That's what I'm doing for now. But the whole point of a format string, is to... specify the format of the string you want to parse. It's seems we've missed the mark if we have cases like that that it is impossible to format for.

Please, I would ask you, give the issue some more consideration before dropping it again. I'm not expecting this will change tomorrow or anything like that, but isn't it better in the long term if the format strings are able to cover a wider variety of cases ?

This general theme seems to be a recurring issue, to wit:
#6189
#27746
#26002

Couldn't some consideration be given to this? Alternatively, maybe this is a plus point for the priority of localization support

@agnivade
Copy link
Contributor

It seems like you want to file a proposal, rather than an issue. And from what I understand, your proposal is very similar to #6189. I'd suggest that you can subscribe to that issue for updates. Or if you feel the colon as a separator deserves its own proposal, please feel free to go ahead and file one.

@josharian josharian reopened this Dec 16, 2019
@josharian josharian changed the title time: Colon not supported in fractional seconds proposal: time: support Colon for fractional seconds Dec 16, 2019
@gopherbot gopherbot added this to the Proposal milestone Dec 16, 2019
@josharian
Copy link
Contributor

All the details are here already—reopened and converted to proposal.

@tengattack
Copy link

tengattack commented Jun 16, 2020

Maybe you can try this package: https://github.com/tengattack/jodatime ;) for arbitrary format

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jan 6, 2021
@rsc
Copy link
Contributor

rsc commented Jan 13, 2021

Non-standard things happen "in the wild", and the library needs to be able to handle them.

Unfortunately, many things happen in the wild, and it is not a goal to handle all of them. That would lead to enormous libraries that are hard to learn to use.

@rsc rsc moved this from Incoming to Active in Proposals (old) Jan 13, 2021
@rsc
Copy link
Contributor

rsc commented Jan 13, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@djaglowski
Copy link

Thank you for considering this further. I very much agree with exactly the sentiment expressed by @trbroyles1 above.

Unfortunately, many things happen in the wild, and it is not a goal to handle all of them.

I believe there is a very specific ask here, which is to make one specific separator configurable. It could likely be limited to a very small set of possible values, perhaps ., ,, ;, :, -.

While it is trivial to write a string replacement, it is a major performance hit when attempting to process millions of timestamps per second, for example.

Admittedly, it should be difficult to justify a general performance hit for all users of time.Parse, but it would seem reasonable to at least explore this use case further before ruling it out. Perhaps there is an implementation that is essentially performance neutral in most cases but also solves #6189, #27746, and #26002.

@ianlancetaylor
Copy link
Contributor

If timestamp processing is a bottleneck, I would strongly encourage you to write a custom parser rather than using time.Parse. time.Parse is not optimized for speed. Writing for a specific known layout should be measurably faster. Just feed the results into time.Date.

@djaglowski
Copy link

My primary use case is distributed log processing, which involves consuming a wide variety of timestamp layouts. time.Parse is almost a perfect solution, with the current proposal covering the only gap I've seen.

Performance-wise, time.Parse has been fine. Timestamp parsing is not a bottleneck, but it is one step in a series of transformations that must be performed on each log. A string replacement operation on every timestamp would be purely additive, hence why I'd like to avoid it if at all possible.

Details of my use case aside, I mostly just wanted to draw focus to the fact that this proposal is a narrowly defined ask, and that is aligned with what I understand to be the purpose of time.Parse - to provide a mechanism for parsing many real-world timestamp layouts. Clearly, there are limitations to what layouts should be supported, but it would seem that the issues linked above collectively make a case for the notion that this particular separator should be configurable.

@rsc
Copy link
Contributor

rsc commented Jan 20, 2021

We still can't do everything. To be worth making a special case here we'd need to know this was a very common thing to run into. A priori it sounds like it's not. But do you perhaps know what is generating these odd times? Is there some commonly-used spec we are unaware of?

@djaglowski
Copy link

Log4J has a default timestamp format that uses , as the separator. Applications that use the library commonly emit such timestamps. I've seen this format coming from Cassandra, Hadoop, Kafka, and Zookeeper, to name a few.

According to Wikipedia, ISO_8601 states that a comma is acceptable, even preferred. Presumably that may be why Log4J chose this default.

I have personally only run into the need for a comma as an alternate separator, but as an additional data point, here is a list of timestamp formats recognized by Sumo Logic, which appears to include at least one other separator.

@ianlancetaylor
Copy link
Contributor

Comma seems plausible, because of course many countries use decimal comma. But that is not this proposal.

One thing to consider is that if we recognize other characters as introducing a fractional number of seconds, it will be possible to write formats whose behavior will change unexpectedly. For example, time.Now().Format("05,000"). This doesn't prevent us from making a change, but it's worth bearing in mind.

@rsc
Copy link
Contributor

rsc commented Jan 27, 2021

In the absence of some kind of widely-used standard that uses : for a decimal point in this context, it seems like we should not add the complexity here. Comma is a separate issue (#43823).

@rsc
Copy link
Contributor

rsc commented Jan 27, 2021

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Decline in Proposals (old) Jan 27, 2021
@rsc rsc moved this from Likely Decline to Declined in Proposals (old) Feb 3, 2021
@rsc
Copy link
Contributor

rsc commented Feb 3, 2021

No change in consensus, so declined.
— rsc for the proposal review group

@gopherbot
Copy link

Change https://golang.org/cl/300996 mentions this issue: time: support "," as separator for fractional seconds

gopherbot pushed a commit that referenced this issue Mar 16, 2021
Accepts comma "," as a separator for fractional seconds
hence we now accept:
* 2006-01-02 15:04:05,999999999 -0700 MST
* Mon Jan _2 15:04:05,120007 2006
* Mon Jan 2 15:04:05,120007 2006

This change follows the recommendations of ISO 8601 per

   https://en.wikipedia.org/wiki/ISO_8601#cite_note-26

which states

   ISO 8601:2004(E), ISO, 2004-12-01, "4.2.2.4 ...
   the decimal fraction shall be divided from the integer
   part by the decimal sign specified in ISO 31-0, i.e.
   the comma [,] or full stop [.]. Of these, the comma
   is the preferred sign."

Unfortunately, I couldn't directly access the ISO 8601 document
because suddenly it is behind a paywall on the ISO website,
charging CHF 158 (USD 179) for 38 pages :-(

However, this follows publicly available cited literature, as well
as the recommendations from the proposal approval.

Fixes #6189
Updates #27746
Updates #26002
Updates #36145
Updates #43813
Fixes #43823

Change-Id: Ibe96064e8ee27c239be78c880fa561a1a41e190c
Reviewed-on: https://go-review.googlesource.com/c/go/+/300996
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@mikerochip
Copy link

mikerochip commented Jan 20, 2022

I get that this issue is closed and sorry in advance for the necro but searching led me here, and I wanted to add a data point since the only data point that appears in this thread is log4j.

My use case is that Unreal Engine 4 formats log timestamps in a custom way e.g. 2022.01.17-00.40.39:660 with the :660 part being milliseconds. I'll also emphasize here that this case isn't a loc issue, it's just a custom format.

The reason this is problematic is that I'm piping logs from a UE4 game server to a SaaS called Honeycomb (written in go, and it more or less passes timestamps through to time.ParseInLocation()). The closest I can get to configuring Honeycomb to make this work out of the box is 2006.01.02-15.04.05 but I wish I could've done 2006.01.02-15.04.05:000 and moved on.

Trying to make this work isn't as simple as modifying some of my application code which seems to be the workaround folks have settled on. To solve this, I'd either have to open a PR to Honeycomb, modify Unreal Engine's source, or create an entirely new log sink in my UE game code. So in my mind this issue represents a developer experience issue (not a loc issue) since all of those approaches are much higher effort than adding more flexible time parsing support.

I'm not intending to re-open this, I'll just say that as a person that's learning Go to investigate this issue, it feels like an odd choice to not allow decoupling of the seconds from the milliseconds in a time format string. For context, I'm coming from the .net world where you would use the string yyyy.MM.dd-HH.mm.ss:fff out of the box to parse those UE4 logs.

@golang golang locked and limited conversation to collaborators Jan 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests