time: RFC3339Nano does not guarantee natural ordering

[arussellsaw]

Hi!

When the time.Time used in Format(time.RFC3339Nano) is rounded to the nearest second (or anything lower than nanosecond), any trailing 0s are not formatted due to the formatting string.
RFC3339Nano should format to a natural/string sortable value. This is mentioned in section 5.1 of RFC3339 https://www.ietf.org/rfc/rfc3339.txt

5.1. Ordering

   If date and time components are ordered from least precise to most
   precise, then a useful property is achieved.  Assuming that the time
   zones of the dates and times are the same (e.g., all in UTC),
   expressed using the same string (e.g., all "Z" or all "+00:00"), and
   all times have the same number of fractional second digits, then the
   date and time strings may be sorted as strings (e.g., using the
   strcmp() function in C) and a time-ordered sequence will result.  The
   presence of optional punctuation would violate this characteristic. 

I propose that time.RFC3339Nano is updated to the following formatting string:

"2006-01-02T15:04:05.000000000Z07:00"

or that a time.RFC3339NanoNatural (or similar) constant is declared

thanks for your time!

What version of Go are you using (go version)?

go version go1.8 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/alex/dev/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/l5/x43qw4sd10j_459gxsy9h3j80000gn/T/go-build164235018=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I formatted two time.Time values using RFC3339Nano
https://play.golang.org/p/qY81JM6Np8

What did you expect to see?

Hello, playground
2009-11-10T23:00:00.00000000Z
2009-11-10T23:00:00.00000001Z

What did you see instead?

Hello, playground
2009-11-10T23:00:00Z
2009-11-10T23:00:00.00000001Z

[ianlancetaylor]

We can't change RFC3339Nano now, because that would break the Go 1 compatibility guarantee (https://golang.org/doc/go1compat).

We could introduce RFC3339NanoNatural as you suggest, but it's not clear to me that it's worth doing. Clearly you can write it yourself. Does this come up often enough to justify adding it to the standard library?

[arussellsaw]

I'm using my own constant defined right now, yes. Perhaps it's not worth the change, i just feel as though this should be raised or considered as we're declaring RFC3339 but not fully implementing it. My biggest issue with the new constant name is just that it doesn't look right 😹 if anyone has a better name/solution (other than defining your own) i welcome it!

i also think it's worth noting that https://golang.org/doc/go1compat mentions that the team reserves the right to fix bugs, and i would consider this a bug

[bradfitz]

We could also just document the difference and/or add an example.

[gopherbot]

CL https://golang.org/cl/47090 mentions this issue.

[ljubomirb]

We could introduce RFC3339NanoNatural as you suggest, but it's not clear to me that it's worth doing. Clearly you can write it yourself. Does this come up often enough to justify adding it to the standard library?

When using time as key in time-series databases such as BoltDB, the key is internally sorted by DB. This approach allows very fast search, however, a key has to be sortable.
Having time string that is sortable allows this to work, and yes, it can be (and it is) all done by hand (using custom format string), but the problem is always systematic - if something is defined by Go team, it's less likely to later cause hard to trace bugs because of using different format string at different parts of code by different developers.

If you come to accept this arguments, I also propose RFC3339MilliNatural, since it is sufficient in most real world cases (especially in IoT world) and on few million entries it saves a lot.

[ianlancetaylor]

@ljubomirb This issue was closed months ago.

To change that decision I think requires a practical argument, not a theoretical one. For example, show us several unrelated packages that are working around this problem.

[ljubomirb]

I understand that is is closed, and that you gave it a thought before, but I ask you to consider it maybe one more time.
The most practical argument (I think) is, for example, BoltDB. It is one of the most popular DB packages for Go, written entirely in Go. It is just a bare bone core key-value store with low footprint, used by database ecosystems like InfluxDB or other (check the list at the end of that page).
The author of BoltDB gives example of how to store key as time and explans limitations here

Another common use case is scanning over a range such as a time range. If you use a sortable time encoding such as RFC3339 then you can query a specific date range like this:

db.View(func(tx *bolt.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 90's decade.
	min := []byte("1990-01-01T00:00:00Z")
	max := []byte("2000-01-01T00:00:00Z")

	// Iterate over the 90's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

You may argue that it's all just a simple matter of inserting format string, so one can simply introduce his own format const like const RFC3339MillisFormat = "2006-01-02T15:04:05.000Z07:00"
However, you will end up writing code like this, but it is the systematic problem here that must be emphasized: if not communicated well, a developer on the other side (since this is a matter of a simple format string) may create his own const with 999 at the end instead of 000 (the reason why I'm posting this). If this simple const was to be within Go, then it's less likely situations like this to happen.

As for my proposal of adding Millis (natural), I have nothing but theoretical argument: most IoT devices work in sub 40MHz clock range. When they capture, process and at the end send event of interest to database backend (often using speeds like 9600bps), then using nano precision during logging is just wrong. If, on the other hand, device is capable to internally capture event related in time difference to previous one (like GPIO rising edge times), then Nano will be used anyway.

P.S. another problem one will encounter is that if you have variable number of digits of unpredictable length, then sending the same timestamp to different databases will result in different times being saved, due to the way trim or round is performed in different versions of same databases, and thus you cannot rely to query DB by exact date as noted here for MySQL

Noted in 8.0.1 changelog.
Inserting a TIME, DATE, or TIMESTAMP value with a fractional seconds
part into a column having the same type but fewer fractional digits
resulted in rounding. This differs from MySQL 5.5, which used
truncation rather than rounding. To enable control over this
behavior, a new TIME_TRUNCATE_FRACTIONAL SQL mode is available. The
default is to use rounding. If this mode is enabled, truncation
occurs instead.

If you, however, do know the length of your time string, than it's easier to approach this problem.

[whilei]

Alternatively you can use time.UnixNano() or Nano(), parsing those int64's to []byte as demo'd here: boltdb/bolt#338 (comment).

https://play.golang.org/p/fYSsSdNGznC