syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

calmh · 2016-07-23T20:24:51Z

ZFS on Solaris retains nanosecond precision file timestamps. These can be seen in the information returned by os.Stat. However os.Chtimes seems limited to setting microsecond precision. This means, among other things, that a file can't be copied with the timestamp fully retained.

jb@zlogin4:~/s/g/s/s/test $ go version
go version go1.7rc3 solaris/amd64

# Nanoseond precision before running program:
jb@zlogin4:~/s/g/s/s/test $ touch test.file 
jb@zlogin4:~/s/g/s/s/test $ stat test.file
  File: ‘test.file’
  Size: 0               Blocks: 1          IO Block: 131072 regular empty file
Device: 5a00010023h/386547122211d       Inode: 96951       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/      jb)   Gid: (    1/   other)
Access: 2016-07-23 20:57:53.896596091 +0200
Modify: 2016-07-23 20:57:53.896596091 +0200
Change: 2016-07-23 20:57:53.896613387 +0200
 Birth: -

# Go program sees correct precision, attempts to set the same:
jb@zlogin4:~/s/g/s/s/test $ go run testchtimes.go test.file
2016-07-23 20:57:53.896596091 +0200 CEST

# Precision was reduced
jb@zlogin4:~/s/g/s/s/test $ stat test.file
  File: ‘test.file’
  Size: 0               Blocks: 1          IO Block: 131072 regular empty file
Device: 5a00010023h/386547122211d       Inode: 96951       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  502/      jb)   Gid: (    1/   other)
Access: 2016-07-23 20:57:53.896596000 +0200
Modify: 2016-07-23 20:57:53.896596000 +0200
Change: 2016-07-23 20:58:02.133653117 +0200
 Birth: -

# The program:
jb@zlogin4:~/s/g/s/s/test $ cat testchtimes.go 
package main

import (
        "fmt"
        "os"
)

func main() {
        info, err := os.Stat(os.Args[1])
        if err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
        t := info.ModTime()
        fmt.Println(t)
        if err := os.Chtimes(os.Args[1], t, t); err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
}

The text was updated successfully, but these errors were encountered:

bradfitz · 2016-07-24T00:05:02Z

I don't think there's anything Solaris-specific here.

The os package does the same thing on all POSIX systems (file_posix.go):

func Chtimes(name string, atime time.Time, mtime time.Time) error {
        var utimes [2]syscall.Timespec
        utimes[0] = syscall.NsecToTimespec(atime.UnixNano())
        utimes[1] = syscall.NsecToTimespec(mtime.UnixNano())
        if e := syscall.UtimesNano(name, utimes[0:]); e != nil {
                return &PathError{"chtimes", name, e}
        }
        return nil
}

... where syscall.UtimesNano always converts to microseconds. Many operating systems have utimens etc support but we don't appear to ever use it.

The os.Chtimes documentation is a little misleading:

The underlying filesystem may truncate or round the values to a less precise
time unit. If there is an error, it will be of type *PathError.

... since it's not really the filesystem truncating things.

This seems fixable.

odeke-em · 2016-09-27T08:23:22Z

Currently only the syscall Linux-es' packages use utimensat in syscall.UtimesNano so that
nano-second precision is kept for the Linux-es but not other OSes. You can see that this is true on the Linuxes from the playground here https://play.golang.org/p/Jwg_7NW2gx and below from this search

Emmanuels-MBP-2:syscall emmanuelodeke$ go version
go version devel +5df7f52 Sun Sep 25 23:42:59 2016 +0000 darwin/amd64
Emmanuels-MBP-2:syscall emmanuelodeke$ grep -Rn 'func utimensat' *
zsyscall_linux_386.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_amd64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_arm.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_arm64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_mips64.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_mips64le.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_ppc64.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_ppc64le.go:128:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {
zsyscall_linux_s390x.go:126:func utimensat(dirfd int, path string, times *[2]Timespec) (err error) {

Unfortunately doesn't seem like darwin doesn't support utimensat and there is a comment in the syscall code about it

However, it seems like we can do this for Solaris and perhaps the other BSDs.
@mdempsky could you help me confirm whether this is still true and if utimensat can be supported on the other BSDs?

rsc · 2016-10-18T16:23:49Z

I think the package docs for os.Chtimes are correct. For example you can use a nanosecond-specific system call but if you're writing to FAT32, it's still going to get truncated to 2-second granularity.

The expectation is that the implementation of syscall.UtimesNano uses the most precise system call available. That's true on Linux: it tries utimensat and falls back to utimes when utimensat is not available. It appears to be true on Windows too: it uses SetFileTime which works at 100ns granularity, but that's true of everything on Windows.

On BSD systems, there is a TODO in syscall_bsd.go to use a higher precision call.

On Solaris, there is no TODO but still the implementation uses plain syscall.Utimes. If Solaris has a more precise call, the Solaris implementation of syscall.UtimesNano should use it.

mdempsky · 2016-10-18T16:58:35Z

@odeke-em utimensat has been supported since OpenBSD 5.0 (Nov 2011), NetBSD 6.0 (Oct 2012), and FreeBSD 11.0 (Oct 2016). It looks like it's still unsupported on Darwin though.

odeke-em · 2016-10-29T20:31:29Z

Thanks for the clarification @mdempsky!

@bradfitz I don't know who to ask but might you access to some Solaris and BSD machines?

odeke-em · 2016-10-29T22:04:09Z

Actually ama let someone who is an expert on syscalls and BSD handle this,
instead of me holding up the freeze cut-off that is in less than 30 hours.

binarycrusader · 2016-10-29T22:49:30Z

All actively supported versions of Solaris from Solaris 10 onwards support utimensat since roughly 2010.

binarycrusader · 2016-10-29T22:52:17Z

You can see that I added support (changed UtimesNano) to use this in the sys/unix package as part of the changes for #8609:

golang/sys@68a71b6

https://golang.org/cl/14643

binarycrusader · 2016-10-29T22:57:37Z

FreeBSD actually first added support for utimensat in FreeBSD 10 as part of this review in January 2015:

https://reviews.freebsd.org/D1426

http://fxr.watson.org/fxr/source/sys/stat.h?v=FREEBSD10#L331

https://www.freebsd.org/cgi/man.cgi?query=utimensat&apropos=0&sektion=0&manpath=FreeBSD+10.3-RELEASE+and+Ports&arch=default&format=html

odeke-em · 2017-03-21T04:25:39Z

Hello there @binarycrusader, the Go1.9 tree is open. Any interest/bandwidth for getting this in? Thanks.

tklauser · 2017-08-11T14:38:06Z

The same problem also exists in x/sys/unix. I'll have a look at it and send CLs for both.

All the BSDs and Solaris support the utimensat syscall, but Darwin doesn't. Account for that by adding the //sys lines not to syscall_bsd.go but the individual OS's syscall_*.go files and implement utimensat on Darwin as just returning ENOSYS, such that UtimesNano will fall back to use utimes as it currently does unconditionally. This also adds the previously missing utimensat syscall number for FreeBSD and Dragonfly. Fixes golang#16480 Change-Id: I367454c6168eb1f7150b988fa16cf02abff42f34

tklauser · 2017-08-11T17:02:30Z

Submitted https://golang.org/cl/55130 attempting to fix this issue in the syscall package. So far, I was only able to do some rudimentary testing on FreeBSD so far and I'd appreciate if anyone running Solaris with a nanosecond-precision file system could give it a try (/cc @calmh @binarycrusader). The change is also available in tklauser/go@a17c114.

gopherbot · 2017-08-11T17:10:33Z

Change https://golang.org/cl/55130 mentions this issue: syscall: add utimensat and use it for UtimesNano on BSD and Solaris

gopherbot · 2017-08-14T08:04:02Z

Change https://golang.org/cl/55141 mentions this issue: syscall: really use utimensat for UtimesNano on Solaris

golang.org/cl/55130 added utimensat for Solaris but didn't use it in UtimesNano (despite indicating otherwise in the commit message). Fix this by also using utimensat for UtimesNano on Solaris. Because all versions of Solaris suppported by Go support utimensat, there is no need for the fallback logic and utimensat can be called unconditionally. This issue was pointed out by Shawn Walker-Salas. Updates #16480 Change-Id: I114338113a6da3cfcb8bca950674bdc8f5a7a9e5 Reviewed-on: https://go-review.googlesource.com/55141 Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>

quentinmit added the OS-Solaris label Jul 23, 2016

quentinmit added this to the Go1.8 milestone Jul 23, 2016

bradfitz removed the OS-Solaris label Jul 24, 2016

bradfitz changed the title ~~os: Chtimes doesn't retain full timestamp granularity on Solaris~~ os: Chtimes doesn't use utimens, retain full timestamp granularity Jul 24, 2016

quentinmit added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 10, 2016

rsc changed the title ~~os: Chtimes doesn't use utimens, retain full timestamp granularity~~ syscall: UtimesNano does not use nanosecond system call on BSD, Solaris Oct 18, 2016

rsc modified the milestones: Go1.9, Go1.8 Nov 11, 2016

bradfitz modified the milestones: Go1.10, Go1.9 Jun 7, 2017

bradfitz added help wanted OS-Solaris labels Jun 7, 2017

gopherbot closed this as completed in 67c3605 Aug 14, 2017

wojas mentioned this issue Nov 5, 2017

test fails on OpenBSD restic/restic#1307

Closed

golang locked and limited conversation to collaborators Aug 14, 2018

gopherbot added the FrozenDueToAge label Aug 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

calmh commented Jul 23, 2016

bradfitz commented Jul 24, 2016

odeke-em commented Sep 27, 2016

rsc commented Oct 18, 2016

mdempsky commented Oct 18, 2016

odeke-em commented Oct 29, 2016

odeke-em commented Oct 29, 2016

binarycrusader commented Oct 29, 2016

binarycrusader commented Oct 29, 2016 •

edited

binarycrusader commented Oct 29, 2016

odeke-em commented Mar 21, 2017

tklauser commented Aug 11, 2017

tklauser commented Aug 11, 2017

gopherbot commented Aug 11, 2017

gopherbot commented Aug 14, 2017

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480

Comments

calmh commented Jul 23, 2016

bradfitz commented Jul 24, 2016

odeke-em commented Sep 27, 2016

rsc commented Oct 18, 2016

mdempsky commented Oct 18, 2016

odeke-em commented Oct 29, 2016

odeke-em commented Oct 29, 2016

binarycrusader commented Oct 29, 2016

binarycrusader commented Oct 29, 2016 • edited

binarycrusader commented Oct 29, 2016

odeke-em commented Mar 21, 2017

tklauser commented Aug 11, 2017

tklauser commented Aug 11, 2017

gopherbot commented Aug 11, 2017

gopherbot commented Aug 14, 2017

binarycrusader commented Oct 29, 2016 •

edited