os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always #42184

kfsone · 2020-10-24T07:43:58Z

What version of Go are you using (`go version`)?

$ go version
go version go1.15.2 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\oliver\AppData\Local\go-build
set GOENV=C:\Users\oliver\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\oliver\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\oliver\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\oliver\AppData\Local\Temp\go-build844512546=/tmp/go-build -gno-record-gcc-switches

What did you do?

Called os.FileInfo.Mode.IsRegular and got the wrong result.

This is caused by a WSL symlink:

[D:\Dev\github.com\kfsone\archive\smugl-src]
> gci compile | format-list

    Directory: D:\Dev\github.com\kfsone\archive\smugl-src

Name           : compile
Length         : 0
CreationTime   : 7/28/2019 10:36:58 PM
LastWriteTime  : 7/28/2019 10:36:58 PM
LastAccessTime : 7/28/2019 10:36:58 PM
Mode           : la---
LinkType       :
Target         :
VersionInfo    : File:             D:\Dev\github.com\kfsone\archive\smugl-src\compile
                 InternalName:
                 OriginalFilename:
                 FileVersion:
                 FileDescription:
                 Product:
                 ProductVersion:
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:

Running the following go code:

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    path := "D:/dev/github.com/kfsone/archive/smugl-src/compile"
    filepath.Walk(path, func (path string, info os.FileInfo, err error) error {
        fmt.Printf("%s %s %#+v\n", path, info.Mode().IsRegular(), info)
        return nil
    })
}

produces:

D:/dev/github.com/kfsone/archive/smugl-src/compile %!s(bool=true) &os.fileStat{name:"compile", FileAttributes:0x420, CreationTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, LastAccessTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, LastWriteTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, FileSizeHigh:0x0, FileSizeLow:0x0, Reserved0:0xa000001d, filetype:0x0, Mutex:sync.Mutex{state:0, sema:0x0}, path:"", vol:0x5ef35831, idxhi:0xb0000, idxlo:0x3b47, appendNameToPath:false}

Windows is definitely telling us it's a reparse point: FileAttributes: 0x420 includes the 1024 required. The issue here is the value of Reserved0, which is not being tested for:

According to https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4


0xA000001D
Used by the Windows Subsystem for Linux (WSL) to represent a UNIX symbolic link. Server-side interpretation only, not meaningful over the wire.

This appears it would require a change to src/os/types_windows.go lines 104,105.

The text was updated successfully, but these errors were encountered:

odeke-em · 2020-10-25T20:51:01Z

Thank you for filing this issue @kfsone and welcome to the Go project!

Interesting bug, and thank you for the background. Please feel free to send a change to fix this as per https://golang.org/doc/contribute.html or even just a PR on Github after signing the CLA, it would be
awesome to have your contributions!

Just for full disclosure we are about 6 days away from a code freeze for Go1.16, thus if you can, please send a change,
with a regression test, and I shall also tag some Windows experts @alexbrainman @mattn. No rush though, because even
if not meeting the deadline, in about 2 months, if/when the change is accepted, we can always backport it.

mattn · 2020-10-26T01:13:38Z

CreateFile fail to open symlink created on WSL2 without FILE_FLAG_OPEN_REPARSE_POINT. To fix this, we should add FILE_FLAG_OPEN_REPARSE_POINT to ~~second~~ third argument of stat.

diff --git a/src/os/stat_windows.go b/src/os/stat_windows.go
index da4c49090e..43f176376c 100644
--- a/src/os/stat_windows.go
+++ b/src/os/stat_windows.go
@@ -103,7 +103,7 @@ func stat(funcname, name string, createFileAttrs uint32) (FileInfo, error) {
 
 // statNolog implements Stat for Windows.
 func statNolog(name string) (FileInfo, error) {
-	return stat("Stat", name, syscall.FILE_FLAG_BACKUP_SEMANTICS)
+	return stat("Stat", name, syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS)
 }
 
 // lstatNolog implements Lstat for Windows.

mattn · 2020-10-26T01:22:47Z

Hmm, TestDirectoryJunction fail if adding FILE_FLAG_OPEN_REPARSE_POINT. So it should do fallback if CreateFile fail.

gopherbot · 2020-10-26T01:31:30Z

Change https://golang.org/cl/265037 mentions this issue: os: fallback to open symlinks

kfsone · 2020-10-28T19:15:54Z

CreateFile fail to open symlink created on WSL2 without FILE_FLAG_OPEN_REPARSE_POINT. To fix this, we should add FILE_FLAG_OPEN_REPARSE_POINT to ~~second~~ third argument of stat.

diff --git a/src/os/stat_windows.go b/src/os/stat_windows.go
index da4c49090e..43f176376c 100644
--- a/src/os/stat_windows.go
+++ b/src/os/stat_windows.go
@@ -103,7 +103,7 @@ func stat(funcname, name string, createFileAttrs uint32) (FileInfo, error) {
 
 // statNolog implements Stat for Windows.
 func statNolog(name string) (FileInfo, error) {
-	return stat("Stat", name, syscall.FILE_FLAG_BACKUP_SEMANTICS)
+	return stat("Stat", name, syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS)
 }
 
 // lstatNolog implements Lstat for Windows.

Note the distinction between IO_REPARSE_TAG_SYMLINK and io_reparse_tag_LX_symlink in IO_REPARSE_TAG_LX_SYMLINK (lx for 'linux' maybe)

kfsone · 2020-10-28T20:03:57Z

Notes from looking at implementing handling for this:

Microsoft themselves have not fully supported this yet:

> cd $env:TEMP ; rm -recurs -force symlinks ; cd (mkdir symlinks)
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> wsl ln -s target.txt relative.lnk
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> echo this is target >target.txt
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> get-content target.txt
this
is
target
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> get-content relative.lnk
Get-Content: The file cannot be accessed by the system. : 'C:\Users\oliver\AppData\Local\Temp\symlinks\relative.lnk'
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> wsl cat relative.lnk
this
is
target
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> fsutil reparsePoint query relative.lnk
Reparse Tag Value : 0xa000001d
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0xe
Reparse Data:
0000:  02 00 00 00 74 61 72 67  65 74 2e 74 78 74        ....target.txt

It also appears this does not use the same type of reparse buffer that non-lx reparses do:
reactos handling: https://doxygen.reactos.org/d9/d78/reparse_8c_source.html
cygwin: https://www.mail-archive.com/cygwin@cygwin.com/msg164048.html, https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=44da5e4b8c89bfcee79eb83fd3b7142720489940 (I'm puzzled by the fact that the buffer seems to be char-based instead of mbcs or utf16).

Perhaps for now, rather than appearing as a symlink, these entities should at least appear as not being regular files?

kfsone · 2020-10-28T21:32:46Z

@odeke-em Taking a look; previously signed the CLA for grpc in 2014, setup a password. The windows tests appear to be failing tho as a result of #40604 but I also experience a seemingly obvious failure of TestLookPath:

    lp_windows_test.go:147: test={rootDir:C:\Users\oliver\AppData\Local\Temp\TestLookPath336921111\d16 PATH:p1;p2 PATHEXT:.COM;.EXE;.BAT files:[p1\a.bat p1\a.exe p2\a.bat p2\a.exe] searchFor:a fails:false} failed: expected to find "p1\\a.bat", but found "p1\\a.exe"```

I'll ticket that separately, but the error looks backwards.

We were previously using an incorrect (or at least not completely correct) symbolic link detection mechanism that differed from that of the Go standard library. On Windows, where we use the os package to drive some of our filesystem implementation, this led to spurious symbolic link reports. This commit unifies Mutagen's symbolic link definition with that of the Go standard library. Note that the Go standard library still has an open issue regarding symbolic link classification with WSL symbolic links (golang/go#42184), so we inherit that for now, but we should be able to adjust our definition once that issue is resolved. Fixes #248. Signed-off-by: Jacob Howard <jacob@havoc.io>

gopherbot · 2023-08-08T09:29:07Z

Change https://go.dev/cl/516555 mentions this issue: os: Stat/LStat to treat all name surrogates as symlinks on Windows

qmuntal · 2023-08-08T11:05:21Z

I don't think we should special-case WSL symlinks (aka IO_REPARSE_TAG_LX_SYMLINK). I haven't seen any important framework doing so, even WSL is trying to limit their occurrence by creating normal symlinks whenever possible to improve compatibility (see WSL 17406 release notes).

Also, WSL symlinks can't be opened from Windows, only from within WSL, so if we identify them as fs.ModeSymlink, then functions like filepath.Walk would fail when trying to follow the link.

@bcmills

Previously, os.Stat only followed IO_REPARSE_TAG_SYMLINK and IO_REPARSE_TAG_MOUNT_POINT reparse points. This CL generalize the logic to detect which reparse points to follow by using the reparse tag value to determine whether the reparse point refers to another named entity, as documented in https://learn.microsoft.com/en-us/windows/win32/fileio/reparse-point-tags. The new behavior adds implicit support for correctly stat-ing reparse points other than mount points and symlinks, e.g., IO_REPARSE_TAG_WCI_LINK and IO_REPARSE_TAG_IIS_CACHE. Updates #42184 Change-Id: I51f56127d4dc6c0f43eb5dfa3bfa6d9e3922d000 Reviewed-on: https://go-review.googlesource.com/c/go/+/516555 Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Quim Muntal <quimmuntal@gmail.com>

qmuntal · 2024-03-07T08:58:29Z

This issue should be fixed since go1.21 thanks to CL 516555 and CL 460595.

odeke-em changed the title ~~os FileInfo Mode on Windows incorrectly interpreting WSL symlinks.~~ os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always Oct 25, 2020

cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 26, 2020

cagedmantis added this to the Backlog milestone Oct 26, 2020

deajan mentioned this issue Jan 17, 2023

Windows - Reparse points / cloud files creating a lot of errors in logs restic/restic#4155

Open

yacoob mentioned this issue Jun 12, 2023

restic on Windows cannot backup certain symlinks restic/restic#3614

Open

qmuntal added the OS-Windows label Aug 8, 2023

qmuntal added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 8, 2023

bcmills mentioned this issue Oct 12, 2023

os: NTFS deduped file changed from regular to irregular #63429

Closed

patrickhuber mentioned this issue Jan 17, 2024

Not working on Windows using Powershell direnv/direnv#1214

Open

qmuntal closed this as completed Mar 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always #42184

os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always #42184

kfsone commented Oct 24, 2020

odeke-em commented Oct 25, 2020

mattn commented Oct 26, 2020 •

edited

mattn commented Oct 26, 2020

gopherbot commented Oct 26, 2020

kfsone commented Oct 28, 2020

kfsone commented Oct 28, 2020 •

edited

kfsone commented Oct 28, 2020

gopherbot commented Aug 8, 2023

qmuntal commented Aug 8, 2023

qmuntal commented Mar 7, 2024

os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always #42184

os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always #42184

Comments

kfsone commented Oct 24, 2020

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

odeke-em commented Oct 25, 2020

mattn commented Oct 26, 2020 • edited

mattn commented Oct 26, 2020

gopherbot commented Oct 26, 2020

kfsone commented Oct 28, 2020

kfsone commented Oct 28, 2020 • edited

kfsone commented Oct 28, 2020

gopherbot commented Aug 8, 2023

qmuntal commented Aug 8, 2023

qmuntal commented Mar 7, 2024

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

mattn commented Oct 26, 2020 •

edited

kfsone commented Oct 28, 2020 •

edited