path/filepath: Walk walks recursive NTFS junction points #10424

ClusterCat · 2015-04-11T19:21:03Z

go Version 1.4.2
Windows 7, AMD64

When using filepath.Walk on an NTFS Volume it follows NTFS junction points.
If there are junction points which point to a parent directory, this leads to recursion and following errors.

Example:
Windows 7 creates the junction points:
C:\Users\username\Local Settings -> C:\Users\username\AppData\local
C:\Users\username\Local Settings\Application Data -> C:\Users\username\AppData\local

which leads to errors like:
GetFileAttributesEx C:\Users\username\Local Settings\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Temp\go-build168935239: The system cannot find the file specified.

I expected Walk to not follow junction points, in accordance to the behavior of not following symlinks.

ianlancetaylor · 2015-04-11T19:35:32Z

CC @alexbrainman

alexbrainman · 2015-04-13T04:27:18Z

@ClusterCat do you have a small program that I can run to see your problem? Thank you.

Alex

ClusterCat · 2015-04-13T18:28:09Z

Sure @alexbrainman , a minimal example would be

package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\USERNAME\\AppData\\Local", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}

Just replace USERNAME and run on Win7 (don't know about the directory structure of later versions).

Thanks,
CC

alexbrainman · 2015-04-14T05:57:12Z

@ClusterCat thank you for the program. But you didn't not explain what is wrong with your program. What does your program output look like? What do you expect to see instead? Why?

Thank you.

Alex

ClusterCat · 2015-04-14T20:23:52Z

Ok, I'll try to explain more clearly...

The starting path (C:\Users\username\AppData\Local) contains the ntfs junction "Application Data" which points to the starting path again (loop/recursion).

When the program gets to the ntfs junction it walks into it and begins looping/recursion. It then creates about one path error for every file in the starting path, which is about 14000 path errors.

Example output:

GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\BrokenLink.html: The system can not find the path specified.
GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\Browse.html: The system can not find the path specified.
GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\Classes: The system can not find the path specified.

I expected Walk to skip the ntfs junction ("Application Data") and go on with the next real directory, avoiding many path errors because it does so for symbolic links.
Accoording to #4759 (comment) Walk avoids following symbolic links by design to avoid looping. It would be consistent if Walk would avoid following ntfs junctions for the same reasons.

mattn · 2015-04-15T03:48:25Z

for about the argument FileInfo, user can handle FileMode to check symlink(on windows, it's reparse point).

alexbrainman · 2015-04-15T05:34:04Z

@ClusterCat I cannot reproduce the output you see:

C:\go\path\src\issue10424>type main.go
package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\brainman\\AppData\\Local", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}
C:\go\path\src\issue10424>go version
go version go1.4.2 windows/amd64

C:\go\path\src\issue10424>go run main.go
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\Application Data: Access is denied.
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\History: Access is denied.
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\Temporary Internet Files: Access is denied.

C:\go\path\src\issue10424>

Am I missing something?

I don't see problem with my output. Do you see problem with my output?

Alex

ClusterCat · 2015-04-15T20:29:27Z

@alexbrainman your output looks fine. I don't see any problem with it either.
I have one last idea, let's test a self-made loop:

C:\Users\username>mkdir test
C:\Users\username>cd test
C:\Users\username>mklink /j test2 C:\Users\username\test

and then point the program to C:\\Users\\username\\test

My output looks like

open C:\Users\username\test\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2: Acess is denied

If your version stops at the first test2 then it seems to be some weird quirk on my machine and not reproducible.

alexbrainman · 2015-04-16T06:26:00Z

My output looks like:

C:\go\path\src\issue10424>type main.go
package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\brainman\\test", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}
C:\go\path\src\issue10424>go version
go version go1.4.2 windows/amd64

C:\go\path\src\issue10424>go run main.go
2015/04/16 16:14:07 open C:\Users\brainman\test\test2: Access is denied.

C:\go\path\src\issue10424>

Are you sure you're using go1.4.2? Windows symlink support has changed significantly between 1.3 and 1.4. What does your "go version" command output?

... it seems to be some weird quirk on my machine and not reproducible.

I don't believe in magic, there is always a reason. You should be able to work out what is going on. I would add new tmp test to path/filepath package that does something similar to your program above. Then add println or fmt.Printf statements in correspondent library code here and there (while running "go test ..." command) to see why your output is different from mine. You would obviously need Go source code for that.

Alex

ClusterCat · 2015-04-23T19:04:04Z

Output of go version is: go version go1.4.2 windows/amd64

Ok, I will get the source code and will try to debug it like you said. This might take some time due to my time constraints atm.

Just for the sake of completeness this are the ACLs for the test2junction on my system. I assume yours are corresponding.

C:\Users\clustercat\test>icacls test2
test2 NT-AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
      tetrahedron\clustercat:(I)(OI)(CI)(F)
      PREDEFINED\Administrators:(I)(OI)(CI)(F)

rsc · 2015-06-29T15:34:10Z

Too late for Go 1.5.

alexbrainman · 2017-02-12T07:31:02Z

@ClusterCat I think I can see what the problem is. Please see https://go-review.googlesource.com/36851
for new test.

Alex

gopherbot · 2017-02-12T07:35:37Z

CL https://golang.org/cl/36851 mentions this issue.

For #10424. Change-Id: Ie4e87503b0ed04f65d2444652bd1db647d3529f4 Reviewed-on: https://go-review.googlesource.com/36851 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

gopherbot · 2017-04-26T06:38:01Z

CL https://golang.org/cl/41830 mentions this issue.

minux changed the title ~~filepath.Walk walks recursive NTFS junction points~~ path/filepath: Walk walks recursive NTFS junction points Apr 12, 2015

minux added the repo-main label Apr 12, 2015

minux added this to the Go1.5Maybe milestone Apr 12, 2015

rsc removed the repo-main label Apr 14, 2015

rsc modified the milestones: Unplanned, Go1.5Maybe Jun 29, 2015

jim-minter mentioned this issue Oct 21, 2016

path/filepath: Walk recurses on directory symlinks on Windows #17540

Closed

alexbrainman mentioned this issue Apr 11, 2017

os: Stat on symlink fails on windows #19922

Closed

gopherbot closed this as completed in 1989921 Apr 26, 2017

golang locked and limited conversation to collaborators Apr 26, 2018

gopherbot added the FrozenDueToAge label Apr 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

path/filepath: Walk walks recursive NTFS junction points #10424

path/filepath: Walk walks recursive NTFS junction points #10424

ClusterCat commented Apr 11, 2015

ianlancetaylor commented Apr 11, 2015

alexbrainman commented Apr 13, 2015

ClusterCat commented Apr 13, 2015

alexbrainman commented Apr 14, 2015

ClusterCat commented Apr 14, 2015

mattn commented Apr 15, 2015

alexbrainman commented Apr 15, 2015

ClusterCat commented Apr 15, 2015

alexbrainman commented Apr 16, 2015

ClusterCat commented Apr 23, 2015

rsc commented Jun 29, 2015

alexbrainman commented Feb 12, 2017

gopherbot commented Feb 12, 2017

gopherbot commented Apr 26, 2017

path/filepath: Walk walks recursive NTFS junction points #10424

path/filepath: Walk walks recursive NTFS junction points #10424

Comments

ClusterCat commented Apr 11, 2015

ianlancetaylor commented Apr 11, 2015

alexbrainman commented Apr 13, 2015

ClusterCat commented Apr 13, 2015

alexbrainman commented Apr 14, 2015

ClusterCat commented Apr 14, 2015

mattn commented Apr 15, 2015

alexbrainman commented Apr 15, 2015

ClusterCat commented Apr 15, 2015

alexbrainman commented Apr 16, 2015

ClusterCat commented Apr 23, 2015

rsc commented Jun 29, 2015

alexbrainman commented Feb 12, 2017

gopherbot commented Feb 12, 2017

gopherbot commented Apr 26, 2017