path/filepath: Walk walks recursive NTFS junction points

[ClusterCat]

go Version 1.4.2
Windows 7, AMD64

When using filepath.Walk on an NTFS Volume it follows NTFS junction points.
If there are junction points which point to a parent directory, this leads to recursion and following errors.

Example:
Windows 7 creates the junction points:
C:\Users\username\Local Settings -> C:\Users\username\AppData\local
C:\Users\username\Local Settings\Application Data -> C:\Users\username\AppData\local

which leads to errors like:
GetFileAttributesEx C:\Users\username\Local Settings\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Temp\go-build168935239: The system cannot find the file specified.

I expected Walk to not follow junction points, in accordance to the behavior of not following symlinks.

[ianlancetaylor]

CC @alexbrainman

[alexbrainman]

@ClusterCat do you have a small program that I can run to see your problem? Thank you.

Alex

[ClusterCat]

Sure @alexbrainman , a minimal example would be

package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\USERNAME\\AppData\\Local", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}

Just replace USERNAME and run on Win7 (don't know about the directory structure of later versions).

Thanks,
CC

[alexbrainman]

@ClusterCat thank you for the program. But you didn't not explain what is wrong with your program. What does your program output look like? What do you expect to see instead? Why?

Thank you.

Alex

[ClusterCat]

Ok, I'll try to explain more clearly...

The starting path (C:\Users\username\AppData\Local) contains the ntfs junction "Application Data" which points to the starting path again (loop/recursion).

When the program gets to the ntfs junction it walks into it and begins looping/recursion. It then creates about one path error for every file in the starting path, which is about 14000 path errors.

Example output:

GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\BrokenLink.html: The system can not find the path specified.
GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\Browse.html: The system can not find the path specified.
GetFileAttributesEx C:\Users\userName\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\SuperCollider\Help\Classes: The system can not find the path specified.

I expected Walk to skip the ntfs junction ("Application Data") and go on with the next real directory, avoiding many path errors because it does so for symbolic links.
Accoording to #4759 (comment) Walk avoids following symbolic links by design to avoid looping. It would be consistent if Walk would avoid following ntfs junctions for the same reasons.

[mattn]

for about the argument FileInfo, user can handle FileMode to check symlink(on windows, it's reparse point).

[alexbrainman]

@ClusterCat I cannot reproduce the output you see:

C:\go\path\src\issue10424>type main.go
package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\brainman\\AppData\\Local", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}
C:\go\path\src\issue10424>go version
go version go1.4.2 windows/amd64

C:\go\path\src\issue10424>go run main.go
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\Application Data: Access is denied.
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\History: Access is denied.
2015/04/15 15:29:40 open C:\Users\brainman\AppData\Local\Temporary Internet Files: Access is denied.

C:\go\path\src\issue10424>

Am I missing something?

I don't see problem with my output. Do you see problem with my output?

Alex

[ClusterCat]

@alexbrainman your output looks fine. I don't see any problem with it either.
I have one last idea, let's test a self-made loop:

C:\Users\username>mkdir test
C:\Users\username>cd test
C:\Users\username>mklink /j test2 C:\Users\username\test

and then point the program to C:\\Users\\username\\test

My output looks like

open C:\Users\username\test\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2\test2: Acess is denied

If your version stops at the first test2 then it seems to be some weird quirk on my machine and not reproducible.

[alexbrainman]

My output looks like:

C:\go\path\src\issue10424>type main.go
package main

import (
    "log"
    "os"
    "path/filepath"
)

func main() {
    err := filepath.Walk("C:\\Users\\brainman\\test", walkFunc)
    if err != nil {
        log.Fatalln(err)
    }
}

func walkFunc(path string, info os.FileInfo, err error) error {
    if err != nil {
        log.Println(err)
    }

    return nil
}
C:\go\path\src\issue10424>go version
go version go1.4.2 windows/amd64

C:\go\path\src\issue10424>go run main.go
2015/04/16 16:14:07 open C:\Users\brainman\test\test2: Access is denied.

C:\go\path\src\issue10424>

Are you sure you're using go1.4.2? Windows symlink support has changed significantly between 1.3 and 1.4. What does your "go version" command output?

... it seems to be some weird quirk on my machine and not reproducible.

I don't believe in magic, there is always a reason. You should be able to work out what is going on. I would add new tmp test to path/filepath package that does something similar to your program above. Then add println or fmt.Printf statements in correspondent library code here and there (while running "go test ..." command) to see why your output is different from mine. You would obviously need Go source code for that.

Alex

[ClusterCat]

Output of go version is: go version go1.4.2 windows/amd64

Ok, I will get the source code and will try to debug it like you said. This might take some time due to my time constraints atm.

Just for the sake of completeness this are the ACLs for the test2junction on my system. I assume yours are corresponding.

C:\Users\clustercat\test>icacls test2
test2 NT-AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
      tetrahedron\clustercat:(I)(OI)(CI)(F)
      PREDEFINED\Administrators:(I)(OI)(CI)(F)

[rsc]

Too late for Go 1.5.