os/exec: calling Command on file with UWP reparse point fails with "file does not exist" #42919

viveklak · 2020-12-01T05:49:23Z

Windows 10 ships with some 0 length files in C:\Users\<User>\AppData\Local\Microsoft\WindowsApps with special reparse points which when run trigger the windows app store app to be launched.

Once installed, the same files result in the installed app to be run going forward. For instance, Windows 10 ships with python3.exe which is in fact a link to python3.8 available on the Windows App Store. C:\Users\<User>\AppData\Local\Microsoft\WindowsApps happens to be on the PATH by default so once installed, the python3 interpreter should be invoked. However, when trying to run the same through Go's exec.Cmd we get a file does not exist error (see below for simple repro).

The main issue seems to be a call to os.Stat from the exec.Cmd run path. os.Stat on windows doesn't pass syscall.FILE_FLAG_OPEN_REPARSE_POINT resulting in the error (unlike Lstat).

What version of Go are you using (`go version`)?

$ go version
1.15.5

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

Windows 10

go env Output

$ go env
set GO111MODULE=

set GOARCH=amd64

set GOBIN=

set GOCACHE=C:\Users\User\AppData\Local\go-build

set GOENV=C:\Users\User\AppData\Roaming\go\env

set GOEXE=.exe

set GOFLAGS=

set GOHOSTARCH=amd64

set GOHOSTOS=windows

set GOINSECURE=

set GOMODCACHE=C:\Users\User\go\pkg\mod

set GONOPROXY=

set GONOSUMDB=

set GOOS=windows

set GOPATH=C:\Users\User\go

set GOPRIVATE=

set GOPROXY=https://proxy.golang.org,direct

set GOROOT=c:\go

set GOSUMDB=sum.golang.org

set GOTMPDIR=

set GOTOOLDIR=c:\go\pkg\tool\windows_amd64

set GCCGO=gccgo

set AR=ar

set CC=gcc

set CXX=g++

set CGO_ENABLED=1

set GOMOD=

set CGO_CFLAGS=-g -O2

set CGO_CPPFLAGS=

set CGO_CXXFLAGS=-g -O2

set CGO_FFLAGS=-g -O2

set CGO_LDFLAGS=-g -O2

set PKG_CONFIG=pkg-config

set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\User\AppData\Local\Temp\go-build056150306=/tmp/go-build -gno-record-gcc-switches

What did you do?

$ cat main.go
package main

import (
	"fmt"
	"os"
	"os/exec"
)

func do() error {
	res, err := os.Lstat("C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe")
	if err != nil {
		return err
	}
	fmt.Println("python3 exists!")
	fmt.Printf("%#v\n", res)
	cmd := exec.Command("C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe", "-c", "print('hello')")
	btes, err := cmd.CombinedOutput()
	if err != nil {
		return err
	}
	fmt.Printf("%v\n", btes)
	return nil
}

func main() {
	err := do()
	if err != nil {
		panic(err)
	}
}

PS C:\Users\User> go run main.go

What did you expect to see?

python3 exists!
&os.fileStat{name:"python3.exe", FileAttributes:0x420, CreationTime:syscall.Filetime{LowDateTime:0x5094a4bc, HighDateTime:0x1d6a0b0}, LastAccessTime:syscall.Filetime{LowDateTime:0xb3aada47, HighDateTime:0x1d6be0b}, LastWriteTime:syscall.Filetime{LowDateTime:0xb3aada47, HighDateTime:0x1d6be0b}, FileSizeHigh:0x0, FileSizeLow:0x0, Reserved0:0x8000001b, filetype:0x0, Mutex:sync.Mutex{state:0, sema:0x0}, path:"", vol:0x16999dcb, idxhi:0x30000, idxlo:0x14bc9, appendNameToPath:false}
hello

What did you see instead?

python3 exists!
&os.fileStat{name:"python3.exe", FileAttributes:0x420, CreationTime:syscall.Filetime{LowDateTime:0x5094a4bc, HighDateTime:0x1d6a0b0}, LastAccessTime:syscall.Filetime{LowDateTime:0xb3aada47, HighDateTime:0x1d6be0b}, LastWriteTime:syscall.Filetime{LowDateTime:0xb3aada47, HighDateTime:0x1d6be0b}, FileSizeHigh:0x0, FileSizeLow:0x0, Reserved0:0x8000001b, filetype:0x0, Mutex:sync.Mutex{state:0, sema:0x0}, path:"", vol:0x16999dcb, idxhi:0x30000, idxlo:0x14bc9, appendNameToPath:false}
panic: exec: "C:\\Users\\User\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe": file does not exist

goroutine 1 [running]:
main.main()
        C:/Users/User/main.go:28 +0x4b
exit status 2

The text was updated successfully, but these errors were encountered:

networkimprov · 2020-12-05T19:46:27Z

cc @alexbrainman @mattn

@gopherbot add OS-Windows

cagedmantis · 2020-12-08T01:48:07Z

/cc /cc @bradfitz @ianlancetaylor @alexbrainman

mattn · 2020-12-08T04:10:41Z

It caused by that findExecutable uses os.Stat instead of os.Lstat. os.Lstat resolve reparse-points.

diff --git a/src/os/exec/lp_windows.go b/src/os/exec/lp_windows.go
index 9ea3d76575..12c7e04e63 100644
--- a/src/os/exec/lp_windows.go
+++ b/src/os/exec/lp_windows.go
@@ -15,7 +15,7 @@ import (
 var ErrNotFound = errors.New("executable file not found in %PATH%")
 
 func chkStat(file string) error {
-	d, err := os.Stat(file)
+	d, err := os.Lstat(file)
 	if err != nil {
 		return err
 	}

This fixes the issue the program can not execute python.exe, but the python.exe exit with 9009 exit code. Probably, it is another issue.

viveklak · 2020-12-08T04:42:24Z

This fixes the issue the program can not execute python.exe, but the python.exe exit with 9009 exit code. Probably, it is another issue.

Yes - I alluded to that in the original report but should have called it out better:

The main issue seems to be a call to os.Stat from the exec.Cmd run path. os.Stat on windows doesn't pass syscall.FILE_FLAG_OPEN_REPARSE_POINT resulting in the error (unlike Lstat).

A naive thought here would be to consider passing syscall.FILE_FLAG_OPEN_REPARSE_POINT to CreateFile from os.Stat similar to what os.Lstat does but I am not familiar enough with the semantics of Windows to judge the implications of that change.

As such, I see the following issues:

Running os.Stat on reparse point files returns an unexpected/misleading error
os.Exec is not able to exec a reparse point file due to the above.

sachinjoseph · 2022-04-14T20:09:03Z

I ran into this the other day; worked around it using os.StartProcess() (more details here). Golang team, it'd be nice to have this bug fixed as this effectively makes the os.Exec() API unreliable and hence unusable on Windows.

alexbrainman · 2022-04-18T07:33:56Z

@sachinjoseph

Does

https://go-review.googlesource.com/c/go/+/384160

fixes this bug?

If yes, please, help review the CL 384160.

Alternatively send your own fix (if you know how to fix it)

https://go.dev/doc/contribute

Thank you.

Alex

IO_REPARSE_TAG_APPEXECLINK are types of links that are created when installing an application via the Windows App Store. The location of these links is added to the Windows PATH and the system can resolve them. This adds support for these links so go can resolve them. Fixes golang#42919

sachinjoseph · 2022-04-19T18:22:07Z

@alexbrainman @JanDeDobbeleer The PR modifies os.Exec() to specially handle files that contain the substring /AppData/Local/Microsoft/WindowsApps in their paths. I don't think specially handling files that have a certain substring in its path is the right approach.

Changing os.Stat() to os.Lstat() in chkStat() (called by findExecutable() here) works for executing regular files (say notepad.exe) and files with reparse points (say winget.exe) - I verified this on my machine and also in a VM. Now I am not an expert in this area, and so I am not sure of any potential implications of this change though. But maybe the actual fix needs to go a level deeper into os.Stat() itself like in @JanDeDobbeleer's PR (but without the special handling for certain paths) because today on Windows, os.Stat() fails on winget.exe but succeeds on notepad.exe.

JanDeDobbeleer · 2022-04-19T18:25:30Z

@sachinjoseph I agree. I'm looking for a new way to handle this in the correct spot but truth be told, it's a bit of a maze 😃. The problem is that in go lookpath can't resolve this. So it needs to be fixed somewhere over there ideally.

IO_REPARSE_TAG_APPEXECLINK are types of links that are created when installing an application via the Windows App Store. The location of these links is added to the Windows PATH and the system can resolve them. This adds support for these links so go can resolve them. Fixes golang#42919

JanDeDobbeleer · 2022-04-19T19:31:30Z

@sachinjoseph decided to add an explicit call to Readlink rather than changing existing code around too much. Same result, no explicit locations mentioned, this will also support any other symbolic link.

sachinjoseph · 2022-04-19T19:42:54Z

@JanDeDobbeleer Great! How do I test this? Can I simply clone your fork/branch and build it? Do I need to enable CGO?

Were you able to launch winget.exe using os.Exec() with your changes?

JanDeDobbeleer · 2022-04-19T20:44:04Z

@sachinjoseph yes, you can simply take that branch and build it, doesn't need cgo. I didn't validate winget but theoretically that should just work.

sachinjoseph · 2022-04-19T20:45:43Z

@JanDeDobbeleer Let me try that

sachinjoseph · 2022-04-19T23:15:52Z

@JanDeDobbeleer ~~I still see issues, although I don't see an error message running exec.Cmd.Output() anymore.~~

exec.Output() works, but os.Stat() fails - is that how it is supposed to be?

Test.go

package main

import (
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
)

func main() {
	path := filepath.Join(os.Getenv("LOCALAPPDATA"), "Microsoft\\WindowsApps\\winget.exe")
	// path = "C:\\Program Files\\Git\\cmd\\git.exe"

	cmd := exec.Command(path, "--version")
	output, err := cmd.Output()
	fmt.Println(string(output))
	fmt.Println(err)

	fmt.Println("\nStat-ing the file")

	f, err := os.Stat(path)
	fmt.Println(f)
	fmt.Println(err)
}

Go 1.18

pwsh> go version        
go version go1.18 windows/amd64

pwsh> go run test.go    

exec: "C:\\Users\\username\\AppData\\Local\\Microsoft\\WindowsApps\\winget.exe": file does not exist

Stat-ing the file
<nil>
CreateFile C:\Users\username\AppData\Local\Microsoft\WindowsApps\winget.exe: The file cannot be accessed by the system.

pwsh>

@JanDeDobbeleer's branch

pwsh> .\go.exe version
go version devel go1.19-aaf7044240 Tue Apr 19 21:19:47 2022 +0200 windows/amd64

pwsh> .\go.exe run test.go
v1.3.692-preview
<nil>

Stat-ing the file
<nil>
CreateFile C:\Users\username\AppData\Local\Microsoft\WindowsApps\winget.exe: The file cannot be accessed by the system.

pwsh>

This change adds support for AppExecLinks files, like C:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.exe The python3.exe can be installed by following these https://www.microsoft.com/store/productId/9PJPW5LDXLZ5 instructions. The executable is added to your PATH and can be called from command line, like `python3 --version`. Calling GetFileAttributesEx on python3.exe returns FileAttributes with FILE_ATTRIBUTE_REPARSE_POINT set. And os.Stat attempts to follow the link. But Microsoft does not provide any link target for AppExecLinks files (see dotnet/runtime#58233 (comment) ), and Go should treat AppExecLinks as normal files and not symlinks. This CL adjusts os.Stat implementation to return normal file os.FileInfo for AppExecLinks files instead of symlinks. The AppExecLinks files are recognised as they return ERROR_CANT_ACCESS_FILE from CreateFile call. The trick is not documented anywhere. Jan De Dobbeleer discovered the trick. Also dotnet/runtime#58233 appears to also use ERROR_CANT_ACCESS_FILE to distinguish AppExecLinks files. The CL also adds new tests. The CL is an extended copy of the Jan De Dobbeleer https://go-review.googlesource.com/c/go/+/384160 CL. Fixes golang#42919 Change-Id: I8b5a26d0cac7882d3445393d26b182ad31cd753b

alexbrainman · 2022-11-25T21:13:38Z

@qmuntal can you, please, see if

https://go-review.googlesource.com/c/go/+/452375

looks reasonable to you?

Rob approved my CL, but I would not trust myself with that change. The CL uses some undocumented logic that @JanDeDobbeleer discovered. It appears to work on my Windows 10.

This is not urgent since we have to wait for Go source tree to open for Go 1.21 development anyway.

Thank you.

Alex

alexbrainman · 2022-11-25T21:34:59Z

@qmuntal re #42919 (comment)

Please, review @JanDeDobbeleer https://go-review.googlesource.com/c/go/+/384160 instead.

He copied all changes from my https://go-review.googlesource.com/c/go/+/452375 and I abandoned my CL.

Thank you.

Alex

qmuntal · 2022-11-27T18:20:34Z

Sure!

This change adds support for AppExecLinks files, like C:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.exe The python3.exe can be installed by following these https://www.microsoft.com/store/productId/9PJPW5LDXLZ5 instructions. The executable is added to your PATH and can be called from command line, like `python3 --version`. Calling GetFileAttributesEx on python3.exe returns FileAttributes with FILE_ATTRIBUTE_REPARSE_POINT set. And os.Stat attempts to follow the link. But Microsoft does not provide any link target for AppExecLinks files (see dotnet/runtime#58233 (comment) ), and Go should treat AppExecLinks as normal files and not symlinks. This CL adjusts os.Stat implementation to return normal file os.FileInfo for AppExecLinks files instead of symlinks. The AppExecLinks files are recognised as they return ERROR_CANT_ACCESS_FILE from CreateFile call. The trick is not documented anywhere. Jan De Dobbeleer discovered the trick. Also dotnet/runtime#58233 appears to also use ERROR_CANT_ACCESS_FILE to distinguish AppExecLinks files. The CL also adds new tests. The CL is an extended copy of the Jan De Dobbeleer https://go-review.googlesource.com/c/go/+/384160 CL. Fixes golang#42919 Change-Id: I8b5a26d0cac7882d3445393d26b182ad31cd753b

gopherbot · 2023-01-04T22:30:00Z

Change https://go.dev/cl/460595 mentions this issue: os: refactor and simplify Stat

bcmills · 2023-01-05T16:30:55Z

I think the problem here is more fundamental than just APPEXECLINK reparse points — the APPEXECLINK special cases in CL 384160 are treating a symptom but not the root cause.

I have mailed an alternative approach in CL 460595, which attempts to generalize the solution by treating non-symlink reparse points as irregular files (instead of treating all reparse points like symlinks as we currently do).

This change adds support for AppExecLinks files, like C:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.exe The python3.exe can be installed by following these https://www.microsoft.com/store/productId/9PJPW5LDXLZ5 instructions. The executable is added to your PATH and can be called from command line, like `python3 --version`. Calling GetFileAttributesEx on python3.exe returns FileAttributes with FILE_ATTRIBUTE_REPARSE_POINT set. And os.Stat attempts to follow the link. But Microsoft does not provide any link target for AppExecLinks files (see dotnet/runtime#58233 (comment) ), and Go should treat AppExecLinks as normal files and not symlinks. This CL adjusts os.Stat implementation to return normal file os.FileInfo for AppExecLinks files instead of symlinks. The AppExecLinks files are recognised as they return ERROR_CANT_ACCESS_FILE from CreateFile call. The trick is not documented anywhere. Jan De Dobbeleer discovered the trick. Also dotnet/runtime#58233 appears to also use ERROR_CANT_ACCESS_FILE to distinguish AppExecLinks files. The CL also adds new tests. The CL is an extended copy of the Jan De Dobbeleer https://go-review.googlesource.com/c/go/+/384160 CL. Fixes golang#42919 Change-Id: I8b5a26d0cac7882d3445393d26b182ad31cd753b

viveklak changed the title ~~exec.Command on file with UWP reparse point links fails with "file does not exist"~~ exec.Command on file with UWP reparse point fails with "file does not exist" Dec 1, 2020

This was referenced Dec 1, 2020

Check for python among execution aliases on Windows pulumi/pulumi#5775

Closed

Windows 10: Failed to locate any of ["python3" "python"] on your PATH pulumi/pulumi#5388

Closed

gopherbot added the OS-Windows label Dec 5, 2020

viveklak pushed a commit to pulumi/pulumi that referenced this issue Dec 7, 2020

Add a python shim script to work around golang/go#42919

acdde47

viveklak mentioned this issue Dec 7, 2020

Shim python interpreter for windows if interpreter uses reparse points pulumi/pulumi#5874

Merged

viveklak pushed a commit to pulumi/pulumi that referenced this issue Dec 7, 2020

Add a python shim script to work around golang/go#42919

6dd4dfc

viveklak pushed a commit to pulumi/pulumi that referenced this issue Dec 7, 2020

Add a python shim script to work around golang/go#42919

ed0d427

cagedmantis changed the title ~~exec.Command on file with UWP reparse point fails with "file does not exist"~~ os/exec: calling Command on file with UWP reparse point fails with "file does not exist" Dec 8, 2020

cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 8, 2020

cagedmantis added this to the Backlog milestone Dec 8, 2020

wcs1only mentioned this issue Dec 22, 2020

Attempting to run the helloworld sample dapr/quickstarts#240

Closed

dact221 mentioned this issue Nov 21, 2021

Julia Segment unable to find Julia executable installed through Microsoft Store JanDeDobbeleer/oh-my-posh#1269

Closed

1 task

bcmills self-assigned this Jan 10, 2023

bcmills modified the milestones: Backlog, Go1.21 Jan 10, 2023

gopherbot closed this as completed in 3e44b7d Jan 23, 2023

golang locked and limited conversation to collaborators Jan 23, 2024

gopherbot added the FrozenDueToAge label Jan 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/exec: calling Command on file with UWP reparse point fails with "file does not exist" #42919

os/exec: calling Command on file with UWP reparse point fails with "file does not exist" #42919

viveklak commented Dec 1, 2020 •

edited

networkimprov commented Dec 5, 2020

cagedmantis commented Dec 8, 2020

mattn commented Dec 8, 2020

viveklak commented Dec 8, 2020 •

edited

sachinjoseph commented Apr 14, 2022

alexbrainman commented Apr 18, 2022

sachinjoseph commented Apr 19, 2022 •

edited

JanDeDobbeleer commented Apr 19, 2022

JanDeDobbeleer commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022

JanDeDobbeleer commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022 •

edited

alexbrainman commented Nov 25, 2022

alexbrainman commented Nov 25, 2022

qmuntal commented Nov 27, 2022

gopherbot commented Jan 4, 2023

bcmills commented Jan 5, 2023

os/exec: calling Command on file with UWP reparse point fails with "file does not exist" #42919

os/exec: calling Command on file with UWP reparse point fails with "file does not exist" #42919

Comments

viveklak commented Dec 1, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

networkimprov commented Dec 5, 2020

cagedmantis commented Dec 8, 2020

mattn commented Dec 8, 2020

viveklak commented Dec 8, 2020 • edited

sachinjoseph commented Apr 14, 2022

alexbrainman commented Apr 18, 2022

sachinjoseph commented Apr 19, 2022 • edited

JanDeDobbeleer commented Apr 19, 2022

JanDeDobbeleer commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022

JanDeDobbeleer commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022

sachinjoseph commented Apr 19, 2022 • edited

Test.go

Go 1.18

@JanDeDobbeleer's branch

alexbrainman commented Nov 25, 2022

alexbrainman commented Nov 25, 2022

qmuntal commented Nov 27, 2022

gopherbot commented Jan 4, 2023

bcmills commented Jan 5, 2023

viveklak commented Dec 1, 2020 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

viveklak commented Dec 8, 2020 •

edited

sachinjoseph commented Apr 19, 2022 •

edited

sachinjoseph commented Apr 19, 2022 •

edited