os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable #66654

ClydeByrdIII · 2024-04-02T21:30:19Z

Go version

go 1.21.8 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/user/.cache/go-build'
GOENV='/home/user/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/user/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/user/go'
GOPRIVATE=''
GOPROXY=''
GOROOT='/home/user/go/v1.21.8'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/user/go/v1.21.8/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.8'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK='/home/user/project/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3303884229=/tmp/go-build -gno-record-gcc-switches'

What did you do?

The following program attempts to Fork+Exec a binary (/usr/local/bin/play-sandbox) via an open file descriptor (FD) in the current process and have the child inherit additional FDs and print the usage information (-h / --help)

go playground link https://go.dev/play/p/MXQGzqeVJK0

package main

import (
	"fmt"
	"os"
	"os/exec"

	"golang.org/x/sys/unix"
)

func main() {
	// open up a binary as O_PATH (more or less just a pointer to the file in the filesystem with no r/w support)
	binPtr, err := os.OpenFile("/usr/local/bin/play-sandbox", unix.O_PATH, 0)
	if err != nil {
		panic(err)
	}
	defer binPtr.Close()
	// refer to the binary via the processes FD table entry
	saved := fmt.Sprintf("/proc/self/fd/%d", int(binPtr.Fd()))

	cmd := exec.Command(saved, "-h")
	// direct child's output to parents stdout
	cmd.Stderr = os.Stdout
	cmd.Stdout = os.Stdout

	// add one or more files for the child to inherit (and thus cause potential fd stomping)
	cmd.ExtraFiles = append(cmd.ExtraFiles, os.Stdout)

	fmt.Printf("Running cmd: %v\n", cmd)
	err = cmd.Run()
	if err != nil {
		panic(err)
	}
}

What did you see happen?

Running cmd: /proc/self/fd/3 -h
panic: fork/exec /proc/self/fd/3: permission denied

goroutine 1 [running]:
main.main()
	/tmp/sandbox3277445564/prog.go:52 +0x2d3

Program exited.

What did you expect to see?

Running cmd: /proc/self/fd/4 -h
Usage of /proc/self/fd/4:
  -dev
    	run in dev mode (show help messages)
  -listen string
    	HTTP server listen address. Only applicable when --mode=server (default ":80")
  -mode string
    	Whether to run in "server" mode or "contained" mode. The contained mode is used internally by the server mode. (default "server")
  -untrusted-container string
    	container image name that hosts the untrusted binary under gvisor (default "gcr.io/golang-org/playground-sandbox-gvisor:latest")
  -workers int
    	number of parallel gvisor containers to pre-spin up & let run concurrently (default 8)

Program exited.

The problem can be circumvented by simply pushing the exe FD in the parent process to a FD integer high enough that it won't get clobbered by the file descriptors in cmd.ExtraFiles.

e.g
situation:
4 FDs are intended to be inherited by the child after exec [stdin(0), stdout(1), stderr(2), stdin_dup(4)] and the fd pointing to the exe is 3.
Problem:
exe(3) get's clobbered by stdin_dup (4) in the child before exec.

Workaround: By making the fd of exe, point to higher then len(ProcAttr.Files) (4 in this case), the exe isn't clobbered (and if CLO_EXEC is set, it'll also be closed in the exec'ed child. Which is my desired end-state)

https://go.dev/play/p/gF5zi4g2fTq

I wasn't sure whether to mark this as a syscall issue or not; I originally ran in to this using libcap's cap.Launcher and was able to reproduce with the equivalent os/exec's Command setup. Both of them use syscall.ForkExec, which is where I believe the FD stomping occurs based on some strace + print statements.

I see in exec_linux.go that there's special handling to avoid stomping on needed FDs like pipes. Perhaps some special casing for /proc/self/fd/<fd> as argv[0] could be done too? Not sure, no expert here.

The text was updated successfully, but these errors were encountered:

dmitshur · 2024-04-02T22:04:38Z

CC @bradfitz, @ianlancetaylor.

ianlancetaylor · 2024-04-02T22:04:42Z

That seems like a very unusual special case. The only way I see to fix it is to recognize the use of /proc/self/fd or /proc/MYPID/fd, dup the file to a large number, make sure that O_CLOEXEC is set, and exec that. As you've discovered the program doing this can do that anyhow. Right now that seems preferable to me: force the program that wants to handle this very unusual case to handle it itself.

ClydeByrdIII · 2024-04-03T01:47:56Z

That seems like a very unusual special case.

I agree; It's definitely unusual, although perhaps not unexpected in Linux.
It's an expected use case of the O_PATH flag used with open(2)

One use of O_PATH for regular files is to provide the
equivalent of POSIX.1's O_EXEC functionality. This
permits us to open a file for which we have execute
permission but not read permission, and then execute that
file, with steps something like the following:

              char buf[PATH_MAX];
              fd = open("some_prog", O_PATH);
              snprintf(buf, PATH_MAX, "/proc/self/fd/%d", fd);
              execl(buf, "some_prog", (char *) NULL);

or similar reasons for fexecve(3)

The idea behind fexecve() is to allow the caller to verify
(checksum) the contents of an executable before executing it.
Simply opening the file, checksumming the contents, and then
doing an execve(2) would not suffice, since, between the two
steps, the filename, or a directory prefix of the pathname, could
have been exchanged (by, for example, modifying the target of a
symbolic link).

My use-case just happens to also incorporate inheriting IPC primitives with the parent and child that lead me to this issue.

The only way I see to fix it is to recognize the use of /proc/self/fd or /proc/MYPID/fd, dup the file to a large number, make sure that O_CLOEXEC is set, and exec that.

I feared as such.

As you've discovered the program doing this can do that anyhow. Right now that seems preferable to me: force the program that wants to handle this very unusual case to handle it itself.

Understandable; one would hope the issue would be more readily presentable though. It had a non-ideal amount of digging needed to figure out it was occurring in golang source code and not the higher-level ForkExec wrappers.

There's a non-zero possibility of someone using this pattern and invoking an unintended binary rather than getting various ERRNO (EPERM, ENXIO, etc) errors that I found.

If not an in-place fix, would you folks be open to any (or all) of the below:

if /proc/self/fd or /proc/MYPID/fd are passed as a path, fail-fast and return an error in ForkExec()
documenting the special case in the syscall.ForkExec docs
creating a near ForkExec duplicate call that does a fork(2) + fexecve(3) like functionality
- I know fexecve(3) isn't a syscall, but instead a glibc wrapper that utilizes /proc or execveat(2) depending on linux kernel version. Still would be neat though

As I have workarounds to my specific situation, I won't go crazy, if nothing changes, but just figured I'd raise the awareness.

Thanks for the quick look and response!

ianlancetaylor · 2024-04-03T04:47:27Z

Documentation would certainly be fine.

ClydeByrdIII changed the title ~~os/exec: Can't use os.Command that uses /proc/self/fd/<exe fd> as the executable~~ os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable Apr 2, 2024

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 2, 2024

dmitshur added this to the Backlog milestone Apr 2, 2024

dmitshur added the FeatureRequest label Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable #66654

os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable #66654

ClydeByrdIII commented Apr 2, 2024 •

edited

dmitshur commented Apr 2, 2024

ianlancetaylor commented Apr 2, 2024

ClydeByrdIII commented Apr 3, 2024 •

edited

ianlancetaylor commented Apr 3, 2024

os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable #66654

os/exec: Can't use os.Command (with ExtraFiles) that uses /proc/self/fd/<exe fd> as the executable #66654

Comments

ClydeByrdIII commented Apr 2, 2024 • edited

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

dmitshur commented Apr 2, 2024

ianlancetaylor commented Apr 2, 2024

ClydeByrdIII commented Apr 3, 2024 • edited

ianlancetaylor commented Apr 3, 2024

ClydeByrdIII commented Apr 2, 2024 •

edited

Output of `go env` in your module/workspace:

ClydeByrdIII commented Apr 3, 2024 •

edited