Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os/exec: Piping to a command blocks under freebsd when input string has length >= 8192 #9512

Closed
AlexanderThaller opened this issue Jan 5, 2015 · 3 comments
Milestone

Comments

@AlexanderThaller
Copy link

  1. Go Version: go version go1.4 freebsd/amd64
  2. OS Version/Architecture: 10.0-RELEASE-p6 FreeBSD 10.0-RELEASE-p6 #0: Tue Jun 24 07:47:37 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
  3. I'm trying to pipe arbitrary strings to an external command which sometimes hangs for some reason.
  4. I want to get the CombinedOutput of the command after executing.
  5. The application hangs while piping.

I investigated and found out that the piping to the command won't finish sometimes. I wrote a program to check under which circumstances the piping would hang and found out that if the string is 8192 chars long (or longer) the piping would hang. Here is the program I used which just pipes to cat and uses increasing sizes for the string:

package main

import (
    "io"
    "log"
    "math/rand"
    "os/exec"
)

var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")

func randSeq(n int) string {
    b := make([]rune, n)
    for i := range b {
        b[i] = letters[rand.Intn(len(letters))]
    }
    return string(b)
}

func main() {
    for i := 8190; i < 8195; i++ {
        log.Println("length: ", i)
        data := randSeq(i)

        command := exec.Command("/bin/cat")

        pipe, err := command.StdinPipe()
        if err != nil {
            log.Fatal("can not get pipe from command: ", err)
        }

        log.Println("writing to pipe")
        _, err = io.WriteString(pipe, data)
        if err != nil {
            log.Fatal("can not write to pipe: ", err)
        }
        log.Println("finished writing to pipe")
        pipe.Close()

        _, err = command.CombinedOutput()
        if err != nil {
            log.Fatal("problem while running command: ", err)
        }

    }
}

I tried the same under Linux amd64 with the same go version and the program worked as expected.

@AlexanderThaller
Copy link
Author

When I do the same in the shell the command will execute as expected:

$ openssl rand -base64 8192 | tr -d '\n' | cat

@bradfitz bradfitz added this to the Go1.5 milestone Jan 5, 2015
@minux
Copy link
Member

minux commented Jan 5, 2015

Your program is expecting that you can write a pipe 8KB worth of data
without blocking, but that's not universally true on Unixes.

For example, if I run your program on Linux and change to write 128KB of
data to the pipe, the program will also block.

The reason your shell command works is that the cat process is running and
consuming content from the pipe concurrently.

Your Go program, however, doesn't start the cat process until the whole
WriteString is completed.

@minux
Copy link
Member

minux commented Jan 5, 2015

Closing as working as intended.

@minux minux closed this as completed Jan 5, 2015
@mikioh mikioh changed the title Piping to a command blocks under freebsd when input string has length >= 8192 os/exec: Piping to a command blocks under freebsd when input string has length >= 8192 Jan 5, 2015
@golang golang locked and limited conversation to collaborators Jun 25, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants