Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: closing Darwin FIFO writer does not trigger kqueue event for reader #24164

Closed
stjohnjohnson opened this issue Feb 27, 2018 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@stjohnjohnson
Copy link

What version of Go are you using (go version)?

go version go1.10 linux/amd64
go version go1.10 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

Ran this script here that reads/writes from a FIFO. On Linux the reader receives the EOF. On Darwin it does not.

package main

import (
    "bufio"
    "fmt"
    "os"
    "time"
)

// @note This is used to get around bufio.Scanner erroring at large lines
func readln(r *bufio.Reader) (string, error) {
    var (
        isPrefix = true
        err      error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return string(ln), err
}

func reader(filePath string) {
    f, err := os.Open(filePath)
    if err != nil {
        panic(err)
    }

    reader := bufio.NewReader(f)
    fmt.Print("READER >> created\n")

    line, readErr := readln(reader)

    for readErr == nil {
        fmt.Printf("READER >> read 1 line: %+v\n", line)
        line, readErr = readln(reader)
    }

    fmt.Printf("READER >> read finished: %+v\n", readErr)
}

func writer(filePath string) {
    f, err := os.OpenFile(filePath, os.O_WRONLY, 0600)

    fmt.Printf("WRITER << opened: %+v|%+v\n", f, err)
    if err != nil {
        panic(err)
    }

    fmt.Printf("WRITER << encoder created\n")

    for i := 0; i < 3; i++ {
        time.Sleep(1 * time.Second)
        _, err = f.WriteString(fmt.Sprint("line", i, "\n"))
        fmt.Printf("WRITER << written line%d, %+v\n", i, err)
    }

    time.Sleep(1 * time.Second)
    err = f.Close()
    fmt.Printf("WRITER << closed: %+v\n", err)
}

func main() {
    fifo := os.Args[1]

    fmt.Printf("STARTED %s\n", fifo)

    go writer(fifo)
    reader(fifo)

    fmt.Print("ALL DONE\n")
}

Linux

go version go1.10 linux/amd64
STARTED /tmp/tmp.6if4ZSWnZ8/fifo
READER >> created
WRITER << opened: &{file:0xc42008c000}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> read 1 line: line0
WRITER << written line1, <nil>
READER >> read 1 line: line1
WRITER << written line2, <nil>
READER >> read 1 line: line2
WRITER << closed: <nil>
READER >> read finished: EOF
ALL DONE

Darwin

go version go1.10 darwin/amd64
STARTED /var/folders/k_/zdt3jb2j7gx1rnf379k_51gh0000gn/T/tmp.ksHGLSvs/fifo
READER >> created
WRITER << opened: &{file:0xc42009e000}|<nil>
WRITER << encoder created
WRITER << written line0, <nil>
READER >> read 1 line: line0
WRITER << written line1, <nil>
READER >> read 1 line: line1
WRITER << written line2, <nil>
READER >> read 1 line: line2
WRITER << closed: <nil>
^C
signal: interrupt

Repo for reproduction: https://github.com/stjohnjohnson/golang-eof-fifo

Notes

Was able to reproduce it as far back as go1.9 darwin/amd64 using asdf. Last working version was go1.8.7 darwin/amd64.

@ianlancetaylor ianlancetaylor changed the title bufio.Reader.ReadLine not receiving EOF on OSX bufio: Reader.ReadLine from FIFO not receiving EOF on OSX Feb 28, 2018
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 28, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 28, 2018
@as
Copy link
Contributor

as commented Feb 28, 2018

This is caused by the poller. Related issue #22024 that happened on Windows. Confirmed with git bisect

c05b06a12d005f50e4776095a60d6bd9c2c91fac is the first bad commit
commit c05b06a12d005f50e4776095a60d6bd9c2c91fac
Author: Ian Lance Taylor <iant@golang.org>
Date:   Fri Feb 10 15:17:38 2017 -0800

    os: use poller for file I/O

@as
Copy link
Contributor

as commented Mar 3, 2018

Adding darwin to the conditional statement here makes your example work, but I don't know if that's a good idea.

@stjohnjohnson
You might be able to use this workaround https://play.golang.org/p/G8YIi1E7j4x. It calls Fd() on the reader's side of the pipe and enables blocking IO again... made the example work on my darwin system without modifying the go source.

@stjohnjohnson
Copy link
Author

@as that works! Thanks!

APShirley added a commit to APShirley/reconfigure-pipeline that referenced this issue Mar 15, 2018
golang/go#24164

Signed-off-by: Adrian Zankich <azankich@pivotal.io>
@ianlancetaylor ianlancetaylor changed the title bufio: Reader.ReadLine from FIFO not receiving EOF on OSX runtime: closing Darwin FIFO writer does not trigger kqueue event for reader Jun 13, 2018
@ianlancetaylor
Copy link
Contributor

I can confirm that on MacOS 16.3.0 closing the last writer to a FIFO does not trigger a kqueue event for the reader. In Go's implementation, that causes the reader to block indefinitely.

There is a Stack Overflow questions that mentions this, with no solution offered: https://stackoverflow.com/questions/49481723/difference-in-kqueue-handling-of-fifos-between-mac-os-and-freebsd .

My best suggestion at this point is that on Darwin we will have to avoid treating fifos as pollable.

@gopherbot
Copy link

Change https://golang.org/cl/118566 mentions this issue: os: don't poll fifos on Darwin

dna2github pushed a commit to dna2fork/go that referenced this issue Jun 14, 2018
The Darwin kqueue implementation doesn't report any event when the
last writer for a fifo is closed.

Fixes golang#24164

Change-Id: Ic2c47018ef1284bf2e26379f8dd7646edaad4d05
Reviewed-on: https://go-review.googlesource.com/118566
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@golang golang locked and limited conversation to collaborators Jun 13, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests

4 participants