Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: fatal error polling some file descriptors on macOS Mojave #37726

Closed
jbardin opened this issue Mar 7, 2020 · 13 comments
Closed

runtime: fatal error polling some file descriptors on macOS Mojave #37726

jbardin opened this issue Mar 7, 2020 · 13 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@jbardin
Copy link
Contributor

jbardin commented Mar 7, 2020 •
edited

What version of Go are you using (go version)?

go1.14
master

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin_amd64
macOS Mojave

What did you do?

The following test file crashes on macOS Mojave

package main

import (
	"os"
	"testing"
	"time"
)

func TestFD(t *testing.T) {
	os.NewFile(4, "")
	time.Sleep(time.Second)
}

With the output:

$ go test fd_test.go
runtime: netpoll: break fd ready for -2
fatal error: runtime: netpoll: break fd ready for something unexpected

runtime stack:
runtime.throw(0x29c482a, 0x39)
/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:1112 +0x72
runtime.netpoll(0x176414, 0xc00004c001)
/usr/local/Cellar/go/1.14/libexec/src/runtime/netpoll_kqueue.go:140 +0x30a
runtime.findrunnable(0xc00004e800, 0x0)
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2323 +0x72b
runtime.schedule()
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2520 +0x2fc
runtime.park_m(0xc000182600)
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x80000)
/usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:318 +0x5b
...

The same error was seen in #35566, but that fix was only for the test in question.
This isn't directly related to testing however, as we've seen this with normal production binaries.
@jbardin jbardin mentioned this issue Mar 7, 2020
Closed
@chenweiyj
Copy link

chenweiyj commented Mar 9, 2020 •
edited

I have found that if the _test file is compiled first with -c flag as running by go test -c, and run the executable object, the runtime error won't occur.

However, when run the go source file directly by go test, the runtime error occurs.

@chenweiyj
Copy link

Also, in macos when calling os.NewFile(4, ""), it will crash if and only if the fd equals 4.

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 9, 2020
@toothrot toothrot added this to the Backlog milestone Mar 9, 2020
@toothrot
Copy link
Contributor

toothrot commented Mar 9, 2020

/cc @ianlancetaylor for maybe related issue #35566

@maurorappa
Copy link

nothing yet ? :(

@toothrot
Copy link
Contributor

@maurorappa While every issue is important to us, please try to be patient as many of us are also juggling personal health and responses to the pandemic.

@ianlancetaylor
Copy link
Contributor

What is the real code doing that causes this problem? While clearly using os.NewFile(4, "") ought to work, why is this an important case?

@ianlancetaylor
Copy link
Contributor

If the problem only happens with file descriptor 4, then it's because file descriptor 4 is being used by the runtime package. The runtime package treats the descriptor specially, and only expects it to be polled for read. Using os.NewFile(4, ''") is expected to fail.

@ianlancetaylor ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 31, 2020
@jbardin
Copy link
Contributor Author

jbardin commented Apr 1, 2020

A number of our tools run primarily in a subprocess, and the main process passes stdin, stdout and stderr in via Cmd.ExtraFiles in order to provide access to things like the raw terminal. This of course puts the original stdout in FD 4, which we need to convert into a File to use.

@ianlancetaylor
Copy link
Contributor

@jbardin Can you show a complete example that demonstrates the problem, and show what happens when you run it? The example above is different in relevant ways from what you are describing. Thanks.

@jbardin
Copy link
Contributor Author

jbardin commented Apr 1, 2020

Thanks @ianlancetaylor, I'll try to put together a standalone example, (need to find a Mojave machine to debug on). The test example above has the exact same fatal error output as what we're experiencing, so thought it might be close enough.

I did however confirm that simply passing stdin, stdout, stderr on 3, 4, 5 works, so there must be something else involved here, and it also seems strange it's related to a specific OS version as well.

@ianlancetaylor
Copy link
Contributor

I'm fairly certain that the error break fd ready for something unexpected can only happen if a program erroneously uses the file descriptor number that happens to have been created to use to wake up a sleeping kevent.

@jbardin
Copy link
Contributor Author

jbardin commented Apr 4, 2020

Thanks again Ian, I see what's going on now. I had time to track this down and find where the file descriptors were being used in the incorrect process.

@jbardin jbardin closed this as completed Apr 4, 2020
@ianlancetaylor
Copy link
Contributor

Thanks for following up.

@trombik trombik mentioned this issue Apr 9, 2020
Closed
@golang golang locked and limited conversation to collaborators Apr 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
6 participants
@toothrot @jbardin @chenweiyj @ianlancetaylor @maurorappa @gopherbot