although we have not yet been able to reduce this to something simpler

So is the whole kill -9 thing just a theory or is that part for sure?

Have you tried a repeated loop of dial a child process in a loop while kill -9'ing the child and it yielded no results, or ...?

It's a theory. The test cluster where we've observed this is used for "chaos" testing where we have a cron job that kills random processes periodically. I've tried to simply connect to a port where nothing is listening and haven't reproduced the problem, so that's where we get the theory that racing with the death of the remote process matters (or at least the closing of the listening socket). I also believe this requires a real network connection and not just two processes on localhost (some time needs to pass between the connect() and the connection refusal).

It's also possible that the missing link is something going on in the client process instead of specific network activity and timing. I had a theory back in #14548 that spurious wakeups were caused by reuse of runtime.pollDesc objects (and specifically the comment that "we can get ready notification from epoll/kqueue after the descriptor is closed/reused. Stale notifications are detected using seq variable", but seq is only consulted for deadlines, and not when netpoll_epoll.go calls netpollready). I haven't grokked this code well enough to understand how exactly stale notifications could happen.

/cc @ianlancetaylor

OK, I have a repro: https://gist.github.com/bdarnell/2d37a812368bb83090ab60d36ceae3c4

In the repro, we start a bunch of goroutines that dial an address where nothing is listening. We then cancel these dials, attempting to race the cancellation against the arrival of the "connection refused" packet. When we hit the race, here's what I think is going on:

1. The "connection refused" packet arrives and is processed by the kernel.
2. In the poller thread, epollwait returns, indicating that the socket FD is ready.
3. In the cancellation goroutine, we cancel the context.
4. The dialer's interrupter goroutine sees that the context is canceled and cancels polling by setting a deadline in the past.
5. pd.WaitWrite returns an error which is passed up the stack. This dial attempt fails.
6. When we close the file descriptor, the runtime.pollDesc (which the epoll thread above still holds a pointer to) is put back on the freelist.
7. The next dial attempt starts, and gets the pollDesc from the freelist. We call connect() and start waiting for the results.
8. The epoll thread from earlier finally gets scheduled again and wakes up the FD that is currently using this pollDesc
9. The dialer goroutine checks the status of the connection by calling getsockopt. No error has yet occurred, so it returns nil. The dial succeeds.

I see two possible solutions:

1. Prevent spurious wakeups at the poller level, by storing a pair (*pollDesc, seq) instead of just a *pollDesc in epollevent.data. Doing this naively would require a lot of additional allocations, but it might be possible to use another freelist.
2. Tolerate spurious wakeups during the connection process. I believe other uses of the poller are generally safe, since they'll just get EAGAIN and try again; only the getsockopt in connect appears to be vulnerable to this issue. DJB suggests getpeername as a potential alternative to getsockopt.

@ianlancetaylor, any theories? Or punt to Go 1.10.

@bdarnell Thanks for test case and analysis. I am able to recreate the problem. One thing I don't yet understand, though, is that even with a spurious wakeup, the call to fetch SO_ERROR only happens after connect has been called on the socket and has returned EINPROGRESS. I don't yet understand why the SO_ERROR would return no error when the socket is not actually connected.

OK, I understand now. SO_ERROR only returns a meaningful value after the socket is ready for writing.

CL https://golang.org/cl/45815 mentions this issue.

Nice find.

Thanks for this fix, we've been chasing the same thing googleapis/google-api-go-client#220