Looks like this is caused by connection pooling. I adjusted your sleep to 30s so I could run netstat. You can see that the listening socket is closed, but there are still connections open to the server which will get re-used to that server process.

Calling transport.CloseIdleConnections() in the client (or server?) should fix it.

tcp        0      0 127.0.0.1:56000         127.0.0.1:39670         ESTABLISHED 4568/z          
tcp        0      0 127.0.0.1:39670         127.0.0.1:56000         ESTABLISHED 4568/z          
tcp6       0      0 2a02:24e0:ff1:ce0:38118 2a00:1450:4009:810:::80 ESTABLISHED 4568/z          

It would be nice to close all existing connections after server.Serve returns, but I'm not sure how to do that!

Could it potentially be fixed by disabling keep-alive on the server? (Not a general solution, but as a temporary workaround)

It seems to be related to keepalives, where conenctions that are kept alive are served by the "old" server, even if the server has exited.

Putting this in place of the time.Sleep makes the problem go away

    closer := http.DefaultTransport.(interface {
        CloseIdleConnections()
    })
    closer.CloseIdleConnections()

@ncw - but isn't that a client-side fix?

I would expect some way of terminating keep-alive connections when the server exits. In this specific case we can just disable keepalive altogether, but it seems like a big gotcha.

related to: #9478

@klauspost yes the CloseIdleConnections hack is a client side fix.

I had a look through the net and net/http code and I can't see a way to do the equivalent on the server side.

Putting server.SetKeepAlivesEnabled(false) in Start also fixes the problem, but at the loss of no keepalives which might be a signficant performance hit.

If as @jbardin suggests #9478 was fixed, then you could put server.SetKeepAlivesEnabled(false) after server.Listen returns to fix the problem.

However I think that server.Listen should really do this tidying up itself when it returns.

There's also the old issue of graceful shutdown at #4674, but there are enough hooks to implement that externally (and some packages that do). I'd like it to be supported directly in net/http myself; maybe that's where we should focus.

Yeah, this is basically a dup of #4674 and/or #9478.

The confusion in your repro is that you thought your (*serv).Stop actually closed everything, but it only closed the Listener. The net/http.*Server does not have a way to close all open connections and does not even track the currently-open connections. Doing so would involve a new map and mutex, which is a cost everybody would have to pay for a little-used feature. It's unclear whether it should be opt-in, but that's kinda gross.

ok, since this is now linked to the other cases, I will close this.

It was quite unexpected behavior for me - and from #9478 I can see for you as well. We should maybe add some documentation, that this is the current behaviour - SetKeepAlivesEnabled isn't very clear, and the rest of the http.Server documetation doesn't mention it either.