Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: netpollWaiters typically not decremented #33624

Open
rutsky opened this issue Aug 13, 2019 · 1 comment
Open

runtime: netpollWaiters typically not decremented #33624

rutsky opened this issue Aug 13, 2019 · 1 comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@rutsky
Copy link

rutsky commented Aug 13, 2019

What is the semantic of runtime.netpollWaiters?
If it should track number of goroutines waiting for poll result, then it is implemented incorrectly (at least on Linux with epoll).

I see that runtime.netpollWaiters is incremented every time new goroutine getting blocked on polling:

go/src/runtime/netpoll.go

Lines 354 to 363 in f686a28

func netpollblockcommit(gp *g, gpp unsafe.Pointer) bool {
r := atomic.Casuintptr((*uintptr)(gpp), pdWait, uintptr(unsafe.Pointer(gp)))
if r {
// Bump the count of goroutines waiting for the poller.
// The scheduler uses this to decide whether to block
// waiting for the poller if there is nothing else to do.
atomic.Xadd(&netpollWaiters, 1)
}
return r
}

and decremented only in func netpollgoready(gp *g, traceskip int):

go/src/runtime/netpoll.go

Lines 365 to 368 in f686a28

func netpollgoready(gp *g, traceskip int) {
atomic.Xadd(&netpollWaiters, -1)
goready(gp, traceskip+1)
}

Looks like netpollgoready() is called only from internal/poll.runtime_pollSetDeadline(), i.e. in some codepaths related to setting polling deadlines:

go/src/runtime/netpoll.go

Lines 204 to 205 in f686a28

//go:linkname poll_runtime_pollSetDeadline internal/poll.runtime_pollSetDeadline
func poll_runtime_pollSetDeadline(pd *pollDesc, d int64, mode int) {

And most frequently parked goroutine waiting for poll result is being awakened somewhere in runtime.findrunnable():

go/src/runtime/proc.go

Lines 2210 to 2221 in 61bb56a

// Poll network.
// This netpoll is only an optimization before we resort to stealing.
// We can safely skip it if there are no waiters or a thread is blocked
// in netpoll already. If there is any kind of logical race with that
// blocked thread (e.g. it has already returned from netpoll, but does
// not set lastpoll yet), this thread will do blocking netpoll below
// anyway.
if netpollinited() && atomic.Load(&netpollWaiters) > 0 && atomic.Load64(&sched.lastpoll) != 0 {
if list := netpoll(false); !list.empty() { // non-blocking
gp := list.pop()
injectglist(&list)
casgstatus(gp, _Gwaiting, _Grunnable)

or runtime.pollWork():

go/src/runtime/proc.go

Lines 2395 to 2409 in 61bb56a

// pollWork reports whether there is non-background work this P could
// be doing. This is a fairly lightweight check to be used for
// background work loops, like idle GC. It checks a subset of the
// conditions checked by the actual scheduler.
func pollWork() bool {
if sched.runqsize != 0 {
return true
}
p := getg().m.p.ptr()
if !runqempty(p) {
return true
}
if netpollinited() && atomic.Load(&netpollWaiters) > 0 && sched.lastpoll != 0 {
if list := netpoll(false); !list.empty() {
injectglist(&list)

Apparently atomic.Load(&netpollWaiters) > 0 condition in the referenced above runtime.findrunnable() and runtime.pollWork() functions is always true, as soon as as single goroutine will wait for poll result and get awakened from those functions.

I verified that runtime.netpollWaiters is increased with each wait of a goroutine on network in an example of handling TCP conection:

// tcp-server.go

package main

import (
        "bufio"
        "fmt"
        "log"
        "net"
        "strings"
)

func main() {

        fmt.Println("Launching server...")

        ln, _ := net.Listen("tcp", ":8081")

        conn, _ := ln.Accept()

        for {
                message, err := bufio.NewReader(conn).ReadString('\n')
                if err != nil {
                        log.Fatal(err)
                }
                fmt.Print("Message Received:", string(message))
                newmessage := strings.ToUpper(message)
                conn.Write([]byte(newmessage + "\n"))
        }
}

$ dlv debug tcp-socket.go
Type 'help' for list of commands.
(dlv) p runtime.netpollWaiters
0
(dlv) c
Launching server...
received SIGINT, stopping process (will not forward signal)
> runtime.epollwait() /usr/lib/golang/src/runtime/sys_linux_amd64.s:675 (PC: 0x4619e0)                                                                               
Warning: debugging optimized function
(dlv) p runtime.netpollWaiters
1
(dlv) c
Message Received:message 1 from netcat
Message Received:message 2 from netcat
received SIGINT, stopping process (will not forward signal)
> runtime.epollwait() /usr/lib/golang/src/runtime/sys_linux_amd64.s:675 (PC: 0x4619e0)
Warning: debugging optimized function
(dlv) p runtime.netpollWaiters
4
(dlv) 

$ go version
go version go1.12.5 linux/amd64

Snippet from go env:

$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
@andybons
Copy link
Member

@aclements

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 13, 2019
@andybons andybons added this to the Unplanned milestone Aug 13, 2019
@andybons andybons changed the title runtime.netpollWaiters typically not decremented runtime: netpollWaiters typically not decremented Aug 13, 2019
@ianlancetaylor ianlancetaylor modified the milestones: Unplanned, Go1.14 Aug 13, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@mknyszek mknyszek moved this to Triage Backlog in Go Compiler / Runtime Jul 15, 2022
@ALTree ALTree mentioned this issue Jun 14, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Compiler / Runtime
Status: Triage Backlog
Milestone
Backlog
Development

No branches or pull requests
5 participants
@rutsky @rsc @andybons @ianlancetaylor @gopherbot