Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: scheduler: go-routine starvation #21053

Closed
prasannavl opened this issue Jul 17, 2017 · 7 comments
Closed

runtime: scheduler: go-routine starvation #21053

prasannavl opened this issue Jul 17, 2017 · 7 comments

Comments

@prasannavl
Copy link

prasannavl commented Jul 17, 2017

What version of Go are you using (go version)?

go version go1.8.3 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Workspace\Golang
set GORACE=
set GOROOT=D:\Apps\Scoop\apps\go\current
set GOTOOLDIR=D:\Apps\Scoop\apps\go\current\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Consider the below program:

package main

import (
	"fmt"
	"sync/atomic"
	"time"
)

func main() {

	var ops uint64 = 0
        // This is the key value. When this is >= to the num 
        // of physical cores, the program will never terminate. 
        // Otherwise it should work as expected.
        iters := 4
	for i := 0; i < iters; i++ {
		go func() {
			for {
				atomic.AddUint64(&ops, 1)
			}
		}()
	}

	// Wait a second to allow some ops to accumulate.
	time.Sleep(time.Second)

	opsFinal := atomic.LoadUint64(&ops)
	fmt.Println("ops:", opsFinal)
}

I was trying to get an estimate of the ops of atomics and mutexes. I wrote the above example. When instead of atomics, mutexes are used, it works as expected. However, when using atomics, when the number of go-routines are greater than the number of the actual physical threads on the system, the goroutine that's waiting on the timer (the main go-routine) is starved - I'm guessing is because of the CAS semantics of atomics that staves the runtime due to SpinWaits.

What did you expect to see?

The program actually completes in a near a second.

What did you see instead?

The program never ends using 100% of the CPU.

This tells me that the scheduler ends up starving the goroutine that's waiting on the timer. Is it possible to improve the scheduler to detect this somehow and not end up starving? This is a unique problem due to the nature of go-routines. Coming from a .NET eco-system, where the Task units are run from a thread pool, which uses physical threads, the OS scheduler handles this smoothly, and this was quite a surprise to me - but makes sense considering the nature of go-routines.

UPDATE

Surprisingly, looks like it doesn't actually have anything to do with atomics. Just turning it into an empty loop, causes the main go-routine to be starved. Quite confused on what to make of this. Is this some kind of a scheduler bug?

@prasannavl prasannavl changed the title Scheduler starvation when using atomics scheduler: go-routine starvation when using atomics Jul 17, 2017
@ALTree
Copy link
Member

ALTree commented Jul 17, 2017

This is #10958.

Note that if you build with GOEXPERIMENT=preemptibleloops, the program terminates. On my 4-cores machine:

go1.8.3:

$ go run prova.go
  [does not terminate]

go1.9beta2, built with preemptibleloops:

$ ~/go/bin/go run prova.go
ops: 30388899

This will be fixed when (if) preemptibleloops-by-default lands.

@prasannavl prasannavl changed the title scheduler: go-routine starvation when using atomics scheduler: go-routine starvation Jul 17, 2017
@prasannavl
Copy link
Author

@ALTree - Ah. Glad to see it work in progress. Shall I close this as dup?

@bradfitz
Copy link
Contributor

Yes, this is a dup. It's targeted to Go 1.10.

@prasannavl
Copy link
Author

@ALTree, does the GOEXPERIMENT apply to all platforms? I tried it in windows - doesn't seem to work. Still hangs.

PS D:\Workspace\Labs\go-play> go version
go version go1.9beta2 windows/amd64
PS D:\Workspace\Labs\go-play> $env:GOEXPERIMENT
preemptibleloops
PS D:\Workspace\Labs\go-play> go run .\play.go

@bradfitz
Copy link
Contributor

You can't just set the environment variable, IIRC. You have to set the variable and then rebuild the Go toolchain.

@ALTree
Copy link
Member

ALTree commented Jul 17, 2017

Yeah you need to rebuild the toolchain from source with the variable enabled, e.g. GOEXPERIMENT=preemptibleloops ./make.bash in go/src

@prasannavl
Copy link
Author

@bradfitz, ah - thanks. Thought it was a runtime config. Rebuilding the toolchain now.

@mikioh mikioh changed the title scheduler: go-routine starvation runtime: scheduler: go-routine starvation Jul 21, 2017
@golang golang locked and limited conversation to collaborators Jul 21, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants