Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: NumCPU() does not match "the number of cpus usable by the current process" in FreeBSD #15206

Closed
wheelcomplex opened this issue Apr 9, 2016 · 4 comments
Labels
FrozenDueToAge OS-FreeBSD Suggested Issues that may be good for new contributors looking for work to do.
Milestone

Comments

@wheelcomplex
Copy link
Contributor

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version

    go version devel +9d4efdf Sat Apr 9 07:15:22 2016 +0000 freebsd/amd64
    
  2. What operating system and processor architecture are you using (go env)?
    go env

    GOARCH="amd64"
    GOBIN="/home/rhinofly/home/sandspace/go/bin"
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="freebsd"
    GOOS="freebsd"
    GOPATH="/home/rhinofly/home/sandspace/external/:/home/rhinofly/home/sandspace/golibs/"
    GORACE=""
    GOROOT="/home/rhinofly/home/sandspace/go"
    GOTOOLDIR="/home/rhinofly/home/sandspace/go/pkg/tool/freebsd_amd64"
    CC="clang"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -gno-record-gcc-switches"
    CXX="clang++"
    CGO_ENABLED="1"
    

    uname -a

    FreeBSD freebsd-i5-box.localdomain 10.2-STABLE FreeBSD 10.2-STABLE #0 r292855: Tue Dec 29 06:17:50 UTC 2015     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
    
  3. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

cpuset is a util to configure/limit processes available for a processor, just like taskset in Linux.

DESCRIPTION
     The cpuset command can be used to assign processor sets to processes, run
     commands constrained to a given set or list of processors, and query
     information about processor binding, sets, and available processors in
     the system.

save follow code to freebsdncpu.go

go build -o freebsdncpu freebsdncpu.go and run as:

/usr/bin/cpuset -l 0 ./freebsdncpu
//
package main

import (
    "fmt"
    "os"
    "os/exec"
    "runtime"
    "strconv"
    "sync/atomic"
)

func main() {
    fmt.Printf("runtime.GOMAXPROCS(-1) return %d CPUs\n", runtime.GOMAXPROCS(-1))
    fmt.Printf("runtime.NumCPU() return %d CPUs\n", runtime.NumCPU())
    cpuset := exec.Command("cpuset", "-g", "-p", strconv.Itoa(os.Getpid()))
    output, err := cpuset.CombinedOutput()
    if err != nil {
        fmt.Printf("exec cpuset failed: %s\n", err.Error())
        os.Exit(1)
    }
    fmt.Printf("cpuset return: %s\n", output)
    var cnt int64
    for i := 0; i < runtime.NumCPU(); i++ {
        go func() {
            for {
                atomic.AddInt64(&cnt, 1)
            }
        }()
    }
    fmt.Printf("%d CPU burnner running ...\n", runtime.NumCPU())
    fmt.Printf("Press +C to exit ...\n")
    select {}
}
  1. What did you expect to see?
runtime.GOMAXPROCS(-1) return 1 CPUs
runtime.NumCPU() return 1 CPUs
cpuset return: pid 40291 mask: 0

1 CPU burnner running ...
Press +C to exit ...

from top -IaP we see Go processor using only CPU0

  1. What did you see instead?
runtime.GOMAXPROCS(-1) return 2 CPUs
runtime.NumCPU() return 2 CPUs
cpuset return: pid 40291 mask: 0

2 CPU burnner running ...
Press +C to exit ...

from top -IaP we see Go processor using only CPU0

  1. more information:

as https://github.com/golang/go/blob/master/src/runtime/os1_freebsd.go#L20 showed, NumCPU()/ncpu is direct probe from sysctl item hw.ncpu, but in
https://www.freebsd.org/cgi/man.cgi?sysctl(3) and /usr/include/sys/sysctl.h, it's say "hw.ncpu" is "The number of cpus", is not the "usable by the current process" in https://github.com/golang/go/blob/master/src/runtime/debug.go#L32

And please note, cpuset already limit the CPU list "befor" Go processes start, not the case in #11609 (change usable cpu list on the fly).

wheelcomplex added a commit to wheelcomplex/wander that referenced this issue Apr 9, 2016
@wheelcomplex
Copy link
Contributor Author

FYI:
equivalent of sched_getaffinity(Linux) in FreeBSD
https://github.com/freebsd/freebsd/blob/master/lib/libc/sys/cpuset_getaffinity.2

@bradfitz bradfitz added this to the Unplanned milestone Apr 9, 2016
@bradfitz bradfitz added the Suggested Issues that may be good for new contributors looking for work to do. label Apr 9, 2016
@wheelcomplex
Copy link
Contributor Author

wheelcomplex commented Apr 12, 2016

Hi folks,
I have make a amd64 only fix for this issue at https://github.com/wheelcomplex/official-go/commit/5f02246310b4c775da979a7433c920fcb1f98331, but I do not know how to port this change to 386/arm so it can not submit for approval.

@mikioh
Copy link
Contributor

mikioh commented Apr 12, 2016

You also need to take care of the system call number between major releases on BSD variants. Luckily, it's still the same as 8 years ago; https://svnweb.freebsd.org/base/head/sys/kern/syscalls.master?view=log.

wheelcomplex pushed a commit to wheelcomplex/smith-go that referenced this issue Aug 31, 2016
    In FreeBSD when run Go proc under a given sub-list of processors(e.g. 'cpuset -l 0 ./goutil' in multicore system),
    runtime.NumCPU() still return all actived CPUs instead of the number of sub-list.

    Fix by use syscall cpuset_getaffinity to account the number of sub-list.

    Fixes golang#15206

    Tested:
    FreeBSD amd64 10/11/12
    FreeBSD i386 10/11/12
    FreeBSD arm 10/11/12

    This is inspired by golang@4f308ed
    (runtime: use sched_getaffinity for runtime.NumCPU() on Linux)
wheelcomplex pushed a commit to wheelcomplex/smith-go that referenced this issue Sep 5, 2016
In FreeBSD when run Go proc under a given sub-list of processors(e.g. 'cpuset -l 0 ./goutil' in multicore system),
runtime.NumCPU() still return all actived CPUs instead of the number of sub-list.

Fix by use syscall cpuset_getaffinity to account the number of sub-list.

Fixes golang#15206

Tested:
FreeBSD amd64 10/11/12
FreeBSD i386 10/11/12
FreeBSD arm 10/11/12

Inspired by golang@4f308ed
(runtime: use sched_getaffinity for runtime.NumCPU() on Linux)
@gopherbot
Copy link

CL https://golang.org/cl/29341 mentions this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge OS-FreeBSD Suggested Issues that may be good for new contributors looking for work to do.
Projects
None yet
Development

No branches or pull requests

4 participants