Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: Support make jobserver for dynamic negotiable concurreny during compilation #52387

Open
kunaltyagi opened this issue Apr 17, 2022 · 2 comments
Open

cmd/compile: Support make jobserver for dynamic negotiable concurreny during compilation #52387

kunaltyagi opened this issue Apr 17, 2022 · 2 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@kunaltyagi
Copy link

What version of Go are you using (go version)?

$ go version
1.12.2 gccgo

Does this issue reproduce with the latest release?

Not tested

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/builder/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/builder/go"
GOPROXY=""
GORACE=""
GOROOT="/usr"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/gcc/x86_64-linux-gnu/9"
GCCGO="/usr/bin/x86_64-linux-gnu-gccgo-9"
CC="/opt/miniconda3/envs/builder/bin/x86_64-conda-linux-gnu-cc"
CXX="/opt/miniconda3/envs/builder/bin/x86_64-conda-linux-gnu-c++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build646202173=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

What did you do?

  • Clone five independent packages
  • Compile each with say 2 threads
  • Observer number of threads
  • It goes down from 10 (max possible) to 5 (minimum)

What did you expect to see?

  • Using a single workstation as CI server allows orchestration of concurrency during compilation of multiple independent packages
  • Compiling go with other languages plays well in regards to optimal utilization of CPU cores

What did you see instead?

  • Controlling max threads in a single go compile process is possible
  • Sharing of threads between independent go compile process is not possible
  • Ensuring that 5 or so independent compile processes are keeping 8 CPU's fully stocked is not feasible.

Possible solution

Implement support for GNU make jobserver for compilation. This doesn't have to be exposed for 80-90% of users whose primary use-case doesn't involve parallel independent compilation of go packages.

Bonus: Both compilation and testing can benefit from this

Pro:

  • Works with make, ninja-cmake and other related build and meta build systems
  • Works cross-platform on systems with sockets (all modern systems targeted by go as compilation platforms based on expectation of pulling dependencies automatically)
  • Parity with compilation process of compilers like C/C++ (clang, gcc), rust (cargo)
  • Allows power users to perform quick context switches during compilation and testing
  • Allows people working on polyglot code bases a significant boost (a lot of tech stacks are not pure go)

Con:

  • jobserver has drawbacks (the pipe could have been closed, env variables need to be passed into sub-shells (not a concern here, but often a source of errors))
  • more complicated process for spawning sub-processes
  • maybe low utilization in go community
@kunaltyagi kunaltyagi changed the title affected/package: affected/package: Support make jobserver for concurreny during compilation Apr 17, 2022
@kunaltyagi kunaltyagi changed the title affected/package: Support make jobserver for concurreny during compilation affected/package: Support make jobserver for dynamic concurreny during compilation Apr 17, 2022
@kunaltyagi kunaltyagi changed the title affected/package: Support make jobserver for dynamic concurreny during compilation cmd/compile: Support make jobserver for dynamic concurreny during compilation Apr 17, 2022
@kunaltyagi kunaltyagi changed the title cmd/compile: Support make jobserver for dynamic concurreny during compilation cmd/compile: Support make jobserver for dynamic negotiable concurreny during compilation Apr 17, 2022
@kunaltyagi kunaltyagi mentioned this issue Apr 17, 2022
Open
@mdempsky
Copy link
Member

Concretely and technically, what does it mean to support the "make jobserver"? Is there a spec or something that explains what this means?

Also, my impression is most Go users use "go build" or Bazel. Will addressing this issue help those users? It sounds like it would only help Go users that use GNU make, which I didn't think there were that many.

@kunaltyagi
Copy link
Author

kunaltyagi commented Apr 17, 2022 •
edited

Make Jobserver is more concretely defined in:

Sadly, I don't think there is a "spec", but this is unlikely to change in a non-backward compatible way since GNU based on historic data.

(There are some subtleties involved, but) Essentially

  1. CLI support: Read --jobserver-auth=R,W from MAKEFLAGS environment variable
  2. During the build/test stage:
    1. Read N character (or more) from fd to get a "token"/permission to utilize N compute resources (aka CPU core)
    2. Create N processes (ideally. You can create more or less processes, if required. Eg: make creates N+1 processes, with the extra process for the orchestration master process)
    3. Write back the same character as soon as a sub-process completes, and goto 1 again

Does this only support users using GNU make

Please see foreign jobserver support for bazel for more details how this would help even a wider audience who are already using this for their mixed language bazel setup.
Please note that the third party script is needed not just due to issue still being open, but also to orchestrate multiple bazel builds together

In a wider context, as stated in the original comment, this would essentially allow much better utilization of the CI for pure go build style builds where the different builds don't explicitly know about each other (and are most likely running inside dockers/jails for isolation).

As a fringe benefit, this would also allow go build to play nice with meta build systems (eg: catkin and colcon, primarily used in robotics community) which target distributed and polyglot code bases.

Sadly, I don't know how the workflows of gophers look like (I'd be a very poor judge), so can't comment more generically.

EDIT:
Long-form post by Paul Smith explaining the design rationale for the interested

@thanm thanm added this to the Backlog milestone Apr 18, 2022
@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 18, 2022
@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Unplanned Apr 19, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
5 participants
@mdempsky @kunaltyagi @ianlancetaylor @gopherbot @thanm