Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go build should be able to write multiple executables #14295

Closed
kardianos opened this issue Feb 10, 2016 · 35 comments
Closed

cmd/go: go build should be able to write multiple executables #14295

kardianos opened this issue Feb 10, 2016 · 35 comments
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@kardianos
Copy link
Contributor

Right now there is no easy cross platform way to output multiple artifacts from multiple packages into a single directory.

Right now you can do env GOBIN=bin/ go install cmd/... on many platforms, but not when cross compiling or on windows.

Suggesting making go build -o bin/ pkg1 pkg2 build both packages into the "bin" folder.

@robpike thinks this would make the meaning different based on if it is a file or directory and suggests using a different flag like "-odir". That seems fine however I'm not sure how -o would interact with -odir. If they were forbidden from being set at the same time, I would say they should be the same flag, namely -o.

Reference thread: https://groups.google.com/forum/#!topic/golang-dev/a1wvIZdnic0

@robpike
Copy link
Contributor

robpike commented Feb 11, 2016

Flag -o would name the file and -odir would name the directory. Seems clear to me. If both are set one could complain if -o is a rooted path.

Not sure this is worth doing at all, but it seems easy to design and understand.

@kardianos
Copy link
Contributor Author

@robpike That's a fair design. I think it would be useful to have, but I'm unsure if it would pay for a new flag.

I'm curious if others would also find it useful or if it is just me :).

@lmb
Copy link
Contributor

lmb commented Feb 12, 2016

FWIW I have been looking for something like this as well. For company policy reasons I need to build a .deb on a build box. There are several binaries, which I currently build individually and then package up. Not a terrible nuisance, but could probably save me some compile time since the binaries share most code.

@minux
Copy link
Member

minux commented Feb 12, 2016 via email

@kardianos
Copy link
Contributor Author

@minux There are times when I want the functionality of "go install", but placed into a local folder.

  • project specific tools that I don't want to install globally, but are needed for a project.
  • project specific binaries where due to the nature of resources or other design, should be built and ran locally.

The above may come up more often for me then others as I work on a contract basis and juggle different customers accounts. I use a single GOPATH, but often architect projects to run within a non-global context. (Sorry, that might not make complete sense).

@bradfitz bradfitz added this to the Unplanned milestone Apr 9, 2016
@rsc
Copy link
Contributor

rsc commented Aug 9, 2018

This is so close to working. I'm not sure why it doesn't. We should fix it for Go 1.12.
Clearly -o can't be used with multiple binary targets, but the implicit "write to the current directory" should work, and instead it does nothing.

$ cd /tmp/zzz
$ go build cmd/doc cmd/gofmt
$ ls -l
$ go build cmd/doc
$ go build cmd/gofmt
$ ls -l
total 14808
-rwxr-xr-x  1 rsc  eng  4266200 Aug  9 15:00 doc
-rwxr-xr-x  1 rsc  eng  3310440 Aug  9 15:00 gofmt
$ 

@rsc rsc modified the milestones: Unplanned, Go1.12 Aug 9, 2018
@rsc rsc changed the title cmd/go: allow "go build <flag>" to output multiple packages cmd/go: go build should be able to write multiple executables Aug 9, 2018
@robpike
Copy link
Contributor

robpike commented Aug 9, 2018

You could imagine -o or a variant specifying a directory rather than or as well as a file.

@pd93
Copy link

pd93 commented Sep 3, 2018

This isn't exactly a necessity, but it would be a huge convenience in my workflow - not to mention it would bring it in line with the install command. I never looked this up in the go docs. I just assumed it would work, but it doesn't 😞

I have a few projects with numerous small tools/utils and it would greatly simplify my testing if I could build them all at once to my cwd. Currently I have a bash script that runs consecutive go build ./path/to/tool commands which is less than ideal.

@neilpa
Copy link

neilpa commented Sep 4, 2018

Same here, we do most of our local dev on mac but than build and package for our standard linux image so have have similar problems since the GOBIN=... go install ... trick doesn't work. I've hacked together Makefiles to support cross-compiling with explicit go build -o ... for each target binary as a workaround. You could probably take this further with wildcard magic over the cmd/ directory but I decided I like readable makefiles.

This works for now but would prefer a go build or go install solution.

@gopherbot
Copy link

Change https://golang.org/cl/143139 mentions this issue: cmd/go: make go build write multiple executables

@rasky
Copy link
Member

rasky commented Oct 18, 2018

One side effect of this CL is that it breaks the venerable "go build -a std cmd" for Go maintainers (when run from $GOROOT/src, it will complain that "go" is already a directory; when run from any different directory, it will drop many binaries in it). The current workaround is "go build -o /dev/null -a std cmd".

Comments? /cc @rsc @robpike

@kardianos
Copy link
Contributor Author

There is a bit of discussion on the proposed CL, as well s the comment that the CL as implemented breaks go build -a std cmd. I would also comment the current CL breaks a personal use case I sometimes run go build ./cmd/... just to make sure everything my cmds rely on still build as a quick smoke test while in development.

I would make the following proposal:

Make go build without -o and multiple packages specified behave as before. Only output binaries from go build with multiple packages are specified when -o is specified. This prevents go build -a std cmd from breaking, preserves all existing behavior, while adding useful method such as go build -o ./bin/ ./cmd/.... Either that, or we could teach go install a -o switch that points to the output directory.


In the CL there are comments that suggest either:
go build $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...)
or for pkg in $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...); go build $pkg; end . Yeah, that's missing the point. I can get around this behavior, On *nix I can have a couple of helper scripts and on Windows another helper scripts, or I could make it part of my build system for individual projects. Again, that's kinda missing the point.

Sometimes it would be really useful to install a bunch of commands into a specific directory. Most of the semantics are already defined.

@rasky
Copy link
Member

rasky commented Oct 24, 2018

Another option is to add an explicit -dryrun option (or similar name) to avoid writing binaries; similar to the undocumented -o /dev/null but more explicit. This would help those who want to “just test it compiles”, in an explicit way.

Then, building multiple executables would just work, by default. This would make the behavior of go build more consistent; by default, it would always build executables, rather than changing behavior depending on how many packages were specified. It would be easier to explain and document, and less surprising for beginners.

@bcmills bcmills added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 25, 2018
@rasky
Copy link
Member

rasky commented Nov 4, 2018

Ping, can we get this unstuck? It just needs a decision on a small detail, we all want the main feature in.
/cc @andybons

@bcmills
Copy link
Contributor

bcmills commented Nov 13, 2018

go build ./... is useful today, and its meaning should not change. We can either take the -o approach that @kardianos suggests, or require an explicit list of targets per my comment on the CL. I'll talk to @rsc to decide which.

@bcmills
Copy link
Contributor

bcmills commented Nov 16, 2018

@kardianos In module mode, it may be useful to issue a go get command outside of any module (#24250) with the path to a binary plus some specific versions of dependencies to upgrade (or downgrade). In those cases, it seems redundant to require the user to pass -o $GOBIN or -o $GOPATH/bin explicitly just to get the binary to install in the usual place.

@kardianos
Copy link
Contributor Author

@bcmills Thank you for working on defining how go get works outside of any module. I think I may be missing something. Here's how I could envision the go command working in module mode:

## Outside of a module

# In all cases, if $GOBIN is not defined, it assumes $GOBIN is $HOME/bin

# Outside a module. Fetches cmd1 and cmd2, compiles it, and installs it in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Outside a module. Fetches cmd1 and cmd2 installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Outside a module. Fails.
go build github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to ./bin
go build -o ./bin /home/user1/code/moduleroot/cmd/...

# Outside a module. Fails.
go install github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to $GOBIN.
go install /home/user1/code/moduleroot/cmd/...

## Inside a module.

# Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Inside a module, builds the local commands cmd1 and cmd2, and discards the results
go build ./cmd/...

# Inside a module. Installs cmd1 and cmd2 in ./bin.
go build -o ./bin ./cmd/...

# (maybe) Inside a module. Installs cmd1 in ./bin (assuming bin is a folder).
# This would be useful in reusing the same script for windows and linux.
go build -o ./bin ./cmd/cmd1

# Inside a module, builds the local commands cmd1 and cmd2, installs in $GOBIN.
go install ./cmd/...

Can you explain why you would need to pass in -o $GOBIN to go get with this change?

@rsc
Copy link
Contributor

rsc commented Nov 20, 2018

We may all want this "in", but it's too late for Go 1.12.

@rsc rsc modified the milestones: Go1.12, Go1.13 Nov 20, 2018
@rsc rsc added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Nov 20, 2018
@rasky
Copy link
Member

rasky commented Nov 20, 2018

This is very sad.

It is the second release that this feature is postponed because the community doesn't get answers from the Go team regarding it in time for the release cycle (see also #23616 where I didn't get an answer for months). I have even sent a CL well in time, to try to unblock things.

How can we break this cycle? Is there a better channel than this? The community can't do development for things we care about if the Go team doesn't even prioritize answering to questions. /cc @andybons @bradfitz @bcmills @FiloSottile @cassandraoid

@ianlancetaylor
Copy link
Contributor

I'm sorry this one got stuck. As you know there has been a great deal of work on cmd/go last cycle and this one, and that has made looking at CLs for cmd/go even slower than usual. Also it seems that this feature interacts with modules, which makes it even worse since modules is in a steady state of flux.

@rsc
Copy link
Contributor

rsc commented Mar 12, 2019

To move forward with this, let's make -o with an argument that is a directory write a new executable to that directory. Note that the key thing is that the argument is a directory, not that it syntactically ends in slash. That is, go build -o becomes more like mv, not rsync.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

@gopherbot
Copy link

Change https://golang.org/cl/167679 mentions this issue: cmd/go: allow -o to point to a folder that writes multiple execs

@FiloSottile
Copy link
Contributor

It looks like only -o behavior was changed, but go build doesn't yet build multiple packages to the current directory by default, so reopening this.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

@FiloSottile FiloSottile reopened this Aug 19, 2019
@FiloSottile FiloSottile modified the milestones: Go1.13, Go1.14 Aug 19, 2019
@gopherbot
Copy link

Change https://golang.org/cl/190839 mentions this issue: cmd/go: fix "go help build -o" docs

@rasky
Copy link
Member

rasky commented Aug 19, 2019

For reference, my CL 143139 did work also without -o. Not sure why it was redone from scratch.

@kardianos
Copy link
Contributor Author

Sorry for the scattered discussion. CL comment https://go-review.googlesource.com/c/go/+/143139/4#message-fb8adc4368e18ba081fe2fe221448bce17d990bc showed that go build ./... that outputs executables (no -o) breaks things both in current std lib tests, and I'm sure in many tools that rely on that to discard output. Requiring -o . to output to the current directory keeps the current command line compatibility, while still gaining the functionality.

@FiloSottile
Copy link
Contributor

Makes sense, thanks for closing the loop.

gopherbot pushed a commit that referenced this issue Aug 20, 2019
The docs refer to "the last two paragraphs", but in fact should refer to
the first two of the previous three paragraphs. Moved up the out of place
paragraph.

Updates #14295

Change-Id: I066da7a665bc6754d246782b941af214a385017a
Reviewed-on: https://go-review.googlesource.com/c/go/+/190839
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/190907 mentions this issue: doc/go1.13: mention '-o <directory>' support for 'go build'

gopherbot pushed a commit that referenced this issue Aug 20, 2019
Fixes #33720
Updates #14295

Change-Id: I9cb6e02bcaccd7971057315163d8810157d465bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/190907
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
t4n6a1ka pushed a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
The docs refer to "the last two paragraphs", but in fact should refer to
the first two of the previous three paragraphs. Moved up the out of place
paragraph.

Updates golang#14295

Change-Id: I066da7a665bc6754d246782b941af214a385017a
Reviewed-on: https://go-review.googlesource.com/c/go/+/190839
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
t4n6a1ka pushed a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
Fixes golang#33720
Updates golang#14295

Change-Id: I9cb6e02bcaccd7971057315163d8810157d465bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/190907
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@odiferousmint
Copy link

odiferousmint commented Dec 18, 2019

Is go build -o build/ ./cmd/foo ./cmd/bar main.go going to be supported? Workaround is:

$ go build -o build/
$ go build -o build/ ./cmd/foo ./cmd/bar

Note that it does not work without ./. Probably intentional.

@jayconrod
Copy link
Contributor

@odiferousmint If a package is specified as a list of .go files on the command line, all the positional arguments must be .go files. So main.go can't be mixed with ./cmd/foo ./cmd/bar in the same command.

@Gregory-Ledray
Copy link

Gregory-Ledray commented Feb 26, 2020

I just got 1.14 and I'm confused. This works great with regular GOOS and GOARCH on non-empty directories (nice work!):

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ go build -o ./cmd ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
cmd.exe  hello  hello.exe  main.go  wasm1  wasm2  wasm3

But doesn't work with GOOS=js GOARCH=wasm:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./cmd ./...
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1: build output "cmd\\wasm1" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2: build output "cmd\\wasm2" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3: build output "cmd\\wasm3" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/hello: build output "cmd\\hello" already exists and is a directory


It works great with GOOS=js GOARCH=wasm if I make a new directory:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ ls ./bin
ls: cannot access './bin': No such file or directory

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ mkdir ./bin

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./bin
cmd  hello  wasm1  wasm2  wasm3

And it works great if I rebuild into the bin directory:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ rm ./bin/cmd ./bin/hello ./bin/wasm2

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

Is this intended to work with non-empty parent directories of the binaries? Why does it work with GOOS=windows GOARCH=amd64 but not GOOS=js GOARCH=wasm?

@bcmills
Copy link
Contributor

bcmills commented Feb 26, 2020

@Gregory-Ledray, please open a new issue with steps to reproduce. Probably there is something special about wasm binaries that is interfering with the feature.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests