I would humbly submit that this is not a Go 2 proposal, since the current standard compiler behaviour is not actually called for in the Go Programming Language Specification, and even if it was intended to be, this proposal is not a backwards compatibility breaking change to current practise.

Can't we already effectively do that ?

I mean, ok you can't literally do what you are proposing, but you could have the Main function but in it's own non main package and have your program and have both the mega-main and unique main packages both import that.

I imagine something like this:

// cmd/go/go/go.go
package main

import gocmd "cmd/go"

func main() {
 gocmd.Main()
}

The examples you gave would be identical.
That would allows to not complexify the current simple-ish rules around importing main.

This would require some automated rewrite step similar to what gobusybox do.

I think it is a language change, as it defines new visibility rules for main packages (an alias Main, no other "exported" identifiers actually exported).

I think this may be an unsafe thing to do in general:

• main programs may often be written to assume it has full control, e.g. registering global resources in package initialization, just importing the packages can lead to conflict.
• extended backwards compatibility for Go #56986 allows main control over //go:debug, but this proposal means the program may run under unintended configurations by default.

While it's true that similar issues can arise with poorly written importable packages, I'd argue that the importable / unimportable separation of intent is important, and for the effort needed to fix those main packages it isn't much more work to move the code into an importable one.

Hi @Jorropo,

Can't we already effectively do that ?

I completely agree! For my own programs I also like to use the structure of a main function in cmd/program/main.go that just calls a single exported identifier from cmd/program/programcmd.

But what about third party programs, and being able to bundle them into a single monolithic binary? If they could be changed to adopt that structure, all the better. But what if that's not possible?

I'd even be happy if there were some kind of compiler-specific mechanism to support this use case:

go build -bundlemain cmd/go -bundlemain cmd/gofmt -bundlemain you/pkg/cmd/program ... multitool.go

... and then some kind of func RunMain(package string) call to actually invoke them. Even if there were lots of further restrictions, such as RunMain will never return to the calling function (so more like an exec style syscall), if that made things easier to implement.

@griesemer ever wanted to support this, but @rsc denied it.
So the possibility is gone now.

I couldn't find the related issue now.
[edit]: found that issue. But sorry it looks that issue doesn't talk about whether or not main packages can be imported or not.

Hi @seankhliao,

I think it is a language change, as it defines new visibility rules for main packages (an alias Main, no other "exported" identifiers actually exported).

I agree that it is a language change, but I believe it could be backwards compatibility preserving change to Go 1. Maybe I am misunderstanding the meaning of "Go 2" in this context (which I thought would refer to truly breaking changes, such as if I were proposing that the main function should in practise be renamed to Main, which I am most certainly not).

I think it would be helpful to clarify the current intention of the language spec:

1. Have I simply missed the part where it talks about the non-importability of package main?
2. Is the non-importability of main something that the language spec intentionally has no opinion about, and hence implementations are free to impose additional restrictions if they wish (as the standard compiler does)?
3. Setting this proposal aside for the moment, should the spec be updated to explicitly mention the non-importability of main, to align with current practise?

I think this may be an unsafe thing to do in general:

I completely agree! There are zero guarantees that, once your program calls into another main function (if it were possible to do so), that the function will ever return back to the caller, or even if it did, that the system won't be in a very strange state. There are also no guarantees that importing two different main's won't just by themselves have unintended consequence, such as with respect to init execution order and global mutable state.

But those issues would be the callers responsibility to sort through. It would not affect the behaviour of any existing programs.

If an open source project received a bug report about "strange behaviour when importing your main" they could quite fairly say "we don't support that, won't fix".

@pdmccormick The "Go 2" term in issue titles, as well as the v2 label, are to facilitate triage and search. They don't imply that a proposal is not backward compatible.

@pdmccormick The "Go 2" term in issue titles, as well as the v2 label, are to facilitate triage and search. They don't imply that a proposal is not backward compatible.

Thank you for the clarification @zephyrtronium ! I appreciate it.

Thank you all for you input.

Starting from the origin point of the current import "cmd/foo" is a program, not an importable package error message, and searching back through the history, I came across issue #4210 . It looks like in Go pre-1.5, it was possible to import main, as @kevinwallace commented:

I'm currently using this to build busybox-style binaries. Each command's main package exports a reference to its main function, and a wrapper command imports them all and selectively calls one, switching on os.Args[0].

Further comments in the thread refer to solving the newly introduced restriction by reorganizing programs to split between the package main and an importable implementation package.

Later in that thread, @ianlancetaylor commented:

CL 4126053 is a change to the spec, which describes the language. The language permits importing a package named main. That can be used, for example, when writing unit tests for functions in commands that use package main.

This issue here is about the go tool, not the language. The question is whether the go tool should permit packages to import a package that defines a command. The general consensus is that it should not.

CL 4126053 specifically removed a spec-level prohibition against importing a main package. Therefore I would conclude that what I am proposing is actually a behaviour change to the standard compiler, not a language specification change.

Given this, would it be worth creating a new issue proposing some kind of "busybox-style multi-program build" support in the standard compiler? Whether by permitting restricted main imports, or by some other way of invoking another programs main. Would there be any willingness to consider such a proposal, @rsc ?

As you say, this seems to be a reversal of #4210. The discussion there included the busybox case. What has changed that we should reconsider that earlier decision?

As you say, this seems to be a reversal of #4210. The discussion there included the busybox case. What has changed that we should reconsider that earlier decision?

The comment in #4210 that first mentioned Busybox-style binaries also mentions that they had the alternative remedy of splitting their main function implementations out into importable packages, which has always been an option when working within a project you have some influence over.

But what about the case of bundling together existing & unmodified third party programs, to share the cost of common code that would otherwise be duplicated across multiple binaries? With the rise of Go-based Linux userlands such as u-root and gokrazy, a Go-based Linux distribution (or embedded system image) could pull together an entire suite of system tools and useful applications with a single go build command.

I probably jumped too quickly to suggesting a specific solution (allow importing main packages again) when really the question I'm grappling with is, could the Go compiler help me in my quest to build multiple programs into a single executable, and offer a Go exec(3)-like mechanism to replace the currently running program with a new one?

At the risk of jumping to solutions again, perhaps something like the combination of allowing import _ "some/pkg/cmd/foo" from main (to pull the contents of some/pkg/cmd/foo into the current program build, and to help ensure that go mod tidy will include referenced packages in go.mod) plus a runtime API such as func ExecProgram(package string, args, environ []string) error to reboot and reinitialize the runtime?

It seems like this proposal hinges on a more philosophical question about the design goals of the language:

Should it be possible for someone to embed someone else's standalone Go program as if it were a library in someone else's Go program without first negotiating an explicit library API with the author of the first program?

Everything else in Go so far seems to have tilted toward giving authors control over their exported API: package symbols can be either exported or unexported, and the internal directory convention allows entire packages to be either exported or not. The language does not offer any explicit means for a caller to override those decisions, although I will concede that there are some seemingly-accidental workarounds available for those willing to depend on implementation details of the Go toolchain.

As an author of one such program that has intentionally not exported any library API, I would say that I would prefer this not to be accepted in the current form. I would not be opposed to an alternative that makes it opt-in for the author of the program being embedded, but I'm not really convinced such a thing is needed because it's already possible for a Go module to offer both a package main entry point and a library endpoint at the same time if an author decides to do so. This proposal seems to be largely motivated by doing something in spite of the author of the program it's being done to, which doesn't seem consistent with the design goals of the language so far.

For the specific use-case of "build something like BusyBox", I would suggest that the author of such a thing ought to design an explicit API through which a Go module can expose one or more entry points with a specified signature, and then encourage authors of programs that would make sense to include in such a bundle to expose those entry points. Not all authors would want to participate in such a thing, but I expect that many would if the argument for it were compelling enough.

Should it be possible for someone to embed someone else's standalone Go program as if it were a library in someone else's Go program without first negotiating an explicit library API with the author of the first program?

Thank you @apparentlymart for the thoughtful comments. You have helped persuade me, importing main is probably not the right approach for achieving my goals here, and it is certainly not a popular option either.

In the case of the program you mentioned, the one that you did not want to expose a package-level API for: you wouldn't have any objection to it be invoked as a program though, would you? i.e. as an OS-level process, with os.Args and os.Environ, possibly stdin/stdout/stderr, with the usual Go runtime startup process that calls your main function. If it were possible to interact with that runtime machinery directly, it could be used to implement a Busybox-style multi-tool executable (as opposed to dispatching to other main's through a regular function call).

Indeed, my concern is that running a program that expects to be in full control of its process as if it were a library inside another process would likely cause it to behave differently in that context, and that those different behaviors would then implicitly become part of the functionality that program is offering and create (possibly undesirable) pressure to keep it working in that context, despite the differences.

If you were able to create an execution environment that is completely indistinguishable from the program being launched directly as a child process then I would not consider that to be problem, but I'm not convinced that the proposal as currently written would guarantee that outcome.

From what you've described in your latest comment, I'm imagining something that is more like a replacement for the Go runtime's startup code, presumably not actually written in Go so that it can run before a Go execution environment is available, which does whatever special logic it needs to decide which of many package main to use, and only then prepares the runtime in the same way that the current implementation does, taking care to ensure that there is no way that the program being launched could observe that it's being launched by that different runtime once it's up and running.

In particular (but not limited to):

• The multiple linked programs are unlikely to agree on exactly which version of each dependency they are linking. Therefore the linker and loader would need to tolerate having multiple versions of the same symbol in the program and select the appropriate versions to dynamically link based on which specific program was selected.

  (I'm assuming it would be a goal to be able to share the code for dependencies between multiple programs in cases where they do agree, because otherwise this would not have significant advantages over just executing the target programs directly using the facilities provided by the OS, such as exec on Unix systems.)

• All of the init functions must run in exactly the same sequence as they would when the program is executed normally, including the implicit init steps that result from assigning an expression to a package-level variable.

• No additional Goroutines or OS threads are running compared to what would happen if the program were executed directly.

• panic behaves in exactly the same way when not intercepted by a recover, printing exactly the same messages to the same handles and exiting with the same exit code.

I realize that this is quite a pedantic set of requirements and that many programs would not actually be affected by differences here in practice, but nonetheless all of these things could materially affect a specific program's behavior, in ways that the author of that program may not be aware of until it's too late.

@apparentlymart Great points, especially about the need to preserve init execution order.

Keeping with the idea of a program invocation style interface, what about a mechanism for restarting the current Go runtime into a different program? Similar to how an exec(3) syscall replaces the currently running OS level process with a new one. In the runtime package, something like func ExecProgram(package string, args, environ []string) error. The only data you can pass across the call would be values for os.Args and os.Environ (plus OS level process state such as open file descriptors and namespaces).

You raise an excellent point about multiple package versions. If a project opted-in to this kind of arrangement, and split out an importable cmd package for its programs, the module system would have to work out the dependency versioning when combining multiple cmds. There's no guarantee that a program would work with those requirements compared to its own local go.mod. If the compiler needed to build multiple versions of the same packages, the compiled code sharing and executable size savings start to go out the window, so why bother with any of this.

Assuming that the bundled programs would still function correctly when their package requirements were resolved together, specifying the programs to bundle could be provided as flags to go build:

go build -bundle pkg1/cmd/foo -bundle pkg2/cmd/bar ./cmd/gopherbox/

In this program, pkg1/cmd/foo, pkg2/cmd/bar and main could be passed to ExecProgram.

Alternatively, at the risk of circling back to this topic, maybe a special case of main importing could be permitted, not for initializing side effects but just to declare that it should be bundled into the given build:

package main

import (
	"os"
	"runtime"
	_ "cmd/go"
)

func main() { runtime.ExecProgram("cmd/go", os.Args, os.Environ()) }

This is introducing an irregularity to how imports are to be treated, and that may require a change to the Language Specification (although, as I have learned in this process, the Go language spec does not actually disallow main imports, just the standard compiler). At least this way go mod tidy and tooling would be aware of the dependencies.

Note that at least on Unix systems we do have https://pkg.go.dev/syscall#Exec .

Note that at least on Unix systems we do have https://pkg.go.dev/syscall#Exec .

If I'm not mistaken, the Go runtime does not currently have a notion of multiple main entrypoints within a single executable, or does it have something along those lines?

No, it does not. It always starts at main.main. Perhaps I misunderstood the suggestion.

No, it does not. It always starts at main.main. Perhaps I misunderstood the suggestion.

I'm sorry for jumping around with ideas in this thread, but thank you for engaging with me on this.

To summarize, I'm wondering if it would be possible to enable:

1. go build to include multiple program entrypoints, and
2. Give programs the ability to teardown the current Go runtime and restart it back up again fresh but from a different entrypoint (and with new argv/envv)

So a Go runtime level exec call, if you will, all from within the same executable and same running process.

I think I get what you're thinking about here, and maybe the following set of analogies would both allow you to confirm whether I've understood correctly, and if so help others in this thread reach a similar point of understanding.

• kexec is a Linux kernel system call that effectively makes Linux behave as a bootloader for another version of itself, effectively replacing the running kernel with a new kernel but without the overhead of rebooting the computer. The intent is that nothing about the original kernel survives, and the new kernel believes it was booted normally.

• exec is a system call on many Unix-derived systems that effectively replaces the current process memory image with a new process image and begins executing it. The intent is that nothing about the original process image survives, and the new program believes it was started normally. (In this case, it was started normally, because exec is the normal way of starting programs!)

• The evolved form of this proposal (now quite different than the original writeup) is a Go runtime function runtime.ExecProgram that replaces the running Go program with another Go program that was somehow linked into the same executable image as the caller, with the new program believing it was launched directly.

  However, this new idea differs from the previous one by promising that if two of the programs linked into the same executable agree on a particular dependency version then that dependency will be included in the executable only once, as an executable file size optimization. So it's a sort of Go-specific exec that allows the target program(s) to be bundled inside the same executable.

  Presumably that means that this new runtime function is subsuming some of the responsibilities that would normally be handled by the linker at build time, since it'll need to decide what to map into memory and fix up calls between packages at runtime instead of at build time.

(I'm intending this comment only as an attempt to confirm that I'm understanding the new proposal correctly, not as any specific opinion on the new proposal.)

In terms of a concrete example for this Busybox-style multitool executable, and possible storage savings:

Two Docker plugins that are typically installed together are buildx and compose (I have no affiliation with Docker). When I build each of these, the binaries are 77 MiB and 75 MiB respectively. Next, I have locally modified each of these projects to move the contents of their primary main.main into an importable subpackage, github.com/docker/buildx/cmd/buildx/buildxcmd and github.com/docker/compose/v2/cmd/composecmd. Then, in a new program I have imported both of these, and dispatch to each depending on the contents of os.Args[0]. Finally, after compiling, I created symlinks, buildx => multitool and compose => multitool, which I use to invoke the specific program. This resultant binary is only 80 MiB, for a savings of 72 MiB compared to two separate executables.

This might be a bad idea for other reasons as discussed above, but it's basically those savings that I am chasing. I will be trying to see if the authors of the particular programs relevant to me would be willing to accept that kind of main/cmd split, but the motivation for this entire proposal was, could this be supported in the toolchain itself, to allow bundling together collections of Go programs into single executables.

I think I get what you're thinking about here, and maybe the following set of analogies would both allow you to confirm whether I've understood correctly, and if so help others in this thread reach a similar point of understanding.

Yes, you've understood me perfectly, thank you!

And you're absolutely right, the discussion has moved on quite a bit from the original proposal. I am happy to withdraw it, or have it withdrawn by someone else, if that is the appropriate course of action for this process?

• Presumably that means that this new runtime function is subsuming some of the responsibilities that would normally be handled by the linker at build time, since it'll need to decide what to map into memory and fix up calls between packages at runtime instead of at build time.

This is a great point: if any kind of runtime patching would be required to switch between programs then I'm definitely thinking that this is starting to seem like a bad idea.

If the results were not the equivalent to what I did in my Docker example above, existing programs which are modified to explicitly opt-in by adding a boilerplate main/importable cmd split, and then having those program packages pulled into a new program as dependencies, if it would not be possible to get some level of compiler support for achieving that without the boilerplate, with the least possible changes to anything else in the ecosystem or runtime, then I think this idea should be abandoned.

Importing main packages was in the distant past permitted, but the tools were changed to disallow it, and the rationale for that change applies to this proposal too.

Therefore, this is a likely decline. Leaving open for four weeks for final comments.
— adonovan for the language proposal review group

No further comments.