This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

I really want to use this buildmode=shared mode.

@rsc how has it been broken since modules?

Just to clarify, this is not related to --buildmode=plugin, right?

@electricface Why do you want to use it? Are you using it today?

@scott-cotton You can't use -buildmode=shared with a module and then use -linkshared to link against it from other modules. The whole idea of -buildmode=shared doesn't fit very well with modules. Each separate program can be linked against different versions of dependent modules, so if it did work people would in practice wind up with many different versions of the shared libraries of the same modules.

@ainar-g Correct, this is not about -buildmode=plugin, although plugins have their own, different, problems.

@ianlancetaylor
I am a debian-baseed Linux Desktop Environment developer. Our team have developed lot of daemon programs with golang. But these programs have significantly larger binary file size and use more ram than C or C++ programs. We currently working on these two problems. An idea is making some modules pluggable, which means only load modules that we need.

As other C or C++ programs are using shared-library, we are researching on some shared-library-like mechanism of golang. Inspired by buildmode=plugin and goloader, we found a way that bases on buildmode=shared to make modules pluggable recently, you can found a demo here. This solution has not been using in production environment, because of some problems hasn't been solved yet. I found out that programs built with -linkeshared can not be debugged using gdb and dlv.

After this commit, our "unoffical plugin" solution can work fine with go 1.14.9~1.16.7. Programs built in this way can run stably. But this won't work on go 1.17. Program will exit with a "fatal error: unreachable method called. linker bug?"

Plan of remove buildmode=shared make me feel bad.

@electricface It sounds like you want plugins, which are supported (though not very well) by -buildmode=plugin. We aren't going to support -buildmode=shared just to support plugins, since we already have -buildmode=plugin.

@ianlancetaylor
I know that using static linking can eliminate the problems caused by the mismatched version of the dependent dynamic library.
But in the linux desktop system ecosystem, C or C++ languages are mostly used. These languages generally use dynamic libraries to save memory. The main reason for not using -buildmode=plugin is that static linking cannot save memory. For example, if the modules are divided into smaller ones and a lot of modules are added, each module occupies a relatively large amount of memory, and the overall occupancy will eventually become unacceptably large. Compared with C or C++ languages, they only use less memory to implement these functions, while the golang implementation uses several times the memory.
There is even an idea in our team that we must change the development language if we can't reduce the memory. I personally like to use golang for development, so I plan to study this part of the technology.

"I want to use it" is different from "I am using it today".
We believe that it is not possible to use it today, and furthermore it is difficult to provide.
We shouldn't advertise something that doesn't actually work.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

Just to confirm, this proposal suggests to remove --buildmode=shared while keeping --buildmode=c-shared? I am using --buildmode=c-shared for a number of projects and would be very sad to see it go.

This proposal is only about -buildmode=shared.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc not sure if this is a good place to give feedback about the proposal process, but this is the first time (since the proposal process exists) that I missed an issue. I was mildly interested in this (not enough now to vehemently protest), but even if I'm subscribed to the proposal process issue and I normally read minutes every week, I missed this. I took 10 days off for vacation, and those 10 days covered the only 2 emails where this issue was mentioned, before today when it was approved. Again, I don't mind about this specific issue at this point but I wanted to raise a general point.

Maybe a proposal should stay a minimum time in the proposal process before being closed, so that even people that don't work on Go full time have time to notice and react even with personal life happening in-between.

To limit binary file size, I use command go install -buildmode=shared std to install libstd.so. And I build my application by go build -linkshared. I have been using this in production environment for a long time. I hope I can still use it in the furture.
Can we keep -buildmode=shared std, it's useful.

Our company has been using the shared mode to reduce the binary size for a long time. The plugin mode is not useful in some scenarios cause the entire runtime package needs to be compiled and packed. So we strongly hope that shared mode can be reserved.

Writing a wrapper around LGPL C libraries is a "work that uses the library." The C libraries can continue to be shared libraries. There is no need for the Go code itself to be in a shared library. That works today and will continue to work.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

That is not our intent. It is fine for Go programs to use GPL libraries. It is fine for Go programs to use LGPL libraries, as shared libraries, if those libraries are written in other languages. It's true that it's hard for Go code to be licensed under the LGPL; it is in effect the same as being licensed uder the GPL. But that doesn't seem so bad as even the FSF recommends against using the LGPL (https://www.gnu.org/licenses/why-not-lgpl.html).

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

And for what it's worth I'm a lot more than 46 years of age.

@ianlancetaylor thanks for addressing our concern but,
A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

A solution would be directly using that C-shared library with cgo and avoid the wrapper
The difference is the programmers should now know the inner workings of the C library.

the ultimate solution is just to write the entire LGPL/GPL code in Go
Too much work I suppose

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

very well understood, Thank you

A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

Yes, but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license. The C code, under the LGPL, will be in a shared library, and the Go wrapper, not under the LGPL, will not.

@ianlancetaylor

but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license

No sir, that's your technical opinion as an engineer, not a legal opinion

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

This is based on LGPL 2.1 otherwise 3.0 is even more difficult.

A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application 
programs (which use some of those functions and data) to form executables.

...

 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) The modified work must itself be a software library.

b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change.

c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License.

d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful.

If Go does not support shared library for Go libraries, it is legally safe to say, not to write wrappers around LGPL C code and statically link it, it is very easy to show the user is violating LGPL

No sir, that's your technical opinion as an engineer, not a legal opinion

That is correct. However, it is also my informed opinion due to having worked on free software licensing issues for decades. I was one of many people who contributed to the design of the LGPL in the first place.

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

Yes. In the scenario I described, the LGPL code continues to be dynamically linked. Absolutely nothing changes with respect to the LGPL code. No part of the LGPL code is modified in any way. It continues to be under the LGPL, it continues to be distributed as a dynamically linked shared library. It continues to be possible for the end user to substitute in a different version of the shared library when running the executable.

@ianlancetaylor

Thanks

My problem: Lawyers at our company: Neither GPL nor LGPL has been explicitly tested in a court of law, the linking issue subject to MySQL vs. NuSphere case was settled outside of the court.

So we take the most conservative way; defending in court is too expensive. exceeding the boundaries of compliance to a license is not.

@ianlancetaylor

Some folks contacted me from ITF USA
Saying the GPL / LGPL problem with Go can easily be fixed in the most conservative way:

Sending messages from process A (EULA or any license) to process B (GPL licensed), embracing 100% separation

I guess that's doable,

Thanks for @ianlancetaylor 's help and other folks for their input.

@rsc Is there any issue or discussion to track about your -o proposal here?

My so far hypothetical use case is a multiprocess architecture, with a bunch of small helper programs that possibly depend on C libraries and can be installed separately. Again, it's about size.

I get a feeling that Linux distributions could make use of a "go-rt" package for "std" and link their stuff against that. It would save a bit of bandwidth for utilities.

I've been working on plugin loading and splitting components cleanly (and automatically) across multiple IPC-connected processes, the code is available here: controller-bus.

It could sure use some improvement on the c-shared plugin loading side, (although I know that's unrelated to this issue)

After reading this whole thread it's not clear to me about -buildmode=shared 's future. Can I summarize it as:

1. plugin and c-shared will stay.
2. shared will be limited to use cases as below only:

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

@laoshaw There has not been a final decision but what you describe is likely. More precisely, it's likely that we will only permit a single -buildmode=shared library to be included in a build; it doesn't necessarily have to be the Go standard library although that will presumably be the common case.

Just wanted to check on the status of this. Since the issue is closed, is there another issue/location where I can follow the evolution of this? I don't see it explicitly in the 1.19 beta 1 change log so I'm guessing at best it'll be for 1.20?

@mlaventure I think the status is that we developed an idea (a single shared library) but stalled out on implementing it. Nothing has changed yet and as far as I know there is no new proposal.

@ianlancetaylor thanks for the update. Is there somewhere I can track the progress on this? In the meantime we'll just have to transition to a different language for our systems with stricter memory constraints unfortunately.

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

Unfortunately it doesn't work for me, the compiler panics when I try to build a package with -buildmode=shared -linkshared:

panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).LoadSyms(0xc00079a000, 0x670a56)
	/usr/local/go/src/cmd/link/internal/loader/loader.go:2210 +0x9f
cmd/link/internal/ld.(*Link).loadlib(0xc0000da000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:577 +0x46e
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0x679c5b, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:249 +0xd8b
main.main()
	/usr/local/go/src/cmd/link/main.go:69 +0x1005

This is using go 1.17.11.

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

That is #47455, which could perhaps be reopened — but, to be honest, I would be surprised if it gets much debugging effort until we figure out the future of -buildmode=shared.

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

It used to be the only way I had found to build a shared library and avoid the /usr/local/go/pkg/tool/linux_amd64/link: cannot implicitly include runtime/cgo in a shared library error. I think there was an issue for it around go 1.7, but the issue seems to still be there. The packages I am building do not depend on CGO.

Change https://go.dev/cl/417194 mentions this issue: misc/cgo/testshared: run tests only in GOPATH mode

I've been looking at -linkshared application building recently to see if it's a viable solution for Linux distribution packaging of Go software. The main workflow concept is this:

• Distributions provide a -buildmode=shared package that includes libstd.so (in Fedora this is golang-shared).

  GOROOT=$(pwd) PATH=$(pwd)/bin:$PATH go install -buildmode=shared -v -x std
  

• Applications are built with -linkshared to link against this global libstd.so for the Go standard library.

The idea behind this is to enable updates to Go within the distribution without requiring mass rebuilds of applications due to stdlib patches/fixes. It has the side benefit of creating smaller binaries, but the gains are dependent on other factors as well (e.g. minimal dependencies). However, I do not fully understand the Go build process and the way the compiler works to know if this is even a feasible goal.

• As it stands right now this isn't possible because Go attempts to rebuild libstd.so on each build run, which doesn't lend confidence to a stable shared object being produced, and users building software as non-root generally do not have write access to the global GOROOT.
• I've been told that it's possible for things to change between patch releases of Go (e.g. 1.19.0->1.19.1) that might introduce reliability issues in this process.
• Dependencies using assembly may run into issues
  • For example, trying to build terraform:

    $ GOROOT=/tmp/golang go build -o ./bin/terraform -mod=vendor -linkshared .
    # github.com/klauspost/compress/zstd/internal/xxhash
    asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/mroche/Documents/code/tools/terraform/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s:120)ADDQ	R15, AX
    asm: assembly failed
    

• If my understanding is correct, this wouldn't remove the need to rebuild applications if there's a security fix to the compiler itself, only the stdlib.

Does this sound like a use case that could be (or maybe shouldn't be at all) supported by Go tooling?

@omenos, to my knowledge the Go compiler doesn't currently have any notion of “ABI boundaries”, and in particular it can do inlining of function bodies across packages. So there is a substantial risk of subtle bugs when picking up changes to the standard library and toolchain — particularly if a function fixed in a patch release was inlined into the caller when it was compiled.

Distributions might choose to do

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

This sounds like a great solution. I know I'm late to the game and perhaps it's a bit much to ask ;) What would really help in my use case (single repo/module many separate applications) is a build command that can build multiple binaries in one go, and intelligently bundles all shared dependencies (or just anything not in their respective main packages) into a shared library.
So basically your solution, with automation so the risk of operator mistake is limited.

+1 for shared libraries support with module aware versions

I'm using plugins with their dependencies. And they are very fat. I just want to move the dependencies to a shared library. And I don't want to care about a compatibility etc.

Also, root installation is not acceptable for me too. Currently I'm using separate Go compiler (unlike /usr/lib/go).

P.S. only linux + amd64, don't care any other OS or architecture