What is the specific change? Cgo already understands response files for command line arguments; the problem here is that the info is being passed to cgo in $CGO_LDFLAGS, and that's exacerbated by the go command and Bazel rules_go both not deduplicating the CGO_LDFLAGS info they gather from every cgo-using package in the build. (If 10 packages all say the same CGO_LDFLAGS, no need to say that part 10 times, just say it once.) Perhaps they should both deduplicate and then the length limit is not a problem anymore.

We may need to deduplicate some linker arguments to appease the Mac linker anyway.

Our CGO_LDFLAGS got huge, using a response file avoided the overflow. Can post the PR we used, this fixed the problem.

I had tried de-duplication, but there were some ordering dependencies in the linker command.

How did you use a response file at all? Response files are for command-line arguments, not environment variables.

If we need to do something, we should add a -ldflags argument to the cgo command, and then that can be put in a response file. Interpreting @ signs in environment variables is a big scary door to open.

The linked bazelbuild/rules_go#2654 has an actual CGO_LDFLAGS line in the top comment. It is one line but it is very very long and GitHub has added a scroll bar, so you don't notice how long it is.

That CGO_LDFLAGS has:

• 843 .lo files, each wrapped in -Wl,--whole-archive ... -Wl,--no-whole-archive.
• 207 .a files, without that wrapping.
• 75 -pthread
• 4 -lpthread
• 3 -lm
• 2 -ldl
• 1 -static-libstdc++
• 1 -static-libgcc
• 1 -lrt
• 1 -l:libstdc++.a
• 1 -fuse-ld=/usr/bin/ld.gold
• 1 -Wl,-z,relro,-z,now
• 1 -Wl,-no-as-needed
• 1 -B/usr/bin

So deduplicating is not going to cut it. My suggestion is we add -ldflags to cgo, and then the go command and Bazel can use the support cgo already has for response files on its command line.

How did you use a response file at all? Response files are for command-line arguments, not environment variables.

In short, CGO_LDFLAGS accepts @file in the same way as an executable would , expanding the arguments from the given file.

Support for this in cgo is needed to fix the corresponding open problem in @rules_go (bazelbuild/rules_go#2654):

• @rules_go collects all the linker flags into CGO_LDFLAGS and then
• calls cgo with this environment in go/tools/builders/cgo2.go.

The fixed environment-variable length limit on Linux/Windows makes it currently impossible to compile larger projects.

Thinking about whether -ldflags would help - there is also a maximum commandline length.

Thinking about whether -ldflags would help - there is also a maximum commandline length.

It would help because cgo accepts response files on the command line already.

Sorry for taking so long to understand - if adding -ldflags is an equivalent solution to passing via CGO_LDFLAGS in @rules_go, then this is clearly preferable.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Thank you.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long.

Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long.

Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem.