This is exactly what I wanted to do ad-hoc for SQLite, would love to see it!

https://twitter.com/FiloSottile/status/1245831899290820608

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

This actually means it could be a fully external tool, at least for prototyping. Indeed, the thing that needs toolchain support is #38918, which is where things get interesting.

@FiloSottile I had your SQLite and x509 work in mind when filing this proposal :) Perhaps you'd like to join the next golang-tools call May 13th for further discussion?

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

This actually means it could be a fully external tool, at least for prototyping.

Indeed. What I'd like to achieve first is some indication that work towards pre-generation may end up in the main repository. I don't fancy maintaining an out-of-tree tool.

I don't really understand how to do this in general.

cgo packages can and do refer to external libraries; presumably we would have to find all of those external libraries and make them available somewhere. But the libraries are going to vary a lot. How do we find them and where do we put them?

Some of those libraries are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

Maybe I'm missing something, but while I can see how this could work in the simplest cases, I don't see how it could work in general.

I don't really understand how to do this in general.

cgo packages can and do refer to external libraries; presumably we would have to find all of those external libraries and make them available somewhere. But the libraries are going to vary a lot. How do we find them and where do we put them?

Can we ask the native toolchain while pre-pregenerating, in a similar way Cgo currently asks the toolchain about the structure of native types, functions etc.?

Some of those libraries are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

Maybe I'm missing something, but while I can see how this could work in the simplest cases, I don't see how it could work in general.

My use-case is system libraries, which are usually designed to have simple C (or Objective-C) interfaces as well as being impossible to replace with Go by definition. Having pre-generation work on a best-effort basis would be ok for me, I would simply not provide pre-generated files for the platforms/libraries that are not (yet) supported.

I am looking forward to this feature, but there seems to be no progress.

Is there a simple way to implement it? Pack the intermediate files generated by gcc into a zip, and continue to use a third-party linker. The main trouble with C language cross-compilation is various dependencies and libraries. I don't know much about compiling and linking, and I might understand it incorrectly.

If you have to have the third party linker and the third party libraries anyhow, then I don't think this idea is going to help very much.

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

Even this may be unnecessary - it should be possible to precompile CGO code to Go assembly. This would then reduce the scope of the problem to creating a CGO -> C .S assembly -> Go .S assembly tool, which would presumably be run via go generate. The only change to the build toolchain would probably then be a magic comment that defines shared libraries to link to.

This way, CGO can be moved from the build process to the go generate process and become an internal detail purely for developers of CGO packages, making it more in line with other code generation projects. This would also resolve the difficulties of reproducible builds when using system libraries.

I don't really understand how to do this in general.

cgo packages can and do refer to external libraries; presumably we would have to find all of those external libraries and make them available somewhere. But the libraries are going to vary a lot. How do we find them and where do we put them?

Some of those libraries are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

Maybe I'm missing something, but while I can see how this could work in the simplest cases, I don't see how it could work in general.

I think those simplest cases are all that's needed, at least as far as C system libraries (arguably the most important use case) are concerned.

For context, there are plenty of Go projects that involve automatically generated bindings to C libraries, such as go-gl. It would be amazing if they also gained the ability to precompile themselves to all supported systems from CI, giving Go developers the newfound ability to cross-compile windowed applications.

I think there wouldn't even be any need for new .cgo file types, it should be possible to just generate .go and .syso files that build with the current toolchain. It might even be relatively stable: the ABI is fixed, so the only unstable internal is asmcgocall.

Even this may be unnecessary - it should be possible to precompile CGO code to Go assembly. This would then reduce the scope of the problem to creating a CGO -> C .S assembly -> Go .S assembly tool, which would presumably be run via go generate.

Some prior art for this approach can be found at c2goasm, with an article describing it here: https://medium.com/@frankwessels_nl/c2goasm-c-to-go-assembly-bb723d2f777f

The only build system change required would probably then be a magic comment that defines shared libraries to link to.

After some further investigation, it looks like this is already how CGO does linking today:

With that in mind, how feasible would it be to implement precompiled CGO packages via a go tool that takes a list of files that import "C", invokes the system C compiler on them, and then translates the resulting C assembly into Go assembly with //go:cgo_dynamic_linker and //go:cgo_import_dynamic directives?

👋 Hey. Coming from the dd-trace-go library where we wanted to add a C library bindings at some point where we had already a large user base and adding CGO requirements for all these users where out of question. We studied a lot of different options and we decided to use a tool called purego that uses a lot of internal mechanisms of the runtime directly. This has been working for the last 6 months already but having an official method to not impact users in case a library decides to start using CGO certainly seems more reliable. Looking forward to see this happening.

In general this is very difficult / impossible. One of many difficulties is keeping both static and dynamic builds working correctly. If you need C code, use cgo.

This proposal has been declined as infeasible.
— rsc for the proposal review group

As @eliottness shared, this is more or less already feasible and this is how we make it in dd-trace-go for the record:

1. Compile your CGO bindings with the options -x -work to be able to copy the CGO-generated files, especially the Go type definitions generated out of the C ones.
2. Copy and adapt them to work without CGO, since a lot of things in there will rely on compiler directives that don't work without CGO_ENABLED=1.
3. Build your C library as a shared library that you can embed in your binary with go:embed.
4. Use purego, which is a higher-level helper runtime/cgo package, to open the shared lib and call it, the same way as CGO would, as it relies on runtime/cgo too (present with or without CGO_ENABLED).