Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: optimize CGO caching and dead code removal #38886

Open
Keithcat1 opened this issue May 5, 2020 · 9 comments
Open

cmd/go: optimize CGO caching and dead code removal #38886

Keithcat1 opened this issue May 5, 2020 · 9 comments
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@Keithcat1
Copy link

I'm writing this with the assumption that each CGO-generated wrapper for each C function is stored in its own C file.
Editing and compiling a CGO package can take a minute or 2, even for one little change.
It should be possible to cache the generated .o files and only recompile them if the code for that function changes.
When linking the CGO wrappers, Go should avoid linking unused wrappers to reduce size.

@ianlancetaylor ianlancetaylor changed the title Optimize CGO caching and dead code removal cmd/cgo: optimize CGO caching and dead code removal May 5, 2020
@ianlancetaylor ianlancetaylor changed the title cmd/cgo: optimize CGO caching and dead code removal cmd/go: optimize CGO caching and dead code removal May 5, 2020
@ianlancetaylor
Copy link
Contributor

One C file is generated for each Go file that uses import "C". It is not the case that we use a separate C file for each C function wrapper. Though I suppose we could if it seems useful.

CC @bcmills @jayconrod

@ianlancetaylor ianlancetaylor added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 5, 2020
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone May 5, 2020
@jayconrod
Copy link
Contributor

Related #9887

Currently, I think we cache the generated .c files but not the .o files produced by the C compiler.

We should have a separate action for each .c file compilation, and we should cache the outputs of those actions.

@ianlancetaylor
Copy link
Contributor

Note that proper caching of C object files require detecting whether any included .h files change. This does arise when people update to a newer version of some C dependency.

@jayconrod
Copy link
Contributor

Good point. We should be incorporating hashes of included .h files into the cache keys for compiled .a files at least. I don't believe we currently are.

@Keithcat1
Copy link
Author

Maybe we could only run the C compiler when the package is actually being compiled in order to get the values of C constants and make sure C functions are called properly.
Then, at link time, the Go linker linkes the entire program instead of using the C linker. We could probably use syscall to call C functions at runtime from CGO Go code wrappers.
This would have several benefits. Faster linking speed since no C linker is being invoked, the Go linker probably wouldn't need to spend time building a C linker compatible object file, dead code elimination could probably be done even for statically linked C code (at least on a per object file basis), and cross compiling might be supported if you have static libraries for other platforms. The Go package "github.com/veandco/go-sdl2/sdl" provides static libraries for all the platforms SDL2 runs on.
The linker would need support for reading a few different formats of object files though.

@ianlancetaylor
Copy link
Contributor

We run the C linker, rather than using only the Go linker, because the Go linker doesn't support C++ global constructors or C++ exception handling or a few other cases that the C linker handles. We could support those things in the Go linker, but it's not trivial.

@docxplusgmoon
Copy link

@Keithcat1 How you solved this issue?

@Keithcat1
Copy link
Author

By putting CGO code in sub-packages so if you make a change to the main package, it won't recompile those packages that use CGO unless you changed them

@cherrymui
Copy link
Member

The Go linker does support reading C object files in various formats (ELF, Mach-O, PE, etc.) on some platforms. But as @ianlancetaylor pointed out, the support of C linking is limited, and it doesn't support fancy C/C++ features. If you're sure your C code doesn't use any of those, and you're targeting a platform that supports internal linking, you can try passing -ldflags=-linkmode=internal, then the Go linker will link the C code and produce an executable without using the C linker. (Note: that linking may fail, use with caution.)

That does not get deadcode elimination on C code automatically, as C object semantics is section based, not function based. Perhaps you could try using C compiler options -ffunction-sections and -fdata-sections. (Go linker's support for it is not well-tested, use with caution.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants