Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: generating C header files for transitive dependencies is awkward #35715

Open
bcmills opened this issue Nov 20, 2019 · 7 comments
Open
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 20, 2019

As discovered in CL 208117 (#30316), cmd/go (as of Go 1.14) lacks a good way to generate C header files for Go libraries in order to use them from C.

go install -buildmode=c-archive and go install -buildmode=c-shared will, as a side-effect, generate header files for the requested package. However, -buildmode=c-archive and -buildmode=c-shared today require the named package to be a package main, and those commands do not generate headers for the (transitive) Go dependencies of the named package.

The -i flag does generate headers for transitive dependencies, but it also has a second, unwanted side effect: it builds (and attempts to install) libraries for the transitive dependencies of the named package, including transitive dependencies in the standard library. That fails if the user cannot write to GOROOT/pkg, such as when a non-root user is working with a go tool installed as root.

As a workaround, one can run go tool cgo -exportheader, but go tool cgo accepts a list of files rather than a list of packages, and generates a number of additional outputs (normally in an _obj subdirectory) that the C caller does not directly need. So we're still doing extra work, just in a different location.

We should see what we can do to make it easier to generate C headers without unnecessary or unwanted side-effects.

CC @ianlancetaylor @jayconrod

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go labels Nov 20, 2019
@bcmills bcmills added this to the Backlog milestone Nov 20, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Nov 20, 2019

One possibility might be to add another -buildmode, such as -buildmode=c-header, that does whatever work is needed (and possibly writes extra artifacts into GOCACHE) but only emits (or installs) header files rather than entire archives.

That could be used as a more direct substitute for the go tool cgo call with specific package arguments.

@bcmills
Copy link
Contributor Author

bcmills commented Nov 20, 2019

Another option might be to make the -i flag suppress errors for standard-library targets, so that the go install -i command would work even if GOROOT is not writable.

@gopherbot
Copy link

Change https://golang.org/cl/208117 mentions this issue: misc/cgo/testcarchive: avoid writing to GOROOT in tests

@gopherbot
Copy link

Change https://golang.org/cl/208119 mentions this issue: misc/cgo/testcshared: avoid writing to GOROOT in tests

@ianlancetaylor
Copy link
Contributor

We could have go install install all headers for imported packages, even without -i. That would be a change in behavior, but only for packages that are not main but have exported functions, which is an uncommon case.

@bcmills
Copy link
Contributor Author

bcmills commented Nov 22, 2019

Maybe, but the import itself seems like a bit of a non-sequitur: if the C code is using identifiers from package a, then it seems like the build process should mention package a rather than some other package b that happens to import a.

On the other hand, I suppose that the C library must link against some (shared or archive) file generated from a main package, so having go install generate all of the headers for identifiers linked into that main package doesn't seem terrible.

gopherbot pushed a commit that referenced this issue Nov 22, 2019
Also add a -testwork flag to facilitate debugging the test itself.

Three of the tests of this package invoked 'go install -i
-buildmode=c-archive' in order to generate an archive as well as
multiple C header files.

Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_shared, which may not be writable — for
example, if GOROOT is owned by the root user but the test is being run
by a non-root user.

Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.

In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.

Updates #28387
Updates #30316
Updates #35715

Change-Id: I3d483f84e22058561efe740aa4885fc3f26137b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/208117
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Nov 22, 2019
The tests in this package invoked 'go install -i -buildmode=c-shared'
in order to generate an archive as well as multiple C header files.

Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_testcshared_shared, which may not be writable
— for example, if GOROOT is owned by the root user but the test is
being run by a non-root user.

Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.

In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.

Updates #28387
Updates #30316
Updates #35715

Change-Id: I622426a860828020d98f7040636f374e5c766d28
Reviewed-on: https://go-review.googlesource.com/c/go/+/208119
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@crockeea
Copy link

crockeea commented Mar 4, 2021

Any movement on this? I'm trying the workaround suggested by bcmills@ of using go tool cgo -exportheader, so my build script includes:

go tool cgo -exportheader ${CGO_BUILD_ROOT}/foo.h foo.go
&& go tool cgo -exportheader ${CGO_BUILD_ROOT}/bar.h bar.go
&& go build -buildmode=c-shared -o ${LIB_FULL_PATH}
&& rm -rf _obj

where both foo.go and bar.go are in the same package. This was working fine until I tried to use a type alias.

I'd like to define a type alias type T1 = T2 and use that alias in both foo.go and bar.go.

If I define the alias in foo.go, go build works fine, as expected (T1 is in scope in bar.go). However, exporting headers for bar.go fails with unrecognized Go type T1.

If I define the same alias in foo.go and bar.go, exporting headers succeeds for both files, but I get a redeclaration error with go build (not surprising).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants