Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: go tries to recompile already installed packages: missing headers #27027

Closed
pebbe opened this issue Aug 16, 2018 · 5 comments
Closed

cmd/go: go tries to recompile already installed packages: missing headers #27027

pebbe opened this issue Aug 16, 2018 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@pebbe
Copy link
Contributor

pebbe commented Aug 16, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/tmp/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/peter/go"
GOPROXY=""
GORACE=""
GOROOT="/my/opt/go"
GOTMPDIR="/tmp"
GOTOOLDIR="/my/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build728326710=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to build this program:

package main

import (
	"fmt"
	"github.com/pebbe/dbxml"
)

func main() {
	fmt.Println(dbxml.Version())
}

What did you expect to see?

I expected to get no errors. The package github.com/pebbe/dbxml is installed in GOPATH.

What did you see instead?

# github.com/pebbe/dbxml
c_dbxml.cc:2:27: fatal error: dbxml/DbXml.hpp: Bestand of map bestaat niet
 #include <dbxml/DbXml.hpp>
                           ^
compilation terminated.

Go tries to recompile the package that is already installed, but it fails because the header files are in a non-standard location. The package itself was installed with a modified environment, like this:

DBXML=/path/to/dbxml # location of Oracle Berkeley DB XML
export CGO_LDFLAGS="-L$DBXML/lib -Wl,-rpath=$DBXML/lib"
export LIBRARY_PATH=$DBXML/lib
export CPATH=$DBXML/include
go install github.com/pebbe/dbxml

When I try to compile the example program, these environment variables are unavailable.

This all used to work fine in older versions of Go. I don't when this changed. Go 1.10? Go 1.11?
@mvdan
Copy link
Member

mvdan commented Aug 16, 2018

This is likely because of GOCACHE. If I remember correctly, the go env used to build each package is recorded too, so that changing any settings (such as CGO_LDFLAGS) will properly trigger a rebuild.

You might have better luck using this package as a binary package. I assume the other option is to not change go env.

@andybons andybons changed the title Go tries to recompile already installed packages: missing headers cmd/go: go tries to recompile already installed packages: missing headers Aug 16, 2018
@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 16, 2018
@andybons andybons added this to the Unplanned milestone Aug 16, 2018
@andybons
Copy link
Member

@bcmills @ianlancetaylor @rsc

@pebbe
Copy link
Contributor Author

pebbe commented Aug 17, 2018

Compiling with GOCACHE=off doesn't solve the problem.

@rsc
Copy link
Contributor

rsc commented Aug 18, 2018

The package itself was installed with a modified environment

GOPATH/pkg is not something you can micromanage anymore. It's just a set of 1-entry caches. If they're up-to-date for the requested build, they'll be used; otherwise not. The go command can now tell that relevant environment variables have changed since the package was built (in particular, $CGO_LDFLAGS changed), so it does not reuse the (stale) installed package. In general, a change to CGO_LDFLAGS should trigger a rebuild, so sorry but this is working as intended.

I'm not sure what to say except that you now need to keep those environment variables set the same way from build to build.

@rsc rsc closed this as completed Aug 18, 2018
@pwaller
Copy link
Contributor

pwaller commented Aug 30, 2018

@rsc has a blog post somewhere (which I have failed to find) which introduces the notion of content-based staleness - the closest I could find was this golang-nuts post where he introduces it. But I recall an 'aha' moment when I read that. go install is now just a function of its inputs, giving you a reproducible build, and the cache is "just a cache", not something you can poke. This means that the binaries you get from a build should not depend on what commands happened to run previously.

If I recall correctly, in the post I can't find, @rsc mentioned that initially he thought - and you might think - that 'depending on what happened to run before' was a feature which enabled you to do various things (as in this issue). He came around (and in his writing, I came around) to the idea that actually those problems can be solved through other means, giving us the very useful property that go install is just a function of the inputs you present it.

@golang golang locked and limited conversation to collaborators Aug 30, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@rsc @andybons @pwaller @pebbe @mvdan @gopherbot