Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/ld: relative #cgo LDFLAGS -L does not work #5428

Closed
gopherbot opened this issue May 8, 2013 · 41 comments
Closed

cmd/ld: relative #cgo LDFLAGS -L does not work #5428

gopherbot opened this issue May 8, 2013 · 41 comments
Labels
FrozenDueToAge Suggested Issues that may be good for new contributors looking for work to do.

Comments

@gopherbot
Copy link

by manoj.dayaram@moovweb.com:

go version go1.1rc2 linux/amd64

The symptoms of this are very similar to this issue:
https://golang.org/issue/5205

However, the difference here is that when running "go build my/project" from
within a project directory in GOPATH, everything compiles fine, however, if I'm outside
the GOPATH directories, I get the following errors:

/home/noj $ go build sassy
# sassy
/usr/bin/ld: cannot find -lsass
collect2: ld returned 1 exit status
/home/noj/.gvm/gos/go1.1rc2/pkg/tool/linux_amd64/6l: running gcc failed: unsuccessful
exit status 0x100

If I actually go into the sassy directory, the project compiles fine:

/home/noj/dev/src/sassy $ go build sassy
/home/noj/dev/src/sassy $

Both of these methods of compiling used to work fine with go1.0.3.


What steps will reproduce the problem?
This is a little bit involved, but I'll try my best to provide a full example.
/home/noj $ mkdir -p dev/src
/home/noj $ export GOPATH=/home/noj/dev
/home/noj/dev/clibs/src $ mkdir -p dev/clibs/src; cd dev/clibs/src
/home/noj/dev/clibs/src $ git clone git@github.com:hcatlin/libsass.git; cd libsass
/home/noj/dev/clibs/src/libsass $ ./configure --prefix=/home/noj/dev/clibs
/home/noj/dev/clibs/src/libsass $ make install; cd /home/noj/dev/src
/home/noj/dev/src $ git clone git@github.com:moovweb/gosass.git
/home/noj/dev/src $ git clone git@github.com:mdayaram/sassy.git; cd sassy

# The following should succeed
/home/noj/dev/src/sassy $ go build sassy

# The following should fail
/home/noj/dev/src/sassy $ cd ~
/home/noj $ go build sassy


What is the expected output?
To compile normally.

What do you see instead?
Errors described above:
/usr/bin/ld: cannot find -lsass

Which compiler are you using (5g, 6g, 8g, gccgo)?
go build

Which operating system are you using?
Ubuntu 12.04 LTS

Which version are you using?  (run 'go version')
1.1rc2
@axw
Copy link
Contributor

axw commented May 8, 2013

Comment 1:

Looks like gosass is private, so... what're the cgo LDFLAGS set by gosass? Specifically,
what's the value of "-L"? A relative path?

@gopherbot
Copy link
Author

Comment 2 by manoj.dayaram@moovweb.com:

Sorry about that, I've gone ahead and made gosass public.

@gopherbot
Copy link
Author

Comment 3 by manoj.dayaram@moovweb.com:

And yes, -L has a relative path.

@axw
Copy link
Contributor

axw commented May 8, 2013

Comment 4:

Thanks. I would say this is expected/intended behaviour; -L is being interpreted
relative to the location of the package being built, not the imported package.
(But I'm not really in a position to make a statement like that - just wanted to make
sure I didn't break anything.)

@gopherbot
Copy link
Author

Comment 5 by manoj.dayaram@moovweb.com:

Yeah, that made sense, but I guess what threw me off is that the packages compile fine
when using go version 1.0.3 and below given the same scenario.
I've made a few build scripts which have stopped working because of this.  It wouldn't
be the end of the world to add fixes to them, but this bug did strike me as a
regression, so I figured I'd post an issue about it.

@gopherbot
Copy link
Author

Comment 6 by manoj.dayaram@moovweb.com:

It looks like it has to do with the fact that my paths for LDFLAGS are relative.
So prior in go1.0.3, it looks like relative paths were resolved relative to the
directory the go file with the flags was in.
In go1.1 it looks like relative paths are resolved relative to the location where 'go
build' is being run.

@gopherbot
Copy link
Author

Comment 7 by manoj.dayaram@moovweb.com:

This problem still exists using go1.1

@gopherbot
Copy link
Author

Comment 8 by fritz@flux.io:

I am experiencing the same problem.  Of note:
- rolling back to go1.0.3 fixes the issue
- I do not run into the problem on the darwin_amd64 platform; only on the linux_amd64
platform. 
Having the LDFLAGS within cgo comments in a go src file in a package be interpreted
differently depending on where the client build was started seems pretty problematic for
creating packages that bundle and use external dso's via cgo.
How should a developer arrange the files for such a package so that it can be reliably
incorporated into a client go application using go's internal build system?  In
particular, the relative lib search paths needed to find the external dso when the
package is built in its own directory vs. as part of a client build must necessarily be
different?

@robpike
Copy link
Contributor

robpike commented May 19, 2013

Comment 9:

The change is almost certainly related to the addition of external linking in Go version
1.1.

Labels changed: added priority-later, toolbug, removed priority-triage.

Status changed to Accepted.

@robpike
Copy link
Contributor

robpike commented May 19, 2013

Comment 10:

Owner changed to @ianlancetaylor.

@axw
Copy link
Contributor

axw commented May 19, 2013

Comment 11:

@r: yes, it definitely is.
IIUC, internal linking:
 - records the path to required shared libraries in shared packages,
 - permits unresolved symbols in the program linked, and sets DT_NEEDED to include the recorded shared library name.
External linking simply records the LDFLAGS and passes those along to the host linker.

@augustoroman
Copy link
Contributor

Comment 12:

In go1.1.1 this shows up on OSX as well.

@gopherbot
Copy link
Author

Comment 13 by manoj.dayaram@moovweb.com:

Yep, problem still exists in linux go1.1.1

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 14:

Labels changed: added go1.2.

@lukescott
Copy link

Comment 15:

I'm having the same issue. Would like to avoid having an absolute path in my #cgo
LDFLAGS. If I cd to the package's src directory it works. Anywhere else, like cd ~, and
ld cannot find the library to link.
go version devel +49d897ab8e2f Fri Aug 16 22:42:54 2013 +0400 darwin/amd64

@ianlancetaylor
Copy link
Contributor

Comment 16:

The test case doesn't work any more--can anybody add a simple valid test case for this? 
Thanks.

@ianlancetaylor
Copy link
Contributor

Comment 17:

To clarify, what I get from the test case in the original issue is:
> git clone git@github.com:hcatlin/libsass.git
Cloning into 'libsass'...
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

@ianlancetaylor
Copy link
Contributor

Comment 18:

Can't fix this without a test case.

Status changed to WaitingForReply.

@gopherbot
Copy link
Author

Comment 19 by manoj.dayaram@moovweb.com:

That's odd, the repo is public in github:  https://github.com/hcatlin/libsass
However, if the ssh method for git cloning doesn't work, you could try using https:
replace each clone with the equivalent https url, so for libsass:
git clone https://github.com/hcatlin/libsass
Since the repo is public, it should not ask for any credentials.

@rsc
Copy link
Contributor

rsc commented Sep 9, 2013

Comment 20:

This is the problem:
https://github.com/moovweb/gosass/blob/master/gosass.go
#cgo LDFLAGS: -L../../clibs/lib -lsass -lstdc++
#cgo CFLAGS: -I../../clibs/include
In the old world we used those to build the .o file that we eventually handed to the
internal linker. We ran the external linker with those flags in the directory for the
package.
In the new world, we just remember them and hand them to the eventual final link
command, which runs in a different directory. 
The only fix I see is to start parsing those strings in order to recognize and rewrite
relative paths.
I think that's too big a change for Go 1.2 at this point.

Labels changed: added go1.3, removed go1.2.

@rsc
Copy link
Contributor

rsc commented Sep 9, 2013

Comment 21:

To be clear, the CFLAGS one is still fine, because we still do the C compilation
per-package.
It's only LDFLAGS that we'd need to parse.

@lukescott
Copy link

Comment 22:

Would it be possible to temporarily change the working directly before and after the
external linker is used? Because it works if you cd into the directory before running go
install/build. It just fails when you are outside, like in ~/.
Another alternative is support $VAR's. If there was a var for the package path, I could
do something like:
-L$PKGPATH/path/to/lib

@rsc
Copy link
Contributor

rsc commented Sep 9, 2013

Comment 23:

The old way invoked the system linker once for each package in the final binary. It
could do each invocation in a different directory.
The new way only has a single link, which can only have a single current directory.
If we could magically figure out which directory to use for one package, we'd still have
a problem if two different packages wanted magical treatment.

@gopherbot
Copy link
Author

Comment 24 by manoj.dayaram@moovweb.com:

Sounds like the only way to do it would be to absolutize the LDFLAGS path upon entering
each package before storing it for later use, which you're right, it would require
parsing it at that time.  
Perhaps a simple option that might not require parsing it though, maybe store both the
value of LDFLAGS as well as the current working directory associated with the package it
came from, then join those together at link time?

@rsc
Copy link
Contributor

rsc commented Sep 10, 2013

Comment 25:

Issue #5319 has been merged into this issue.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 26:

Labels changed: added release-go1.3, removed go1.3.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 27:

Labels changed: added repo-main.

@rsc
Copy link
Contributor

rsc commented Apr 3, 2014

Comment 28:

I'd really like to avoid parsing these paths. Use absolute paths as a workaround.

Labels changed: added suggested, removed release-go1.3.

Status changed to Accepted.

@gopherbot
Copy link
Author

Comment 29 by camilo@buglabs.net:

the problem with absolute paths is that it is not portable as you already know. On the
other side, trying to use environment variables in CGO directives, inside the Go source,
doesn't work either. So, we are not left with good workarounds at this moment.

@rsc
Copy link
Contributor

rsc commented Apr 3, 2014

Comment 30:

Labels changed: added release-none.

@gopherbot
Copy link
Author

Comment 31 by kyle.j.conroy:

Could we support relative paths? That way I can put a .a file in my module and any other
module that imports it will still work.

@slimsag
Copy link

slimsag commented Apr 3, 2014

Comment 32:

Absolute paths won't make very portable Go packages. I see two easy solutions:
add support for a GOPATH environment variable:
#cgo LDFLAGS: $GOPATH/src/my/pkg/mylib.a
OR
copy any .a library files found into the working directory such that this works:
#cgo LDFLAGS: mylib.a

@gopherbot
Copy link
Author

Comment 33 by camilo@buglabs.net:

The problem with using $GOPATH is that it could contain multiple paths.

@gopherbot
Copy link
Author

Comment 34 by manoj.dayaram@moovweb.com:

If relative paths can't be used, using shell variables could be a good compromise.  It
doesn't have to be $GOPATH specifically since, as mentioned, that could contain multiple
paths, but allowing shell variable expansion for those flags would solve the problem,
and arguable, be better than supporting relative paths.

@slimsag
Copy link

slimsag commented Apr 4, 2014

Comment 35:

You can already do 'shell variable'-like things with the CGO_LDFLAGS environment
variable, but it still requires people to set environment variables to install your
package.
Are we strongly opposed to copying all .a files from the source dir to build dir when
compiling so we can just use same-directory paths (#cgo LDFLAGS: mylib.a)?

@gopherbot
Copy link
Author

Comment 36 by camilo@buglabs.net:

In my use case, I don't really want to statically compile the library into my result
binary. So, ideally for me, this should work exactly like it used to work in Go 1.0.3.
Keeping backwards compatibility should be a priority in Go.

@gopherbot
Copy link
Author

Comment 37 by manoj.dayaram@moovweb.com:

I have a very similar usecase as Camilo.  We're dynamically linking libraries.  Several
of these individual libraries are used all over the Go code in different locations,
however, we dump all the libraries into one "lib" folder that we provide relative paths
to during the compile process with LDFLAGS.
I would also like to mention that this affects "go test" as well.  Perhaps obvious, I
feel like a lot of the alternative solutions provided in this thread ignore the fact
that the workflow for go test (specially for the "go test ..." command) would not work
since it would require the command to be ran from within the directory using the cgo
flags.  If you're using two projects that use different cgo flags, then the alternatives
fail.
As mentioned, using relative paths worked fine in Go1.0.3.  This issue has now been
bumped for nearly three versions of Go since it's been filed because of its difficult to
implement I presume.  I work in software as well, I understand that sometimes things
will take longer than expected, so this does not bother me that much.
However, if we're going to talk about discarding this as an issue all together, please
provide a more powerful alternative.  As mentioned by others, using absolute paths is
not a viable alternative as it forces the code to be non-portable.  Please don't take
features away without at least providing a new more powerful alternative.

@gopherbot
Copy link
Author

Comment 38 by rusnakp:

+1 for copying all static libraries from source dir to build dir so "#cgo LDFLAGS:
mylib.a" works

@gopherbot
Copy link
Author

Comment 39 by christopher@climatemonkeys.com:

Confirmed here as well that copying the static libraries from another directory into the
build directory works with a simple "#cgo LDFLAGS". 
Is there a recommended elegant way to do this? Makefile? Shell script wrapper that
executes either "go build" or "go install"?

@ianlancetaylor
Copy link
Contributor

Comment 40:

One possible approach in issue #7891.

Labels changed: removed priority-later, toolbug.

@gopherbot
Copy link
Author

Comment 41:

CL https://golang.org/cl/114640044 mentions this issue.

@gopherbot gopherbot added accepted Suggested Issues that may be good for new contributors looking for work to do. labels Aug 18, 2014
@golang golang locked and limited conversation to collaborators Jun 24, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge Suggested Issues that may be good for new contributors looking for work to do.
Projects
None yet
Development

No branches or pull requests

8 participants