@laboger, what's the status here? I thought these were all fixed. Guess not?

@cherrymui This problem is related to the big text problem and I suspect it would happen in arm too unless you did something to fix it. I need help on this one because there is a lot about golang linking I don't understand.

In this case, the problem occurs because there is a call off to something in glibc, so it gets a call stub in the PLT, but the PLT is getting placed too far in the binary so that the bl to the call stub is too far.

If we use -linkmode=external on this one, the error doesn't occur. Actually I didn't even realize that you could call something in glibc when using internal linking.

Any suggestions on solutions for this one? Even if the PLT was put up front there could still be code later on in the binary that is too far from the PLT.

@ianlancetaylor for linker expertise.

@bradfitz I thought they were all fixed too :(
This one is related to a call to something in the PLT which I haven't seen before.
This just started happening when Kubernetes updated to a new beta version. Didn't happen in 1.6.0-beta.0 but now fails in 1.6.0-beta1.

I tried go build -ldflags=-linkmode=internal k8s.io/kubernetes/federation/cmd/genfeddocs with cgo enabled, for both GOARCH=arm and GOARCH=ppc64le. Both succeeded.
Could you let me know how to reproduce it?

The PLT stubs are generated before the trampoline insertion pass. So the trampoline insertion pass should be able to take care of it. Maybe a relocation case is missing?

The problem happens at the instruction that does the bl to the PLT stub. At the point of trampoline insertion, it doesn't know where the PLT will be, does it? That isn't know until the linker links it all together because I believe the PLT stubs are in a separate section.

This only fails with the latest Kubernetes (after v1.6.0-beta1). Here is what I see after the build has failed:

~/kub/kublatest/kubernetes/_output/local/go$ go build k8s.io/kubernetes/federation/cmd/genfeddocs
//# k8s.io/kubernetes/federation/cmd/genfeddocs
net(.text): direct call from: net(.text) too far: net(.text).malloc 20b3e00
net(.text): direct call from: net(.text) too far: net(.text).malloc 20b3dbc
net(.text): direct call from: net(.text) too far: net(.text).__errno_location 20b3d7c
net(.text): direct call from: net(.text) too far: net(.text).getnameinfo 20b3d64
net(.text): direct call from: net(.text) too far: net(.text).getnameinfo 20b3cbc
net(.text): direct call from: net(.text) too far: net(.text).__errno_location 20b3c1c
net(.text): direct call from: net(.text) too far: net(.text).getaddrinfo 20b3c28
net(.text): direct call from: net(.text) too far: net(.text).free 20b3bd0
net(.text): direct call from: net(.text) too far: net(.text).freeaddrinfo 20b3ba8
net(.text): direct call from: net(.text) too far: net(.text).gai_strerror 20b3b64
net(.text): direct call from: net(.text) too far: net(.text).getaddrinfo 20b3a90
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).malloc 20b3a78
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).malloc 20b3a34
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).getgrouplist 20b39e8
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).free 20b3998
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).getgrgid_r 20b3944
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).getgrnam_r 20b38cc
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).getpwnam_r 20b3854
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).getpwuid_r 20b37dc
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).realloc 20b3770
os/user(.text): direct call from: os/user(.text) too far: os/user(.text).sysconf 20b370c

OK now I see what you were saying earlier, the trampoline code is not handling the SDYNIMPORT type correctly. I'll look into that.

I found that the problem happens as a result of there being C code calling functions in glibc, where the plt call stub is too far and no trampoline is generated for that case. For example, this happens in net.a because C code is generated through cgo for the call to C.malloc. When the C code is processed, the function IsDirectJump() must be true before it will call trampoline(), but that function only checks for Go relocation types and not C relocation types so returns false for C code. I think this same problem could happen on arm at some point, but since ppc64le is 64 bit the programs are in general larger so we are hitting it first.

Here is a simpler summary:

1. net.a is read in from ldelf.go. This generates a symbol called net(.text) which contains various functions such as _cgo_cb4050e05860_Cfunc__Cmalloc, and some of those make calls to functions like malloc which require a call stub. These symbols are added to Textp first.
2. All the other packages needed by the program are read in and their symbols are added to Textp.
3. The call stubs are generated, and one must be created to be used by the function mentioned above, because it does a bl malloc. It is called net(.text).malloc. The call stubs are added after all the packages have been added, and since Kubernetes has a very large number of very large packages, the call stubs end up too far for the code in net(.text). The trampoline processing doesn't handle this because the call stub (target of the bl) hasn't had its code address assigned at the point the code for the bl is processed.

I think the solution would be to just put the call stubs closer to where they are used, and actually I think it should be OK to just put them within the range of the outer sym, i.e., as a sub of net(.text) since the way these call stubs are named, they are unique within the outer sym. net(.text).malloc can only be called within net(.text).

@cherrymui or @ianlancetaylor I need some direction how to do this. I've made a few naive attempts at getting the callstubs in a different location but I can't get it to build.

@laboger thank you for looking into it.

Would it be possible to add call stubs to Textp before other packages?

If that doesn't work, maybe you can pre-compute the size of the program before assigning the addresses, and insert trampolines when assigning addresses for functions in 1. I think the size of the text is known except for trampolines inserted when assigning addresses. You could make some conservative estimate of the trampoline sizes.

I'll take a careful look in the weekend or early next week.

Workaround for this is to add -ldflags '-linkmode=external' for the build of nonstatic binaries built by kubernetes. The static binaries built by kubernetes should not be affected by this problem and should not have this option added.

@laboger CL https://go-review.googlesource.com/c/38131/ puts the stubs to the beginning of Textp. Could you try that? By moving the stubs, the calls to the stubs are actually not too far now in that binary. But I think it should work even if it is too far.

CL https://golang.org/cl/38131 mentions this issue.

Yes I will try it.

Is it possible that https://go-review.googlesource.com/c/38131/ can be cherry-picked into the go1.8 release branch? The risk seems to be quite low, and the gain pretty high if it fixes net and os/user problems with external linking for ppc64le and maybe ARM.

WDYT?

Issue #19578 is opened for cherry-picking.

I see this same issue on arm (also building kubernetes; go 1.8; works with -linkmode=external).
Is there a similar patch around for arm? Can it also be backported to 1.8?

@anguslees Please open a new issue with more details and cc me.
Really want to pin that down if there's an issue; it's not known I think