Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: unexpected trampoline when cross-compiling to ppc64le #45564

Closed
dbenoit17 opened this issue Apr 14, 2021 · 11 comments
Closed

cmd/link: unexpected trampoline when cross-compiling to ppc64le #45564

dbenoit17 opened this issue Apr 14, 2021 · 11 comments
Labels
arch-ppc64x FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@dbenoit17
Copy link

What version of Go are you using (go version)?

$ go version
1.16.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOVERSION="go1.16.3"

What did you do?

Cross compile OpenShift's oc command from x86_64 to ppc64le.

git clone https://github.com/openshift/oc.git && pushd oc
make cross-build-linux-ppc64le

What did you expect to see?

Program builds.

What did you see instead?

Linker error at

go/src/cmd/link/internal/ppc64/asm.go

Line 708 in f12cf76

ctxt.Errorf(s, "unexpected trampoline for shared or dynamic linking") 

# github.com/openshift/oc/cmd/oc
github.com/openshift/oc/pkg/cli/observe.(*ObserveOptions).Run.func3: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.newlineTrailingWriter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.injectUserVars: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.processTemplateLocally: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).Complete.func2: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/process.processPrinter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/project.ProjectOptions.Run: unexpected trampoline for shared or dynamic linking
/root/go/pkg/tool/linux_amd64/link: too many errors

Additional Notes

  • This issue is not seen when compiling natively on ppc64le, only during cross-compilation from x86_64.
  • The issue was not seen prior to upgrading openshift/oc to use kubernetes 1.21.
  • The issue also occurs in Go versions 1.15.8 through 1.15.10, which are the only versions of 1.15 I've tested. However, it produces the error against different oc functions in Go 1.15 vs. Go 1.16.
  • Interestingly, 1.16 and 1.16.1 seem unaffected. The issue appears again in 1.16.2, and seems to be triggered by a9547ad#diff-aa02dd8eebb2420ba09bee48832b239fc5f612a07cb7b1709595511fa1d8d0ca. This commit might just uncover the issue rather than be it's direct cause, though, I'm not certain.
@mknyszek mknyszek changed the title Unexpected trampoline when cross-compiling to ppc64le cmd/link: unexpected trampoline when cross-compiling to ppc64le Apr 14, 2021
@mknyszek
Copy link
Contributor

CC @cherrymui @mdempsky via https://dev.golang.org/owners

@mknyszek mknyszek added arch-ppc64x NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 14, 2021
@mknyszek mknyszek added this to the Backlog milestone Apr 14, 2021
@cherrymui
Copy link
Member

It is on my plan to rework trampoline insertion in the linker, that should address this. Hopefully I can get it in for 1.17.

@dbenoit17
Copy link
Author

Thanks for the quick feedback!

I'm curious if there might be also a possibility for an interim patch to 1.16 in the meantime. One thing that does seem to get the linker past the issue is adjusting this condition to check for ctxt.IsExternal(), to closer match the earlier conditional here. I'm not sure if this is the correct thing to do though. I think it depends whether it's valid for the linker to arrive at line 706 with any of those configuration options when using internal linking.

@cherrymui
Copy link
Member

@dbenoit17 thanks for the comment. I'll take a careful look.

@yselkowitz yselkowitz mentioned this issue Apr 26, 2021
Merged
yselkowitz added a commit to yselkowitz/openshift-installer that referenced this issue Apr 26, 2021
@yselkowitz
Link ppc64le binary dynamically
24ac0a1 
Recent versions of Go have had issues statically linking some ppc64le
binaries:

golang/go#45564
@cherrymui
Copy link
Member

@dbenoit17 sorry for the delay. As you said, I don't think we use internal linking for any of those configurations. I.e. IsExternal should always be true at that point.

Could you share how the "go" command or the linker is invoked in your build process, and what build mode are you building for? Thanks.

@pmur
Copy link
Contributor

pmur commented Apr 27, 2021

This seems to be caused by the combination of:
CGO_ENABLED=0 and something in openshift/installer using the plugin package causing canUsePlugins to be set, which causes (*Link).DynlinkingGo() to evaulate true, and causes the panic noted above.

Given that plugins are effectively disabled in the above case, shouldn't canUsePlugins also be set to false when linking?

@gopherbot
Copy link

Change https://golang.org/cl/314449 mentions this issue: cmd/link: disable plugin support if cgo is explicitly disabled

@dbenoit17
Copy link
Author

To hit the issue, go build is called against openshift/oc commit 90af8a5d7a1ae91e3e675607c4f76a403e6ec817 by the Makefile using CGO_ENABLED=0, with the invocation:

go build -mod=vendor -tags 'include_gcs include_oss containers_image_openpgp' -ldflags "-X github.com/openshift/oc/pkg/version.versionFromGit="v4.2.0-alpha.0-1048-g90af8a5" -X github.com/openshift/oc/pkg/version.commitFromGit="90af8a5d7" -X github.com/openshift/oc/pkg/version.gitTreeState="clean" -X github.com/openshift/oc/pkg/version.buildDate="<date>" -X k8s.io/component-base/version.gitMajor="1" -X k8s.io/component-base/version.gitMinor="21" -X k8s.io/component-base/version.gitVersion="v0.21.0-beta.1" -X k8s.io/component-base/version.gitCommit="90af8a5d7" -X k8s.io/component-base/version.buildDate="<date>" -X k8s.io/component-base/version.gitTreeState="clean" -X k8s.io/client-go/pkg/version.gitVersion="v4.2.0-alpha.0-1048-g90af8a5" -X k8s.io/client-go/pkg/version.gitCommit="90af8a5d7" -X k8s.io/client-go/pkg/version.buildDate="<date>" -X k8s.io/client-go/pkg/version.gitTreeState="clean"" -o '_output/bin/linux_ppc64le/oc' github.com/openshift/oc/cmd/oc

I think this is using using buildmode=default since there is no buildmode specified.

As a follow-up, I've had a chance to test https://golang.org/cl/314449 and can confirm it resolves the issue.

@laboger
Copy link
Contributor

laboger commented Apr 27, 2021

I think this is using using buildmode=default since there is no buildmode specified.

It was originally thought that the messages about dynamic linking were due to a buildmode setting, but that turned out not to be the case. Code that uses plugins is usually compiled as PIC (i.e., like shared or dynamic linking) but if CGO_ENABLED=0 then plugins are disabled and then not compiled as PIC so that led to confusion in the linker and the errors.

@pmur
Copy link
Contributor

pmur commented Apr 28, 2021

@gopherbot please consider a backport to 1.16. This seems to a be a regression from go 1.16.1 to 1.16.2.

I don't think a backport to earlier versions is necessary since the issue likely affects all 1.15 releases. It might be possible to workaround by not disabling cgo, or using external linking.

This was referenced Apr 28, 2021
Closed
Closed
@gopherbot
Copy link

gopherbot commented Apr 28, 2021 •
edited by dmitshur

Backport issue(s) opened: #45831 (for 1.15), #45832 (for 1.16).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@cherrymui cherrymui mentioned this issue Apr 29, 2021
Closed
@laboger laboger mentioned this issue May 3, 2021
Open
AnnaZivkovic pushed a commit to AnnaZivkovic/installer that referenced this issue Apr 1, 2022
@yselkowitz
Link ppc64le binary dynamically
c74951d 
Recent versions of Go have had issues statically linking some ppc64le
binaries:

golang/go#45564
@golang golang locked and limited conversation to collaborators May 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-ppc64x FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
6 participants
@mknyszek @gopherbot @laboger @pmur @cherrymui @dbenoit17