Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: build failure with Openshift on ppc64le with recent merge of dev.link into master #38849

Closed
laboger opened this issue May 4, 2020 · 10 comments
Closed

cmd/link: build failure with Openshift on ppc64le with recent merge of dev.link into master #38849

laboger opened this issue May 4, 2020 · 10 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.15

Comments

@laboger
Copy link
Contributor

laboger commented May 4, 2020

What version of Go are you using (go version)?

$ go version
go version devel +53f27474a4 Sun May 3 07:23:32 2020 +0000 linux/ppc64le.

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
linux/ppc64le
power8

Build of Openshift with latest golang on ppc64le:
mkdir $HOME/openshift // Any directory name is fine
export GOPATH=$HOME/openshift
mkdir -p $GOPATH/src/github.com/openshift; cd $GOPATH/src/github.com/openshift
git clone https://github.com/openshift/origin
cd origin
export PERMISSIVE_GO=y
make all

What did you expect to see?

This is a recent commit that works:

~/openshift/src/github.com/openshift/origin$ make all
hack/build-go.sh  
[INFO] Detected go version: go version devel +0f8fecaba7 Thu Apr 30 20:18:38 2020 +0000 linux/ppc64le.
[WARNING] Detected golang version doesn't match required Go version.
[WARNING] This version mismatch could lead to differences in execution between this run and the CI systems.
++ Building go targets for linux/ppc64le: vendor/k8s.io/kubernetes/cmd/kube-apiserver vendor/k8s.io/kubernetes/cmd/kube-controller-manager vendor/k8s.io/kubernetes/cmd/kube-scheduler vendor/k8s.io/kubernetes/cmd/kubelet
[INFO] hack/build-go.sh exited with code 0 after 00h 00m 47s

What did you see instead?

~/openshift/src/github.com/openshift/origin$ make all
hack/build-go.sh  
[INFO] Detected go version: go version devel +53f27474a4 Sun May 3 07:23:32 2020 +0000 linux/ppc64le.
[WARNING] Detected golang version doesn't match required Go version.
[WARNING] This version mismatch could lead to differences in execution between this run and the CI systems.
++ Building go targets for linux/ppc64le: vendor/k8s.io/kubernetes/cmd/kube-apiserver vendor/k8s.io/kubernetes/cmd/kube-controller-manager vendor/k8s.io/kubernetes/cmd/kube-scheduler vendor/k8s.io/kubernetes/cmd/kubelet
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util/util.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities/capabilities.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1
vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1/defaults.go:20:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
# github.com/openshift/origin/vendor/k8s.io/client-go/listers/policy/v1beta1
vendor/k8s.io/client-go/listers/policy/v1beta1/eviction.go:22:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
# github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/scheme
vendor/k8s.io/client-go/kubernetes/scheme/register.go:51:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
[ERROR] hack/lib/build/binaries.sh:224: `GOOS=${platform%/*} GOARCH=${platform##*/} go install -tags "${OS_GOFLAGS_TAGS-} ${!platform_gotags_envvar:-}" -ldflags="${local_ldflags}" "${goflags[@]:+${goflags[@]}}" -gcflags "${gogcflags}" "${nonstatics[@]}"` exited with status 2.
[ERROR] hack/lib/build/binaries.sh:138: `( os::build::internal::build_binaries "${binaries[@]+"${binaries[@]}"}" )` exited with status 2.
[ERROR] hack/build-go.sh exited with code 2 after 00h 00m 27s
Makefile:39: recipe for target 'all' failed
make: *** [all] Error 2

@cherrymui @jeremyfaller
@cherrymui
Copy link
Member

How does it build the binary exactly? The error reading fingerprint is likely due to something weird happened in the build procedure. Could you let it print the exact command that make runs?

@laboger
Copy link
Contributor Author

laboger commented May 4, 2020

Not sure how much you want. I generated the full log by doing this:
export GOFLAGS="-x -work"
hack/build-go.sh

Here are the first two spots where this error occurs:

cd /home/boger/openshift/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/client-go/security/clientset/versioned/typed/security/v1
/home/boger/golang/link/go/pkg/tool/linux_ppc64le/compile -o $WORK/b1332/_pkg_.a -trimpath "$WORK/b1332=>" -p github.com/openshift/origin/vendor/github.com/openshift/client-go/security/clientset/versioned/typed/security/v1 -complete -buildid yB9rdJHqkKPJQndDIyYo/yB9rdJHqkKPJQndDIyYo -D "" -importcfg $WORK/b1332/importcfg -pack -c=4 ./doc.go ./generated_expansion.go ./podsecuritypolicyreview.go ./podsecuritypolicyselfsubjectreview.go ./podsecuritypolicysubjectreview.go ./rangeallocation.go ./security_client.go ./securitycontextconstraints.go
/home/boger/golang/link/go/pkg/tool/linux_ppc64le/buildid -w $WORK/b1825/_pkg_.a # internal
cp $WORK/b1825/_pkg_.a /home/boger/.cache/go-build/32/321885c62f142b4d9b4d7497410bcaf06e76a3dafb85aeabbe99a7fea085c636-d # internal
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util/util.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint

...
cd /home/boger/openshift/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/scale/scheme/appsv1beta1
/home/boger/golang/link/go/pkg/tool/linux_ppc64le/compile -o $WORK/b769/_pkg_.a -trimpath "$WORK/b769=>" -p github.com/openshift/origin/vendor/k8s.io/client-go/scale/scheme/appsv1beta1 -complete -buildid pU2vzf4VlOg8YokC-WTV/pU2vzf4VlOg8YokC-WTV -D "" -importcfg $WORK/b769/importcfg -pack -c=4 ./conversion.go ./doc.go ./register.go ./zz_generated.conversion.go
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities/capabilities.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
cd /home/boger/openshift/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/listers/apps/v1beta1
/home/boger/golang/link/go/pkg/tool/linux_ppc64le/compile -o $WORK/b368/_pkg_.a -trimpath "$WORK/b368=>" -p github.com/openshift/origin/vendor/k8s.io/client-go/listers/apps/v1beta1 -complete -buildid wpdRHdiZkYQ1fA6Ot88f/wpdRHdiZkYQ1fA6Ot88f -D "" -importcfg $WORK/b368/importcfg -pack -c=4 ./controllerrevision.go ./deployment.go ./expansion_generated.go ./statefulset.go ./statefulset_expansion.go
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1
vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1/defaults.go:20:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint
# github.com/openshift/origin/vendor/k8s.io/client-go/listers/policy/v1beta1
vendor/k8s.io/client-go/listers/policy/v1beta1/eviction.go:22:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint

@cherrymui
Copy link
Member

I mostly need to know how the shell script invokes the go command. Maybe you could run it with sh -x?

Also it might be useful if you could attach one of the object file it complains, like v1beta1.a.

@laboger
Copy link
Contributor Author

laboger commented May 4, 2020

Here's what seems to be the main go command, but it doesn't fail if I run it manually (outside of script).

go install -tags 'include_gcs include_oss containers_image_openpgp gssapi selinux' '-ldflags=-s -w -X github.com/openshift/origin/pkg/version.majorFromGit=4 -X github.com/openshift/origin/pkg/version.minorFromGit=1+ -X github.com/openshift/origin/pkg/version.versionFromGit=v4.1.0+6c88aa2-2657-dirty -X github.com/openshift/origin/pkg/version.commitFromGit=6c88aa23a5 -X github.com/openshift/origin/pkg/version.gitTreeState=dirty -X github.com/openshift/origin/pkg/version.buildDate=2020-05-04T17:59:39Z -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitMajor=1 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitMinor=18+ -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitCommit=6c88aa23a5 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitVersion=v1.18.0-rc.1 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.buildDate=2020-05-04T17:59:39Z -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitTreeState=clean -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitMajor=1 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitMinor=18+ -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitCommit=6c88aa23a5 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitVersion=v1.18.0-rc.1 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.buildDate=2020-05-04T17:59:39Z -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitTreeState=clean' -gcflags '' github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-controller-manager github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-scheduler github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kubelet

Build of package.

cd /home/boger/openshift/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1
/home/boger/golang/link/go/pkg/tool/linux_ppc64le/compile -o $WORK/b263/_pkg_.a -trimpath "$WORK/b263=>" -p github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1 -complete -buildid igZv2DQ2c6wFddKzutdM/igZv2DQ2c6wFddKzutdM -D "" -importcfg $WORK/b263/importcfg -pack -c=4 ./doc.go ./generated.pb.go ./register.go ./types.go ./types_swagger_doc_generated.go ./zz_generated.deepcopy.go

pkg.a.gz

@cherrymui
Copy link
Member

Here's what seems to be the main go command, but it doesn't fail if I run it manually (outside of script).

Thanks. Looks like something weird going on with that script.

The .a file looks fine to me, and it does contain a fingerprint that the compiler expects.

Maybe try this patch and see what error it prints?

diff --git a/src/cmd/compile/internal/gc/iimport.go b/src/cmd/compile/internal/gc/iimport.go
index 7c0764c677..07f8ba06e3 100644
--- a/src/cmd/compile/internal/gc/iimport.go
+++ b/src/cmd/compile/internal/gc/iimport.go
@@ -193,7 +193,7 @@ func iimport(pkg *types.Pkg, in *bio.Reader) (fingerprint goobj2.FingerprintType
 	// Fingerprint
 	n, err := in.Read(fingerprint[:])
 	if err != nil || n != len(fingerprint) {
-		yyerror("import %s: error reading fingerprint", pkg.Path)
+		yyerror("import %s: error reading fingerprint %v %d %d", pkg.Path, err, n, in.Offset())
 		errorexit()
 	}
 	return fingerprint

@laboger
Copy link
Contributor Author

laboger commented May 5, 2020

I also printed out len(fingerprint) after n since that is what is being tested before printing the error.

+ go install -tags 'include_gcs include_oss containers_image_openpgp gssapi selinux' '-ldflags=-s -w -X github.com/openshift/origin/pkg/version.majorFromGit=4 -X github.com/openshift/origin/pkg/version.minorFromGit=1+ -X github.com/openshift/origin/pkg/version.versionFromGit=v4.1.0+6c88aa2-2657-dirty -X github.com/openshift/origin/pkg/version.commitFromGit=6c88aa23a5 -X github.com/openshift/origin/pkg/version.gitTreeState=dirty -X github.com/openshift/origin/pkg/version.buildDate=2020-05-05T12:28:40Z -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitMajor=1 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitMinor=18+ -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitCommit=6c88aa23a5 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitVersion=v1.18.0-rc.1 -X github.com/openshift/origin/vendor/k8s.io/component-base/version.buildDate=2020-05-05T12:28:40Z -X github.com/openshift/origin/vendor/k8s.io/component-base/version.gitTreeState=clean -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitMajor=1 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitMinor=18+ -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitCommit=6c88aa23a5 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitVersion=v1.18.0-rc.1 -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.buildDate=2020-05-05T12:28:40Z -X github.com/openshift/origin/vendor/k8s.io/client-go/pkg/version.gitTreeState=clean' -gcflags '' github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-apiserver github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-controller-manager github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kube-scheduler github.com/openshift/origin/vendor/k8s.io/kubernetes/cmd/kubelet
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/util/util.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint <nil> 4 8 109999
# github.com/openshift/origin/vendor/k8s.io/client-go/listers/policy/v1beta1
vendor/k8s.io/client-go/listers/policy/v1beta1/eviction.go:22:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint <nil> 4 8 109999
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1
vendor/k8s.io/kubernetes/pkg/apis/policy/v1beta1/defaults.go:20:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint <nil> 4 8 109999
# github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities
vendor/k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities/capabilities.go:23:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint <nil> 4 8 109999
# github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/scheme
vendor/k8s.io/client-go/kubernetes/scheme/register.go:51:2: import github.com/openshift/origin/vendor/k8s.io/api/policy/v1beta1: error reading fingerprint <nil> 4 8 109999

@gopherbot
Copy link

Change https://golang.org/cl/232317 mentions this issue: cmd/compile: use ReadFull to read fingerprint

@cherrymui
Copy link
Member

Thanks! That is useful. Does the CL above fix it?

(fingerprint is [8]byte so len(fingerprint) is constant 8. So I didn't ask for it.)

@laboger
Copy link
Contributor Author

laboger commented May 5, 2020

The CL fixes the build error. I'm running the testcases and I am getting a few SIGILLs:

SIGILL: illegal instruction
PC=0x11e303d0 m=0 sigcode=1
instruction bytes: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

Trying to track down where they are.
I built and ran the test suite with the commit before the recent merge of dev.link and these errors did not occur then.

@cherrymui
Copy link
Member

Thanks. The CL is in.

My hunch is that the SIGILL is a different problem. Could you try running it with the old linker? go build -gcflags=all=-go115newobj=false -asmflags=all=-go115newobj=false -ldflags=all=-go115newobj=false

@dmitshur dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label May 8, 2020
@dmitshur dmitshur added this to the Go1.15 milestone May 8, 2020
xujianhai666 pushed a commit to xujianhai666/go-1 that referenced this issue May 21, 2020
@cherrymui @xujianhai666
cmd/compile: use ReadFull to read fingerprint
670b09c 
Don't fail on partial read.

May fix golang#38849.

Change-Id: Icf075d454e1bfe9299b07eea47bbc4d448c3bd5e
Reviewed-on: https://go-review.googlesource.com/c/go/+/232317
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@golang golang locked and limited conversation to collaborators May 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.15
Development

No branches or pull requests
4 participants
@dmitshur @gopherbot @laboger @cherrymui