Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: permission issue building CGO with Go 1.10 #24674

Closed
jaricftw opened this issue Apr 4, 2018 · 28 comments
Closed

cmd/go: permission issue building CGO with Go 1.10 #24674

jaricftw opened this issue Apr 4, 2018 · 28 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@jaricftw
Copy link

jaricftw commented Apr 4, 2018

What version of Go are you using (go version)?

1.10

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/<foo>/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/<foo>/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build884921568=/tmp/go-build -gno-record-gcc-switches"

Intel(R) Xeon(R) CPU E5-2680

What did you do?

We have been running a go program that access rocksdb (written in c) via cgo, and it has been running fine under go 1.9. However, after upgrading to 1.10, we get the following error:

go build runtime/cgo: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/cgo.a: permission denied

And the permissions of the cgo files:

ls -l /usr/lib/go-1.10/pkg/linux_amd64/runtime/
total 720
-rw-r--r-- 1 root root  93876 Mar 29 23:49 cgo.a
-rw-r--r-- 1 root root  15674 Mar 29 23:49 debug.a
drwxr-xr-x 2 root root   4096 Mar 29 23:49 internal
drwxr-xr-x 3 root root   4096 Mar 29 23:49 pprof
-rw-r--r-- 1 root root 231096 Mar 29 23:49 pprof.a
-rw-r--r-- 1 root root 378548 Mar 29 23:49 race.a
-rw-r--r-- 1 root root   2614 Mar 29 23:49 trace.a
@AlexRouSg
Copy link
Contributor

Have you checked the read permissions of the /usr/lib/go-1.10 folder and that specific file?

@davecheney
Copy link
Contributor

davecheney commented Apr 4, 2018 via email

@jaricftw
Copy link
Author

jaricftw commented Apr 4, 2018

@AlexRouSg updated the description. It has read permission so probably something is going on here to write it.

@davecheney
Copy link
Contributor

The reason the go tool would be trying to write to this directory is if the files are considered stale (ie, the .a files do not match the source). In most cases this is because of a corrupt Go installation.

How did you install Go? It looks like it was provided by your operating system distribution.

@davecheney davecheney added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 4, 2018
@bcmills bcmills changed the title Permission issue building CGO with Go 1.10 cmd/go: permission issue building CGO with Go 1.10 Apr 4, 2018
@jaricftw
Copy link
Author

jaricftw commented Apr 5, 2018

@davecheney So I've found out the issue and a possible fix, but don't know why it only happens after our go upgrade (I don't have enough details about our go upgrade process yet). Basically when we build the program, we've included the -i flags for go build as well as go test, and that somehow tries to recompile cgo and causes the permission issue. After deleting all -i from our build script, everything works fine now. What's your thoughts on this?

@davecheney
Copy link
Contributor

I recommend avoiding the package provided by your os district and using the tarball from the golang.org website, they are well tested and are known to work.

@jaricftw
Copy link
Author

jaricftw commented Apr 6, 2018

@davecheney thanks. I am not convinced that it's the issue with the go installation, as removing the -i flag in go build and go test solves the issues. And we are only have issues with cgo users, not any other Go programs so far.

Based on what I read so far, The -i flag installs the packages that are dependencies of the target, discussed here.

I also noticed with go 1.10, it suggest not to use -i anymore with the introduction of go build cache

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i.

Given these suggestions, I consider the immediately fix on my side is to remove the -i flags, but I wonder what has changed to go 1.10 that causes writing to cgo in GOROOT.

@AlexRouSg
Copy link
Contributor

I am unable to reproduce this on linux amd64 with 1.10 and 1.10.1 with freshly downloaded binaries and the entire folder set to read only.

@davecheney
Copy link
Contributor

davecheney commented Apr 6, 2018 via email

@funny-falcon
Copy link
Contributor

Same issue on Ubuntu 17.10 MATE with snap installation of go:

$ go install -i somepackage
go install runtime/cgo: open /snap/go/current/pkg/linux_amd64/runtime/cgo.a: read-only file system

Note: it is not variant to remove "-i" because build cache somewhat doesn't work with testing.
We have huge project with numerous packages. With "-i" whole testing consumes 20 minutes. Without "-i" it consumes 50 minutes. Why build cache doesn't work? I don't know :-( Installation of dependencies works well, because there is a lot of common dependencies between packages.
But flag "-i" is wired into our build command utility, and it runs with the same flag on my developer machine leading to silly error above.

@funny-falcon-at-joomcode

Note: there were no problem with go 1.9 although it also installed dependencies.
So, why it fails with 1.10?

@davecheney
Copy link
Contributor

@funny-falcon the issue lies with process that produced the snap, it has messed up the timestamps causing the go tool to try to rebuild the runtime. There isn't anything we can do. Sorry.

@davecheney
Copy link
Contributor

@funny-falcon

Why build cache doesn't work? I don't know

Please raise a new issue with steps to reproduce the test caching failure. Thank you.

@funny-falcon-at-joomcode

@davecheney

it has messed up the timestamps causing the go tool to try to rebuild the runtime.

How this can be checked? Is there way to trace how decision to recompile cgo were accepted?
Why cgo.a triggers recompilation in both cases (original issue case, and mine)? Is it because it is first file in folder "linux_amd64/runtime/" ?
Why it didn't happen with go 1.9 ?

@davecheney
Copy link
Contributor

How this can be checked? Is there way to trace how decision to recompile cgo were accepted?

Try

% go list -f '{{.Stale}}' runtime
false
% go list -f '{{.StaleReason }}' runtime

If .Stale reports true the .StaleReason may indicate why

Why cgo.a triggers recompilation in both cases (original issue case, and mine)?

Possibly the Go installation was compiled with cgo explicitly disabled, or may have happened as a side effect of cross compilation.

Is it because it is first file in folder "linux_amd64/runtime/" ?

That's just the base of almost all dependency trees, so it's the first one to be compiled and thus hits the permission error.

Why it didn't happen with go 1.9 ?

You'll have to ask the people who packaged 1.9.

@funny-falcon-at-joomcode

Nope:

$ go list -f '{{.Stale}}' runtime
false
$ go list -f '{{.StaleReason }}' runtime

@funny-falcon-at-joomcode
Copy link

funny-falcon-at-joomcode commented Apr 9, 2018

Ops, that is for runtime/cgo:

$ go list -f '{{.Stale}}' runtime/cgo
true
$ go list -f '{{.StaleReason}}' runtime/cgo
build ID mismatch

@funny-falcon
Copy link
Contributor

I've build go 1.10.1 in a temporary directory, and its cgo.a differs from cgo.a from snap (~100kb in temp dir, and ~ 32kb in snap dir), while other files in runtime directory matches.
So, looks like builder of snap package really does strange thing with it.

@AlexRouSg
Copy link
Contributor

AlexRouSg commented Apr 9, 2018

Found a way to contact someone who I hope is still the snap maintainer or at least in contact with them. My post is still under review since it's a google groups post on golang-dev.

https://groups.google.com/forum/#!topic/golang-dev/iRauvZsPrNE%5B1-25%5D

@ianlancetaylor
Copy link
Contributor

It's not obvious to me that there is anything the Go project can do to fix this problem.

@mwhudson
Copy link
Contributor

mwhudson commented Apr 9, 2018

Feel free to ping me in any bugs to do with debian/ubuntu/snap packaging!

For the snap packaging, I do know why runtime/cgo appears stale. It's fairly obscure though: the snap packaging builds runtime/cgo with particular C compiler flags to ensure that the generated .o files can be handled by older compilers. The go tool now keeps track of these, so unless you happen to use exactly the same flags, the go tool considers runtime/cgo to be stale:

mwhudson@aeglos:~$ /snap/bin/go list  -f '{{ .ImportPath }}: {{ if .Stale }}stale because "{{ .StaleReason }}"{{ else }}not stale {{ end }}' runtime/cgo
runtime/cgo: stale because "build ID mismatch"
mwhudson@aeglos:~$ CGO_CFLAGS='-Wa,-mrelax-relocations=no' /snap/bin/go list  -f '{{ .ImportPath }}: {{ if .Stale }}stale because "{{ .StaleReason }}"{{ else }}not stale {{ end }}' runtime/cgo
runtime/cgo: not stale 

In a sense, runtime/cgo being stale is not actually a bad thing because it means that the C parts are always going to be built with your local C compiler (once, and then saved into the cache, so this doesn't even have much of a time cost).

It's not obvious to me that there is anything the Go project can do to fix this problem.

Well, I think the go tool could swallow failures to write to GOROOT, at least when they are being made because of a -i flag. I presume the initial variant of the report (i.e. not the one for the snap) is something like this, which I don't think is terribly sensible behaviour:

ubuntu@bionic:~$ go build -i -gcflags all=-N trivial.go 
go build runtime/internal/sys: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/internal/sys.a: permission denied
go build runtime/internal/atomic: open /usr/lib/go-1.10/pkg/linux_amd64/runtime/internal/atomic.a: permission denied

(the alternative is training people to stop using -i, I guess)

@ianlancetaylor
Copy link
Contributor

Swallowing failures sounds like a bad idea, but we could consider changing -i to not install to GOROOT at all.

But likely better is to convince people to stop using -i unnecessarily, as with 1.10 it's no longer particularly useful.

@mwhudson
Copy link
Contributor

mwhudson commented Apr 9, 2018

Swallowing failures sounds like a bad idea, but we could consider changing -i to not install to GOROOT at all.

That works for me (maybe only for release versions?)

But likely better is to convince people to stop using -i unnecessarily, as with 1.10 it's no longer particularly useful.

Well yes, but it takes a while for people's scripts and tooling to catch up I guess.

@akshayjshah
Copy link
Contributor

I'm so sorry that this has bubbled up to all of you.

This is purely an artifact of our internal build system, which has to support Go 1.9, installing multiple Go compilers on the same host, and a handful of other charming quirks. There's nothing broken in Go, and there's nothing that the Go project or Debian package maintainers can or should do about this. Slightly altering the internal build configuration for this particular project fully resolves the issue.

Again, I'm sorry that our internal triaging process failed to catch this before it surfaced here.

@ianlancetaylor
Copy link
Contributor

Thanks, closing.

sameersbn pushed a commit to sameersbn/kubeapps that referenced this issue May 29, 2018
sameersbn pushed a commit to vmware-tanzu/kubeapps that referenced this issue May 29, 2018
joestringer added a commit to joestringer/cilium that referenced this issue Jun 22, 2018
Using `-i` in the "go build" arguments with Golang 1.10 on Ubuntu using
a snap package results in this error:

    $ make -C daemon
      CHECK contrib/scripts/bindata.sh
      GO    daemon/cilium-agent
    go build runtime/cgo: open /snap/go/2130/pkg/linux_amd64/runtime/cgo.a: read-only file system

According to the upstream issue, this is because it's attempting to rebuild
the core Golang packages on the system, and snap is preventing this via a
read-only file system to ensure consistency for the base Golang version.

The workaround suggested in the following issue is to remove `-i`.

golang/go#24674

Signed-off-by: Joe Stringer <joe@covalent.io>
tgraf pushed a commit to cilium/cilium that referenced this issue Jun 22, 2018
Using `-i` in the "go build" arguments with Golang 1.10 on Ubuntu using
a snap package results in this error:

    $ make -C daemon
      CHECK contrib/scripts/bindata.sh
      GO    daemon/cilium-agent
    go build runtime/cgo: open /snap/go/2130/pkg/linux_amd64/runtime/cgo.a: read-only file system

According to the upstream issue, this is because it's attempting to rebuild
the core Golang packages on the system, and snap is preventing this via a
read-only file system to ensure consistency for the base Golang version.

The workaround suggested in the following issue is to remove `-i`.

golang/go#24674

Signed-off-by: Joe Stringer <joe@covalent.io>
tgraf pushed a commit to cilium/cilium that referenced this issue Jun 23, 2018
[ upstream commit e542412 ]

Using `-i` in the "go build" arguments with Golang 1.10 on Ubuntu using
a snap package results in this error:

    $ make -C daemon
      CHECK contrib/scripts/bindata.sh
      GO    daemon/cilium-agent
    go build runtime/cgo: open /snap/go/2130/pkg/linux_amd64/runtime/cgo.a: read-only file system

According to the upstream issue, this is because it's attempting to rebuild
the core Golang packages on the system, and snap is preventing this via a
read-only file system to ensure consistency for the base Golang version.

The workaround suggested in the following issue is to remove `-i`.

golang/go#24674

Signed-off-by: Joe Stringer <joe@covalent.io>
Signed-off-by: Thomas Graf <thomas@cilium.io>
tgraf pushed a commit to cilium/cilium that referenced this issue Jun 23, 2018
[ upstream commit e542412 ]

Using `-i` in the "go build" arguments with Golang 1.10 on Ubuntu using
a snap package results in this error:

    $ make -C daemon
      CHECK contrib/scripts/bindata.sh
      GO    daemon/cilium-agent
    go build runtime/cgo: open /snap/go/2130/pkg/linux_amd64/runtime/cgo.a: read-only file system

According to the upstream issue, this is because it's attempting to rebuild
the core Golang packages on the system, and snap is preventing this via a
read-only file system to ensure consistency for the base Golang version.

The workaround suggested in the following issue is to remove `-i`.

golang/go#24674

Signed-off-by: Joe Stringer <joe@covalent.io>
Signed-off-by: Thomas Graf <thomas@cilium.io>
@NatoBoram
Copy link

I still have this issue with Go 1.11.

neofetch --off

OS: Ubuntu 18.04.1 LTS x86_64 
Kernel: 4.15.0-34-generic 
Uptime: 1 day, 47 mins 
Packages: 2213 
Shell: bash 4.4.19 
Resolution: 1920x1080 
DE: GNOME 3.28.3 
WM: GNOME Shell 
WM Theme: Adwaita 
Theme: Adwaita-dark [GTK2/3] 
Icons: Ubuntu-mono-dark [GTK2/3] 
Terminal: gnome-terminal 
CPU: Intel i5-3570K (4) @ 4.100GHz 
GPU: NVIDIA GeForce GTX 660 Ti 
Memory: 4952MiB / 7922MiB 

snap info go

name:      go
summary:   Go programming language compiler, linker, stdlib
publisher: Michael Hudson-Doyle (mwhudson)
contact:   michael.hudson@ubuntu.com
license:   unset
description: |
  This snap provides an assembler, compiler, linker, and compiled libraries
  for the Go programming language.
commands:
  - go
  - go.gofmt
snap-id:      Md1HBASHzP4i0bniScAjXGnOII9cEK6e
tracking:     candidate
refresh-date: 25 days ago, at 05:30 EDT
installed:        1.11          (2635) 82MB classic

go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nato/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/nato/go"
GOPROXY=""
GORACE=""
GOROOT="/snap/go/2635"
GOTMPDIR=""
GOTOOLDIR="/snap/go/2635/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build648960057=/tmp/go-build -gno-record-gcc-switches"
go build runtime/cgo: open /snap/go/2635/pkg/linux_amd64/runtime/cgo.a: read-only file system

@ghost
Copy link

ghost commented Sep 21, 2018

Same issue

go build runtime/cgo: open /snap/go/2635/pkg/linux_amd64/runtime/cgo.a: read-only file system

@golang golang locked as resolved and limited conversation to collaborators Sep 22, 2018
@davecheney
Copy link
Contributor

@NatoBoram @x-t0xid this issue relates to Go 1.10 and is closed.

Please open a new issue for the problems you are having with Go 1.11. You can open seperate issues if you like, we'll take care of handling any duplication.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

9 participants