cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

yunabe · 2018-02-22T13:05:21Z

What version of Go are you using (`go version`)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yunabe/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/yunabe/local/gocode"
GORACE=""
GOROOT="/home/yunabe/go1.10"
GOTMPDIR=""
GOTOOLDIR="/home/yunabe/go1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build331277736=/tmp/go-build -gno-record-gcc-switches"

What did you do?

cd $TMPDIR
go get -d golang.org/x/net/context
go install -buildmode=shared -linkshared -pkgdir `pwd`/tmppkg -linkshared std
go install -x -buildmode=shared -pkgdir `pwd`/tmppkg -linkshared golang.org/x/net/context

What did you expect to see?

The second go install command compiles context.a and link libgolang.org-x-net-context.so. It takes less than 1 sec to finish.

What did you see instead?

From go1.10, the second go install run link command to create libstd.so though libstd.so already exists in -pkgdir.
It makes the second go install command 3-4 times slower (0.77sec in go1.9.4 vs 2.8sec in go1.10)

I debugged the behavior of go tools and it seems like b.useCache in linkShared returns false to libstd.so target (https://github.com/golang/go/blob/master/src/cmd/go/internal/work/exec.go#L1032).

The text was updated successfully, but these errors were encountered:

FiloSottile · 2018-04-24T00:56:06Z

@gopherbot please open a backport tracking issue, as this is a 1.10 regression, although just a performance one.

gopherbot · 2018-04-24T00:56:11Z

Backport issue(s) opened: #25044 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

Foxboron · 2018-05-22T19:04:03Z

This isn't actually just a performance issue i believe. Currently this prevents users from running -buildmode=shared -linkshared as we don't have write privileges inn $GOROOT, and it attempts to write towards several files there.

With go 1.10

==> Starting build()...
WORK=/tmp/go-build138655936
rm -r $WORK/b003/
rm -r $WORK/b011/
rm -r $WORK/b007/
rm -r $WORK/b022/
mkdir -p $WORK/b010/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/
cp /home/fox/.cache/go-build/20/203799d24b5628cee748f251b4361e400b7392a9e65f21f6345ecb0ac1e99ca2-d /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a
rm -r $WORK/b010/
go install runtime/internal/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a: permission denied
rm -r $WORK/b023/
rm -r $WORK/b037/
mkdir -p $WORK/b012/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/sync/
cp /home/fox/.cache/go-build/59/59efc6e118c62c218e3e59d3e843f2e6895024fd8cfe6cba4d14ad046db4f7c4-d /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a
rm -r $WORK/b047/
rm -r $WORK/b012/
go install sync/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a: permission denied
rm -r $WORK/b048/
rm -r $WORK/b052/
mkdir -p $WORK/b014/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/internal/
cp /home/fox/.cache/go-build/5c/5cad130c59015083c83300f528455ed005db9a7862397945d4740e5e35e8f044-d /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a
rm -r $WORK/b014/
go install internal/cpu: open /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a: permission denied
mkdir -p $WORK/b053/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/
cp /home/fox/.cache/go-build/ae/aee318ece67832c036cdbc8be92aef902422e29e53942e16e2935a2dd7a7f0e7-d /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a
rm -r $WORK/b053/
rm -r $WORK/b077/
go install crypto/internal/cipherhw: open /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a: permission denied
rm -r $WORK/b079/
rm -r $WORK/b094/
rm -r $WORK/b100/
mkdir -p $WORK/b085/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/
cp /home/fox/.cache/go-build/27/278eecfb4ee5a9a34edbfc24a41c030a7f28e489c9a1edcc610606a7684baf40-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a
rm -r $WORK/b085/
mkdir -p $WORK/b084/
rm -r $WORK/b146/
cp /home/fox/.cache/go-build/a8/a883a42b6f584a222b93fd8c5df99499fd4dba6efcf90a2eb564ea90cdad6611-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a
rm -r $WORK/b084/
go install vendor/golang_org/x/crypto/poly1305: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a: permission denied
rm -r $WORK/b154/
rm -r $WORK/b155/
rm -r $WORK/b147/
rm -r $WORK/b156/
go install vendor/golang_org/x/crypto/curve25519: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a: permission denied
rm -r $WORK/b187/
rm -r $WORK/b186/
rm -r $WORK/b025/
rm -r $WORK/b180/

Meanwhile go 1.9.4

==> Starting build()...
WORK=/tmp/go-build073266895
golang.org/x/net/context
mkdir -p $WORK/golang.org/x/net/context/_obj/
mkdir -p $WORK/golang.org/x/net/
cd /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context
/usr/lib/go/pkg/tool/linux_amd64/compile -o $WORK/golang.org/x/net/context.a -trimpath $WORK -dynlink -goversion go1.9.4 -p golang.org/x/net/context -complete -installsuffix dynlink -buildid 3d063c354b94a000bba5463988c84e19a35173d6 -D _/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context -I $WORK -pack ./context.go ./go17.go ./go19.go
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/
cp $WORK/golang.org/x/net/context.a /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
cd .
/usr/lib/go/pkg/tool/linux_amd64/link -o $WORK/libgolang.org-x-net-context.so -L $WORK -L /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink -installsuffix dynlink -buildmode=shared -linkshared -w -extld=gcc golang.org/x/net/context=/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
echo 'libgolang.org-x-net-context.so' > /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.shlibname # internal
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/
cp $WORK/libgolang.org-x-net-context.so /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/libgolang.org-x-net-context.so

rsc · 2018-08-10T02:37:59Z

Sorry but not going to get to this for Go 1.11. Linkshared is admittedly a mess.

yunabe · 2019-02-28T15:19:16Z

Congratulations for go1.12 launch, Go team.

Do you think this linkShared related performance regression will be fixed in Go1.13?
https://github.com/yunabe/lgo, an interactive Go programming environment with Jupyter, heavily relies on go install -linkshared and it has a critical performance issue since go1.10 due to this bug.

Now, go1.12 was released and I want to stop using go1.9 in the project. I would appreciate it if this bug is fixed in the near future.

Thanks,
yunabe

tbolsh · 2019-09-26T14:54:01Z

Was this bug fixed in go 1.13 or 1.14?
Was the #30768 fixed in 1.13 or 1.14?

mvdan · 2019-12-03T13:51:40Z

@tbolsh this bug isn't fixed. #30768 was fixed in 1.13, which you can see via the milestone and commit.

rezaalavi · 2020-05-15T14:43:21Z

The bug still exists in 1.14.3

mvdan · 2020-05-15T14:45:46Z

@rezaalavi yes, the issue is open. Please don't comment in the issue if this is all you have to add.

ldemailly · 2020-08-25T17:54:10Z

being able to use golang in interactive notebooks is critical for both having people learn and teach go as well as build interactive science and quick prototypes and simulation, maybe the team can prioritize this regression a bit more?

seankhliao · 2021-10-27T21:51:18Z

Obsoleted by #47788

ianlancetaylor changed the title ~~go install -linkshared invokes link commands for existing .so files since go1.10~~ cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 Feb 22, 2018

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 22, 2018

ianlancetaylor added this to the Go1.10.1 milestone Feb 22, 2018

yunabe closed this as completed Feb 22, 2018

yunabe reopened this Feb 22, 2018

andybons mentioned this issue Mar 27, 2018

all: Go 1.10.1 release status #24563

Closed

andybons modified the milestones: Go1.10.1, Go1.10.2 Mar 27, 2018

yunabe mentioned this issue Apr 10, 2018

lgo kernel exits if GOPATH is unset yunabe/lgo#40

Closed

AlphaWong mentioned this issue Apr 22, 2018

Slow startup cjbassi/gotop#23

Closed

FiloSottile modified the milestones: Go1.10.2, Go1.11 Apr 24, 2018

gopherbot mentioned this issue Apr 24, 2018

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 [1.10 backport] #25044

Closed

yunabe mentioned this issue Apr 25, 2018

go1.10 support yunabe/lgo#32

Closed

rsc modified the milestones: Go1.11, Go1.12 Aug 10, 2018

Foxboron mentioned this issue Oct 22, 2018

cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT #27285

Closed

rsc modified the milestones: Go1.12, Go1.13 Nov 20, 2018

FiloSottile added the Performance label Dec 14, 2018

yunabe mentioned this issue Feb 28, 2019

can't work on termux Arch Linux in Android aarch64 yunabe/lgo#72

Closed

yunabe mentioned this issue Mar 22, 2019

strings.Builder is not available in yunabe/lgo:20190218 yunabe/lgo#80

Open

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

yunabe mentioned this issue Jul 9, 2019

Installation failure on go1.12 yunabe/lgo#82

Open

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

seankhliao closed this as completed Oct 27, 2021

rsc mentioned this issue Oct 29, 2021

cmd/go: remove -buildmode=shared (not c-shared) #47788

Closed

golang locked and limited conversation to collaborators Oct 27, 2022

gopherbot added the FrozenDueToAge label Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

yunabe commented Feb 22, 2018

FiloSottile commented Apr 24, 2018

gopherbot commented Apr 24, 2018

Foxboron commented May 22, 2018

rsc commented Aug 10, 2018

yunabe commented Feb 28, 2019

tbolsh commented Sep 26, 2019

mvdan commented Dec 3, 2019

rezaalavi commented May 15, 2020

mvdan commented May 15, 2020

ldemailly commented Aug 25, 2020

seankhliao commented Oct 27, 2021

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

cmd/go: install -linkshared invokes link commands for existing .so files since go1.10 #24034

Comments

yunabe commented Feb 22, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

FiloSottile commented Apr 24, 2018

gopherbot commented Apr 24, 2018

Foxboron commented May 22, 2018

rsc commented Aug 10, 2018

yunabe commented Feb 28, 2019

tbolsh commented Sep 26, 2019

mvdan commented Dec 3, 2019

rezaalavi commented May 15, 2020

mvdan commented May 15, 2020

ldemailly commented Aug 25, 2020

seankhliao commented Oct 27, 2021

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?