Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: buildmode=plugin always recompile all dependencies #19707

Closed
LionNatsu opened this issue Mar 25, 2017 · 3 comments
Closed

cmd/go: buildmode=plugin always recompile all dependencies #19707

LionNatsu opened this issue Mar 25, 2017 · 3 comments
Labels
GoCommand cmd/go help wanted NeedsFix The path to resolution is known, but the work has not been done. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@LionNatsu
Copy link
Contributor

go version go1.8 linux/amd64

What did you do?

For example,

package main
var V int
go build -buildmode=plugin -v test/main.go

What did you expect to see?

lion@Lion-Laptop [ test ] $ go build -buildmode=plugin -v test/main.go
command-line-arguments
lion@Lion-Laptop [ test ] $

What did you see instead?

lion@Lion-Laptop [ test ] $ go build -buildmode=plugin -v test/main.go
runtime/internal/sys
runtime/internal/atomic
runtime
runtime/cgo
command-line-arguments
lion@Lion-Laptop [ test ] $

The same as building with -a
@crawshaw
Copy link
Member

crawshaw commented Sep 2, 2017

This is because all packages need to be rebuilt by the toolchain configured for dynamic linking, and by default go build does not cache intermediate package builds. I believe work on cmd/go will change that, and this problem will go away.

Until then, you can use -i when building plugins to cache intermediate package builds, so this only happens the first time:

$ go build -buildmode=plugin -i -v plugin1
runtime/internal/sys
runtime/internal/atomic
runtime
common
internal/cpu
errors
unicode/utf8
sync/atomic
unicode
internal/race
math
sync
syscall
strconv
reflect
runtime/cgo
plugin1
$ go build -buildmode=plugin -i -v plugin1
plugin1
$

samcday added a commit to samcday/aws-lambda-go-shim that referenced this issue Oct 9, 2017
@samcday
Ensure dynlink cache is preserved between builds
bf66b52 
First up, awesome project. Huge fan! :)

I started using it but quickly noticed that the builds for my Lambda function were getting crazy slow as I imported more packages. I did a bit of digging and it turns out running `go build buildmode=plugin` doesn't properly cache the `pkg/` artifacts unless you tell it to with the `-i` option. That limitation is kinda hidden away in golang/go#19707 and I didn't find it until the third or fourth frustrated google search ;)

So what this does is adds the `-i` option to the build command, and ensures we have a data volume attached to `/usr/local/go/pkg/linux_amd64_dynlink` in the container so that the pkg cache is reused between builds.

Before this change:

```
$ time make docker
docker run --rm\
	-e HANDLER=handler\
	-e PACKAGE=handler\
	-e GOPATH=/home/sam\
	-e LDFLAGS=''\
	-v /home/sam/src/github.com/samcday/scrobbify/backend:/home/sam/src/github.com/samcday/scrobbify/backend\
	-v /home/sam:/home/sam\
	-w /home/sam/src/github.com/samcday/scrobbify/backend\
	eawsy/aws-lambda-go-shim:latest make -f Makefile all
go build -v -buildmode=plugin -ldflags='-w -s ' -o handler.so
runtime/internal/sys
runtime/internal/atomic
runtime
errors
internal/race
sync/atomic
internal/cpu
unicode/utf8
unicode
encoding
unicode/utf16
math/bits
container/list
sync
crypto/subtle
crypto/internal/cipherhw
math
internal/nettrace
vendor/golang_org/x/crypto/poly1305
vendor/golang_org/x/crypto/curve25519
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/client/metadata
github.com/samcday/scrobbify/vendor/github.com/eawsy/aws-lambda-go-core/service/lambda/runtime
io
syscall
internal/singleflight
hash
crypto/cipher
hash/crc32
crypto/hmac
strings
bytes
bufio
vendor/golang_org/x/text/transform
strconv
path
math/rand
time
internal/syscall/unix
reflect
crypto/rc4
crypto/aes
crypto
encoding/base64
crypto/sha512
crypto/sha1
crypto/md5
crypto/sha256
internal/poll
os
fmt
encoding/binary
sort
path/filepath
regexp/syntax
encoding/pem
crypto/des
vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20
vendor/golang_org/x/crypto/chacha20poly1305
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/internal/shareddefaults
io/ioutil
context
compress/flate
encoding/json
log
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/awserr
math/big
encoding/hex
vendor/golang_org/x/net/http2/hpack
vendor/golang_org/x/text/unicode/norm
net/url
vendor/golang_org/x/text/unicode/bidi
regexp
mime
mime/quotedprintable
net/http/internal
compress/gzip
runtime/cgo
vendor/golang_org/x/text/secure/bidirule
encoding/xml
github.com/samcday/scrobbify/vendor/github.com/go-ini/ini
github.com/samcday/scrobbify/vendor/golang.org/x/net/context
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/endpoints
github.com/samcday/scrobbify/vendor/github.com/jmespath/go-jmespath
github.com/samcday/scrobbify/vendor/github.com/eawsy/aws-lambda-go-event/service/lambda/runtime/event/apigatewayproxyevt
vendor/golang_org/x/net/idna
crypto/rand
crypto/elliptic
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/credentials
encoding/asn1
crypto/rsa
crypto/dsa
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/awsutil
crypto/ecdsa
crypto/x509/pkix
net
crypto/x509
vendor/golang_org/x/net/lex/httplex
vendor/golang_org/x/net/proxy
github.com/samcday/scrobbify/vendor/github.com/eawsy/aws-lambda-go-net/service/lambda/runtime/net
net/textproto
mime/multipart
crypto/tls
github.com/samcday/scrobbify/vendor/github.com/dgrijalva/jwt-go
net/http/httptrace
net/http
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws
net/http/httputil
github.com/samcday/scrobbify/vendor/github.com/eawsy/aws-lambda-go-net/service/lambda/runtime/net/apigatewayproxy
github.com/samcday/scrobbify/vendor/golang.org/x/net/context/ctxhttp
github.com/samcday/scrobbify/vendor/golang.org/x/oauth2/internal
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/request
github.com/samcday/scrobbify/vendor/golang.org/x/oauth2
github.com/samcday/scrobbify/vendor/github.com/zmb3/spotify
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/client
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/rest
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/corehandlers
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/query/queryutil
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/ec2metadata
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/signer/v4
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/jsonrpc
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/private/protocol/query
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/defaults
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/service/sts
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/service/dynamodb
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/credentials/stscreds
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/aws/session
github.com/samcday/scrobbify/vendor/github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute
github.com/samcday/scrobbify/backend
pack handler handler.so handler.zip
chown 1000:1000 handler.so handler.zip
make docker 0.01s user 0.03s system 0% cpu 6.252 total
```

After:

```
$ time make docker
docker run --rm\
	-e HANDLER=handler\
	-e PACKAGE=handler\
	-e GOPATH=/home/sam\
	-e LDFLAGS=''\
	-v lambdagoshimcache:/usr/local/go/pkg/linux_amd64_dynlink\
	-v /home/sam/src/github.com/samcday/scrobbify/backend:/home/sam/src/github.com/samcday/scrobbify/backend\
	-v /home/sam:/home/sam\
	-w /home/sam/src/github.com/samcday/scrobbify/backend\
	eawsy/aws-lambda-go-shim:latest make -f Makefile all
go build -v -i -buildmode=plugin -ldflags='-w -s ' -o handler.so
github.com/samcday/scrobbify/backend
pack handler handler.so handler.zip
chown 1000:1000 handler.so handler.zip
make docker 0.04s user 0.02s system 2% cpu 1.904 total
```
@samcday samcday mentioned this issue Oct 9, 2017
Closed
@andybons andybons added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Apr 11, 2018
@andybons andybons added this to the Unplanned milestone Apr 11, 2018
@bcmills bcmills added the GoCommand cmd/go label Sep 7, 2023
@bcmills
Copy link
Contributor

bcmills commented Sep 7, 2023

It's been a long time since this issue was filed, and cmd/go's caching for dependencies has changed a lot. Is this still an issue?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 7, 2023
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
GoCommand cmd/go help wanted NeedsFix The path to resolution is known, but the work has not been done. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
5 participants
@crawshaw @andybons @bcmills @LionNatsu @gopherbot