x/tools/go/packages: Unusably slow when there are packages using cgo in the list #29427

segevfiner · 2018-12-26T18:48:42Z

What version of Go are you using (`go version`)?

$ go version
go version go1.11.4 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=<snip>\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=<snip>\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=<snip>\Temp\go-build664130090=/tmp/go-build -gno-record-gcc-switches

What did you do?

I tried using a Go tool that uses golang.org/x/go/tools/packages under the hood (Specifically gogetdoc, but many more tools are going to use it in order to support Go modules). Since packages unconditionally calls go list -compiled (go/tools/go/packages/golist.go:630), when there are large cgo modules anywhere on the module list, packages will cause them to be compiled, without any caching. This can take a very long time and makes any tool doing so unusably slow.

Steps to reproduce

You need to have ImageMagick available. (See https://github.com/gographics/imagick)

Save the following as main.go in a new go package directory:

package main

import "gopkg.in/gographics/imagick.v2/imagick"

func main() {
	imagick.Initialize()
}

dep init/go get/whatever.
gogetdoc -pos main.go:#87.

What did you expect to see?

The tool works fast.

What did you see instead?

It's extremely slow, to the point of being unusable.

Related issues

zmb3/gogetdoc#47
microsoft/vscode-go#667
microsoft/vscode-go#1025
rogpeppe/godef#103

cc @zmb3 @ramya-rao-a

The text was updated successfully, but these errors were encountered:

ALTree · 2018-12-26T19:11:39Z

cc @matloob

heschi · 2018-12-26T19:54:57Z

#29371 is relevant. Does this reproduce at tip with that bug fixed? There's probably still a discussion to be had, but I imagine it'd be less urgent if it didn't happen every time.

heschi · 2018-12-26T19:58:32Z

Actually, never mind; that issue was specific to -find which wasn't in use until my https://golang.org/cl/155477 last week, and the related issues are much older.
@jayconrod

heschi · 2018-12-26T20:32:31Z

I'm not sure if this is because of cgo, or if it's just more noticeable because it takes so long to build cgo packages. I investigated by doing some go lists on a clean cache.

$ go clean -cache -modcache
$ go list -e -json -compiled -deps=true -- . | grep Stale
	"Stale": true,
	"StaleReason": "build ID mismatch",
	"Stale": true,
	"StaleReason": "stale dependency: gopkg.in/gographics/imagick.v2/imagick/types",
	"Stale": true,
	"StaleReason": "stale dependency: gopkg.in/gographics/imagick.v2/imagick/types",
$ go list -e -json -compiled -deps=true -- . | grep Stale          
	"Stale": true,
	"StaleReason": "build ID mismatch",
	"Stale": true,
	"StaleReason": "stale dependency: gopkg.in/gographics/imagick.v2/imagick/types",
	"Stale": true,
	"StaleReason": "stale dependency: gopkg.in/gographics/imagick.v2/imagick/types",
$ go build .
$ go list -e -json -compiled -deps=true -- . | grep Stale
	"Stale": true,
	"StaleReason": "not installed but available in build cache",
	"Stale": true,
	"StaleReason": "not installed but available in build cache",
	"Stale": true,
	"StaleReason": "not installed but available in build cache",

I would have expected the second command to reuse cached results from the first, but it doesn't. Doing a build manually puts things in the cache and the last list is fast. Seems like a bug to me.

klauspost · 2019-01-02T15:17:30Z

This is quite bad. Loading SSA for a package takes more than 1.5 hours. See referenced issue above.

heschi · 2019-01-03T19:14:50Z

I don't know what's going on in @klauspost's bug but it's not this one.

jayconrod · 2019-01-04T21:50:33Z

Some analysis:

gogetdoc invokes packages.Load in AllSyntax mode, which asks for all available information on the package and its dependencies. Maybe this tool could ask for less. Looks like it wants type information, but maybe it could just get syntax trees?
packages invokes go list -json -compiled -test=false -export=false -deps=true -- $PWD. As @segevfiner points out in the original report, -compiled is unconditional. Ideally, this would not be necessary for all modes, but it sounds like the current contract requires it.
Because -compiled and -deps=true are passed, go list needs cgo results from every package in the dependency graph. cgo is very expensive for imagemagick. I'm not sure if it's doing a huge amount of work or if there's some optimization opportunity there.
For packages that were cached from a previous run of go build, go list will re-use cached cgo files. Since b7451e2 (two weeks ago, not in 1.11), go list will also re-use cached cgo files from previous runs of go list (though go build won't re-use those files because of the way cache keys are derived).

I think this largely has the same root cause as #29371, so maybe this should be closed. However, I think it would be interesting to discuss whether go/packages could be invoked in a way that doesn't require cgo. cc @matloob

matloob · 2019-03-21T19:52:05Z

I'm working on a change (https://go-review.googlesource.com/c/tools/+/162140) that will allow tool to specify that they don't need the CompiledGoFiles field. Once that change is in, if a tool doesn't request CompiledGoFiles, or other Package struct fields that need compiled files, go/packages will invoke go list without -compiled.

gopherbot · 2019-03-21T19:56:24Z

Change https://golang.org/cl/162140 mentions this issue: go/packages: split LoadMode into fine-grained options

The options are all unexported, and this CL is (almost) a no-op: the one difference is that since needImports and needSyntax are now independently specified, LoadSyntax and LoadAllSyntax are equivalent, because LoadSyntax needs both the needImports and needSyntax bits. I want to pin down the options that we want to split into, and future CLs can allow the options to be used individually... Updates golang/go#29429 Updates golang/go#29427 Change-Id: I5b2913e2c53e7ade56905e46912b076ccc339827 Reviewed-on: https://go-review.googlesource.com/c/tools/+/162140 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>

jayconrod · 2021-10-14T16:50:30Z

cc @matloob: is there anything else to do for this issue? It sounds like after https://golang.org/cl/162140, cgo is not always needed, but when it is needed, it still takes a long time.

StevenACoffman · 2021-11-09T15:44:52Z

As an example, a small project that is using cgo (via sqlc) running:

time go list -m -u all

resulted in:

real	11m17.749s
user	0m21.221s
sys	0m22.311s

But for some reason this is speedy:

time go list -m -f '{{if not (or .Indirect .Main)}}go get -u {{.Path}}@latest{{end}}' all

and results in:

real	0m0.243s
user	0m0.448s
sys	0m0.285s

gopherbot added this to the Unreleased milestone Dec 26, 2018

ALTree added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance labels Dec 26, 2018

heschi added GoCommand cmd/go and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 26, 2018

ALTree mentioned this issue Dec 29, 2018

cmd/go: go list has too many (more than zero) side effects #29452

Closed

klauspost mentioned this issue Jan 2, 2019

Package loading time in Go 1.10.x dominikh/go-tools#377

Closed

jayconrod self-assigned this Jan 2, 2019

This was referenced Jan 11, 2019

v1.2.0 enters infinite loop kisielk/errcheck#162

Closed

builder: upgrade to go1.11.4 cockroachdb/cockroach#33668

Merged

segevfiner mentioned this issue Jan 23, 2019

Perfomance downgrade when using "go module" mode microsoft/vscode-go#2251

Closed

anjmao mentioned this issue Mar 9, 2019

x/tools/go/packages: bad performance #30677

Closed

vektah mentioned this issue May 12, 2019

make gqlgen generate 10x faster in some projects 99designs/gqlgen#707

Merged

2 tasks

segevfiner mentioned this issue Jul 22, 2019

x/tools/go/packages: high CPU usage while processing cgo file #32821

Open

rcoreilly mentioned this issue Feb 15, 2020

packages.Load slows *everything* down even though it is in a separate thread (go mod mode) cogentcore/cogent#150

Closed

mgartner mentioned this issue Apr 5, 2020

Alternative to gogetdoc for :GoDoc fatih/vim-go#2808

Closed

rcoreilly mentioned this issue Feb 26, 2021

proposal: cmd/go: keep GOPATH mode #37755

Closed

jayconrod removed their assignment Oct 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/go/packages: Unusably slow when there are packages using cgo in the list #29427

x/tools/go/packages: Unusably slow when there are packages using cgo in the list #29427

segevfiner commented Dec 26, 2018

ALTree commented Dec 26, 2018

heschi commented Dec 26, 2018

heschi commented Dec 26, 2018

heschi commented Dec 26, 2018

klauspost commented Jan 2, 2019

heschi commented Jan 3, 2019

jayconrod commented Jan 4, 2019

matloob commented Mar 21, 2019

gopherbot commented Mar 21, 2019

jayconrod commented Oct 14, 2021

StevenACoffman commented Nov 9, 2021

x/tools/go/packages: Unusably slow when there are packages using cgo in the list #29427

x/tools/go/packages: Unusably slow when there are packages using cgo in the list #29427

Comments

segevfiner commented Dec 26, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

Steps to reproduce

What did you expect to see?

What did you see instead?

Related issues

ALTree commented Dec 26, 2018

heschi commented Dec 26, 2018

heschi commented Dec 26, 2018

heschi commented Dec 26, 2018

klauspost commented Jan 2, 2019

heschi commented Jan 3, 2019

jayconrod commented Jan 4, 2019

matloob commented Mar 21, 2019

gopherbot commented Mar 21, 2019

jayconrod commented Oct 14, 2021

StevenACoffman commented Nov 9, 2021

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?