Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: Load function throws "argument list too long" error #36909

Closed
awelc opened this issue Jan 30, 2020 · 3 comments
Closed

x/tools/go/packages: Load function throws "argument list too long" error #36909

awelc opened this issue Jan 30, 2020 · 3 comments
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@awelc
Copy link

awelc commented Jan 30, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What is the problem?

I am trying to analyze a large application using Go's static analysis support. I am using the "packages" package to load all application packages using the Load function. Unfortunately the application is really big and the list of patterns that the Load function takes the second parameter is really long. This causes a problem on Mac OS as in this case it leads to the "argument list too long" error popping up after the Load function is called. The reason for it is that internally, the Load function executes "go list" command passing it the list of patterns as arguments, which leads to exceeding the limit established by Mac OS (which, as far as I can tell, cannot be changed).

The usual workaround for this problem is to split the argument list and apply a given function incrementally to each list fragment - in this case it would mean calling the Load function multiple times with a different subset of the patterns list at each invocation. This, however, does not quite work, as it messes up the SSA representation that I need to generate afterwards using AllPackages function from the "ssautil" package. The AllPackages function constructs a program that contains a "file set" containing all files constituting the program. As far as I can tell, the only way to access this file set during analysis is via the program itself. And this is the gist of the problem, incremental application of the Load function creates a "file set" per increment, and the resulting program during SSA construction will assign only one to the program, making the remaining ones inaccessible during analysis.

What would be great is to have an API-level solution that would allow incremental loading of packages.

@gopherbot gopherbot added this to the Unreleased milestone Jan 30, 2020
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Jan 30, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 7, 2020
@cagedmantis
Copy link
Contributor

/cc @matloob

@matloob
Copy link
Contributor

matloob commented Feb 13, 2020

I think the solution to this is to have go/packages split the arguments to go list (or to the underlying GOPACKAGESDRIVER if it's set) and make multiple calls to the driver. those results should be able to just be merged (removing duplicates if, for example, -deps is set as a go list argument). The pcakage structures returned by the driver are independent, and the driver should return consistent ids over several calls. Then those can be supplied to the "refining" stage where we parse/typecheck the packages.

@matloob matloob added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Feb 13, 2020
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 13, 2020
@gopherbot
Copy link

Change https://go.dev/cl/550477 mentions this issue: x/tools/go/packages: fix Load's "argument list too long" error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

4 participants