Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: needs more explanations and examples #35872

Open
go101 opened this issue Nov 27, 2019 · 5 comments
Open

x/tools/go/packages: needs more explanations and examples #35872

go101 opened this issue Nov 27, 2019 · 5 comments
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@go101
Copy link

go101 commented Nov 27, 2019

Edit: It looks I used this package wrongly. On the other hand, I think the docs of this package is lack of detailed explanations and examples. I converted this issue to a document enrichment request. For examples, what are "the package as compiled for the test" and "the test binary". And some command line argument examples may be helpful.

And is the paragraph problematic?

As noted earlier, the Config.Mode controls the amount of detail reported about the loaded packages, with each mode returning all the data of the previous mode with some extra added. See the documentation for type LoadMode for details.

Is it intended to explain the deprecated LoadFiles<LoadImports<LoadTypes<LoadSyntax<LoadAllSyntax preset modes?

(The following is the old content)

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

package main

import (
    "fmt"
    "time"
    
    "go/ast"
    "go/parser"
    "go/token"

    "golang.org/x/tools/go/packages"
)

func main() {
	pkgs, err := packages.Load(nil, "std")
	if err != nil {
		panic(err)
	}

	var configForParsing = &packages.Config{
		Mode: packages.NeedName | packages.NeedImports | packages.NeedTypes |
			packages.NeedDeps | packages.NeedExportsFile | packages.NeedFiles |
			packages.NeedCompiledGoFiles | packages.NeedTypesSizes |
			packages.NeedSyntax | packages.NeedTypesInfo,
		Tests: false,

		ParseFile: func(fset *token.FileSet, parseFilename string, _ []byte) (*ast.File, error) {
			var src interface{}
			mode := parser.ParseComments // | parser.AllErrors
			file, err := parser.ParseFile(fset, parseFilename, src, mode)
			if file == nil {
				return nil, err
			}
			for _, decl := range file.Decls {
				if fd, ok := decl.(*ast.FuncDecl); ok {
					fd.Body = nil
				}
			}
			return file, nil
		},
	}

	parsedPkgs := make([]*packages.Package, 0, 1000)
	for _, pkg := range pkgs {

		fmt.Println(pkg.PkgPath)
		
		pkgs, err := packages.Load(configForParsing, pkg.PkgPath)
		if err != nil {
			fmt.Println("packages.Load error:", err)
		}
		parsedPkgs = append(parsedPkgs, pkgs...)
	}
	
	fmt.Println("[done]")
    
	for {
		time.Sleep(time.Hour)
	}
	
	fmt.Println(parsedPkgs)
}

What did you expect to see?

<1G memory is consumed.

What did you see instead?

7G memory is consumed.

Initially I posted this result here: https://groups.google.com/forum/#!topic/golang-tools/cLIwae8rx1g
Alan Donovan said this is not normal and Josh Bleecher Snyder suggested me post an issue here, so I do.

@go101
Copy link
Author

go101 commented Nov 27, 2019

BTW, if the ParseFile option is not set, 18G memory will be consumed.

@go101
Copy link
Author

go101 commented Nov 27, 2019

Er, maybe I haven't got how to use the go/packages package.
It looks the following program also parses all std packages, but only uses 200M memory.

package main

import (
    "fmt"
    "time"
    
    "go/ast"
    "go/parser"
    "go/token"

    "golang.org/x/tools/go/packages"
)

func main() {
	var configForParsing = &packages.Config{
		Mode: packages.NeedName | packages.NeedImports | packages.NeedTypes |
			packages.NeedDeps | packages.NeedExportsFile | packages.NeedFiles |
			packages.NeedCompiledGoFiles | packages.NeedTypesSizes |
			packages.NeedSyntax | packages.NeedTypesInfo,
		Tests: false,

		ParseFile: func(fset *token.FileSet, parseFilename string, _ []byte) (*ast.File, error) {
			var src interface{}
			mode := parser.ParseComments // | parser.AllErrors
			file, err := parser.ParseFile(fset, parseFilename, src, mode)
			if file == nil {
				return nil, err
			}
			for _, decl := range file.Decls {
				if fd, ok := decl.(*ast.FuncDecl); ok {
					fd.Body = nil
				}
			}
			return file, nil
		},
	}

		
	pkgs, err := packages.Load(configForParsing, "std")
	if err != nil {
		fmt.Println("packages.Load error:", err)
	}

	for _, pkg := range pkgs {
		fmt.Println(pkg.PkgPath, len(pkg.Syntax), len(pkg.Imports))
	}
	
	fmt.Println("[done]")
    
	for {
		time.Sleep(time.Hour)
	}
	
	fmt.Println(pkgs)
}

Are there any other docs/tutorials on how to use this package, except the package docs (https://godoc.org/golang.org/x/tools/go/packages)?

@mvdan
Copy link
Member

mvdan commented Nov 27, 2019

It looks like in the original program, you were loading all of std first, then each of its packages again, including all of their dependencies. So you'd probably end up loading a low-level package like runtime over fifty times.

It's not surprising that it used over ten times more memory :)

@go101
Copy link
Author

go101 commented Nov 27, 2019

Yes, it is 35 times! ;D

I will modify this issue to a document enrichment request.

@go101 go101 changed the title go/packages: memory hungry in loading packages go/packages: needs more explanations and examples Nov 27, 2019
@ALTree ALTree added this to the Unplanned milestone Nov 28, 2019
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 2, 2019
@dmitshur dmitshur changed the title go/packages: needs more explanations and examples x/tools/go/packages: needs more explanations and examples Dec 2, 2019
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Dec 2, 2019
@gopherbot
Copy link

Change https://golang.org/cl/209977 mentions this issue: go/packages: remove obsolete comment about LoadMode in doc.go

gopherbot pushed a commit to golang/tools that referenced this issue Dec 5, 2019
The comment referred to each mode returning more data about previous modes,
which was true about the old LoadMode hierarchy, but doesn't apply to the
new bits system.

Updates golang/go#35872

Change-Id: Id8354f49fdebcb231df8e5ece58644a29d678e4a
Reviewed-on: https://go-review.googlesource.com/c/tools/+/209977
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

5 participants