Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/go/packages: memory hungry in parsing the github.com/go-fonts/latin-modern module packages #57985

Open
go101 opened this issue Jan 25, 2023 · 3 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@go101
Copy link

go101 commented Jan 25, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20rc3 linux/amd64

The version of the golang.org/x/tools module is v0.5.0.

Does this issue reproduce with the latest release?

Yes

What did you do?

main.go:

package main

import (
	"log"
	"runtime"
	"runtime/metrics"

	"golang.org/x/tools/go/packages"

	_ "github.com/go-fonts/latin-modern/lmmath"
)

func printUsedMemory(prefix string) {
	runtime.GC()

	samples := []metrics.Sample{
		{Name: "/memory/classes/total:bytes"},
		{Name: "/memory/classes/heap/free:bytes"},
		{Name: "/memory/classes/heap/objects:bytes"},
		{Name: "/gc/heap/goal:bytes"},
	}
	metrics.Read(samples)
	
	log.Println("=========", prefix)
	for _, s := range samples {
		log.Printf("%44s %dMiB", s.Name, s.Value.Uint64()>>20)
	}
}

var allPkgs []*packages.Package

func main() {
	log.SetFlags(0)

	var configForParsing = &packages.Config{
		Mode: packages.NeedName | packages.NeedImports | packages.NeedDeps |
			packages.NeedTypes | packages.NeedExportsFile | packages.NeedFiles |
			packages.NeedCompiledGoFiles | packages.NeedTypesSizes |
			packages.NeedSyntax | packages.NeedTypesInfo,
		Tests: false,
	}

	printUsedMemory("before parsing:")

	pkgs, err := packages.Load(configForParsing, "github.com/go-fonts/latin-modern/...")
	if err != nil {
		log.Println("packages.Load (parse packages): %w", err)
		return
	}

	printUsedMemory("after parsing:")

	allPkgs = pkgs
}

go,mod:

module a.b/c

go 1.20

require golang.org/x/tools v0.5.0

require github.com/go-fonts/latin-modern v0.2.0

require (
	golang.org/x/mod v0.7.0 // indirect
	golang.org/x/sys v0.4.0 // indirect
)

Output:

========= before parsing:
                 /memory/classes/total:bytes 11MiB
             /memory/classes/heap/free:bytes 0MiB
          /memory/classes/heap/objects:bytes 0MiB
                         /gc/heap/goal:bytes 4MiB
========= after parsing:
                 /memory/classes/total:bytes 3079MiB
             /memory/classes/heap/free:bytes 1290MiB
          /memory/classes/heap/objects:bytes 1669MiB
                         /gc/heap/goal:bytes 3338MiB

What did you expect to see?

About several hundred of memory consumed.

What did you see instead?

About 3G memory consumed.

Each of the font source files only contains a var TTF = []byte{...} variable.
It is unexpected to use so much memory.

When parsing the packages in the github.com/tdewolff/canvas module,
which depends on the github.com/go-fonts/latin-modern module,
my computer (8G memory) hangs for virtual memory are swapped in and out frequently.

The github.com/tdewolff/canvas module is a relative small module.
My computer is able to parse much larger modules.

@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Jan 25, 2023
@gopherbot gopherbot added this to the Unreleased milestone Jan 25, 2023
@mknyszek mknyszek added Performance NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 30, 2023
@mknyszek
Copy link
Contributor

CC @golang/tools-team

@findleyr
Copy link
Contributor

CC @griesemer @adonovan

This is almost certainly caused by recording information in types.Info for each element of those byte literals. We could potentially optimize go/types for such cases.

@findleyr
Copy link
Contributor

Tentatively marking this for Go 1.21, to see if we can improve memory consumption in go/types in these cases.

@gopherbot gopherbot modified the milestones: Go1.21, Go1.22 Aug 8, 2023
@gopherbot gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

4 participants