Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/gopls: cannot handle large files #33003

Closed
myitcv opened this issue Jul 9, 2019 · 3 comments
Closed

x/tools/gopls: cannot handle large files #33003

myitcv opened this issue Jul 9, 2019 · 3 comments
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@myitcv
Copy link
Member

myitcv commented Jul 9, 2019

What version of Go are you using (go version)?

$ go version
go version devel +13327f219e Sat Jul 6 13:25:59 2019 +0000 linux/amd64
$ go list -m golang.org/x/tools
golang.org/x/tools v0.0.0-20190708171232-dd7c7173f160
$ go list -m golang.org/x/tools/gopls
golang.org/x/tools/gopls v0.1.2-0.20190708171232-dd7c7173f160

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build907756658=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Open $(go env GOROOT)/src/cmd/compile/internal/ssa/rewriteAMD64.go in your editor, and remove the indentation from every line in the file. Now trigger goimports by whatever means necessary in your editor; you will see gopls run out of memory before the method returns:

 fatal error: runtime: out of memory
 
 runtime stack:
 runtime.throw(0xadb7c1, 0x16)
 	/home/myitcv/gos/src/runtime/panic.go:758 +0x72
 runtime.sysMap(0xc210000000, 0x4000000, 0xfd5438)
 	/home/myitcv/gos/src/runtime/mem_linux.go:169 +0xc5
 runtime.(*mheap).sysAlloc(0xfbcfa0, 0x20e000, 0x18000376a8, 0x8ceaa0)
 	/home/myitcv/gos/src/runtime/malloc.go:681 +0x1cd
 runtime.(*mheap).grow(0xfbcfa0, 0x107, 0xffffffff)
 	/home/myitcv/gos/src/runtime/mheap.go:1244 +0x42
 runtime.(*mheap).allocSpanLocked(0xfbcfa0, 0x107, 0xfd5448, 0xf742c0)
 	/home/myitcv/gos/src/runtime/mheap.go:1155 +0x291
 runtime.(*mheap).alloc_m(0xfbcfa0, 0x107, 0x101, 0x0)
 	/home/myitcv/gos/src/runtime/mheap.go:1007 +0xc2
 runtime.(*mheap).alloc.func1()
 	/home/myitcv/gos/src/runtime/mheap.go:1078 +0x4c
 runtime.(*mheap).alloc(0xfbcfa0, 0x107, 0x7fc6c3010101, 0x4292d0)
 	/home/myitcv/gos/src/runtime/mheap.go:1077 +0x8a
 runtime.largeAlloc(0x20d6c8, 0x450101, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/malloc.go:1103 +0x97
 runtime.mallocgc.func1()
 	/home/myitcv/gos/src/runtime/malloc.go:998 +0x46
 runtime.systemstack(0xc00029c180)
 	/home/myitcv/gos/src/runtime/asm_amd64.s:370 +0x66
 runtime.mstart()
 	/home/myitcv/gos/src/runtime/proc.go:1146
 
 goroutine 2805 [running]:
 runtime.systemstack_switch()
 	/home/myitcv/gos/src/runtime/asm_amd64.s:330 fp=0xc008b19308 sp=0xc008b19300 pc=0x45da60
 runtime.mallocgc(0x20d6c8, 0x9c8c00, 0xfffffffffffff601, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/malloc.go:997 +0x85c fp=0xc008b193a8 sp=0xc008b19308 pc=0x40e19c
 runtime.makeslice(0x9c8c00, 0x41ad9, 0x41ad9, 0xc20fdc4000)
 	/home/myitcv/gos/src/runtime/slice.go:49 +0x6c fp=0xc008b193d8 sp=0xc008b193a8 pc=0x4469fc
 golang.org/x/tools/internal/lsp/diff.shortestEditSequence(0xc047034000, 0x106b6, 0x106b7, 0xc04713c000, 0x106b6, 0x106b7, 0x1, 0x1, 0x7fc6a6638a90, 0x106b6)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/diff/diff.go:177 +0x394 fp=0xc008b19460 sp=0xc008b193d8 pc=0x8cf5f4
 golang.org/x/tools/internal/lsp/diff.Operations(0xc047034000, 0x106b6, 0x106b7, 0xc04713c000, 0x106b6, 0x106b7, 0xc00002a2d0, 0x28, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/diff/diff.go:70 +0x81 fp=0xc008b19528 sp=0xc008b19460 pc=0x8ceaa1
 golang.org/x/tools/internal/lsp/source.computeTextEdits(0xbb9ca0, 0xc0250a66f0, 0xbbc4a0, 0xc0000ca3c0, 0xc046d96000, 0x15db55, 0x0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/source/format.go:145 +0x391 fp=0xc008b19628 sp=0xc008b19528 pc=0x8e66b1
 golang.org/x/tools/internal/lsp/source.Imports(0xbb9ca0, 0xc0250a66f0, 0xbc3380, 0xc000272340, 0xbc2080, 0xc0000ca3c0, 0xc0001abe00, 0xd4d18, 0x2134f4, 0x0, ...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/source/format.go:83 +0x518 fp=0xc008b19748 sp=0xc008b19628 pc=0x8e5b58
 golang.org/x/tools/internal/lsp.organizeImports(0xbb9ca0, 0xc0250a66f0, 0xbc3380, 0xc000272340, 0xc0000301e0, 0x44, 0x1, 0x1, 0x0, 0x1, ...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/code_action.go:103 +0x131 fp=0xc008b19800 sp=0xc008b19748 pc=0x92df41
 golang.org/x/tools/internal/lsp.(*Server).codeAction(0xc000224000, 0xbb9ca0, 0xc0250a66f0, 0xc0244752c0, 0xc0244752c0, 0x0, 0x0, 0x0, 0xc0249aae70)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/code_action.go:48 +0x3ad fp=0xc008b19ba8 sp=0xc008b19800 pc=0x92d27d
 golang.org/x/tools/internal/lsp.(*Server).CodeAction(0xc000224000, 0xbb9ca0, 0xc0250a66f0, 0xc0244752c0, 0xc0244752c0, 0x0, 0x0, 0x493ff2, 0xc0001af860)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/server.go:204 +0x4d fp=0xc008b19c00 sp=0xc008b19ba8 pc=0x938bdd
 golang.org/x/tools/internal/lsp/protocol.serverHandler.func1(0xbb9ca0, 0xc0250a66f0, 0xc02508a060)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/protocol/tsserver.go:355 +0x2b1a fp=0xc008b19f00 sp=0xc008b19c00 pc=0x73929a
 golang.org/x/tools/internal/jsonrpc2.(*Conn).Run.func1(0xc0001d2000, 0xc02508a060, 0xc000226000, 0xbb9ca0, 0xc0250a66f0, 0xc0250956d0, 0xc006df3d00)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:441 +0x15c fp=0xc008b19fa8 sp=0xc008b19f00 pc=0x72cfdc
 runtime.goexit()
 	/home/myitcv/gos/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc008b19fb0 sp=0xc008b19fa8 pc=0x45fb31
 created by golang.org/x/tools/internal/jsonrpc2.(*Conn).Run
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:428 +0x746
 
 goroutine 1 [syscall]:
 syscall.Syscall(0x0, 0x0, 0xc000222000, 0x1000, 0xc000239438, 0x800000, 0x42f74a)
 	/home/myitcv/gos/src/syscall/asm_linux_amd64.s:18 +0x5
 syscall.read(0x0, 0xc000222000, 0x1000, 0x1000, 0x0, 0x0, 0x6)
 	/home/myitcv/gos/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
 syscall.Read(...)
 	/home/myitcv/gos/src/syscall/syscall_unix.go:187
 internal/poll.(*FD).Read(0xc0000b0000, 0xc000222000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
 	/home/myitcv/gos/src/internal/poll/fd_unix.go:165 +0x164
 os.(*File).read(...)
 	/home/myitcv/gos/src/os/file_unix.go:259
 os.(*File).Read(0xc0000ae000, 0xc000222000, 0x1000, 0x1000, 0xc0250a09b5, 0xae7715, 0x1)
 	/home/myitcv/gos/src/os/file.go:112 +0x71
 bufio.(*Reader).fill(0xc0001af8c0)
 	/home/myitcv/gos/src/bufio/bufio.go:100 +0x103
 bufio.(*Reader).ReadSlice(0xc0001af8c0, 0xa, 0x8dfc45d58af49759, 0xc000239628, 0x40a95b, 0xc0250a3070, 0x8dfc45d58af49759)
 	/home/myitcv/gos/src/bufio/bufio.go:359 +0x3d
 bufio.(*Reader).ReadBytes(0xc0001af8c0, 0xa, 0xc000239600, 0xc0001cc7e0, 0xc0001cc8a8, 0xaff1a8, 0x7fc6ddad2b28)
 	/home/myitcv/gos/src/bufio/bufio.go:437 +0x73
 bufio.(*Reader).ReadString(...)
 	/home/myitcv/gos/src/bufio/bufio.go:477
 golang.org/x/tools/internal/jsonrpc2.(*headerStream).Read(0xc0001a89a0, 0xbb9c20, 0xc0000aa000, 0xc02508a060, 0xc000226000, 0xbb9ca0, 0xc0250a66f0, 0xc0250956d0, 0xc006df3d00)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/stream.go:97 +0x92
 golang.org/x/tools/internal/jsonrpc2.(*Conn).Run(0xc000226000, 0xbb9c20, 0xc0000aa000, 0xc0001a89e0, 0xc000224000)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/jsonrpc2/jsonrpc2.go:396 +0xa2
 golang.org/x/tools/internal/lsp.(*Server).Run(...)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/server.go:61
 golang.org/x/tools/internal/lsp/cmd.(*Serve).Run(0xc00021a030, 0xbb9c20, 0xc0000aa000, 0xc0000b2030, 0x0, 0x0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/cmd/serve.go:95 +0x515
 golang.org/x/tools/internal/tool.Main.func2(0x0, 0xbbc520, 0xc00021a030, 0xbb9c20, 0xc0000aa000, 0xc0001af800, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:130 +0xda
 golang.org/x/tools/internal/tool.Main(0xbb9c20, 0xc0000aa000, 0xbbc520, 0xc00021a030, 0xc0000b2030, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:131 +0x241
 golang.org/x/tools/internal/lsp/cmd.(*Application).Run(0xc00021a000, 0xbb9c20, 0xc0000aa000, 0xc0000b2030, 0x0, 0x0, 0x7, 0x199)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/lsp/cmd/cmd.go:106 +0x48c
 golang.org/x/tools/internal/tool.Main.func2(0xc00021a000, 0xbbc4e0, 0xc00021a000, 0xbb9c20, 0xc0000aa000, 0xc0001af7a0, 0x0, 0x0)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:130 +0xda
 golang.org/x/tools/internal/tool.Main(0xbb9c20, 0xc0000aa000, 0xbbc4e0, 0xc00021a000, 0xc0000b2010, 0x3, 0x3)
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools@v0.0.0-20190706070813-72ffa07ba3db/internal/tool/tool.go:131 +0x241
 main.main()
 	/home/myitcv/gostuff/pkg/mod/golang.org/x/tools/gopls@v0.1.2-0.20190706070813-72ffa07ba3db/main.go:20 +0xba

This was discovered whilst I was trying to test out how govim behaved under the "load" of thousands of edits from gopls.

What did you expect to see?

The file to be correctly reformatted.

What did you see instead?

Out of memory error.

cc @stamblerre @ianthehat
@myitcv myitcv added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. gopls Issues related to the Go language server, gopls. labels Jul 9, 2019
@gopherbot gopherbot added this to the Unreleased milestone Jul 9, 2019
@myitcv myitcv mentioned this issue Jul 9, 2019
Open
@stamblerre stamblerre changed the title x/tools/cmd/gopls: out of memory error on large file with lots of required formats x/tools/gopls: cannot handle large files Aug 8, 2019
@stamblerre
Copy link
Contributor

Seems like gopls in general just cannot handle a file of this size. Running gopls -rpc.trace -v check rewriteAMD64.go times out.

@stamblerre stamblerre mentioned this issue Aug 8, 2019
Closed
@ianthehat
Copy link

We seem to be able to handle large files mostly now, the editor works, we can survive them being type checked and generating outlines (although it is slow and we seem to redo it too much even when the file is not changing) but the original test case still fails.
Undenting the whole file causes almost every line to be different, which causes diff.shortestEditSequence to allocate huge amounts of memory, and eventually the machine runs out (it takes a while).
Given that we were looking to change diff algorithms anyway to something that supported sub-line, I am not sure how much effort we should put into fixing this rather than just finding a good diff library and switching to it.
Does anyone know of a good one?

@gopherbot
Copy link

Change https://golang.org/cl/191018 mentions this issue: gopls: use go-diff for edit generation

@stamblerre stamblerre mentioned this issue Aug 24, 2019
Closed
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019
@golang golang locked and limited conversation to collaborators Oct 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@ianthehat @myitcv @stamblerre @gopherbot