cmd/go: "runtime: cannot allocate memory" in workspaces with many non go files #54226
Comments
gopherbot
added
the
compiler/runtime
|
What counts as "a lot"? |
|
sorry for being unspecifc on this. "a lot" can be translated to ~491635 files. |
seankhliao
changed the title
seankhliao
added
NeedsInvestigation
|
That's kind of amazing. 😅 Are these all regular files, or some directories, or some symlinks? Are they used as |
|
Does setting |
just regular text based files. and not used within
yes - it does. with |
|
@florianl What are the extensions of these files? (We save information about some non-go files that can potentially be used by the toolchain, such as C, C++, Obj-C, Fortran, Assembly, C, and object files.) Are the files all located in the same directory? Is there a chance you can share your repository offline so we can try to reproduce and debug on our side? |
|
There are json files with a There are multiple directories with the very same structure like the following: Unfortunately I cannot share these files. |
|
Got it. How many modules are in your workspace and what's their makeup? Is there one module with many of those files or many modules? |
|
A followup clarification question about the use of the word 'workspace': are you referring to the workspaces feature or using the word generically? That is, are these go workspaces using the |
bcmills
added
the
WaitingForInfo
|
Change https://go.dev/cl/439118 mentions this issue: |
We were saving non-go file information in the module index files, leading in an unnecessary increase in memory usage in modules containing many non-go files. This was a bug because this information is never used. Don't save that information. For #54226 Change-Id: I0644064f83f96e3a9f43b7e66ca94d69d9603376 Reviewed-on: https://go-review.googlesource.com/c/go/+/439118 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
bcmills
added
NeedsFix
gopherbot
removed
the
NeedsInvestigation
|
We've submitted CL 439118. @florianl could you verify if this CL fixes your issue? |
matloob
added
WaitingForInfo
|
sorry @matloob 🙏 I was out for some time. Will test it beginning of next week and provide feedback. Hope this is fine.
my bad - in this context |
|
Thanks for the clarification, let me know what the results of the test are! |
|
I did run a test with Thanks for the fix @matloob |
|
Thanks for confirming! |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
yes. No for
go1.18.5.What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
What did you expect to see?
No changes at all or changes to
go.modand/orgo.sum.What did you see instead?
Findings
With
GODEBUG=gofsystrace=1I was able to trace down the issue. The repository in question contains a lot of non Go files andgo mod tidyparses them all. Moving these files into a directory with the prefix_resolves this issue - but I think this might be more a workaround for this particular case. Unfortunately this happens on a non-public repository, so I can't share it directly.The text was updated successfully, but these errors were encountered: