cmd/compile: 'internal compiler error: bvbulkalloc too big' when compiling a file containing a large map #33437
Comments
bcmills
added
the
NeedsInvestigation
bcmills
changed the title
|
Does this also reproduce using Go 1.12.7, or is it a regression in 1.13? |
|
Yes, same issue with 1.12.7 |
|
AFAIK this is not new. See #26560 (comment). It can be triggered by huge, auto-generated literals or functions (I've triggered this a few times while fuzzing the compiler with really big autogenerated functions). The standard workaround is to copy the data in the map in |
|
Clearly it still would be interesting to know if there was a regression on this, i.e. if a map of the same size compiled fine in Go1.10 and it doesn't now. |
|
@ALTree I'm not sure what you mean by "copy the data in the map in |
|
I used a smaller version of your reproducer with ~5000 map entries. Your code does this: Compare with this: Note how the second version initializes the map in Compiling the first version: Compiling the second version: |
|
@ALTree thanks. This does work, but increases the size of my binary by about 18MB compared to the prior runtime initialisation method I used e.g. rather than Looks like I'll take the runtime parsing hit and stick with my existing method. Should this be left open for someone to address the issue of the compiler failure? |
|
Yes, you can leave this open. The complete example will help. |
|
@randall77, @mcdee posted a full example which you can access at https://raw.githubusercontent.com/wealdtech/compilebug/master/signatures.go, it is 12.5MB though :) so I've derived for you a repro that generates the offending code, and then invokes package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
)
var code = fmt.Sprintf(`
package main
type function struct {
name string
params []string
}
var functions map[uint32]function
func main() {
functions = make(map[uint32]function)
%s
}`, makeFunctions(140000))
func makeFunctions(n int) string {
sb := new(strings.Builder)
for i := 0; i < n; i++ {
name := fmt.Sprintf("fn-%d", i)
fmt.Fprintf(sb, "\tfunctions[%d] = function{name: %q, params: []string{%q}}\n", i, name, name)
}
return sb.String()
}
func main() {
tmpDir, err := ioutil.TempDir(os.TempDir(), "issue-33437")
if err != nil {
log.Fatalf("Failed to create tempdir: %v", err)
}
defer os.Remove(tmpDir)
generatedMainGoFile := filepath.Join(tmpDir, "main.go")
if err := ioutil.WriteFile(generatedMainGoFile, []byte(code), 0655); err != nil {
log.Printf("Failed to write generated main.go file: %v", err)
return
}
ctx := context.Background() // Perhaps set it to a timer.
cmd := exec.CommandContext(ctx, "go", "run", generatedMainGoFile)
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("Failed to run the command: %v\n%s", err, output)
}
} |
|
On my Darwin machine, it took about 50 minutes to run but it did so successfully, and here is my Darwin Emmanuels-MacBook-Pro-2.local 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64and go version $ go version
go version devel +30da79d958 Mon Oct 7 17:19:13 2019 +0000 darwin/amd64 |
As described in golang/go#33437 (comment) maps of too large size cannot be defined declaratively. So I split the map in a Key (ΛEnumTypesKeys) and a Value (ΛEnumTypesValues) Slice and compose the ΛEnumTypes map via init().
As described in golang/go#33437 (comment) maps of too large size cannot be defined declaratively. So I split the map in a Key (ΛEnumTypesKeys) and a Value (ΛEnumTypesValues) Slice and compose the ΛEnumTypes map via init().
Dlv was crashing trying to debug a large number of code-generated test cases. I believe it's caused by golang/go#33437 Workaround by grouping the test cases into slices of at most 256 items.
gopherbot
added
the
compiler/runtime
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?What did you do?
Attempting to
go buildon a file that creates a large map fails:This contains approximately 140K additions, carried out as individual calls (rather than initialising the map during declaration). Relevant parts of code are:
A full copy of the file is at https://github.com/wealdtech/compilebug
What did you expect to see?
Would expect the build to complete.
What did you see instead?
Error output as above after approximately 50 minutes of building.
The text was updated successfully, but these errors were encountered: