cmd/compile: fails to compile very long files starting go1.17 #51193

sywhang · 2022-02-14T18:39:45Z

What version of Go are you using (`go version`)?

My dev laptop is m1 Mac:

$ go version
go version go1.17.1 darwin/arm64

But we also verified this fails on linux/amd64 as well.

Does this issue reproduce with the latest release?

Yes, including the go1.18 beta.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN="/Users/sungyoon/go/bin/"
GOCACHE="/Users/sungyoon/Library/Caches/go-build"
GOENV="/Users/sungyoon/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/sungyoon/go118/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/sungyoon/go118/go/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/sungyoon/go118/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/sungyoon/go118/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.17.1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/l1/s1n3wjrj7hn5fwgwdn70kzww0000gn/T/go-build3915184577=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Some teams here at Uber were using go-bindata to embed binaries into Go file, which resulted in a very lengthy file (>1 mil lines of Go).

In Go 1.17 these files started seeing build failures, with the following compiler error:

./generated2.go:1048574:3: internal compiler error: non-monotonic scope positions
        /Users/sungyoon/repro/generated2.go:1048574:42: previous scope position

Please file a bug report including a short program that triggers the error.
https://go.dev/issue/new

What did you expect to see?

The file being compiled.

What did you see instead?

The compiler error above.

Additional Details

We dug into the source code and found out that this happens due to an overflow bug in cmd/compile. Specifically, the newly added dwarfgen package does not account for the case where XPos can be non-monotonic increment between two characters due to Lico using only 20 bits to represent the line number:

https://github.com/golang/go/blob/3d7f83612390d913e7e8bb4ffa3dc69c41b3078d/src/cmd/internal/src/pos.go#L313
// A lico is a compact encoding of a LIne and COlumn number.
type lico uint32

// Layout constants: 20 bits for line, 8 bits for column, 2 for isStmt, 2 for pro/epilogue
// (If this is too tight, we can either make lico 64b wide,
// or we can introduce a tiered encoding where we remove column
// information as line numbers grow bigger; similar to what gcc
// does.)
// The bitfield order is chosen to make IsStmt be the least significant
// part of a position; its use is to communicate statement edges through
// instruction scrambling in code generation, not to impose an order.
// TODO: Prologue and epilogue are perhaps better handled as pseudo-ops for the assembler,
// because they have almost no interaction with other uses of the position.

We are unblocked by changing these files to not do this by using go:embed (or opting to not generate DWARF symbols), but I do feel that 20 bits is a bit too limiting since it's possible for code-generated files to reach this length.

We tried a local fix as suggested in the comment - by changing lico to uint64 and changing the layout as 10 bits reserved (empty), 42 bits for line, 8 bits for column, 2 for isStmt, 2 for pro/epilogue and verified that the fix works.

We're happy to submit this fix but I'm posting an issue here before submitting it to discuss potential alternatives. Thanks!

The text was updated successfully, but these errors were encountered:

ALTree · 2022-02-14T18:48:13Z

cc @griesemer for decision.

randall77 · 2022-02-14T21:56:42Z

Another fix is to ensure that when we saturate the line field, we always use column 1. That way incrementing a position never goes backwards.

randall77 · 2022-02-14T22:00:28Z

Does this work?

--- a/src/cmd/internal/src/pos.go
+++ b/src/cmd/internal/src/pos.go
@@ -389,9 +389,10 @@ func makeBogusLico() lico {
 }
 
 func makeLico(line, col uint) lico {
-       if line > lineMax {
+       if line >= lineMax {
                // cannot represent line, use max. line so we have some information
                line = lineMax
+               col = 1
        }
        if col > colMax {
                // cannot represent column, use max. column so we have some information

ianlancetaylor · 2022-02-14T22:30:02Z

CC @thanm because DWARF.

abhinav · 2022-02-14T22:44:11Z

@randall77 just checked; that patch also fixes the issue.

gopherbot · 2022-02-14T23:20:01Z

Change https://go.dev/cl/385795 mentions this issue: cmd/compile: drop column info when line number saturates

thanm · 2022-02-14T23:42:36Z

@sywhang thanks for the report.

Just for my curiosity, what is the advantage of using go-bindata as opposed to Go's "go:embed" feature, e.g. https://pkg.go.dev/embed?

abhinav · 2022-02-14T23:45:49Z

Just for my curiosity, what is the advantage of using go-bindata as opposed to Go's "go:embed" feature, e.g. https://pkg.go.dev/embed?

There is no advantage to using go-bindata over go:embed that we're aware of.
This was just one of those cases that had not yet migrated from go-bindata to go:embed.

sywhang · 2022-02-14T23:46:29Z

@thanm there isn't. This existed prior to go:embed feature.

shawndx · 2022-04-01T03:48:09Z

My apologies for commenting a closed issue, we ran into the error in practice when compiling a large auto-generated file, feasible to make a backport? Thanks.

@gopherbot please consider this for backport to 1.17, it's a compiler error.

gopherbot · 2022-04-01T03:48:31Z

Backport issue(s) opened: #52095 (for 1.17).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

aclements · 2022-04-20T16:15:22Z

@randall77 , do you mind preparing a back-port CL for this/

randall77 · 2022-04-20T16:57:20Z

@aclements Done.

ALTree changed the title ~~affected/package: cmd/compile fails to compile very long files starting go1.17~~ cmd/compile: fails to compile very long files starting go1.17 Feb 14, 2022

ALTree added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Feb 14, 2022

ALTree added this to the Unplanned milestone Feb 14, 2022

gopherbot closed this as completed in 1de2344 Feb 14, 2022

gopherbot mentioned this issue Apr 1, 2022

cmd/compile: fails to compile very long files starting go1.17 [1.17 backport] #52095

Closed

dmitshur modified the milestones: Unplanned, Go1.18 May 4, 2022

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels May 4, 2022

golang locked and limited conversation to collaborators May 4, 2023

gopherbot added the FrozenDueToAge label May 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: fails to compile very long files starting go1.17 #51193

cmd/compile: fails to compile very long files starting go1.17 #51193

sywhang commented Feb 14, 2022 •

edited

ALTree commented Feb 14, 2022

randall77 commented Feb 14, 2022

randall77 commented Feb 14, 2022

ianlancetaylor commented Feb 14, 2022

abhinav commented Feb 14, 2022

gopherbot commented Feb 14, 2022

thanm commented Feb 14, 2022

abhinav commented Feb 14, 2022 •

edited

sywhang commented Feb 14, 2022

shawndx commented Apr 1, 2022

gopherbot commented Apr 1, 2022

aclements commented Apr 20, 2022

randall77 commented Apr 20, 2022

cmd/compile: fails to compile very long files starting go1.17 #51193

cmd/compile: fails to compile very long files starting go1.17 #51193

Comments

sywhang commented Feb 14, 2022 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Additional Details

ALTree commented Feb 14, 2022

randall77 commented Feb 14, 2022

randall77 commented Feb 14, 2022

ianlancetaylor commented Feb 14, 2022

abhinav commented Feb 14, 2022

gopherbot commented Feb 14, 2022

thanm commented Feb 14, 2022

abhinav commented Feb 14, 2022 • edited

sywhang commented Feb 14, 2022

shawndx commented Apr 1, 2022

gopherbot commented Apr 1, 2022

aclements commented Apr 20, 2022

randall77 commented Apr 20, 2022

sywhang commented Feb 14, 2022 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

abhinav commented Feb 14, 2022 •

edited