Well I solved this in my tool,

previously the file structure was like this

package assets

// ImagesDocTXT file
var ImagesDocTXT = []byte{
    0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0x0a, 0x49,
    0x2d, 0x2e, 0xc9, 0xcc, 0x4b, 0x57, 0x30, 0x54, 0x30, 0x52, 0x30, 0xe6,
    0x02, 0x04, 0x00, 0x00, 0xff, 0xff, 0x23, 0xc2, 0x19, 0x35, 0x0e, 0x00,
    0x00, 0x00,
}

And so I thought the lexer has to read a lot of tokens for upto 300K loc and that might be clogging it up so I then rewrote all the data to this

package assets

// ImagesDocTXT file
var ImagesDocTXT = []byte("\x1f\x8b\x08\x00\x00\x09\x6e\x88\x02\xff\x0a\x49\x2d\x2e\xc9\xcc\x4b\x57\x30\x54\x30\x52\x30\xe6\x02\x04\x00\x00\xff\xff\x23\xc2\x19\x35\x0e\x00\x00\x00")

And now the no of lines drop to about 50 and everything works fine.
Well I guess its about how you generate large code bases and mostly dealing with code generators and not the go compiler.
Closing this one for now

I'm reopening this. Even if you worked around it, the issue remains.

iirc, there's a very old issue relating to very large slice literals causing the compiler to use lots of memory and slow down. What that ever fixed?

If you're referring to #11612 no, it's not fixed yet. It's in the 1.7 milestone.

@ALTree much older (known from at least around the 1.0-1.1 days). I'll try to dig it up. I suspect these are all the same issue though, and that fixing one will fix them all.

@extemporalgenome #6643 and d227d68

@dominikh that's the one, thanks!

I'm experiencing some version of this as well. Working on a Go port of HTTPS Everywhere, and the build fails if I try to embed all the HTTPS Everywhere rules here:

https://github.com/getlantern/httpseverywhere/tree/too-large-for-go

Cutting down the size of the targets.go file by about half allows the build to complete, but otherwise just running go build with either 1.7.5 or 1.8 on darwin amd64 gives:

[httpseverywhere (master *)]$ go build
# github.com/getlantern/httpseverywhere
panic: runtime error: makeslice: len out of range

goroutine 1 [running]:
cmd/compile/internal/gc.newliveness(0xc4272221b0, 0xc42011c238, 0xc49bc7e000, 0x4a651, 0x5ac00, 0xc4da1da000, 0x832c, 0x9800, 0xc48ef5f898)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:717 +0x145


cmd/compile/internal/gc.liveness(0xc4272221b0, 0xc42011c238, 0xc4c21321c0, 0xc4c2132230)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1740 +0x14f
cmd/compile/internal/gc.genssa(0xc45d30b440, 0xc42011c238, 0xc4c21321c0, 0xc4c2132230)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:4500 +0x77c
cmd/compile/internal/gc.compile(0xc4272221b0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:443 +0x707
cmd/compile/internal/gc.funccompile(0xc4272221b0)
	/usr/local/go/src/cmd/compile/internal/gc/dcl.go:1292 +0xdc
cmd/compile/internal/gc.fninit(0xc420115c00, 0x3b, 0x70)
	/usr/local/go/src/cmd/compile/internal/gc/init.go:164 +0xb8d
cmd/compile/internal/gc.Main()
	/usr/local/go/src/cmd/compile/internal/gc/main.go:471 +0x23b1
main.main()
	/usr/local/go/src/cmd/compile/main.go:50 +0xfe

@myleshorton please file a new issue and cc me.

I think the original issue here is a straightforward dup of #6643. The other conversations have either been resolved or have their own issues. Closing.