cmd/go: go generate fails if a source file contains lines longer than 64KB #9143
Labels
Milestone
Comments
I wrote a quick hack to fix this at https://golang.org/cl/179090043 Although with that patch, go generate would still fail if the //go:generate line itself is larger than 64KB. |
CL https://golang.org/cl/182970043 mentions this issue. |
|
Owner changed to @robpike. Status changed to Started. |
First, I want to say huge thank you for fixing this issue. It makes me feel happy and relieved! Rob Pike, you said this earlier: > That said, I am unconvinced that this is an actual problem that needs to be addressed. > To be fair, if you're going to have a stupid long line > it's not a bad bet you'll want to run it through go > generate, because it's some embeddable asset that > has been machine generated. (One could ask why > that generation process didn't add a newline or two, > but we should cope anyway.) And I wanted to share why I feel that handling these stupid long lines is so valuable. One of my favorite things about Go is its simplicity, and how few "exceptions to the rules" it has. This is great because it lowers my cognitive load and allows me to spend more resources for the task at hand. I really like that I can have a comma after the last line (unlike in JSON) in slice declarations, because it means the last line is not any different from the first or fifth. Why should it be? It lets me cut and paste lines and not have to fix that last comma. Similarly, before this problem, I could not worry about how long each line is. Most lines are probably less than 100 characters, but if for whatever reason I end up writing valid Go code that spans thousands of characters on one line, I don't have to have the following train of thought: "Oh, wait, is this line too long? What was the limit? 32 or 64 KB? How many characters is this line I pasted? But wait, I'm not using go generate, so it's okay. On the other hand, what if I want to use it tomorrow, I should (spend time and) add some newlines here... just because there's that issue." Here's a silly exaggeration of the situation: imagine in go1.5 a new feature is added and it makes it so that one cannot have a function name a multiple of 79, but only when there are at least three _test.go files. It's a situation that is extremely unlikely, but the cognitive load of having to think and worry about hitting some artificial limit would be there - I'd much rather not have it. I understand long lines are probably hard to deal with for editors, they're not optimized, but I'm okay with that - if I see my editor is sluggish, I will choose to make a line shorter if I want to. But I don't want to worry about an artificial limitation. So with that said, you can see why I think such a small detail matters and why I really appreciate that it's going to be fixed. Thank you (and sorry for my inability to convey small details that happen to be important to me with fewer words)! |
|
This issue was closed by revision dd26fc3. Status changed to Fixed. |
rsc
added a commit
that referenced
this issue
May 11, 2015
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes #9143. Fixes #9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes golang#9143. Fixes golang#9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 26, 2018
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes golang#9143. Fixes golang#9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes golang#9143. Fixes golang#9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 20, 2018
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes golang#9143. Fixes golang#9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 30, 2018
««« CL 182970043 / 573a7b5178c4 cmd/go: avoid use of bufio.Scanner in generate Scanner can't handle stupid long lines and there are reports of stupid long lines in production. Note the issue isn't long "//go:generate" lines, but any long line in any Go source file. To be fair, if you're going to have a stupid long line it's not a bad bet you'll want to run it through go generate, because it's some embeddable asset that has been machine generated. (One could ask why that generation process didn't add a newline or two, but we should cope anyway.) Rewrite the file scanner in "go generate" so it can handle arbitrarily long lines, and only stores in memory those lines that start "//go:generate". Also: Adjust the documentation to make clear that it does not parse the file. Fixes golang#9143. Fixes golang#9196. LGTM=rsc, dominik.honnef R=rsc, cespare, minux, dominik.honnef CC=golang-codereviews https://golang.org/cl/182970043 »»» TBR=r CC=golang-codereviews https://golang.org/cl/183060044
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What does 'go version' print? go version devel +d7d71f7d6f53 Thu Nov 20 09:35:22 2014 -0800 darwin/amd64 What steps reproduce the problem? Run 'go generate longline.go' (attached) or any file with a line longer than 64KB long. What happened? Go generate stops processing the file at that line, and exits with the error message: longline.go:4: error reading longline.go: bufio.Scanner: token too long What should have happened instead? Go generate should have continued to process the file. Please provide any additional information below. While it's unlikely that normal source files have such long lines, it's plausible that generated ones do. E.g. when embedding binary data.Attachments:
The text was updated successfully, but these errors were encountered: