Reading stream with json.NewDecoder fails with "invalid character '\r' in string literal" if large base64 []byte present #39874

fxtentacle · 2020-06-26T16:27:50Z

Dear Go team,

first off, thank you for this amazingly productive language :) I'm writing you because I believe that I have found an issue with the streaming JSON decoder in the standard library.

This crashes with "invalid character '\r' in string literal":

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"io"
	"os"
)

type RequestDataItem struct {
	Name string
	Data []byte
}

type Request struct {
	Data []RequestDataItem
}

func main() {
	dec := json.NewDecoder(os.Stdin)
	for {
		var request Request
		err := dec.Decode(&request)
		if err == io.EOF {
			break
		}
		if err != nil {
			panic(err)
		}
		fmt.Println("OK")
	}
}

if presented with this input data:

{"Data":[{"Name":"test","Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="}]}

In contrast, this main function works, because the JSON is now accumulated in the scanner before parsing:

func main() {
	scanner := bufio.NewScanner(os.Stdin)
	for scanner.Scan() {
		var request Request
		err := json.Unmarshal(scanner.Bytes(), &request)
		if err == io.EOF {
			break
		}
		if err != nil {
			panic(err)
		}
		fmt.Println("OK")
	}
}

Example data that works in both cases would be

{"Data":[{"Name":"test","Data":"dGVzdAo="}]}

In my opinion, the issue is the space character invented here:

go/src/encoding/json/stream.go

Line 114 in ba108c9

if stateEndValue(&dec.scan, ' ') == scanEnd {

in combination with reading the input in 512 byte chunks here:

go/src/encoding/json/stream.go

Line 157 in ba108c9

const minRead = 512

So while parsing with the streaming JSON decoder, it will only read up until

{"Data":[{"Name":"test","Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

(or exactly 512 bytes)
and then crash, because it erroneously tries to base64-decode the "A[..]AA " string generated by appending a space to the unfinished base64 data.

As expected, changing

go/src/encoding/json/stream.go

Line 157 in ba108c9

const minRead = 512

to read const minRead = 5120 fixes the issue for my test case here, but of course it'll still break for larger JSON payloads.

The text was updated successfully, but these errors were encountered:

seankhliao · 2020-06-26T18:08:59Z

please fill out the issue template when filing reports

this is more likely a limitation of your terminal emulator than an issue with json, related: https://unix.stackexchange.com/questions/204815/terminal-does-not-accept-pasted-or-typed-lines-of-more-than-1024-characters

fxtentacle · 2020-06-26T19:23:44Z

Thank you for pointing that out. I agree with you that this is most likely an issue with my terminal when data is being read from it with a small buffer, like the default 512 that json.NewDecoder uses.

Replacing
dec := json.NewDecoder(os.Stdin)
with
dec := json.NewDecoder(bufio.NewReaderSize(os.Stdin, 2048))
will also fix the problem.

In other words, the exact same terminal emulator works if I force Go to read from it with a larger 2048 byte buffer.

I'll close this issue, then. Let's hope that the next person observing the problem will find this thread so that they can pinpoint the cause more quickly than me.

fxtentacle closed this as completed Jun 26, 2020

golang locked and limited conversation to collaborators Jun 26, 2021

gopherbot added the FrozenDueToAge label Jun 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reading stream with json.NewDecoder fails with "invalid character '\r' in string literal" if large base64 []byte present #39874

Reading stream with json.NewDecoder fails with "invalid character '\r' in string literal" if large base64 []byte present #39874

fxtentacle commented Jun 26, 2020 •

edited

seankhliao commented Jun 26, 2020

fxtentacle commented Jun 26, 2020 •

edited

Navigation Menu

Reading stream with json.NewDecoder fails with "invalid character '\r' in string literal" if large base64 []byte present #39874

Reading stream with json.NewDecoder fails with "invalid character '\r' in string literal" if large base64 []byte present #39874

Comments

fxtentacle commented Jun 26, 2020 • edited

seankhliao commented Jun 26, 2020

fxtentacle commented Jun 26, 2020 • edited

fxtentacle commented Jun 26, 2020 •

edited

fxtentacle commented Jun 26, 2020 •

edited