Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: misleading error for end-of-input in an escape sequence #58680

Open
LukeShu opened this issue Feb 24, 2023 · 5 comments
Open

encoding/json: misleading error for end-of-input in an escape sequence #58680

LukeShu opened this issue Feb 24, 2023 · 5 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@LukeShu
Copy link

LukeShu commented Feb 24, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20.1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lukeshu/.cache/go-build"
GOENV="/home/lukeshu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/lukeshu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lukeshu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/run/user/1000/tmpdir/go-build3465712561=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I fed the JSON parser a truncated JSON document that has a backslash as the final rune.

package main

import (
	"encoding/json"
	"fmt"
)

func main() {
	var obj any
	err := json.Unmarshal([]byte(`"\`), &obj)
	fmt.Println(err)
}

https://go.dev/play/p/KxvQ6lpnT7I

What did you expect to see?

I expected the error to be unexpected end of JSON input

What did you see instead?

I saw the error invalid character ' ' in string escape code, which is a nonsense error as there is no character in the input at all!

json.Compact and json.Indent have the same issue, while json.Decoder.Decode correctly returns io.ErrUnexpectedEOF.

@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 24, 2023
@thanm
Copy link
Contributor

thanm commented Feb 24, 2023

@dsnet @bradfitz @mvdan per owners.

@hafus
Copy link

hafus commented Feb 24, 2023

I think the root case of the issue is the function below:

func (s *scanner) eof() int {
	if s.err != nil {
		return scanError
	}
	if s.endTop {
		return scanEnd
	}
	s.step(s, ' ')
	if s.endTop {
		return scanEnd
	}
	if s.err == nil {
		s.err = &SyntaxError{"unexpected end of JSON input", s.bytes}
	}
	return scanError
}

I do update s.err == nil to be s.err != nil but causes some test cases to fail

@seankhliao seankhliao changed the title encoding/json: Misleading error for end-of-input in an escape sequence encoding/json: misleading error for end-of-input in an escape sequence Feb 24, 2023
@mvdan
Copy link
Member

mvdan commented Feb 24, 2023

A similar error in the past: #56332

There are a handful of these "confusing error message" bug reports on encoding/json. My intuition is that we shouldn't try to fix each one of them individually. cc @dsnet

@dsnet
Copy link
Member

dsnet commented Feb 24, 2023

I have an experimental CL in progress that completely switches the "encoding/json" parser to a new implementation, which will fix most (if not all) of these weird error cases.

@LukeShu
Copy link
Author

LukeShu commented Feb 26, 2023

I guess I should mention that I'm finding these issues by fuzzing my own JSON parser against encoding/json. My implementation is GPL, but I'm 100% willing to re-license it for incorporation in to the stdlib if there's anything that you'd like to steal.

In particular, compat/json/compat_test.go is some tests that may be worth stealing; they are tests that I felt necessary to add even though my implementation already passed encoding/json's existing test suite (which I have copied in the borrowed_*_test.go files).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants