Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

encoding/json: Unmarshal should reject malformed UTF-8 #4783

Closed
the42 opened this issue Feb 10, 2013 · 4 comments
Closed

encoding/json: Unmarshal should reject malformed UTF-8 #4783

the42 opened this issue Feb 10, 2013 · 4 comments
Labels
FrozenDueToAge
Milestone
Go1.1

Comments

@the42
Copy link

the42 commented Feb 10, 2013 

What is the expected output?
json.Unmarshal should return json.InvalidUTF8Error for input streams containing
non-Unicode runes

What do you see instead?
Non-Unicode runes replaced by surrogates

Which version are you using?  (run 'go version')
go version devel +439cb8bad388 Wed Feb 06 17:18:53 2013 -0800 linux/amd64


Please provide any additional information below.
This program ought to return an error, though it replaces the input stream with Unicode
surrogates. Maybe Unmarshal deserves a parameter or an UnmarshalWithError - method?

http://play.golang.org/p/r2AWWqdSNY
@minux
Copy link
Member

minux commented Feb 10, 2013

Comment 1:

\ufffd is not a surrogate, it is a unicode replacement character.
we can't change the behavior of unmarshal now, but we should at least document that
InvalidUTF8Error only applies to encoding,
and make it clear that Unmarshal will use \ufffd to replace any incorrect utf-8 bytes.

@rsc
Copy link
Contributor

rsc commented Feb 13, 2013

Comment 2:

Unmarshal is supposed to reject malformed UTF-8. If it is not, that's a bug we should
fix.

Labels changed: added priority-later, go1.1, removed priority-triage, go1.1maybe.

Status changed to Accepted.

@rsc
Copy link
Contributor

rsc commented Feb 14, 2013

Comment 3:

Actually, the source code is working quite hard to substitute the replacement characters
where appropriate, so I will defer to the intent of the original author (me a few years
ago) and leave it as is. But I will add docs and tests.

@rsc
Copy link
Contributor

rsc commented Feb 14, 2013

Comment 4:

This issue was closed by revision 30359a5.

Status changed to Fixed.

@rsc rsc added this to the Go1.1 milestone Apr 14, 2015
@rsc rsc removed the go1.1 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.1
Development

No branches or pull requests
4 participants
@rsc @minux @the42 @gopherbot