Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

encoding/json: unexpected output with SetEscapeHTML(false) #20581

Closed
wxcsdb88 opened this issue Jun 5, 2017 · 4 comments
Closed

encoding/json: unexpected output with SetEscapeHTML(false) #20581

wxcsdb88 opened this issue Jun 5, 2017 · 4 comments
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Go1.10

Comments

@wxcsdb88
Copy link

wxcsdb88 commented Jun 5, 2017 •
edited

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/amd64
and
go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/wangxin/work/golang:/home/wangxin/work/golang/src/unicontract"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build900066718=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

In order to get the same serialize result for json data in java(gson), js(sort keys by alph...) , python(rapaidjson) and golang(encoding/json), I found the golang set the htmlescape flag true and it`s result is not the same as others, so I rewrite the encoder and got the output.

The test code as follows:

func Test_SerializeDisableHtmlEscape(t *testing.T) {
	jsonStr := `{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}`
	default_data := Deserialize(jsonStr)
	default_data_jsonStr := SerializePretty(default_data)
	t.Log("----------default serialize result------------:\n", default_data_jsonStr)

	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	enc.Encode(jsonStr)

	strJson := buf.String()
	t.Log("disableEscapeHTML and output len is", len(strJson), ",content is:\n", strJson)
	strJson = strings.Replace(strJson, "\\", "", -1)
	t.Log("remove the backslash and output len is", len(strJson), ",content is:\n", strJson)
	strJsonWithOutBacklashLen := len(strJson)
	strJson = strings.Trim(strJson, "\"")
	t.Log("after strings trim, the len is", len(strJson), ",expect is", strJsonWithOutBacklashLen-2, ",content is:\n", strJson)

	defaultJsonStr := "\"{\"host\": \"http://localhost:9090\",\"port\": 9090,\"analytics_file\": \"1>=0 && 3+2 <=5 || 4&2 || 2^1\",\"static_file_version\": 1,\"static_dir\": \"E:/Project/goTest/src/\",\"templates_dir\": \"E:/Project/goTest/src/templates/\",\"serTcpSocketHost\": \":12340\",\"serTcpSocketPort\": 12340,\"fruits\": [\"apple\", \"peach\"]}\""
	t.Log("defaultJsonStr len is ", len(defaultJsonStr))
	defaultJsonStr = strings.Replace(defaultJsonStr, "\\", "", -1)
	strdefaultJsonStrWithOutBacklashLen := len(strJson)
	t.Log("defaultJsonStr remove the backslash and output len is", len(defaultJsonStr), ",content is:\n", defaultJsonStr)
	defaultJsonStr = strings.Trim(defaultJsonStr, "\"")
	t.Log("defaultJsonStr after strings trim, the len is", len(defaultJsonStr), ",expect is", strdefaultJsonStrWithOutBacklashLen-2, ",content is:\n", defaultJsonStr)

}

The fully output as follows:

utils_test.go:136: ----------default serialize result------------:
		 {
			"analytics_file": "1\u003e=0 \u0026\u0026 3+2 \u003c=5 || 4\u00262 || 2^1",
			"fruits": [
				"apple",
				"peach"
			],
			"host": "http://localhost:9090",
			"port": 9090,
			"serTcpSocketHost": ":12340",
			"serTcpSocketPort": 12340,
			"static_dir": "E:/Project/goTest/src/",
			"static_file_version": 1,
			"templates_dir": "E:/Project/goTest/src/templates/"
		}
	utils_test.go:144: disableEscapeHTML and output len is 331 ,content is:
		 "{\"host\": \"http://localhost:9090\",\"port\": 9090,\"analytics_file\": \"1>=0 && 3+2 <=5 || 4&2 || 2^1\",\"static_file_version\": 1,\"static_dir\": \"E:/Project/goTest/src/\",\"templates_dir\": \"E:/Project/goTest/src/templates/\",\"serTcpSocketHost\": \":12340\",\"serTcpSocketPort\": 12340,\"fruits\": [\"apple\", \"peach\"]}"
		
	utils_test.go:146: remove the backslash and output len is 299 ,content is:
		 "{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
		
	utils_test.go:149: after strings trim, the len is 298 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
		
	utils_test.go:152: defaultJsonStr len is  298
	utils_test.go:155: defaultJsonStr remove the backslash and output len is 298 ,content is:
		 "{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
	utils_test.go:157: defaultJsonStr after strings trim, the len is 296 ,expect is 296 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}

What did you expect to see?

after strings trim, the len is 297 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"

What did you see instead?

after strings trim, the len is 298 ,expect is 297 ,content is:
		 {"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"
@wxcsdb88 wxcsdb88 changed the title Encode json disableHTMLEscape occur the unexpected output Encode json disableHTMLEscape occured the unexpected output Jun 5, 2017
@bradfitz bradfitz changed the title Encode json disableHTMLEscape occured the unexpected output encoding/json: unexpected output with SetEscapeHTML(false) Jun 5, 2017
@bradfitz bradfitz added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 5, 2017
@bradfitz bradfitz added this to the Go1.10 milestone Jun 5, 2017
@bradfitz
Copy link
Contributor

bradfitz commented Jun 5, 2017

What is SerializePretty? We can't help you with bugs in your own code.

If there's a bug in the standard library, please report what the standard library is doing (with a code example) and what output you expect.

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 5, 2017
@wxcsdb88
Copy link
Author

wxcsdb88 commented Jun 5, 2017

Thanks, maybe our usage error, the test code as follows:

func Test_JsonStrTrim(t *testing.T) {
	jsonStr := `{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}`

	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	// disabled the HTMLEscape for &, <, and > to \u0026, \u003c, and \u003e in json string
	enc.SetEscapeHTML(false)
	/*--------------- 1. trim the result maybe have the extra double quotation in last--------------------*/
	enc.Encode(jsonStr)

	/*---------------- 2. it`s ok change to interface and encode--------------------------*/
	//var jsonStrData interface{}
	//json.Unmarshal([]byte(jsonStr), &jsonStrData)
	//enc.Encode(jsonStrData)

	strJson := buf.String()
	//t.Log("disableEscapeHTML and output len is", len(strJson), ",content is:\n", strJson)
	strJson = strings.Replace(strJson, "\\", "", -1)
	//t.Log("remove the backslash and output len is", len(strJson), ",content is:\n", strJson)
	strJsonWithOutBacklashLen := len(strJson)
	strJson = strings.Trim(strJson, "\"")
	t.Log("after strings trim, the len is", len(strJson), ",expect is", strJsonWithOutBacklashLen-2, ",content is:\n", strJson)
	// the last double quotation except removed

}

output

the last backslash is not removed, with strings.Trim(...)

after strings trim, the len is 298 ,expect is 297 ,content is:
{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}"

expect

after strings trim, the len is 297 ,expect is 297 ,content is:
{"host": "http://localhost:9090","port": 9090,"analytics_file": "1>=0 && 3+2 <=5 || 4&2 || 2^1","static_file_version": 1,"static_dir": "E:/Project/goTest/src/","templates_dir": "E:/Project/goTest/src/templates/","serTcpSocketHost": ":12340","serTcpSocketPort": 12340,"fruits": ["apple", "peach"]}

@bradfitz
Copy link
Contributor

bradfitz commented Jun 5, 2017

There's a newline at the end of end of your string after the double quote, so your strings.Trim isn't trimming the double quote.

I don't see a bug in Go here.

@bradfitz bradfitz closed this as completed Jun 5, 2017
@wxcsdb88
Copy link
Author

wxcsdb88 commented Jun 6, 2017

Thanks a lot. I have found that strings.TrimSpace() will ok, before trim the ".

@golang golang locked and limited conversation to collaborators Jun 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Go1.10
Development

No branches or pull requests
3 participants
@bradfitz @wxcsdb88 @gopherbot