Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/encoding: UTF-16 decoder handles unpaired surrogates incorrectly #39492

Open
abacabadabacaba opened this issue Jun 9, 2020 · 1 comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@abacabadabacaba
Copy link

When decoding some strings containing unpaired surrogates, UTF-16 decoder produces wrong number of \ufffd runes. Some examples:

  • On string \xdc\x00\xdc\x00: expected result \ufffd\ufffd (two copies of \ufffd), actual result \ufffd.
  • On string \xd8\x00\x00: expected result \ufffd, actual result \ufffd\ufffd.

The expected results are derived from a WhatWG spec.

Also, the name of internal function isHighSurrogate is misleading: it actually checks whether the argument is a low surrogate.

Code to reproduce:

package main

import (
	"golang.org/x/text/encoding/unicode"
	"fmt"
)

func main() {
	res, err := unicode.UTF16(unicode.BigEndian, unicode.IgnoreBOM).NewDecoder().String("\xd8\x00 ")
	fmt.Println(res, err)
}
@gopherbot gopherbot added this to the Unreleased milestone Jun 9, 2020
@odeke-em
Copy link
Member

Thank you for this report @abacabadabacaba!

Kindly cc-ing @mpvl.

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants