encoding/xml: unable to handle utf-16-encoded file without manual manipulation of source bytes #38335
Labels
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Comments
andybons
added
the
NeedsInvestigation
|
I know this is old but does the following help @mccolljr ? dec := xml.NewDecoder(reader)
dec.CharsetReader = func(charset string, r io.Reader) (io.Reader, error) {
enc, err := ianaindex.IANA.Encoding(charset)
if err != nil {
return nil, fmt.Errorf("charset %s: %w", charset, err)
}
if enc == nil {
// Assume it's compatible with (a subset of) UTF-8 encoding
// Bug: https://github.com/golang/go/issues/19421
return r, nil
}
return enc.NewDecoder().Reader(r), nil
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I have a file like the following, encoded in
utf-16(BOM is little endian) on disk:When I read the file from disk, the bytes (including the line containing
<?xml version="1.0" encoding="utf-16"?>), are encoded in utf-16.I wanted to parse the full file, with no modification, using the
encoding/xmlpackage.What did you expect to see?
I expected to be able to either A: transform the file's bytes to
utf8and pass that reader toxml.NewDecoderto successfully parse the utf8 data as xml, or B: pass theutf16-encoded bytes toxml.NewDeciderand provide aCharsetReaderto successfully parse theutf16data as XML.What did you see instead?
There were a couple of error cases.
resultOfOsOpendirectly toxml.NewDecoder, with or without settingCharsetReadertocharset.NewReaderLabel:XML syntax error on line 1: invalid UTF-8utf-8reader returned bycharset.NewReader(resultOfOsOpen, "text/xml")toxml.NewDecoder:xml: encoding "utf-16" declared but Decoder.CharsetReader is nilutf-8reader returned bycharset.NewReader(resultOfOsOpen, "text/xml")toxml.NewDecoder, AND setCharsetReadertocharset.NewReaderLabel: The (nowutf-8-encoded) data is interpreted asutf-16and I the decoder reads the file as gibberish.It seems to me that the
encoding/xmlpackage expects the line containing<?xml version="1.0" encoding="utf-16"?>to be in some encoding that resembles validutf-8-encoded text in order to read the encoding line and properly parse the rest of the file, OR for the line to be removed if manual transformation of the input is done beforehand (like withutf-16, which cannot be read as validutf-8text).Am I missing something? Is there a way to do this without modifying the input bytes?
The text was updated successfully, but these errors were encountered: