Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/encoding/unicode: add examples #14133

Open
TomOnTime opened this issue Jan 28, 2016 · 1 comment
Open

x/text/encoding/unicode: add examples #14133

TomOnTime opened this issue Jan 28, 2016 · 1 comment
Milestone

Comments

@TomOnTime
Copy link
Contributor

golang.org/x/text/encoding/unicode's documentation is a bit unclear to the newbie like myself that just wants to read files that Windows folks send them without having to become an expert in UTF-16. I think the docs make sense to someone that already understands transforms, unicode, etc, but not everyone should need to know such plumbing. All I wanted was something like ioutil.ReadFile() that automagically read MS-Windows UTF-16 files and gave me UTF-8.

An example like this would have helped:

func NewReader(rd io.Reader) io.Reader {
    // Make an tranformer that decodes MS-Windows (16LE) UTF files:
    winutf := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
    // Make a transformer that is like winutf, but abides by BOM if found:
    decoder := winutf.NewDecoder()
    // Make a Reader that uses decoder:
    return transform.NewReader(rd, unicode.BOMOverride(decoder))
}

fd, _ := os.Open(filename)
r := NewReader(fd)  // Read from "r" to get UTF-8.
utf8, _ := ioutil.ReadAll(r)
text := string(utf8)

It would be awesome if that example (or one like it) was added to the documentation. I'd be glad to submit a PR.

Furthermore, it would be useful to have a library similar to ioutil but automagically detects UTF-16 is found. I've made an example here: https://github.com/TomOnTime/utfutil

@ianlancetaylor ianlancetaylor changed the title Add examples to golang.org/x/text/encoding/unicode's x/text/encoding/unicode: add examples Jan 28, 2016
@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jan 28, 2016
@TomOnTime
Copy link
Contributor Author

P.S. I think the example might be clearer if the func was inlined.

@rsc rsc unassigned mpvl Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants