Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text: iterator does not handle zero width joiner #60522

Open
l0n3star opened this issue May 30, 2023 · 2 comments
Open

x/text: iterator does not handle zero width joiner #60522

l0n3star opened this issue May 30, 2023 · 2 comments
Labels
ExpertNeeded help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@l0n3star
Copy link

What version of Go are you using (go version)?

$ go version 1.20

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

I'm using the go playground

What did you do?

package main

import (
	"fmt"

	"golang.org/x/text/unicode/norm"
)

func main() {
	var ia norm.Iter
	s := "👩🏾‍🦰👱🏾🧑🏾‍⚖️"

	ia.InitString(norm.NFD, s)
	rev := ""

	for !ia.Done() {
		n := ia.Next()
		rev = string(n) + rev
	}

	fmt.Println(rev)
}

Playground: https://go.dev/play/p/Jq9CnUd2zA8

What did you expect to see?

It should print 🧑🏾‍⚖️👱🏾👩🏾‍🦰

I think it has to do with the library not handling zero width joiners. If I use this string: "😀😂" it works as expected.

What did you see instead?

️⚖‍🏾🧑🏾👱🦰‍🏾👩

@gopherbot gopherbot added this to the Unreleased milestone May 30, 2023
@mknyszek
Copy link
Contributor

CC @mpvl via https://dev.golang.org/owners

@mknyszek mknyszek added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ExpertNeeded help wanted labels May 31, 2023
@elliotwutingfeng
Copy link

The underlying issue is that x/text doesn't handle multi-rune unicode grapheme clusters.

In the meantime, you could use this third-party module uniseg to iterate through grapheme clusters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ExpertNeeded help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants