Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/cases: incorrect case folding for Ꮭ #46101

Open
jdeflander opened this issue May 11, 2021 · 6 comments
Open

x/text/cases: incorrect case folding for Ꮭ #46101

jdeflander opened this issue May 11, 2021 · 6 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@jdeflander
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.16 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jasper/Library/Caches/go-build"
GOENV="/Users/jasper/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jasper/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/jasper/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jasper/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4y/xv17zpy917g587hyy3m6zxrw0000gn/T/go-build2484744143=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

https://play.golang.org/p/N-Ypx8AgIgl

What did you expect to see?

Ꮭ
Ꮭ

What did you see instead?

Ꮭ
ꮭ
@gopherbot gopherbot added this to the Unreleased milestone May 11, 2021
@seankhliao
Copy link
Member

Unicode says is the lowercase form for , is there any reason to expect otherwise?

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 11, 2021
@jdeflander
Copy link
Author

Unicode says is the lowercase form for , is there any reason to expect otherwise?

I expect case folding to produce a case-insensitive result, not necessarily a lowercase form. I could definitely be misunderstanding this, but Python's casefold folds both 'ꮭ' and 'Ꮭ' to 'Ꮭ'.

@seankhliao seankhliao added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 11, 2021
@seankhliao
Copy link
Member

cc @mpvl

@mpvl
Copy link
Contributor

mpvl commented May 13, 2021

In rare cases case folding should map to the upper case variant indeed. Looking at https://www.unicode.org/Public/14.0.0/ucd/UnicodeData-14.0.0d10.txt and https://www.unicode.org/Public/14.0.0/ucd/CaseFolding-14.0.0d8.txt this is indeed the case.

So the correct behavior indeed is to map to U+13DD and this looks like a bug.

FYI: the reason why it folds to the uppercase form is to have consistent behavior with older versions of Unicode. Cherokee originally only had its uppercase letters present in Unicode and only became a bicameral script in 8.0.0.

@BrannonKing
Copy link

@mpvl , you're referencing the v14 tables on purpose? The Unicode v14 tables do have different case fold mappings than v13. It looks like x/text/cases is using Unicode v14 tables even though it claims to be Unicode version 13. That's possibly the culprit. The example in #47755 shows a few other characters that have this struggle.

@gopherbot
Copy link

Change https://go.dev/cl/445115 mentions this issue: cases: fix build, memory leaks, and error checks with -tags=icu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants