Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/text: panic of index out of range in language.Matcher #26580

Open
yuanbai opened this issue Jul 25, 2018 · 5 comments
Open

x/text: panic of index out of range in language.Matcher #26580

yuanbai opened this issue Jul 25, 2018 · 5 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@yuanbai
Copy link

yuanbai commented Jul 25, 2018 •
edited

What version of Go are you using (go version)?

go 1.9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64, darwin and linux.

What did you do?

language.MustParse accepts "lang_ZZ", e.g. en_ZZ, es_ZZ, etc. and returns a valid tag.
but m.Match(such a language tag) triggers a panic of "index out of range" at this lookup table: https://github.com/golang/text/blob/master/language/tables.go#L50

Sample code to reproduce the issue:

package main

import (
	"fmt"
	"golang.org/x/text/language"
)

func main() {
	supportedLanguages := []language.Tag{
		language.English,
		language.Arabic,
		language.BritishEnglish,
		language.Spanish,
		language.French,
		language.Italian,
		language.Japanese,
		language.Korean,
		language.Portuguese,
		language.Russian,
		language.SimplifiedChinese,
		language.TraditionalChinese,
	}

	tl := language.MustParse("en_ZZ")
	fmt.Printf("target language: %v\n", tl)

	m := language.NewMatcher(supportedLanguages)
	_, idx, _ := m.Match(tl)

	fmt.Printf("index matched: %v\n", idx)
}

Output:

target language: en-ZZ
panic: runtime error: index out of range

goroutine 1 [running]:
golang.org/x/text/language.regionGroupDist(0x139005701650135, 0x1650139)
	/Users/yb/go_root/src/golang.org/x/text/language/match.go:677 +0x13d
golang.org/x/text/language.(*bestMatch).update(0xc42003fbf0, 0xc4200722d0, 0x1650139, 0x0, 0x0, 0x101650057)
	/Users/ybi/go_root/src/golang.org/x/text/language/match.go:622 +0x28d
golang.org/x/text/language.(*matcher).getBest(0xc420072270, 0xc42000adc0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/yb/go_root/src/golang.org/x/text/language/match.go:504 +0x4a0
golang.org/x/text/language.(*matcher).Match(0xc420072270, 0xc42000adc0, 0x1, 0x1, 0x0, 0x0, 0x1161220, 0xc420072270, 0x2d952e1a19cc2e08)
	/Users/yb/go_root/src/golang.org/x/text/language/match.go:83 +0xd4
main.main()
	/Users/yb/test.go:28 +0x4d5
exit status 2

What did you expect to see?

either
(1) language.MustParse rejects "lang_ZZ" and throws an error;
or
(2) language.MustParse parses "lang_ZZ" to a tag of just "lang", similar to "lang_UND", e.g. language.MustParse of "en_ZZ" and "en" should return the identical result.
or
(3) language.Matcher supports ZZ country code and works properly.

What did you see instead?

language.MustParse accepts ZZ, but language.Matcher crashes on that.
@gopherbot gopherbot added this to the Unreleased milestone Jul 25, 2018
@schultz9999
Copy link

schultz9999 commented Sep 12, 2018 •
edited

Same reproduces in 1.10.3 darwin/amd64. Fails due to bad index:

image

@alexcons
Copy link

This is still happening even on 1.11.1 darwin/amd64

@agnivade
Copy link
Contributor

Yes, because it is not fixed yet.

/cc @mpvl

@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 28, 2018
@rlk833
Copy link

rlk833 commented Nov 11, 2019

Has this been fixed yet? It is a year later and I'm at go1.12.4 darwin/amd64 and it is occurring to my code.

What is actually wrong with the input? Maybe if I knew what was wrong with the input I could screen for it.

This is our production code used by hundreds of different people so I don't know in each case what is sent in by them.

@streak324
Copy link

streak324 commented Jun 17, 2020 •
edited

Not familiar with the code that well and this could just be a symptom of a larger problem, but it looks like the cause of the panics are because of one byte in the regionToGroups byte array in the language/tables.go file that was removed (accidentally?)
golang/text@b85db4f#diff-9b4a7a0caec1562de30551ab14452668L3346

If anyone encounters this, and wants to try out a hacky fix, just fork the repo, add an extra zero byte to the byte array. https://github.com/golang/text/blob/master/language/tables.go#L101 and use the forked repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
7 participants
@alexcons @agnivade @schultz9999 @rlk833 @yuanbai @streak324 @gopherbot