We'll at least figure this out before Go 1.9 which was slated to depend on this (#17268).

The IDNA used by Go 1.8 doesn't involve tables and probably only covers like 99.9% of the use cases, which is probably sufficient. @mpvl?

Maybe we need some sort of opt-in mechanism for the tables.

It will be hard to reduce the size of these tables. Go 1.8 includes some of these tables, probably adding about 60-80k. I estimate the the full implementation adds about 90k of data plus code.
So the size increase is somewhat surprising.

There are two options to reduce this

1. See why the addition is larger than expected and if it can be fixed. If successful, it will still add about 100k (although further rather complicated optimizations could probably reduce it by another 30k).
2. Make some of the accuracy optional. I should be possible to significantly improve on the current implementation by adding 30-50k, while making the rest optional. The improvement would mean handling most languages properly, rather than handling mostly English properly. :)

I don't think keeping the old implementation and the new on as an option is a good idea. The old implementation differs in hard-to-document ways. Also, it pulls in different tables, meaning that someone using the new implementation will get an additional penalty making it even more expensive to "do the right thing".

I don't have a good idea on how to make the additional accuracy optional, though. Probably something along the lines of having precise and imprecise Profiles in idna and then making the profile an option in Transport. The idna package would then be designed such that tables only associated with profiles that are not used are dropped by the linker.

Correction, I was assuming you were comparing to the current implementation in core. In 1.8, core http wraps idna and adds a few tables to make it more correct. Compared to this, the new implementation adds about 50k.

I ran some tests. Very roughly, the breakdown of bulky stuff used in the new implementation is:
tables in idna: 40k
norm: 124k
bidi(rule): 31k

So norm seems to be the culprit. The bad news is that norm is rather important. The good news is that norm is already used in 1.8 so that effectively this implementation adds only about 50k (it eliminates the width table).

There does seem to be one possibility: the idna tables already contain some of the normalization information. It may be possible to interleave some of the norm information into these tables and then reimplement the normalization algorithm for this purpose. A very rough estimate is that this will save about 60-80k. This is a very complicated optimization, though. It also adds up to 60k of data to applications that are already using norm somewhere anyway. I'm skeptical this is worth it.

The bidirule could be made optional. As it only saves 15% of the total tables used here it seems not worth the trouble. But if people think so anyway it is worth considering.

Please take a look at https://go-review.googlesource.com/#/c/36512/.

I've now adopted the semantics that a zero Profile only does Punycode and that any validation is optional. A few pre-made profiles are provided for Lookup and Registration cases of the IDNA RFC. One could argue that at least a normalization check should at least always be in there, but that's also the largest table. If validation is not used, tables are not linked in.

The default ToUnicode and ToASCII are now wrappers for idna.Punycode.{ToUnicode|ToASCII}. A bit odd, but it is justifiable to take the zero profile as the default. This means the interpretation of these functions is virtually left unchanged with this elaborate update while still providing all the UTS 46 goodness.

CL https://golang.org/cl/36512 mentions this issue.

CL https://golang.org/cl/37050 mentions this issue.