I'm not sure we'll ever do this, since we already have the machinery in place to support UTF-16. https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocuments is a little unclear whether servers must support UTF-16, though it says that all clients must.

The potential benefit of this is the maintenance cost. Most of the Go tools are written to work with utf-8.
As seen in https://go-review.git.corp.google.com/c/tools/+/533755/1/gopls/internal/lsp/fake/editor.go line 1566 and also during the vulncheck analysis logic, it causes confusion and inefficiency (for correct UTF-16 conversion, the gopls has to read the file contents. This can be problematic for workspace-level position computation).

I think Rob's point is that the performance benefit of speaking UTF-8 with willing clients is not worth the complexity cost of having two different ways of doing things, and that UTF-8 only becomes a net win if we can stop supporting UTF-16, which doesn't seem likely ever to happen.

My point was to measure whether how many clients depend on only utf-16. According to the spec, utf-16 support mandate is for the backwards compatibility. If we find most clients support utf-8, we can discuss potentially dropping utf-16 support even though the spec says it is mandatory for backwards compatibility - my argument was - if there is no client depending on it, what would be the point of maintaining it and suffering?

On the other hand, I learned that VS code still supports only utf-16. And assuming conversion requires the source code, I doubt vscode that needs to work with many different extensions and maybe purely on client side without source code, can easily adopt utf-8 will happen near future. lol.

Closing.

Looks like we have a couple of places we don't do the conversion correctly. (e.g. gc details)