While I am sympathetic, I don't think this is the right answer.
At the moment the utf-16 is specified at the protocol level, so switching to utf-8 makes it non compliant, the benefits would have to be truly huge to be worth doing that.
Doing so with a command line flag when you have client server modes would almost inevitably result in two clients operating against the same server in competing modes.
We could make it negotiate per connection (although if we add that we should probably try to add it to the initialization dance and get it added to the spec) but that also means we cannot share cached results between two clients that disagree about what a position means, which would be kind of horrible, or we have to pick one format, and then do the dance at the protocol layer to convert all the positions (in which case we are not any better off than leaving it utf-16 and having the client do it instead).

Thanks @ianthehat. I'm very aware this proposal is not without its downsides! So appreciate the discussion.

At the moment the utf-16 is specified at the protocol level, so switching to utf-8 makes it non compliant, the benefits would have to be truly huge to be worth doing that.

Agreed. The most recent motivation has been the work to add fuzzy-finder driver symbol searching to govim. There we use the Symbol method as an input to the Vim fuzzy finder/selector. Whilst the gopls results are incredibly fast because they are cached in memory, the client needs to read file contents from disk for every match not open in the editor. Whilst not a huge cost on an individual basis, this starts to add up if you have hundreds of results per keystroke.

This same overhead applies to diagnostics returned by LSP (although the numbers there are generally far less).

It also makes it much harder to write a "plain old LSP client" for things like scripting (cc @findleyr).

Doing so with a command line flag when you have client server modes would almost inevitably result in two clients operating against the same server in competing modes.

Under what conditions do you see this as being "inevitable"? Do many people have two editors running against the same LSP?

Would it not be the case that for those people who are not using -remote right now there would be no impact because they are not sharing the same remote instance?

For those people who are using remote, you could also automatically add the result of the -utf8 flag to the key that determines which remote instance to select. Worst case you end up with two remote instances in auto mode.

Whilst it would always be possible for people to find edge cases, it feels like it would be entirely possible to prevent foot-gun incidents entirely if people use the forwarder to connect to gopls, remote or otherwise (that design has, incidentally, been worth its weight in gold).

although if we add that we should probably try to add it to the initialization dance and get it added to the spec

This is a good point. I struggle imagine that I'm the first person ever to bring this up.

I'd add refererence of microsoft/language-server-protocol#376

@mattn thanks, I had not seen that one, very relevant

@myitcv I expect in the future that everyone will be using -remote, it should be the default, and as for picking a different client, we already use far to much memory we don't want to double it.
The common case I would expect is something like you are using vim, it runs up the shared server, but sets it to utf-8 mode, then you run a command line tool, which tries to use the same server because then it gets to run fast on the results already sitting there thanks to vim being open on the same project.... but the tool was compiled only knowing about utf-16 because that is the protocol standard.
I just think that is far too much of a foot gut waiting to happen, if we do it then it absolutely has to go into the protocol negotiation.

I think we should close this issue in favor of microsoft/language-server-protocol#376. Like Ian said, we won't make any changes like this unless they are supported in the protocol.