x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

iNeil77 · 2023-12-28T01:27:52Z

gopls version

v0.11.0

go env

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/ineilpaul/Library/Caches/go-build'
GOENV='/Users/ineilpaul/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/ineilpaul/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ineilpaul/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/cw/p84y_v8x6yg8sk4jfx1sqnh40000gn/T/go-build85885991=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

TLDR
I don't see any way to enable semanticTokens (seems off by default) when running gopls from the CLI. How does one pass configurations equivalent to the settings.json when in such a setting?

Long Version
I am trying to extract semantic tokens from gopls in the terminal. Towards this, I have written up the following rudimentary JsonRPC client that inits the server, opens the document, queries semantic tokens, closes the document and finally shuts down the server connection.

import json
import os
import asyncio

async def lsp_client():
    host = '0.0.0.0'     # The host where gopls is running
    port = 56063         # The port where gopls is listening

    reader, writer = await asyncio.open_connection(host, port)

    # Function to send a JSON RPC message
    async def send_message(message):
        message_json = json.dumps(message).encode('utf-8')
        # Prepend message with Content-Length header required by LSP
        message_length = len(message_json)
        writer.write(f"Content-Length: {message_length}\r\n\r\n".encode('utf-8') + message_json)
        await writer.drain()
    
    async def receive_message():
        while True:
            content_length = int((await reader.readline()).decode('utf-8').split(":")[1].strip())
            await reader.readline()  # Read the next two CRLF characters
            message_text = (await reader.readexactly(content_length)).decode('utf-8')
            message = json.loads(message_text)

            if 'method' in message and message['method'] == 'window/showMessage':
                print("Notification from server:", message['params']['message'])
            elif 'method' in message and message['method'] == 'window/logMessage':
                print("Log from server:", message['params']['message'])
            else:
                return message

    # Initialize the server
    init_msg = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": "initialize",
        "params": {
            "rootUri": f"file://{os.getcwd()}/Go/Tests/",
            "capabilities": {
                "textDocument": {
                    "semanticTokens": {
                        "tokenTypes": [
                            "namespace", "type", "class", "enum", "interface",
                            "struct", "typeParameter", "parameter", "variable",
                            "property", "enumMember", "event", "function", "method",
                            "macro", "keyword", "modifier", "comment", "string",
                            "number", "regexp", "operator"
                        ],
                        "tokenModifiers": [
                            "declaration", "definition", "readonly", "static",
                            "deprecated", "abstract", "async", "modification",
                            "documentation", "defaultLibrary"
                        ],
                        "formats": ["relative"],
                        "requests": {
                            "full": True
                        }
                    },
                    "initializationOptions": {
                        "semanticTokens": True
                    }
                }
            },
        },
    }
    await send_message(init_msg)
    print(f"Initialization Response: {await receive_message()}")

    # Notify the server that we're initialized
    initialized_msg = {
        "jsonrpc": "2.0",
        "id": 2,
        "method": "initialized",
        "params": {},
    }
    await send_message(initialized_msg)
    print(f"Initialized Response: {await receive_message()}")

    # Open a document
    open_doc_msg = {
        "jsonrpc": "2.0",
        "method": "textDocument/didOpen",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go",
                "languageId": "go",
                "version": 1,
                "text": "package main\n\nfunc main() {\n\t// Your Go code here\n}"
            }
        }
    }
    await send_message(open_doc_msg)
    print(f"Open Document Response: {await receive_message()}")

    # Request semantic tokens
    semantic_tokens_msg = {
        "jsonrpc": "2.0",
        "id": 2,
        "method": "textDocument/semanticTokens/full",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go"
            }
        }
    }
    await send_message(semantic_tokens_msg)
    print(f"Semantic Tokens Response: {await receive_message()}")

    # Close the document
    close_doc_msg = {
        "jsonrpc": "2.0",
        "method": "textDocument/didClose",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go"
            }
        }
    }
    await send_message(close_doc_msg)
    print(f"Close Document Response: {await receive_message()}")

    # Close the server
    shutdown_msg = {
        "jsonrpc": "2.0",
        "id": 3,
        "method": "shutdown",
    }
    await send_message(shutdown_msg)
    print(f"Shutdown Response: {await receive_message()}")

    writer.close()
    await writer.wait_closed()

asyncio.run(lsp_client())

However on running this I get the following logs:

Initialization Response: {'jsonrpc': '2.0', 'result': {'capabilities': {'textDocumentSync': {'openClose': True, 'change': 2, 'save': {}}, 'completionProvider': {'triggerCharacters': ['.']}, 'hoverProvider': True, 'signatureHelpProvider': {'triggerCharacters': ['(', ',']}, 'definitionProvider': True, 'typeDefinitionProvider': True, 'implementationProvider': True, 'referencesProvider': True, 'documentHighlightProvider': True, 'documentSymbolProvider': True, 'codeActionProvider': True, 'codeLensProvider': {}, 'documentLinkProvider': {}, 'workspaceSymbolProvider': True, 'documentFormattingProvider': True, 'renameProvider': True, 'foldingRangeProvider': True, 'executeCommandProvider': {'commands': ['gopls.add_dependency', 'gopls.add_import', 'gopls.apply_fix', 'gopls.check_upgrades', 'gopls.edit_go_directive', 'gopls.fetch_vulncheck_result', 'gopls.gc_details', 'gopls.generate', 'gopls.generate_gopls_mod', 'gopls.go_get_package', 'gopls.list_imports', 'gopls.list_known_packages', 'gopls.regenerate_cgo', 'gopls.remove_dependency', 'gopls.reset_go_mod_diagnostics', 'gopls.run_govulncheck', 'gopls.run_tests', 'gopls.start_debugging', 'gopls.test', 'gopls.tidy', 'gopls.toggle_gc_details', 'gopls.update_go_sum', 'gopls.upgrade_dependency', 'gopls.vendor']}, 'callHierarchyProvider': True, 'inlayHintProvider': {}, 'workspace': {'workspaceFolders': {'supported': True, 'changeNotifications': 'workspace/didChangeWorkspaceFolders'}, 'fileOperations': {}}}, 'serverInfo': {'name': 'gopls', 'version': '{"GoVersion":"go1.21.5","Path":"golang.org/x/tools/gopls","Main":{"Path":"golang.org/x/tools/gopls","Version":"v0.11.0","Sum":"h1:/nvKHdTtePQmrv9XN3gIUN9MOdUrKzO/dcqgbG6x8EY=","Replace":null},"Deps":[{"Path":"github.com/BurntSushi/toml","Version":"v1.2.1","Sum":"h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=","Replace":null},{"Path":"github.com/google/go-cmp","Version":"v0.5.9","Sum":"h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=","Replace":null},{"Path":"github.com/sergi/go-diff","Version":"v1.1.0","Sum":"h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=","Replace":null},{"Path":"golang.org/x/exp","Version":"v0.0.0-20221031165847-c99f073a8326","Sum":"h1:QfTh0HpN6hlw6D3vu8DAwC8pBIwikq0AI1evdm+FksE=","Replace":null},{"Path":"golang.org/x/exp/typeparams","Version":"v0.0.0-20221031165847-c99f073a8326","Sum":"h1:fl8k2zg28yA23264d82M4dp+YlJ3ngDcpuB1bewkQi4=","Replace":null},{"Path":"golang.org/x/mod","Version":"v0.7.0","Sum":"h1:LapD9S96VoQRhi/GrNTqeBJFrUjs5UHCAtTlgwA5oZA=","Replace":null},{"Path":"golang.org/x/sync","Version":"v0.1.0","Sum":"h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=","Replace":null},{"Path":"golang.org/x/sys","Version":"v0.2.0","Sum":"h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A=","Replace":null},{"Path":"golang.org/x/text","Version":"v0.4.0","Sum":"h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg=","Replace":null},{"Path":"golang.org/x/tools","Version":"v0.3.1-0.20221213193459-ca17b2c27ca8","Sum":"h1:7/HkGkN/2ktghBCSRRgp31wAww4syfsW52tj7yirjWk=","Replace":null},{"Path":"golang.org/x/vuln","Version":"v0.0.0-20221109205719-3af8368ee4fe","Sum":"h1:qptQiQwEpETwDiz85LKtChqif9xhVkAm8Nhxs0xnTww=","Replace":null},{"Path":"honnef.co/go/tools","Version":"v0.3.3","Sum":"h1:oDx7VAwstgpYpb3wv0oxiZlxY+foCpRAwY7Vk6XpAgA=","Replace":null},{"Path":"mvdan.cc/gofumpt","Version":"v0.4.0","Sum":"h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=","Replace":null},{"Path":"mvdan.cc/xurls/v2","Version":"v2.4.0","Sum":"h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=","Replace":null}],"Settings":[{"Key":"-buildmode","Value":"exe"},{"Key":"-compiler","Value":"gc"},{"Key":"DefaultGODEBUG","Value":"panicnil=1"},{"Key":"CGO_ENABLED","Value":"1"},{"Key":"CGO_CFLAGS","Value":""},{"Key":"CGO_CPPFLAGS","Value":""},{"Key":"CGO_CXXFLAGS","Value":""},{"Key":"CGO_LDFLAGS","Value":""},{"Key":"GOARCH","Value":"arm64"},{"Key":"GOOS","Value":"darwin"}],"Version":"v0.11.0"}'}}, 'id': 1}
Notification from server: Loading packages...
Log from server: 2023/12/28 02:13:01 go env for /Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
(root /Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/)
(go version go version go1.21.5 darwin/arm64)
(valid build configuration = false)
(build flags: [])
GOINSECURE=
GOSUMDB=sum.golang.org
GOMOD=/dev/null
GOPROXY=https://proxy.golang.org,direct
GOROOT=/opt/homebrew/Cellar/go/1.21.5/libexec
GONOSUMDB=
GO111MODULE=
GOCACHE=/Users/ineilpaul/Library/Caches/go-build
GOFLAGS=
GOMODCACHE=/Users/ineilpaul/go/pkg/mod
GOPRIVATE=
GOPATH=/Users/ineilpaul/go
GOWORK=
GONOPROXY=


Log from server: 2023/12/28 02:13:01 go/packages.Load #3
        snapshot=0
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        query=[./ builtin]
        packages=2

Log from server: 2023/12/28 02:13:01 go/packages.Load #3: updating metadata for 1 packages

Notification from server: Finished loading packages.
Initialized Response: {'jsonrpc': '2.0', 'result': None, 'id': 2}
Log from server: 2023/12/28 02:13:01 go/packages.Load #4
        snapshot=1
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        query=[file=/path/to/your/file.go]
        packages=1

Log from server: 2023/12/28 02:13:01 go/packages.Load #4
        snapshot=1
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        package="command-line-arguments"
        files=[/path/to/your/file.go]

Log from server: 2023/12/28 02:13:01 go/packages.Load #4: updating metadata for 1 packages

Open Document Response: {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///path/to/your/file.go', 'version': 1, 'diagnostics': []}}
Semantic Tokens Response: {'jsonrpc': '2.0', 'error': {'code': 0, 'message': 'semantictokens are disabled'}, 'id': 2}
Close Document Response: {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///path/to/your/file.go', 'diagnostics': []}}
Log from server: 2023/12/28 02:13:01 Shutdown session
        shutdown_session=2

Shutdown Response: {'jsonrpc': '2.0', 'result': None, 'id': 3}

It appears the semantic tokens are disabled by default and can only be enabled by editing the settings.json file which only seems to apply when running the language server from within the IDE. How does one go about enabling this when running it on the CLI? I don't see any way to pass a settings file to the CLI executable.

What did you expect to see?

Semantic tokenization of Go source file.

What did you see instead?

{'jsonrpc': '2.0', 'error': {'code': 0, 'message': 'semantictokens are disabled'}, 'id': 2}

Editor and settings

No response

Logs

No response

The text was updated successfully, but these errors were encountered:

findleyr · 2023-12-28T20:24:07Z

Hmm, the command line is really just for debugging, but we shouldn't have it if it doesn't work!

Can I ask why do you want to use this feature? Requesting semantic tokens from the command line will be very inefficient.

iNeil77 · 2023-12-28T22:02:21Z

I wished to use semantic tokenization to test the code-understanding ability of code language models. The unified interface provided by the LSP has been a huge leg up towards this end. So far I have used clangd for C, C++ and rust-analyzer for Rust from the terminal. I was wishing I could do the same for Go without instrumenting a VS code extension.

Would it be possible for you to allow for a way for a settings.json to be passed to the gopls CLI executable?

hyangah · 2023-12-29T13:19:27Z

Is the init_param correct? From LSP spec, initializationOptions is a top-level field of the InitializeParams message, not the TextDocumentClientCapabilities message.

iNeil77 · 2023-12-29T20:01:44Z

@hyangah Thanks for pointing out my silly error!

It works now. It would seem I have completely misunderstood how this capability init worked.
Closing this issue.

iNeil77 added gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. labels Dec 28, 2023

gopherbot added this to the Unreleased milestone Dec 28, 2023

iNeil77 closed this as completed Dec 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

iNeil77 commented Dec 28, 2023

findleyr commented Dec 28, 2023

iNeil77 commented Dec 28, 2023

hyangah commented Dec 29, 2023

iNeil77 commented Dec 29, 2023

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

Comments

iNeil77 commented Dec 28, 2023

gopls version

go env

What did you do?

What did you expect to see?

What did you see instead?

Editor and settings

Logs

findleyr commented Dec 28, 2023

iNeil77 commented Dec 28, 2023

hyangah commented Dec 29, 2023

iNeil77 commented Dec 29, 2023