Skip to content

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
iNeil77 opened this issue Dec 28, 2023 · 4 comments
Closed

x/tools/gopls: Enable semanticTokens when running gopls from the CLI #64884

iNeil77 opened this issue Dec 28, 2023 · 4 comments
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@iNeil77
Copy link

iNeil77 commented Dec 28, 2023

gopls version

v0.11.0

go env

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/ineilpaul/Library/Caches/go-build'
GOENV='/Users/ineilpaul/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/ineilpaul/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ineilpaul/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/cw/p84y_v8x6yg8sk4jfx1sqnh40000gn/T/go-build85885991=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

TLDR
I don't see any way to enable semanticTokens (seems off by default) when running gopls from the CLI. How does one pass configurations equivalent to the settings.json when in such a setting?

Long Version
I am trying to extract semantic tokens from gopls in the terminal. Towards this, I have written up the following rudimentary JsonRPC client that inits the server, opens the document, queries semantic tokens, closes the document and finally shuts down the server connection.

import json
import os
import asyncio

async def lsp_client():
    host = '0.0.0.0'     # The host where gopls is running
    port = 56063         # The port where gopls is listening

    reader, writer = await asyncio.open_connection(host, port)

    # Function to send a JSON RPC message
    async def send_message(message):
        message_json = json.dumps(message).encode('utf-8')
        # Prepend message with Content-Length header required by LSP
        message_length = len(message_json)
        writer.write(f"Content-Length: {message_length}\r\n\r\n".encode('utf-8') + message_json)
        await writer.drain()
    
    async def receive_message():
        while True:
            content_length = int((await reader.readline()).decode('utf-8').split(":")[1].strip())
            await reader.readline()  # Read the next two CRLF characters
            message_text = (await reader.readexactly(content_length)).decode('utf-8')
            message = json.loads(message_text)

            if 'method' in message and message['method'] == 'window/showMessage':
                print("Notification from server:", message['params']['message'])
            elif 'method' in message and message['method'] == 'window/logMessage':
                print("Log from server:", message['params']['message'])
            else:
                return message

    # Initialize the server
    init_msg = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": "initialize",
        "params": {
            "rootUri": f"file://{os.getcwd()}/Go/Tests/",
            "capabilities": {
                "textDocument": {
                    "semanticTokens": {
                        "tokenTypes": [
                            "namespace", "type", "class", "enum", "interface",
                            "struct", "typeParameter", "parameter", "variable",
                            "property", "enumMember", "event", "function", "method",
                            "macro", "keyword", "modifier", "comment", "string",
                            "number", "regexp", "operator"
                        ],
                        "tokenModifiers": [
                            "declaration", "definition", "readonly", "static",
                            "deprecated", "abstract", "async", "modification",
                            "documentation", "defaultLibrary"
                        ],
                        "formats": ["relative"],
                        "requests": {
                            "full": True
                        }
                    },
                    "initializationOptions": {
                        "semanticTokens": True
                    }
                }
            },
        },
    }
    await send_message(init_msg)
    print(f"Initialization Response: {await receive_message()}")

    # Notify the server that we're initialized
    initialized_msg = {
        "jsonrpc": "2.0",
        "id": 2,
        "method": "initialized",
        "params": {},
    }
    await send_message(initialized_msg)
    print(f"Initialized Response: {await receive_message()}")

    # Open a document
    open_doc_msg = {
        "jsonrpc": "2.0",
        "method": "textDocument/didOpen",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go",
                "languageId": "go",
                "version": 1,
                "text": "package main\n\nfunc main() {\n\t// Your Go code here\n}"
            }
        }
    }
    await send_message(open_doc_msg)
    print(f"Open Document Response: {await receive_message()}")

    # Request semantic tokens
    semantic_tokens_msg = {
        "jsonrpc": "2.0",
        "id": 2,
        "method": "textDocument/semanticTokens/full",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go"
            }
        }
    }
    await send_message(semantic_tokens_msg)
    print(f"Semantic Tokens Response: {await receive_message()}")

    # Close the document
    close_doc_msg = {
        "jsonrpc": "2.0",
        "method": "textDocument/didClose",
        "params": {
            "textDocument": {
                "uri": "file:///path/to/your/file.go"
            }
        }
    }
    await send_message(close_doc_msg)
    print(f"Close Document Response: {await receive_message()}")

    # Close the server
    shutdown_msg = {
        "jsonrpc": "2.0",
        "id": 3,
        "method": "shutdown",
    }
    await send_message(shutdown_msg)
    print(f"Shutdown Response: {await receive_message()}")

    writer.close()
    await writer.wait_closed()

asyncio.run(lsp_client())

However on running this I get the following logs:

Initialization Response: {'jsonrpc': '2.0', 'result': {'capabilities': {'textDocumentSync': {'openClose': True, 'change': 2, 'save': {}}, 'completionProvider': {'triggerCharacters': ['.']}, 'hoverProvider': True, 'signatureHelpProvider': {'triggerCharacters': ['(', ',']}, 'definitionProvider': True, 'typeDefinitionProvider': True, 'implementationProvider': True, 'referencesProvider': True, 'documentHighlightProvider': True, 'documentSymbolProvider': True, 'codeActionProvider': True, 'codeLensProvider': {}, 'documentLinkProvider': {}, 'workspaceSymbolProvider': True, 'documentFormattingProvider': True, 'renameProvider': True, 'foldingRangeProvider': True, 'executeCommandProvider': {'commands': ['gopls.add_dependency', 'gopls.add_import', 'gopls.apply_fix', 'gopls.check_upgrades', 'gopls.edit_go_directive', 'gopls.fetch_vulncheck_result', 'gopls.gc_details', 'gopls.generate', 'gopls.generate_gopls_mod', 'gopls.go_get_package', 'gopls.list_imports', 'gopls.list_known_packages', 'gopls.regenerate_cgo', 'gopls.remove_dependency', 'gopls.reset_go_mod_diagnostics', 'gopls.run_govulncheck', 'gopls.run_tests', 'gopls.start_debugging', 'gopls.test', 'gopls.tidy', 'gopls.toggle_gc_details', 'gopls.update_go_sum', 'gopls.upgrade_dependency', 'gopls.vendor']}, 'callHierarchyProvider': True, 'inlayHintProvider': {}, 'workspace': {'workspaceFolders': {'supported': True, 'changeNotifications': 'workspace/didChangeWorkspaceFolders'}, 'fileOperations': {}}}, 'serverInfo': {'name': 'gopls', 'version': '{"GoVersion":"go1.21.5","Path":"golang.org/x/tools/gopls","Main":{"Path":"golang.org/x/tools/gopls","Version":"v0.11.0","Sum":"h1:/nvKHdTtePQmrv9XN3gIUN9MOdUrKzO/dcqgbG6x8EY=","Replace":null},"Deps":[{"Path":"github.com/BurntSushi/toml","Version":"v1.2.1","Sum":"h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=","Replace":null},{"Path":"github.com/google/go-cmp","Version":"v0.5.9","Sum":"h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=","Replace":null},{"Path":"github.com/sergi/go-diff","Version":"v1.1.0","Sum":"h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=","Replace":null},{"Path":"golang.org/x/exp","Version":"v0.0.0-20221031165847-c99f073a8326","Sum":"h1:QfTh0HpN6hlw6D3vu8DAwC8pBIwikq0AI1evdm+FksE=","Replace":null},{"Path":"golang.org/x/exp/typeparams","Version":"v0.0.0-20221031165847-c99f073a8326","Sum":"h1:fl8k2zg28yA23264d82M4dp+YlJ3ngDcpuB1bewkQi4=","Replace":null},{"Path":"golang.org/x/mod","Version":"v0.7.0","Sum":"h1:LapD9S96VoQRhi/GrNTqeBJFrUjs5UHCAtTlgwA5oZA=","Replace":null},{"Path":"golang.org/x/sync","Version":"v0.1.0","Sum":"h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=","Replace":null},{"Path":"golang.org/x/sys","Version":"v0.2.0","Sum":"h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A=","Replace":null},{"Path":"golang.org/x/text","Version":"v0.4.0","Sum":"h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg=","Replace":null},{"Path":"golang.org/x/tools","Version":"v0.3.1-0.20221213193459-ca17b2c27ca8","Sum":"h1:7/HkGkN/2ktghBCSRRgp31wAww4syfsW52tj7yirjWk=","Replace":null},{"Path":"golang.org/x/vuln","Version":"v0.0.0-20221109205719-3af8368ee4fe","Sum":"h1:qptQiQwEpETwDiz85LKtChqif9xhVkAm8Nhxs0xnTww=","Replace":null},{"Path":"honnef.co/go/tools","Version":"v0.3.3","Sum":"h1:oDx7VAwstgpYpb3wv0oxiZlxY+foCpRAwY7Vk6XpAgA=","Replace":null},{"Path":"mvdan.cc/gofumpt","Version":"v0.4.0","Sum":"h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=","Replace":null},{"Path":"mvdan.cc/xurls/v2","Version":"v2.4.0","Sum":"h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=","Replace":null}],"Settings":[{"Key":"-buildmode","Value":"exe"},{"Key":"-compiler","Value":"gc"},{"Key":"DefaultGODEBUG","Value":"panicnil=1"},{"Key":"CGO_ENABLED","Value":"1"},{"Key":"CGO_CFLAGS","Value":""},{"Key":"CGO_CPPFLAGS","Value":""},{"Key":"CGO_CXXFLAGS","Value":""},{"Key":"CGO_LDFLAGS","Value":""},{"Key":"GOARCH","Value":"arm64"},{"Key":"GOOS","Value":"darwin"}],"Version":"v0.11.0"}'}}, 'id': 1}
Notification from server: Loading packages...
Log from server: 2023/12/28 02:13:01 go env for /Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
(root /Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/)
(go version go version go1.21.5 darwin/arm64)
(valid build configuration = false)
(build flags: [])
GOINSECURE=
GOSUMDB=sum.golang.org
GOMOD=/dev/null
GOPROXY=https://proxy.golang.org,direct
GOROOT=/opt/homebrew/Cellar/go/1.21.5/libexec
GONOSUMDB=
GO111MODULE=
GOCACHE=/Users/ineilpaul/Library/Caches/go-build
GOFLAGS=
GOMODCACHE=/Users/ineilpaul/go/pkg/mod
GOPRIVATE=
GOPATH=/Users/ineilpaul/go
GOWORK=
GONOPROXY=


Log from server: 2023/12/28 02:13:01 go/packages.Load #3
        snapshot=0
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        query=[./ builtin]
        packages=2

Log from server: 2023/12/28 02:13:01 go/packages.Load #3: updating metadata for 1 packages

Notification from server: Finished loading packages.
Initialized Response: {'jsonrpc': '2.0', 'result': None, 'id': 2}
Log from server: 2023/12/28 02:13:01 go/packages.Load #4
        snapshot=1
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        query=[file=/path/to/your/file.go]
        packages=1

Log from server: 2023/12/28 02:13:01 go/packages.Load #4
        snapshot=1
        directory=file:///Users/ineilpaul/Downloads/Semantic_LSP/Go/Tests/
        package="command-line-arguments"
        files=[/path/to/your/file.go]

Log from server: 2023/12/28 02:13:01 go/packages.Load #4: updating metadata for 1 packages

Open Document Response: {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///path/to/your/file.go', 'version': 1, 'diagnostics': []}}
Semantic Tokens Response: {'jsonrpc': '2.0', 'error': {'code': 0, 'message': 'semantictokens are disabled'}, 'id': 2}
Close Document Response: {'jsonrpc': '2.0', 'method': 'textDocument/publishDiagnostics', 'params': {'uri': 'file:///path/to/your/file.go', 'diagnostics': []}}
Log from server: 2023/12/28 02:13:01 Shutdown session
        shutdown_session=2

Shutdown Response: {'jsonrpc': '2.0', 'result': None, 'id': 3}

It appears the semantic tokens are disabled by default and can only be enabled by editing the settings.json file which only seems to apply when running the language server from within the IDE. How does one go about enabling this when running it on the CLI? I don't see any way to pass a settings file to the CLI executable.

What did you expect to see?

Semantic tokenization of Go source file.

What did you see instead?

{'jsonrpc': '2.0', 'error': {'code': 0, 'message': 'semantictokens are disabled'}, 'id': 2}

Editor and settings

No response

Logs

No response
@iNeil77 iNeil77 added gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. labels Dec 28, 2023
@gopherbot gopherbot added this to the Unreleased milestone Dec 28, 2023
@findleyr
Copy link
Member

Hmm, the command line is really just for debugging, but we shouldn't have it if it doesn't work!

Can I ask why do you want to use this feature? Requesting semantic tokens from the command line will be very inefficient.

@iNeil77
Copy link
Author

iNeil77 commented Dec 28, 2023

I wished to use semantic tokenization to test the code-understanding ability of code language models. The unified interface provided by the LSP has been a huge leg up towards this end. So far I have used clangd for C, C++ and rust-analyzer for Rust from the terminal. I was wishing I could do the same for Go without instrumenting a VS code extension.

Would it be possible for you to allow for a way for a settings.json to be passed to the gopls CLI executable?

@hyangah
Copy link
Contributor

hyangah commented Dec 29, 2023

Is the init_param correct? From LSP spec, initializationOptions is a top-level field of the InitializeParams message, not the TextDocumentClientCapabilities message.

@iNeil77
Copy link
Author

iNeil77 commented Dec 29, 2023

@hyangah Thanks for pointing out my silly error!

It works now. It would seem I have completely misunderstood how this capability init worked.
Closing this issue.

@iNeil77 iNeil77 closed this as completed Dec 29, 2023
@golang golang locked and limited conversation to collaborators Dec 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@hyangah @gopherbot @iNeil77 @findleyr