Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UI for running fuzz tests #3152

Open
qiulaidongfeng opened this issue Jan 12, 2024 · 12 comments
Open

UI for running fuzz tests #3152

qiulaidongfeng opened this issue Jan 12, 2024 · 12 comments
Labels
FeatureRequest go-test issues related to go test support (test output, test explorer, ...)

Comments

@qiulaidongfeng
Copy link

gopls version

Build info ---------- golang.org/x/tools/gopls v0.14.2 golang.org/x/tools/gopls@v0.14.2 h1:sIw6vjZiuQ9S7s0auUUkHlWgsCkKZFWDHmrge8LYsnc= github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= github.com/google/go-cmp@v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0= golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y= golang.org/x/mod@v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0= golang.org/x/sync@v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ= golang.org/x/sys@v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q= golang.org/x/telemetry@v0.0.0-20231114163143-69313e640400 h1:brbkEFfGwNGAEkykUOcryE/JiHUMMJouzE0fWWmz/QU= golang.org/x/text@v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= golang.org/x/tools@v0.14.1-0.20231114185516-c9d3e7de13fd h1:Oku7E+OCrXHyst1dG1z10etCTxewCHXNFLRlyMPbh3w= golang.org/x/vuln@v1.0.1 h1:KUas02EjQK5LTuIx1OylBQdKKZ9jeugs+HiqO5HormU= honnef.co/go/tools@v0.4.5 h1:YGD4H+SuIOOqsyoLOpZDWcieM28W47/zRO7f+9V3nvo= mvdan.cc/gofumpt@v0.4.0 h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM= mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc= go: devel go1.22-46410b7 Tue Nov 28 05:13:40 2023 +0000

go env

set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\26454\AppData\Local\go-build
set GOENV=C:\Users\26454\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\file\gofile\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\file\gofile
set GOPRIVATE=
set GOPROXY=https://goproxy.cn,direct
set GOROOT=C:\Users\26454\.go\current
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=local
set GOTOOLDIR=C:\Users\26454\.go\current\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.22-b2dbfbfc Fri Jan 12 00:56:20 2024 +0000
set GCCGO=gccgo
set GOAMD64=v3
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=D:\tmp\go-build2134721204=/tmp/go-build -gno-record-gcc-switches

What did you do?

In vscode , open .go file , it have FuzzTest.

What did you see happen?

I see have "run test" on FuzzTest , it can run go test -run ^FuzzTest$

What did you expect to see?

I see have "run fuzz" on FuzzTest , it can run go test -fuzz ^FuzzTest$
I hope such feature requests will enable gopls to better support fuzz test.

Editor and settings

"go.editorContextMenuCommands": {
    "toggleTestFile": false,
    "addTags": false,
    "testFile": true,
    "testPackage": true,
    "generateTestForFile": true,
    "generateTestForPackage": true,
    "playground": false,
    "debugTestAtCursor": false
},

Logs

No response
@qiulaidongfeng qiulaidongfeng added the gopls gopls related issues label Jan 12, 2024
@mauri870
Copy link
Member

cc @golang/tools-team

@hyangah
Copy link
Contributor

hyangah commented Jan 12, 2024

Fuzz tests are supposed to keep running for some time until a crasher is found. How long do you want to run the tests, when, where, how do you you want results to be present to you?

@qiulaidongfeng
Copy link
Author

qiulaidongfeng commented Jan 12, 2024 via email

@qiulaidongfeng
Copy link
Author

Where the result is output.
Any possible way is acceptable within vscode.

@qiulaidongfeng
Copy link
Author

Background information that may be useful.
I created this issue because the first time I clicked "run test" on FuzzTest did go test -run ^FuzzTest$ instead of go test -fuzz ^FuzzTest$
When I need to go test -fuzz ^FuzzTest$ (usually when I finish writing new code to test), I can use the "run package tests" provided by gopls every time go test, and fuzzy test can only start in cmd by itself. This led me to note that it would be easier to use fuzz test if gopls provided a feature to run a specific fuzz test.

@findleyr
Copy link
Contributor

the first time I clicked "run test" on FuzzTest did go test -run ^FuzzTest$ instead of go test -fuzz ^FuzzTest$

Oops, that's just a bug! Thanks for reporting. @hyangah this would be a VS Code bug, right? Shall we just roll this into our grand unification of tests?

I personally think it's easier to run fuzz tests from the terminal -- better for browsing output, having terminal history, and cancelling with ctrl-c. But I tend to prefer the terminal more than others. I'm curious to hear if there is support for the idea of running from VS Code.

@hyangah
Copy link
Contributor

hyangah commented Jan 12, 2024

It's not a bug but intentional behavior for "run test" codelens.
FuzzTest is also a normal test. When it runs with -run, it runs against the corpus. So the intended workflow is:

  • go test -fuzz=FuzzTest and collect a crasher.
  • fix bug.
  • run go test -run=FuzzTest to verify the fix
  • check in crasher dataset, so the FuzzTest can keep serving as a regression detector.

@qiulaidongfeng
Copy link
Author

I'm curious to hear if there is support for the idea of running from VS Code.

There is a scenario where run fuzz is the last step before git commit, used to check if modifications introduce new bugs, which is more convenient to run in vscode.
I encountered this scenario not long ago, and all I need to know is whether run fuzz has not failed. At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

@hyangah
Copy link
Contributor

hyangah commented Jan 17, 2024

At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

Just provide the full function name to -fuzz parameter, not prefix.

The test functionality in vscode go is implemented in the vscode side, not in gopls.
LSP doesn't have methods for test execution. Should we move this discussion to vscode?

If it is about vscode, an alternative approach for fuzzing is to utilize the vscode tasks.

@qiulaidongfeng
Copy link
Author

At the same time, I also encountered two fuzz tests sharing the same prefix, and I don't think it's easier to run one fuzz on the terminal than pressing a "run fuzz" in VScode.

Just provide the full function name to -fuzz parameter, not prefix.

What I'm trying to say is.
If the fuzz test involves typing dozens of letters on the command line, it's easier to click "run fuzz".

The test functionality in vscode go is implemented in the vscode side, not in gopls.
LSP doesn't have methods for test execution. Should we move this discussion to vscode?

I researched this issue in vscode before opening it
"go.useLanguageServer": false
The "run test" will be gone
"go.useLanguageServer": true
"run test" will be available

@hyangah
Copy link
Contributor

hyangah commented Jan 19, 2024 •
edited

I researched this issue in vscode before opening it
"go.useLanguageServer": false
The "run test" will be gone
"go.useLanguageServer": true
"run test" will be available

Vscode go extension uses gopls to extract test, benchmark, fuzz function names, that are needed to implement the code lens provider. Thus, we disabled the test code lens if the language server is disabled. Other than that the implementation is done by the extension, not gopls.

The code is in https://github.com/golang/vscode-go/blob/c5607ecbf6305f208ae2786834c16240662dc53c/extension/src/goRunTestCodelens.ts,

vscode-go/extension/src/testUtils.ts

Line 51 in c5607ec

const fuzzFuncRegx = /^Fuzz$|^Fuzz\P{Ll}.*/u;
, etc.

Gopls can help detect fuzz test, but proxying to run fuzz testing seems to be beyond what can be easily offered by LSP spec. We currently have no plan for it.

I think, it's possible that we can consider vscode-go specific UX.

@findleyr
Copy link
Contributor

Moving this to the VS Code issue tracker. I don't think this type of workflow is appropriate to run from the language server -- better to invoke the test client side, if desired.

@findleyr findleyr transferred this issue from golang/go Jan 30, 2024
@findleyr findleyr removed the gopls gopls related issues label Jan 30, 2024
@findleyr findleyr changed the title x/tools/gopls: support run fuzz test UI for running fuzz tests Jan 30, 2024
@findleyr findleyr added the go-test issues related to go test support (test output, test explorer, ...) label Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
FeatureRequest go-test issues related to go test support (test output, test explorer, ...)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hyangah @mauri870 @findleyr @qiulaidongfeng