Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/website: write a guide for fuzzing #48255

Closed
jayconrod opened this issue Sep 8, 2021 · 16 comments
Closed

x/website: write a guide for fuzzing #48255

jayconrod opened this issue Sep 8, 2021 · 16 comments
Labels
Documentation FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.18

Comments

@jayconrod
Copy link
Contributor

jayconrod commented Sep 8, 2021 •
edited by katiehockman

We expect to have many users that are encountering fuzzing for the first time. We should write a thorough guide on how to use fuzzing effectively with that audience in mind.

An introductory guide should cover, at minimum:

  • What fuzzing is.
  • What fuzzing is good for (and not good for).
  • How to write a fuzz target.
  • How to run fuzzing with go test -fuzz=target, with information about relevant flags.
  • What to do with a crasher if one is found
  • Strategies for helping the fuzzer if it isn't finding any crashers after running for a while
  • Current limitations of the native fuzzing support (e.g. it stops running after the first crash is found, it isn't currently supported to easily integrate with CI)

Some advanced topics that could be covered in other guides:

  • Choosing a good seed corpus.
  • Measuring coverage.
  • Differential fuzzing.
  • Fuzzing continuously.
@jayconrod jayconrod added Documentation release-blocker fuzz Issues related to native fuzzing support labels Sep 8, 2021
@jayconrod jayconrod added this to the Go1.18 milestone Sep 8, 2021
@cagedmantis cagedmantis added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 13, 2021
@jayconrod
Copy link
Contributor Author

We should reconsider what documentation should be part of the go command and where that should be located. Currently, I'm thinking:

  • The testing package documentation should have a section on writing a fuzz target. It should explain the signature for the fuzz target (must take a *testing.F), what it must do (call F.Add then F.Fuzz), and what testing will do with it when fuzzing is enabled and disabled. This is meant to be reference documentation.
  • go help test and go help testflag should explain all flags, including those related to fuzzing. They should be kept brief, pointing to other sources for more explanation.
  • One or more guides on the website should tie this all together. Most users should read those first.
  • We should delete go help fuzz. It will be redundant with the guides.

@rsc rsc changed the title [dev.fuzz] x/website: write a guide for fuzzing x/website: write a guide for fuzzing Sep 21, 2021
@toothrot
Copy link
Contributor

Checking in on this issue as it's labeled a release blocker for Go 1.18. Is there any update?

@jayconrod jayconrod added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Sep 22, 2021
@jayconrod
Copy link
Contributor Author

Not yet. There are a lot of fuzzing issues. We'll get to them as soon as we can.

This one can be done during the freeze.

@gopherbot
Copy link

Change https://golang.org/cl/363754 mentions this issue: _content/ref: create fuzz.md with glossary

gopherbot pushed a commit to golang/website that referenced this issue Nov 30, 2021
@katiehockman
_content/doc/fuzz: create index.md with glossary
b0a4f90 
Link to /doc/fuzz from /doc#learning.

Updates golang/go#48255

Change-Id: Ib93373f8e1fb24144dbcded5d5656be575f6bc2e
Reviewed-on: https://go-review.googlesource.com/c/website/+/363754
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Website-Publish: Dmitri Shuralyov <dmitshur@google.com>
@toothrot
Copy link
Contributor

toothrot commented Dec 8, 2021

@katiehockman Checking in on this as a release-blocking issue. Is there more left to do here?

@katiehockman
Copy link
Contributor

There is more to do on it, and it's in progress. I can remove the release-blocker label though, as we'll keep working on it and get as much done before the release as we can.

@katiehockman katiehockman removed release-blocker okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 labels Dec 8, 2021
@gopherbot
Copy link

Change https://golang.org/cl/370676 mentions this issue: _content/doc/fuzz: add example and restructure page

@gopherbot
Copy link

Change https://golang.org/cl/370635 mentions this issue: _content/doc/fuzz: add rules/suggestions

gopherbot pushed a commit to golang/website that referenced this issue Dec 13, 2021
@katiehockman
_content/doc/fuzz: add example and restructure page
60484b3 
For golang/go#48255

Change-Id: Icd94e0455a3f4c3bfdc8800e89a956d4258b847e
Reviewed-on: https://go-review.googlesource.com/c/website/+/370676
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit to golang/website that referenced this issue Dec 13, 2021
@katiehockman
_content/doc/fuzz: add rules/suggestions
5924dab 
For golang/go#48255

Change-Id: I310ec039345f2c4e3c9592ef92ae748b2b32bd3b
Reviewed-on: https://go-review.googlesource.com/c/website/+/370635
Trust: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <bracewell@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/371394 mentions this issue: _content/doc/fuzz: fix example

gopherbot pushed a commit to golang/website that referenced this issue Dec 14, 2021
@katiehockman
_content/doc/fuzz: fix example
301dcae 
Also fix the formatting of the doc.

For golang/go#48255
Fixes golang/go#50142

Change-Id: I24fdd5ae623fc7653c5034078d7cc080f2011aa9
Reviewed-on: https://go-review.googlesource.com/c/website/+/371394
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/372314 mentions this issue: _content/doc/fuzz: document custom settings

@gopherbot
Copy link

Change https://golang.org/cl/372434 mentions this issue: _content/doc/fuzz: document corpus file format

gopherbot pushed a commit to golang/website that referenced this issue Dec 20, 2021
@katiehockman
_content/doc/fuzz: document custom settings
c471c5c 
For golang/go#48255

Change-Id: I41f6fa5ad6eee14c265bb537fda7e4ca3ef62404
Reviewed-on: https://go-review.googlesource.com/c/website/+/372314
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/website that referenced this issue Jan 5, 2022
@katiehockman
_content/doc/fuzz: document corpus file format
7a18f71 
For golang/go#48255
Fixes golang/go#48717

Change-Id: Ie87c09b9e031a2221ca77d33c4e8140d787a33d2
Reviewed-on: https://go-review.googlesource.com/c/website/+/372434
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Trust: DO NOT USE <katiehockman@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/376434 mentions this issue: _content/doc/fuzz: document command line output

gopherbot pushed a commit to golang/website that referenced this issue Jan 12, 2022
@katiehockman
_content/doc/fuzz: document command line output
06562a2 
For golang/go#48255

Change-Id: I15476144618cce3b0405a4b587aec30452b93bf9
Reviewed-on: https://go-review.googlesource.com/c/website/+/376434
Reviewed-by: Roland Shoemaker <bracewell@google.com>
Trust: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@ianlancetaylor
Copy link
Contributor

@katiehockman Is there more to do here for 1.18? Thanks.

@katiehockman
Copy link
Contributor

Nope I think we're in a pretty good spot at this point, and new things can be added at any time, so it isn't really tied to a release anymore. I'll go ahead and just close this.

@ronaldpetty
Copy link

ronaldpetty commented Feb 4, 2022 •
edited

Hi @katiehockman , I see this is closed, but not sure this warrants opening a new issue.

The go doc testing and this page https://pkg.go.dev/testing@master#hdr-Fuzzing still are pointing to this issue. I see its @master, but unclear if that is 1.18 or not.

TODO(#48255): write and link to documentation that will be helpful to users
who are unfamiliar with fuzzing.

I am using "go version go1.18beta2 darwin/amd64"

I actually found this ticket because I was hoping to find if there is some 'eta' when fuzzing stops running. I am running the FuzzHex example and still running after 22 minutes on 12 cores, 64 GB. I assume we want to test everything, but unclear am I suppose to stop it?

update: I did stop it, and based on other issues (like rust fuzz), I have come full circle (to explain or link to more operational details).

fuzz: elapsed: 36m0s, execs: 535674465 (257494/sec), new interesting: 19 (total: 19)
^Cfuzz: elapsed: 36m0s, execs: 535675693 (10268/sec), new interesting: 19 (total: 19)
--- PASS: FuzzHex (2160.12s)
PASS
ok  	command-line-arguments	2160.332s

@katiehockman
Copy link
Contributor

Feel free to file a separate issue with those documentation suggestions, just in case I can't get to it soon. Thanks for pointing that out.

@ronaldpetty ronaldpetty mentioned this issue Feb 6, 2022
Open
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: create index.md with glossary
b456845 
Link to /doc/fuzz from /doc#learning.

Updates golang/go#48255

Change-Id: Ib93373f8e1fb24144dbcded5d5656be575f6bc2e
Reviewed-on: https://go-review.googlesource.com/c/website/+/363754
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Website-Publish: Dmitri Shuralyov <dmitshur@google.com>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: add example and restructure page
ce1eba0 
For golang/go#48255

Change-Id: Icd94e0455a3f4c3bfdc8800e89a956d4258b847e
Reviewed-on: https://go-review.googlesource.com/c/website/+/370676
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: add rules/suggestions
a111897 
For golang/go#48255

Change-Id: I310ec039345f2c4e3c9592ef92ae748b2b32bd3b
Reviewed-on: https://go-review.googlesource.com/c/website/+/370635
Trust: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <bracewell@google.com>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: fix example
2b4fa1a 
Also fix the formatting of the doc.

For golang/go#48255
Fixes golang/go#50142

Change-Id: I24fdd5ae623fc7653c5034078d7cc080f2011aa9
Reviewed-on: https://go-review.googlesource.com/c/website/+/371394
Reviewed-by: Julie Qiu <julieqiu@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: document custom settings
f23b96d 
For golang/go#48255

Change-Id: I41f6fa5ad6eee14c265bb537fda7e4ca3ef62404
Reviewed-on: https://go-review.googlesource.com/c/website/+/372314
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: document corpus file format
b3a0fe8 
For golang/go#48255
Fixes golang/go#48717

Change-Id: Ie87c09b9e031a2221ca77d33c4e8140d787a33d2
Reviewed-on: https://go-review.googlesource.com/c/website/+/372434
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julieqiu@google.com>
Trust: DO NOT USE <katiehockman@google.com>
passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
@passionSeven
_content/doc/fuzz: document command line output
c65cf69 
For golang/go#48255

Change-Id: I15476144618cce3b0405a4b587aec30452b93bf9
Reviewed-on: https://go-review.googlesource.com/c/website/+/376434
Reviewed-by: Roland Shoemaker <bracewell@google.com>
Trust: Bryan Mills <bcmills@google.com>
Trust: DO NOT USE <katiehockman@google.com>
Run-TryBot: DO NOT USE <katiehockman@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@golang golang locked and limited conversation to collaborators Feb 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Documentation FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Security
Status: No status
Milestone
Go1.18
Development

No branches or pull requests
7 participants
@toothrot @cagedmantis @jayconrod @ronaldpetty @ianlancetaylor @gopherbot @katiehockman