Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/fuzz: test_fuzz_minimize_interesting failures with non-minimal lengths #52285

Closed
bcmills opened this issue Apr 11, 2022 · 2 comments
Closed
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Apr 11, 2022

Seen in a linux-amd64-longtest trybot on CL 399157:

https://storage.googleapis.com/go-build-log/9220dea4/linux-amd64-longtest_767a7d40.log

--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_fuzz_minimize_interesting (34.97s)
        script_test.go:260: 
            # Test that when an interesting value is discovered (one that expands coverage),
            # the fuzzing engine minimizes it before writing it to the cache.
            #
            # The program below starts with a seed value of length 100, but more coverage
            # will be found for any value other than the seed. We should end with a value
            # in the cache of length 1 (the minimizer currently does not produce empty
            # strings). check_cache.go confirms that.
            #
            # We would like to verify that ALL values in the cache were minimized to a
            # length of 1, but this isn't always possible when new coverage is found in
            # functions called by testing or internal/fuzz in the background. (30.828s)
            # Test that minimization occurs for a crash that appears while minimizing a
            # newly found interesting input. There must be only one worker for this test to
            # be flaky like we want. (0.412s)
            # Check that the input written to testdata will reproduce the error, and is the
            # smallest possible. (1.496s)
            # Test that a nonrecoverable error that occurs while minimizing an interesting
            # input is reported correctly. (0.367s)
            # Check that the input written to testdata will reproduce the error. (1.831s)
            > go run check_testdata/check_testdata.go FuzzMinimizerNonrecoverableCrashInMinimization 100
            [stderr]
            expect length <= 100, got 104
            exit status 1
            [exit status 1]
            FAIL: testdata/script/test_fuzz_minimize_interesting.txt:44: unexpected command failure
            
FAIL
FAIL	cmd/go	181.968s

It appears that either something is wrong with the fuzz minimization algorithm, or test_fuzz_minimize_interesting needs to fuzz for an even longer time to successfully minimize. 😕

(attn @golang/fuzzing)

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Apr 11, 2022
@bcmills bcmills added this to the Go1.19 milestone Apr 11, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Apr 11, 2022

I've also seen this in at least one local run of go test cmd/go recently.

This is a release-blocker for Go 1.19 because it causes occasional TryBot failures on a first-class port.
(But if @rolandshoemaker or other relevant owners decide that it's not a priority, we can unblock the release by laxening or skipping the test.)

@gopherbot
Copy link

Change https://go.dev/cl/399814 mentions this issue: cmd/go: fix TetsScript/test_fuzz_minimize_interesting flake

@golang golang locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Development

No branches or pull requests

2 participants