Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/fuzz: test_fuzz_minimize failures missing "got the minimum size" beginning 2021-11-17 #49685

Closed
bcmills opened this issue Nov 19, 2021 · 6 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 19, 2021

    --- FAIL: TestScript/test_fuzz_minimize (46.15s)
        script_test.go:257: 
            # We clean the fuzz cache during this test. Don't clean the user's cache. (0.000s)
            # Test that fuzzminimizetime cannot be negative seconds (18.682s)
            # Test that fuzzminimizetime cannot be negative times (2.829s)
            # Test that fuzzminimizetime can be zero seconds, and minimization is disabled (4.268s)
            # Test that fuzzminimizetime can be zero times, and minimization is disabled (7.046s)
            # Test that minimization is working for recoverable errors. (1.949s)
            # The error message that was printed should be for the one written to testdata. (0.000s)
            # Check that the bytes written to testdata are of length 50 (the minimum size) (1.023s)
            # Test that re-running the minimized value causes a crash. (4.685s)
            # Test that minimization is working for recoverable errors. Run it with -v this
            # time to ensure the command line output still looks right. (1.639s)
            > ! go test -v -fuzz=FuzzMinimizerRecoverable -run=FuzzMinimizerRecoverable -fuzztime=10000x .
            [stdout]
            === FUZZ  FuzzMinimizerRecoverable
            fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
            fuzz: elapsed: 0s, gathering baseline coverage: 1/1 completed, now fuzzing with 16 workers
            fuzz: minimizing 427-byte failing input file
            fuzz: elapsed: 0s, minimizing
            --- FAIL: FuzzMinimizerRecoverable (0.24s)
                --- FAIL: FuzzMinimizerRecoverable (0.00s)
                    y_test.go:38: contains a non-zero byte of length 97
                
                Failing input written to testdata\fuzz\FuzzMinimizerRecoverable\064b38366638e28ac6d4036031db2afcda0528bfb8393486ed0358da0f930fe6
                To re-run:
                go test -run=FuzzMinimizerRecoverable/064b38366638e28ac6d4036031db2afcda0528bfb8393486ed0358da0f930fe6
            FAIL
            exit status 1
            FAIL	example.com/y	0.304s
            [exit status 1]
            > ! stdout '^ok'
            > stdout 'got the minimum size!'
            FAIL: testdata\script\test_fuzz_minimize.txt:54: no match for `(?m)got the minimum size!` found in stdout

greplogs --dashboard -md -l -e '(?ms)FAIL: TestScript/test_fuzz_minimize.*no match for ...m.got the minimum size' --since=2021-06-01

2021-11-19T00:05:59-e8cda0a/linux-386-longtest
2021-11-18T22:40:42-2375b6e/linux-386-longtest
2021-11-18T20:11:45-24898d6/windows-amd64-longtest
2021-11-18T14:00:00-14c3f74/windows-amd64-longtest
2021-11-18T02:15:36-2463b4f/windows-amd64-longtest
2021-11-18T02:10:22-e3b48af/linux-386-longtest
2021-11-17T21:26:25-0440fb8/linux-386-longtest
2021-11-17T21:26:25-0440fb8/windows-amd64-longtest

(CC @rolandshoemaker @katiehockman)

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 fuzz Issues related to native fuzzing support labels Nov 19, 2021
@bcmills bcmills added this to the Go1.18 milestone Nov 19, 2021
@gopherbot
Copy link

Change https://golang.org/cl/365315 mentions this issue: cmd/go: temporarily skip TestScript/test_fuzz_minimize

gopherbot pushed a commit that referenced this issue Nov 19, 2021
This test is failing on the longtest builders.
Adding a skip temporarily until it can be diagnosed and fixed.

For #49685

Change-Id: I0ceaf009f5029d1ad6f667f7cfee1f1605737bf3
Reviewed-on: https://go-review.googlesource.com/c/go/+/365315
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
@rolandshoemaker
Copy link
Member

After submitting CL365175 and CL365294 (the two fuzzing changes from yesterday) I am no longer able to reproduce this (at least on the linux-386-longtest builder).

@bcmills
Copy link
Contributor Author

bcmills commented Nov 19, 2021

I merged CL 365315 (skipping the test) right around then. Are you testing with that rolled back?

@bcmills
Copy link
Contributor Author

bcmills commented Nov 19, 2021

We could always try just rolling back the skip and seeing if the builders remain clear. The failure rate was high enough previously that it should show up within a half-dozen CLs or so if it's still broken.

@bcmills
Copy link
Contributor Author

bcmills commented Nov 22, 2021

Rolled back the skip. Watching the builders to see whether this appears to be fixed.

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 22, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Nov 22, 2021

A fair number of longtest runs so far and no flakes from this test. Barring further evidence to the contrary, I'm calling this fixed.

@bcmills bcmills closed this as completed Nov 22, 2021
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 22, 2021
@golang golang locked and limited conversation to collaborators Nov 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker
Projects
Status: No status
Development

No branches or pull requests

3 participants