Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: fuzzing failed to run if test run with open TCP connections #71560

Closed
shuLhan opened this issue Feb 4, 2025 · 5 comments
Closed

testing: fuzzing failed to run if test run with open TCP connections #71560

shuLhan opened this issue Feb 4, 2025 · 5 comments
Labels
BugReport Issues describing a possible bug in the Go implementation.

Comments

@shuLhan
Copy link
Contributor

shuLhan commented Feb 4, 2025

Go version

go version devel go1.25-cc874072f3 Mon Feb 3 08:25:31 2025 -0800 linux/amd64

Output of go env in your module/workspace:

AR='ar'                                                                                                                 
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN='/home/ms/go/bin'
GOCACHE='/home/ms/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/ms/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2786547649=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/ms/src/go/src/go.mod'
GOMODCACHE='/home/ms/go/pkg/mod'
GONOPROXY='<REDACTED>'
GONOSUMDB='<REDACTED>'
GOOS='linux'
GOPATH='/home/ms/go'
GOPRIVATE='<REDACTED>'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/ms/local/share/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/ms/.config/go/telemetry'
GOTMPDIR=''

What did you do?

Given the following code and fuzzing test based on Fuzzing tutorial [1]

package main

import (
	"fmt"
	"html"
	"log"
	"net"
	"net/http"
	"testing"
	"unicode/utf8"
)

func FuzzReverse(f *testing.F) {
	// Uncomment either runHTTPServer or runTCPServer to make test fuzz
	// fail.
	go runHTTPServer()
	//go runTCPServer()
	testcases := []string{"Hello, world", " ", "!12345"}
	for _, tc := range testcases {
		f.Add(tc) // Use f.Add to provide a seed corpus
	}
	f.Fuzz(func(t *testing.T, orig string) {
		rev, err1 := Reverse(orig)
		if err1 != nil {
			return
		}
		doubleRev, err2 := Reverse(rev)
		if err2 != nil {
			return
		}
		if orig != doubleRev {
			t.Errorf("Before: %q, after: %q", orig, doubleRev)
		}
		if utf8.ValidString(orig) && !utf8.ValidString(rev) {
			t.Errorf("Reverse produced invalid UTF-8 string %q", rev)
		}
	})
}

func runHTTPServer() {
	http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
	})

	log.Fatal(http.ListenAndServe(":18080", nil))
}

func runTCPServer() {
	ln, err := net.Listen("tcp", ":18081")
	if err != nil {
		log.Fatal(`runTCPServer:`, err)
	}
	for {
		_, err := ln.Accept()
		if err != nil {
			log.Fatal(`runTCPServer: Accept`, err)
		}
	}
}

I believe the same issues has been reported several times,

There are other issues but I am not sure if its related,

The only error that I can trace is there is a broken pipe during ping [2].

[1] https://go.dev/doc/tutorial/fuzz
[2]

if err := w.client.ping(ctx); err != nil {

What did you see happen?

The fuzz process terminated with an error EOF,

$ go test -fuzz=. -run=none -fuzztime=10s -v
=== RUN   FuzzReverse
fuzz: elapsed: 0s, gathering baseline coverage: 0/46 completed
fuzz: elapsed: 0s, gathering baseline coverage: 0/46 completed
--- FAIL: FuzzReverse (0.02s)
    fuzzing process terminated without fuzzing: EOF
=== NAME  
FAIL
exit status 1
FAIL    git.sr.ht/~shulhan/sandbox.go/fuzz      0.021s

What did you expect to see?

Uncomment both of the lines in FuzzReverse,

	//go runHTTPServer()
	//go runTCPServer()

and run the test again,

$ go test -fuzz=. -run=none -fuzztime=10s -v                                                                      
=== RUN   FuzzReverse                                                                                                   
fuzz: elapsed: 0s, gathering baseline coverage: 0/46 completed
fuzz: elapsed: 0s, gathering baseline coverage: 46/46 completed, now fuzzing with 4 workers
fuzz: elapsed: 3s, execs: 130236 (43416/sec), new interesting: 0 (total: 46)
fuzz: elapsed: 6s, execs: 255613 (41785/sec), new interesting: 0 (total: 46)
fuzz: elapsed: 9s, execs: 387455 (43941/sec), new interesting: 0 (total: 46)
fuzz: elapsed: 10s, execs: 431389 (40198/sec), new interesting: 0 (total: 46)
--- PASS: FuzzReverse (10.10s) 
=== NAME  
PASS
ok      git.sr.ht/~shulhan/sandbox.go/fuzz      10.107s

@gabyhelp gabyhelp added the BugReport Issues describing a possible bug in the Go implementation. label Feb 4, 2025
@seankhliao
Copy link
Member

fuzzing uses multiple subprocesses (since the processes can crash). that is incompatible with using a fixed port.

https://go.dev/doc/security/fuzz/

closing as working as intended.

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Feb 4, 2025
@shuLhan
Copy link
Contributor Author

shuLhan commented Feb 4, 2025 via email

@ianlancetaylor
Copy link
Member

The error means that the program being fuzzed crashed unexpectedly.

In your case it most likely crashed because you are using fixed ports, and crashing if opening the port fails.

@shuLhan
Copy link
Contributor Author

shuLhan commented Feb 22, 2025

Thank you @seankhliao and @ianlancetaylor for the answers.

I keep thinking about this issue and I think there are some underlying issues with current fuzzing.
I take notes here so someone that new to Go fuzzing, like me, can read it when stumbling with the same problem in the future. Please correct me if I am wrong here.

  1. The behaviour of running go test with fuzzing is different with the rest of test (TestXxx function and BenchXxx function). Unlike test function, go test with fuzzing run the test binary multiple times. This behaviour as far as I know is undocumented.

  2. Any error from the test binary does not printed to the (?) stderr. This caused further confusion when inspecting the fuzzing output. For example, in my case above, it should at least print an error, something like, "listen tcp :18080: bind: address already in use" instead of only "fuzzing process terminated without fuzzing: EOF"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BugReport Issues describing a possible bug in the Go implementation.
Projects
None yet
Development

No branches or pull requests

4 participants