Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: Fuzz corpus header should support CRLF line endings #52268

Closed
moorereason opened this issue Apr 10, 2022 · 6 comments
Closed

testing: Fuzz corpus header should support CRLF line endings #52268

moorereason opened this issue Apr 10, 2022 · 6 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.19

Comments

@moorereason
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.18 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\XXX\AppData\Local\go-build
set GOENV=C:\Users\XXX\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\XXX\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\XXX\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\XXX\src\github.com\pelletier\go-toml\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\XXX\AppData\Local\Temp\go-build3190186843=/tmp/go-build -gno-record-gcc-switches

What did you do?

Checkout a Go project with a fuzzing corpus and begin fuzzing:

git clone https://github.com/pelletier/go-toml.git
cd ./go-toml
checkout fixes
go test -fuzz FuzzUnmarshal

What did you expect to see?

I expected the fuzzer to startup successfully and begin fuzzing.

What did you see instead?

--- FAIL: FuzzUnmarshal (0.04s)
    fuzz_test.go:22: "testdata\\fuzz\\FuzzUnmarshal\\0c08b924aa26af23d16b77029b4f5567a96af93402e0472fef7b837b37580d7c": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\249a017d8ffa0d88d41c594ada5399f764833f64050180cb39f106d12666853f": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\391ce67b866d58464d067c5981e99951fd499896453a2fe2779a6bfe7df11bf5": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\49ae83283c5cfbed874bcacbfb342de107b69c2a3d3c35c2d99f67345fe89946": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\69ea82e85bb4ad966ae86bb05eea3af69ed1a826c846f1bf2282f329b4d8fb36": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\746910fac6deb42e7e62dce60b333c81c39d844a122c9075894c023400e8fdbf": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\844d1171a3adc95e51627f0e9378bbaa193fe2d0284f4ba8ed19a5226a8cbc74": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\900c54b5071113ae2a5365b3156377ddd775b9d38add61629544903cb9c94b00": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\c87e8eb4e8fa40fd3c89d8ed16a049ce81592ded710965ee29a4e506eb5427b3": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d0e5d6fc3b865cfae5480c01f301b87c932722ae0c1d692deea413349ea028be": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d2cecea5b8ee5f148419671cef0644178cdba1ee91d3f295be38da576e89ef40": unmarshal: unknown encoding version: go test fuzz v1
FAIL
exit status 1
FAIL    github.com/pelletier/go-toml/v2 0.455s

Root Cause

The issue is that the corpus file format must use LF line endings. Git was "helpfully" converting everything to CRLF by default on Windows. In order to use the fuzzer, the Git config must contain core.autocrlf=false and core.eol=lf.
@moorereason moorereason mentioned this issue Apr 10, 2022
Closed
@seankhliao
Copy link
Member

I think this is the wrong place to fix things, testdata/ can contain arbitrary data and letting git mess with them can cause all sorts of issues (eg CRLFs in golden files). There are per repo settings available via .gitattributes, which can and should be set.

@pelletier pelletier mentioned this issue Apr 11, 2022
Merged
@moorereason
Copy link
Author

Also, if CRLF will not be supported, the corpus file format documentation should make that explicit.

@dmitshur
Copy link
Contributor

CC @golang/fuzzing.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 11, 2022
@dmitshur dmitshur added this to the Backlog milestone Apr 11, 2022
@dominikh
Copy link
Member

My two cents: Yes, letting git mess with testdata can be bad, but for the large majority of users, it isn't, and fuzzing is the first instance of it being bad. Seeing that the format could trivially support CRLF, and seeing how this issue will affect every multi-OS user of fuzzing, I think this should be fixed in Go. It's an easy fix that helps a lot of users and makes fuzz testdata work automatically, without having to learn about git mechanisms.

It's also in line with the rest of the Go toolchain, which supports CRLF line endings well.

@gopherbot
Copy link

Change https://go.dev/cl/402074 mentions this issue: internal/fuzz: trim carriage return from version line

@katcipis
Copy link

+1

I missed this issue because I searched for issues with the "fuzz" tag, but I also ran into problems when testing things on multiple OS's on GHA. More details here.

@dmitshur dmitshur added the fuzz Issues related to native fuzzing support label May 20, 2022
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 21, 2022
@dmitshur dmitshur modified the milestones: Backlog, Go1.19 May 21, 2022
@greatroar greatroar mentioned this issue Jun 6, 2022
Merged
8 tasks
@tdakkota tdakkota mentioned this issue Jul 15, 2022
Merged
@ahmed-mez ahmed-mez mentioned this issue Aug 30, 2022
Merged
10 tasks
@golang golang locked and limited conversation to collaborators May 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Security
Status: No status
Milestone
Go1.19
Development

No branches or pull requests
6 participants
@dominikh @moorereason @katcipis @dmitshur @gopherbot @seankhliao