Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document exit codes of a process executing go test #25989

Open
kostix opened this issue Jun 21, 2018 · 7 comments
Open

cmd/go: document exit codes of a process executing go test #25989

kostix opened this issue Jun 21, 2018 · 7 comments
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@kostix
Copy link

kostix commented Jun 21, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.3

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/devel/go"
GORACE=""
GOROOT="/home/user/devel/golang1.10"
GOTMPDIR=""
GOTOOLDIR="/home/user/devel/golang1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build141686204=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I run go help test and try to search for keywords "return", "exit", "code" in the output.

What did you expect to see?

I expect the possible exit codes for running go test be documented
in order to make sensible decisions about running this command in scripts
(and from Makefiles, by a CI system etc).

What did you see instead?

No menton of the return/exit code.


I'd expect the following cases to be documented if implemented:

  • Successful run—all tests compiled and passed OK—exit code 0.
  • Failed run—the compilation of at least one package failed or at least a single test failed—a non-zero exit code.
  • Invalid parameters passed and/or one of the listed packages wasn't found—a non-zero exit code,
    different from that of the former case.

I'd also think that the cached results should effectively "memorize" the exit codeused ATM the results were generated (and cached).

See also #23716.

@GTB3NW
Copy link

GTB3NW commented Jun 21, 2018

Ignore me I'm an idiot. t.Error != t.Fail

I agree with the OP however, it should have a none 0 status, 1 for error, 2 for fail maybe?

@ianlancetaylor
Copy link
Contributor

Right now the exit status is 1 for most failures, 2 if the build was interrupted.

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 21, 2018
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jun 21, 2018
@smasher164
Copy link
Member

Do we distinguish between flag parsing errors where different exit codes are output? For example, passing in "-c=invalid" returns a 2

$ go test -c=invalid
go test: illegal bool flag value invalid
run "go help test" or "go help testflag" for more information
$ echo $?
2

while passing in "-covermode=invalid" returns a 1.

$ go test -covermode=invalid
invalid flag argument for -covermode: "invalid"
$ echo $?
1

@kostix
Copy link
Author

kostix commented Jun 22, 2018

@GTB3NW note that your particular problem is most possibly not related to the essense of this issue which is about documenting the implemented behaviour which appears to be OK.
Here at my $dayjob, in our CI pipeline, we call go test multiple times with different command-line options (say, including -race for nightly tests and also -bench … etc), and go test has always exited with a non-zero exit code on error since, I reckon, at least v1.6, and I really have no reason to think it was otherwise before.

I've created this issue merely because such things ought to be documented for reference purposes (say, for DevOps folks which may be not too familiar with Go to implement the CI pipeline).

Try to figure out what may mask the exit code of your go test process.
Note that all /bin/sh-style shells found in commodity OSes by default run in a mode which ignores any errors, that means, if you have a script like

go test ...
echo Whatever

this code won't ever exit if go test fails.
(And these shells did this since forever; and this is even mandated by POSIX.)
You either need to re-write your scripts so that they run with set -e -u (almost always a good thing to do) or explicitly test whether a command of interest failed, like in

go test ...
rc=$?
if [ $rc -ne 0 ]; then
  echo "testing failed" >&2
  exit $rc
fi

Updated @GTB3NW, ah, I see you got sorted it out :-)
Sorry for the noise then—should have re-read the thread rather than relying on comment e-mailing (which ignores edits).

@GTB3NW
Copy link

GTB3NW commented Jun 22, 2018

Okay here's one for you then, I've read the documentation, perhaps I'm missing something here, but the following should FAIL and return error code 1 should it not?

blahtest/blah_test.go:

package blahtest

import "testing"

func TestBlah(t *testing.T) {
	t.Fatal("Blah")
}
go test <redacted>/blahtest -v

=== RUN   TestBlah
--- FAIL: TestBlah (0.00s)
        blah_test.go:6: Blah
FAIL
ok      <redacted>/blahtest      0.004s

Should that not be a FAIL on /blahtest too?

The aim is to just run:

go test -v -short $(go list ${PKG}./... | grep -v /vendor/)

Test all the packages and if any one of them fails the whole test fails, surely that's a common requirement. Either I'm misunderstanding the testing facilities/there's a bug/go test is lacking something critical.

Could anyone set me in the right direction please I'd really appreciate it :)

@bcmills
Copy link
Contributor

bcmills commented May 8, 2019

@GTB3NW, if you're seeing a 0 exit code or ok summary for a failed test, please file a separate issue with steps to reproduce.

@bcmills
Copy link
Contributor

bcmills commented May 8, 2019

Before we document the error codes, I think we will probably need to do some work to make them consistent. That's a pretty big investigation, and probably ought to happen as part of a broader go test cleanup (which I hope to do for Go 1.14 or 1.15, but don't have a specific timeline).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

6 participants