Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: run standard library tests without -short #17472

Closed
robpike opened this issue Oct 16, 2016 · 9 comments
Closed

x/build: run standard library tests without -short #17472

robpike opened this issue Oct 16, 2016 · 9 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@robpike
Copy link
Contributor

robpike commented Oct 16, 2016

I saw some bugs with go test in cmd/go, so I decided to run everything without -short. Need to run go test without -short once in a while on the whole tree. I tried it today and there were some disappointments.

I'll let someone else decide if they want to break out individual bugs for the failures, but the point made in the first paragraph still stands: Before release, test everything.

What version of Go are you using (go version)?

go version devel +7a60a96 Sat Apr 30 05:59:54 2016 +0000 darwin/amd64

What operating system and processor architecture are you using (go env)?

wally=% go env
GOARCH="amd64"
GOBIN="/Users/r/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/r"
GORACE=""
GOROOT="/Users/r/go"
GOTOOLDIR="/Users/r/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/r3/cryt3b891yj6rzqr8vzcn2xc0000gn/T/go-build408997659=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
wally=%

What did you do?

cd $GOROOT/src; go test ./...

What did you expect to see?

Everything passes.

What did you see instead?

Most do well, but some crash and some (math/big for example) are just crazy slow and time out

wally=% go test ./...
ok      archive/tar 0.049s
ok      archive/zip 45.824s
ok      bufio   0.095s
?       builtin [no test files]
ok      bytes   1.378s
ok      cmd/addr2line   0.915s
ok      cmd/api 0.017s
?       cmd/asm [no test files]
?       cmd/asm/internal/arch   [no test files]
ok      cmd/asm/internal/asm    0.214s
?       cmd/asm/internal/flags  [no test files]
ok      cmd/asm/internal/lex    0.013s
?       cmd/cgo [no test files]
ok      cmd/compile 6.745s
?       cmd/compile/internal/amd64  [no test files]
?       cmd/compile/internal/arm    [no test files]
?       cmd/compile/internal/arm64  [no test files]
SIGQUIT: quit
PC=0x5222b m=0

goroutine 0 [idle]:
runtime.mach_semaphore_wait(0x7fff00000803, 0x0, 0x14, 0x0, 0x100437600, 0x358900, 0x7fff5fbff6e8, 0x4c803, 0xffffffffffffffff, 0x4300000000, ...)
    /Users/r/go/src/runtime/sys_darwin_amd64.s:414 +0xb
runtime.semasleep1(0xffffffffffffffff, 0x4300000000)
    /Users/r/go/src/runtime/os_darwin.go:413 +0x4b
runtime.semasleep.func1()
    /Users/r/go/src/runtime/os_darwin.go:429 +0x33
runtime.systemstack(0x7fff5fbff710)
    /Users/r/go/src/runtime/asm_amd64.s:314 +0xab
runtime.semasleep(0xffffffffffffffff, 0x0)
    /Users/r/go/src/runtime/os_darwin.go:430 +0x44
runtime.notesleep(0x358d70)
    /Users/r/go/src/runtime/lock_sema.go:166 +0xb2
runtime.stopm()
    /Users/r/go/src/runtime/proc.go:1598 +0xad
runtime.findrunnable(0xc42001c600, 0x0)
    /Users/r/go/src/runtime/proc.go:2030 +0x241
runtime.schedule()
    /Users/r/go/src/runtime/proc.go:2129 +0x14c
runtime.park_m(0xc420057380)
    /Users/r/go/src/runtime/proc.go:2192 +0xa0
runtime.mcall(0x7fff5fbff8e0)
    /Users/r/go/src/runtime/asm_amd64.s:240 +0x5b

goroutine 1 [chan receive, 3 minutes]:
testing.(*T).Run(0xc420264000, 0x23d6d7, 0x17, 0x24bf90, 0x51f01)
    /Users/r/go/src/testing/testing.go:658 +0x2e6
testing.RunTests.func1(0xc420264000)
    /Users/r/go/src/testing/testing.go:804 +0x67
testing.tRunner(0xc420264000, 0xc420040e30)
    /Users/r/go/src/testing/testing.go:621 +0x81
testing.RunTests(0x24ca88, 0x355360, 0x94, 0x94, 0xc420423800)
    /Users/r/go/src/testing/testing.go:810 +0x2bb
testing.(*M).Run(0xc420040ef8, 0xc42006c230)
    /Users/r/go/src/testing/testing.go:754 +0x85
main.main()
    cmd/compile/internal/big/_test/_testmain.go:452 +0xc6

goroutine 194 [runnable]:
cmd/compile/internal/big.nat.trailingZeroBits(0xc42050e000, 0x30d5, 0x511f, 0xc4204d4140)
    /Users/r/go/src/cmd/compile/internal/big/nat.go:691 +0x91
cmd/compile/internal/big.(*Int).binaryGCD(0xc4204d4140, 0xc42021c080, 0xc42021c0a0, 0x333148)
    /Users/r/go/src/cmd/compile/internal/big/int.go:540 +0x411
cmd/compile/internal/big.(*Rat).norm(0xc42021c080, 0x0)
    /Users/r/go/src/cmd/compile/internal/big/rat.go:425 +0x122
cmd/compile/internal/big.(*Rat).Sub(0xc42021c080, 0xc42021c300, 0xc4200cdb38, 0xc42021c040)
    /Users/r/go/src/cmd/compile/internal/big/rat.go:488 +0x15a
cmd/compile/internal/big.delta(0xc42021c300, 0xb6a0000000000000, 0xc42021c040)
    /Users/r/go/src/cmd/compile/internal/big/rat_test.go:517 +0x95
cmd/compile/internal/big.checkIsBestApprox32(0xc420264300, 0x80000000, 0xc42021c300, 0x8000000000000000)
    /Users/r/go/src/cmd/compile/internal/big/rat_test.go:537 +0x140
cmd/compile/internal/big.TestFloat32SpecialCases(0xc420264300)
    /Users/r/go/src/cmd/compile/internal/big/ratconv_test.go:386 +0x220
testing.tRunner(0xc420264300, 0x24bf90)
    /Users/r/go/src/testing/testing.go:621 +0x81
created by testing.(*T).Run
    /Users/r/go/src/testing/testing.go:657 +0x2bc

rax    0xe
rbx    0x358c60
rcx    0x7fff5fbff688
rdx    0x7fff5fbff710
rdi    0x803
rsi    0x401
rbp    0x7fff5fbff6c0
rsp    0x7fff5fbff688
r8     0x7fff5fbff668
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0xc420000ea0
r14    0x38c40
r15    0x2a18d0
rip    0x5222b
rflags 0x286
cs     0x7
fs     0x0
gs     0x0
*** Test killed with quit: ran too long (10m0s).
FAIL    cmd/compile/internal/big    600.017s
ok      cmd/compile/internal/gc 171.985s
?       cmd/compile/internal/mips64 [no test files]
?       cmd/compile/internal/ppc64  [no test files]
?       cmd/compile/internal/s390x  [no test files]
ok      cmd/compile/internal/ssa    0.253s
ok      cmd/compile/internal/syntax 4.552s
ok      cmd/compile/internal/test   0.013s
?       cmd/compile/internal/x86    [no test files]
ok      cmd/cover   1.919s
?       cmd/dist    [no test files]
ok      cmd/doc 0.265s
ok      cmd/fix 0.025s
--- FAIL: TestRepoRootForImportPath (1.02s)
    vcs_test.go:164: RepoRootForImport("git.openstack.org/openstack/swift.git") = VCS(Git) Repo(https://git.openstack.org/openstack/swift.git), want VCS(Git) Repo(https://git.openstack.org/openstack/swift)
--- FAIL: TestCoverageRuns (4.20s)
    go_test.go:244: running testgo [test -short -coverpkg=strings strings regexp]
    go_test.go:259: standard output:
    go_test.go:260: --- FAIL: TestIndexRune (0.00s)
            strings_test.go:272: expected no allocations, got 1.000000
        FAIL
        coverage: 97.2% of statements in strings
        FAIL    strings 0.168s
        ok      regexp  0.188s  coverage: 15.5% of statements in strings

    go_test.go:273: go [test -short -coverpkg=strings strings regexp] failed unexpectedly: exit status 1
--- FAIL: TestGoGetUpdateInsecure (0.03s)
    go_test.go:2449: cloning github.com/golang/example repo: exit status 128
        fatal: transport 'http' not allowed
--- FAIL: TestGoGetInsecureCustomDomain (3.17s)
    go_test.go:244: running testgo [get -d wh3rd.net/repo]
    go_test.go:263: standard error:
    go_test.go:264: package wh3rd.net/repo: cannot download, http://wh3rd.net/git uses insecure protocol

    go_test.go:283: testgo failed as expected: exit status 1
    go_test.go:244: running testgo [get -d -insecure wh3rd.net/repo]
    go_test.go:263: standard error:
    go_test.go:264: # cd .; git clone http://wh3rd.net/git /var/folders/r3/cryt3b891yj6rzqr8vzcn2xc0000gn/T/gotest969981157/src/wh3rd.net/repo
        Cloning into '/var/folders/r3/cryt3b891yj6rzqr8vzcn2xc0000gn/T/gotest969981157/src/wh3rd.net/repo'...
        fatal: transport 'http' not allowed
        package wh3rd.net/repo: exit status 128

    go_test.go:273: go [get -d -insecure wh3rd.net/repo] failed unexpectedly: exit status 1
FAIL
FAIL    cmd/go  363.020s
ok      cmd/gofmt   43.335s
?       cmd/internal/bio    [no test files]
?       cmd/internal/browser    [no test files]
?       cmd/internal/dwarf  [no test files]
?       cmd/internal/gcprog [no test files]
ok      cmd/internal/goobj  0.013s
ok      cmd/internal/obj    0.014s
?       cmd/internal/obj/arm    [no test files]
?       cmd/internal/obj/arm64  [no test files]
?       cmd/internal/obj/mips   [no test files]
?       cmd/internal/obj/ppc64  [no test files]
?       cmd/internal/obj/s390x  [no test files]
ok      cmd/internal/obj/x86    0.118s
?       cmd/internal/objfile    [no test files]
?       cmd/internal/pprof/commands [no test files]
?       cmd/internal/pprof/driver   [no test files]
?       cmd/internal/pprof/fetch    [no test files]
?       cmd/internal/pprof/plugin   [no test files]
ok      cmd/internal/pprof/profile  0.014s
?       cmd/internal/pprof/report   [no test files]
?       cmd/internal/pprof/svg  [no test files]
?       cmd/internal/pprof/symbolizer   [no test files]
?       cmd/internal/pprof/symbolz  [no test files]
?       cmd/internal/pprof/tempfile [no test files]
?       cmd/internal/sys    [no test files]
ok      cmd/link    0.014s
?       cmd/link/internal/amd64 [no test files]
?       cmd/link/internal/arm   [no test files]
?       cmd/link/internal/arm64 [no test files]
?       cmd/link/internal/ld    [no test files]
?       cmd/link/internal/mips64    [no test files]
ok      cmd/link/internal/pe    0.025s
?       cmd/link/internal/ppc64 [no test files]
?       cmd/link/internal/s390x [no test files]
?       cmd/link/internal/x86   [no test files]
ok      cmd/nm  14.478s
ok      cmd/objdump 39.243s
ok      cmd/pack    9.990s
?       cmd/pprof   [no test files]
ok      cmd/trace   0.032s
ok      cmd/vendor/golang.org/x/arch/arm/armasm 0.020s
ok      cmd/vendor/golang.org/x/arch/ppc64/ppc64asm 0.017s
ok      cmd/vendor/golang.org/x/arch/x86/x86asm 0.453s
ok      cmd/vet 15.959s
ok      cmd/vet/internal/cfg    0.015s
?       cmd/vet/internal/whitelist  [no test files]
ok      compress/bzip2  0.080s
ok      compress/flate  40.458s
ok      compress/gzip   0.141s
ok      compress/lzw    0.087s
ok      compress/zlib   2.196s
ok      container/heap  0.016s
ok      container/list  0.015s
ok      container/ring  0.029s
ok      context 2.509s
?       crypto  [no test files]
ok      crypto/aes  0.111s
ok      crypto/cipher   2.687s
ok      crypto/des  0.029s
ok      crypto/dsa  40.585s
ok      crypto/ecdsa    7.006s
ok      crypto/elliptic 0.753s
ok      crypto/hmac 0.016s
ok      crypto/md5  0.016s
ok      crypto/rand 1.083s
ok      crypto/rc4  0.142s
ok      crypto/rsa  1.186s
ok      crypto/sha1 0.055s
ok      crypto/sha256   0.018s
ok      crypto/sha512   0.018s
ok      crypto/subtle   0.022s
ok      crypto/tls  305.765s
ok      crypto/x509 8.573s
?       crypto/x509/pkix    [no test files]
ok      database/sql    0.537s
ok      database/sql/driver 0.014s
ok      debug/dwarf 0.029s
ok      debug/elf   0.038s
ok      debug/gosym 0.318s
ok      debug/macho 0.011s
ok      debug/pe    0.019s
ok      debug/plan9obj  0.011s
?       encoding    [no test files]
ok      encoding/ascii85    0.010s
ok      encoding/asn1   0.018s
ok      encoding/base32 0.011s
ok      encoding/base64 0.011s
ok      encoding/binary 0.010s
ok      encoding/csv    0.009s
ok      encoding/gob    3.912s
ok      encoding/hex    0.013s
ok      encoding/json   3.027s
ok      encoding/pem    0.020s
ok      encoding/xml    0.016s
ok      errors  0.009s
ok      expvar  0.015s
ok      flag    0.010s
ok      fmt 0.150s
ok      go/ast  0.017s
ok      go/build    0.265s
ok      go/constant 0.015s
ok      go/doc  0.062s
ok      go/format   0.021s
?       go/importer [no test files]
ok      go/internal/gccgoimporter   0.016s
ok      go/internal/gcimporter  1.312s
ok      go/parser   0.051s
ok      go/printer  0.351s
ok      go/scanner  0.015s
ok      go/token    0.026s
ok      go/types    12.281s
?       hash    [no test files]
ok      hash/adler32    0.018s
ok      hash/crc32  0.019s
ok      hash/crc64  0.011s
ok      hash/fnv    0.015s
ok      html    0.014s
ok      html/template   0.054s
ok      image   0.619s
ok      image/color 0.100s
?       image/color/palette [no test files]
ok      image/draw  0.089s
ok      image/gif   0.072s
?       image/internal/imageutil    [no test files]
ok      image/jpeg  0.191s
ok      image/png   0.142s
ok      index/suffixarray   0.022s
?       internal/nettrace   [no test files]
?       internal/race   [no test files]
ok      internal/singleflight   0.025s
?       internal/syscall/windows    [no test files]
?       internal/syscall/windows/registry   [no test files]
?       internal/syscall/windows/sysdll [no test files]
?       internal/testenv    [no test files]
ok      internal/trace  0.043s
ok      io  0.030s
ok      io/ioutil   0.015s
ok      log 0.016s
ok      log/syslog  2.082s
ok      math    0.016s
ok      math/big    2.500s
ok      math/cmplx  0.015s
ok      math/rand   1.170s
ok      mime    0.021s
ok      mime/multipart  0.483s
ok      mime/quotedprintable    0.169s
ok      net 71.162s
ok      net/http    36.075s
ok      net/http/cgi    0.588s
ok      net/http/cookiejar  0.017s
ok      net/http/fcgi   0.014s
ok      net/http/httptest   0.124s
ok      net/http/httptrace  0.012s
ok      net/http/httputil   0.067s
ok      net/http/internal   0.014s
?       net/http/pprof  [no test files]
ok      net/internal/socktest   0.967s
ok      net/mail    0.011s
ok      net/rpc 0.025s
ok      net/rpc/jsonrpc 0.017s
ok      net/smtp    0.019s
ok      net/textproto   0.012s
ok      net/url 0.015s
ok      os  1.380s
ok      os/exec 0.506s
ok      os/signal   7.216s
ok      os/user 0.013s
ok      path    0.010s
ok      path/filepath   0.038s
?       plugin  [no test files]
ok      reflect 0.872s
ok      regexp  33.375s
ok      regexp/syntax   0.477s
ok      runtime 217.252s
?       runtime/cgo [no test files]
ok      runtime/debug   0.018s
ok      runtime/internal/atomic 0.286s
ok      runtime/internal/sys    0.013s
ok      runtime/pprof   25.184s
?       runtime/race    [no test files]
ok      runtime/trace   1.678s
ok      sort    1.099s
ok      strconv 2.417s
ok      strings 0.548s
--- FAIL: TestWaitGroupMisuse2 (1.49s)
    waitgroup_test.go:114: Should panic
    waitgroup_test.go:84: Unexpected panic: <nil>
FAIL
FAIL    sync    1.972s
ok      sync/atomic 2.872s
ok      syscall 0.081s
ok      testing 3.199s
?       testing/iotest  [no test files]
ok      testing/quick   0.052s
ok      text/scanner    0.012s
ok      text/tabwriter  0.011s
ok      text/template   0.454s
ok      text/template/parse 0.016s
ok      time    7.398s
ok      unicode 0.012s
ok      unicode/utf16   0.012s
ok      unicode/utf8    0.013s
?       unsafe  [no test files]
ok      vendor/golang_org/x/crypto/curve25519   0.033s
ok      vendor/golang_org/x/net/http2/hpack 0.060s
ok      vendor/golang_org/x/net/idna    0.014s
ok      vendor/golang_org/x/net/lex/httplex 0.013s
ok      vendor/golang_org/x/net/route   0.057s
?       vendor/golang_org/x/text/transform  [no test files]
?       vendor/golang_org/x/text/unicode/norm   [no test files]
?       vendor/golang_org/x/text/width  [no test files]
wally=%
@robpike robpike added this to the Go1.8 milestone Oct 16, 2016
@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Oct 17, 2016
@bradfitz
Copy link
Contributor

I also noticed the the net package doesn't pass in full mode for GOOS=nacl.

Maybe non-trybot (post-submit) builders should run in non-short mode.

@quentinmit quentinmit added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 17, 2016
@josharian
Copy link
Contributor

We definitely need some builder somewhere running in non-short mode. Another example, just off the top of my head: #15979.

@odeke-em odeke-em mentioned this issue Oct 30, 2016
Closed
@josharian
Copy link
Contributor

I'm going to break out the individual failures into new issues, so that they get looked at. For the moment, I'm going to ignore tests that are merely slow; my list of issues to break out will be generated by go test -timeout=1h ./....

This was referenced Oct 31, 2016
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@josharian
Copy link
Contributor

I've split out all the failing (not merely slow) tests into their own issues. Once those are fixed, we should give this another whirl. And set up a builder to do it regularly.

@gopherbot
Copy link

CL https://golang.org/cl/32585 mentions this issue.

gopherbot pushed a commit that referenced this issue Nov 3, 2016
@josharian
cmd/compile/internal/gc: make tests run faster
6650448 
TestAssembly takes 20s on my machine,
which is too slow for normal operation.
Marking as -short has its dangers (#17472),
but hopefully we'll soon have a builder for that.

All the SSA tests are hermetic and not time sensitive
and can thus be run in parallel.
Reduces the cmd/compile/internal/gc test time during
all.bash on my laptop from 42s to 7s.

Updates #17751

Change-Id: Idd876421db23b9fa3475e8a9b3355a5dc92a5a29
Reviewed-on: https://go-review.googlesource.com/32585
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@ianlancetaylor
Copy link
Contributor

I just ran go test ./... on tip. Everything passed except the sync test TestWaitGroupMisuse2, which is considered flaky as reported in #11443.

I'm going to repurpose this issue as being to run a builder without -short, and move it off the Go 1.8 release milestone.

@ianlancetaylor ianlancetaylor removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 15, 2016
@ianlancetaylor ianlancetaylor changed the title all: various failures when all tests are run without -short builders: run standard library tests without -short Nov 15, 2016
@ianlancetaylor ianlancetaylor modified the milestones: Unreleased, Go1.8 Nov 15, 2016
@josharian josharian mentioned this issue Feb 6, 2017
Closed
@bradfitz bradfitz changed the title builders: run standard library tests without -short x/build: run standard library tests without -short May 12, 2017
@quenelle
Copy link

quenelle commented Jul 20, 2017 •
edited

I see an odd failure in the original test output:

--- FAIL: TestRepoRootForImportPath (1.02s)
vcs_test.go:164: RepoRootForImport("git.openstack.org/openstack/swift.git") = VCS(Git) Repo(https://git.openstack.org/openstack/swift.git), want VCS(Git) Repo(https://git.openstack.org/openstack/swift)

In other words, the test is passing a URL ending in "swift.git" to repoRootForImportPath but the path returned ends in "swift". The test is expecting the returned URL to end with "swift.git".

I'm seeing the same failure in my area, and I was curious if anyone could give me tips
on what's going on?

Was that failure one of the failures that was split out and fixed in this issue?

Whoops. Nevermind. I saw my problem in 1.7ish and It goes away in 1.8.3. Must be due to some vcs.go changes.

@gopherbot
Copy link

Change https://golang.org/cl/58050 mentions this issue: runtime: always rebuild in TestIntendedInlining

gopherbot pushed a commit that referenced this issue Aug 24, 2017
@josharian
runtime: always rebuild in TestIntendedInlining
5df1fe5 
This is necessary when you aren't actively changing the runtime. Oops.

Also, run the tests on the builders, to avoid silent failures (#17472).

Change-Id: I1fc03790cdbddddb07026a772137a79919dcaac7
Reviewed-on: https://go-review.googlesource.com/58050
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ALTree
Copy link
Member

ALTree commented Jan 16, 2018

The specific issues reported here has been fixed and for the general suggestion of running go test without -short this is a dup of the older #10456; closing this.

@ALTree ALTree closed this as completed Jan 16, 2018
@golang golang locked and limited conversation to collaborators Jan 16, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
8 participants
@bradfitz @josharian @quentinmit @quenelle @ianlancetaylor @ALTree @robpike @gopherbot