Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/coordinator: some linux-amd64 trybot builds getting stuck at the end #36629

Closed
dmitshur opened this issue Jan 17, 2020 · 4 comments
Closed

x/build/cmd/coordinator: some linux-amd64 trybot builds getting stuck at the end #36629

dmitshur opened this issue Jan 17, 2020 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@dmitshur
Copy link
Contributor

There are two trybot invocations that were started on January 6th, 2020:

They still haven't finished. They look like this:

image

All builds other than linux-amd64 passed. linux-amd64 gets to the end and then doesn't complete. Its log is:


  builder: linux-amd64
      rev: af9eda2749f3bf51a169b584d98d4bec7482a08a
 buildlet: http://10.240.0.24 GCE VM: buildlet-linux-stretch-rn8350cea
  started: 2020-01-17 15:34:43.678309637 +0000 UTC m=+257501.903709568
   status: still running

Events:
  2020-01-17T15:34:43Z checking_for_snapshot 
  2020-01-17T15:34:43Z finish_checking_for_snapshot after 19.3ms
  2020-01-17T15:34:43Z get_buildlet 
  2020-01-17T15:35:44Z finish_get_buildlet after 1m0.2s
  2020-01-17T15:35:44Z using_buildlet 10.240.0.24:80
  2020-01-17T15:35:44Z write_snapshot_tar 
  2020-01-17T15:35:48Z finish_write_snapshot_tar after 4.45s
  2020-01-17T15:35:48Z make_and_test 
  2020-01-17T15:35:48Z starting_helpers 
  2020-01-17T15:35:48Z get_helper helper 3/4
  2020-01-17T15:35:48Z get_helper helper 1/4
  2020-01-17T15:35:48Z get_helper helper 2/4
  2020-01-17T15:35:48Z get_helper helper 4/4
  2020-01-17T15:35:48Z discovering_tests 
  2020-01-17T15:35:49Z get_test_stats 
  2020-01-17T15:35:49Z finish_get_test_stats after 0s
  2020-01-17T15:35:49Z starting_tests 254 tests
  2020-01-17T15:35:49Z run_tests_multi 10.240.0.24:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2020-01-17T15:35:53Z finish_run_tests_multi after 4.09s; 10.240.0.24:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2020-01-17T15:35:53Z run_tests_multi 10.240.0.24:80: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2020-01-17T15:35:57Z finish_run_tests_multi after 3.69s; 10.240.0.24:80: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2020-01-17T15:35:57Z run_tests_multi 10.240.0.24:80: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
  2020-01-17T15:35:59Z finish_run_tests_multi after 1.75s; 10.240.0.24:80: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
  2020-01-17T15:35:59Z run_tests_multi 10.240.0.24:80: [go_test:container/heap go_test:container/list go_test:container/ring go_test:context]
  2020-01-17T15:36:03Z finish_run_tests_multi after 3.75s; 10.240.0.24:80: [go_test:container/heap go_test:container/list go_test:container/ring go_test:context]
  2020-01-17T15:36:03Z run_tests_multi 10.240.0.24:80: [go_test:crypto go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
  2020-01-17T15:36:05Z finish_run_tests_multi after 1.78s; 10.240.0.24:80: [go_test:crypto go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
  2020-01-17T15:36:05Z run_tests_multi 10.240.0.24:80: [go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/internal/subtle go_test:crypto/md5]
  2020-01-17T15:36:07Z finish_run_tests_multi after 1.88s; 10.240.0.24:80: [go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/internal/subtle go_test:crypto/md5]
  2020-01-17T15:36:07Z run_tests_multi 10.240.0.24:80: [go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256 go_test:crypto/sha512]
  2020-01-17T15:36:09Z finish_run_tests_multi after 1.86s; 10.240.0.24:80: [go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256 go_test:crypto/sha512]
  2020-01-17T15:36:09Z run_tests_multi 10.240.0.24:80: [go_test:crypto/subtle go_test:crypto/tls go_test:crypto/x509]
  2020-01-17T15:36:15Z finish_run_tests_multi after 5.84s; 10.240.0.24:80: [go_test:crypto/subtle go_test:crypto/tls go_test:crypto/x509]
  2020-01-17T15:36:15Z run_tests_multi 10.240.0.24:80: [go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym go_test:debug/macho]
  2020-01-17T15:36:19Z finish_run_tests_multi after 3.87s; 10.240.0.24:80: [go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym go_test:debug/macho]
  2020-01-17T15:36:19Z run_tests_multi 10.240.0.24:80: [go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85 go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob]
  2020-01-17T15:36:22Z finish_run_tests_multi after 3.31s; 10.240.0.24:80: [go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85 go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob]
  2020-01-17T15:36:22Z run_tests_multi 10.240.0.24:80: [go_test:encoding/hex go_test:encoding/json go_test:encoding/pem go_test:encoding/xml go_test:errors go_test:expvar]
  2020-01-17T15:36:26Z finish_run_tests_multi after 3.34s; 10.240.0.24:80: [go_test:encoding/hex go_test:encoding/json go_test:encoding/pem go_test:encoding/xml go_test:errors go_test:expvar]
  2020-01-17T15:36:26Z run_tests_multi 10.240.0.24:80: [go_test:flag go_test:fmt go_test:go/ast]
  2020-01-17T15:36:29Z finish_run_tests_multi after 2.81s; 10.240.0.24:80: [go_test:flag go_test:fmt go_test:go/ast]
  2020-01-17T15:36:29Z run_tests_multi 10.240.0.24:80: [go_test:go/build go_test:go/constant go_test:go/doc go_test:go/format go_test:go/importer go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter]
  2020-01-17T15:36:33Z finish_run_tests_multi after 4.47s; 10.240.0.24:80: [go_test:go/build go_test:go/constant go_test:go/doc go_test:go/format go_test:go/importer go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter]
  2020-01-17T15:36:33Z run_tests_multi 10.240.0.24:80: [go_test:go/internal/srcimporter go_test:go/parser go_test:go/printer go_test:go/scanner go_test:go/token]
  2020-01-17T15:36:35Z finish_get_helper after 47s; helper 3/4
  2020-01-17T15:36:35Z empty_helper_ready 10.240.0.43:80
  2020-01-17T15:36:35Z got_empty_test_helper http://10.240.0.43 GCE VM: buildlet-linux-stretch-rn7e4f049
  2020-01-17T15:36:37Z finish_get_helper after 48.7s; helper 1/4
  2020-01-17T15:36:37Z empty_helper_ready 10.240.0.36:80
  2020-01-17T15:36:37Z got_empty_test_helper http://10.240.0.36 GCE VM: buildlet-linux-stretch-rn9b9fd03
  2020-01-17T15:36:37Z finish_run_tests_multi after 4.31s; 10.240.0.24:80: [go_test:go/internal/srcimporter go_test:go/parser go_test:go/printer go_test:go/scanner go_test:go/token]
  2020-01-17T15:36:37Z run_tests_multi 10.240.0.24:80: [go_test:go/types go_test:hash go_test:hash/adler32 go_test:hash/crc32]
  2020-01-17T15:36:38Z finish_get_helper after 49.6s; helper 2/4
  2020-01-17T15:36:38Z empty_helper_ready 10.240.0.35:80
  2020-01-17T15:36:38Z got_empty_test_helper http://10.240.0.35 GCE VM: buildlet-linux-stretch-rn18fbd4c
  2020-01-17T15:36:38Z finish_get_helper after 50s; helper 4/4
  2020-01-17T15:36:38Z empty_helper_ready 10.240.0.38:80
  2020-01-17T15:36:38Z got_empty_test_helper http://10.240.0.38 GCE VM: buildlet-linux-stretch-rne260e9f
  2020-01-17T15:36:40Z test_helper_set_up 10.240.0.43:80
  2020-01-17T15:36:40Z run_test:go_test:cmd/go 10.240.0.43:80
  2020-01-17T15:36:41Z finish_run_tests_multi after 3.71s; 10.240.0.24:80: [go_test:go/types go_test:hash go_test:hash/adler32 go_test:hash/crc32]
  2020-01-17T15:36:41Z run_tests_multi 10.240.0.24:80: [go_test:hash/crc64 go_test:hash/fnv go_test:hash/maphash go_test:html go_test:html/template]
  2020-01-17T15:36:43Z test_helper_set_up 10.240.0.35:80
  2020-01-17T15:36:43Z run_test:reboot 10.240.0.35:80
  2020-01-17T15:36:43Z test_helper_set_up 10.240.0.36:80
  2020-01-17T15:36:43Z run_test:cgo_test 10.240.0.36:80
  2020-01-17T15:36:43Z test_helper_set_up 10.240.0.38:80
  2020-01-17T15:36:43Z run_test:runtime:cpu124 10.240.0.38:80
  2020-01-17T15:36:43Z finish_run_tests_multi after 1.9s; 10.240.0.24:80: [go_test:hash/crc64 go_test:hash/fnv go_test:hash/maphash go_test:html go_test:html/template]
  2020-01-17T15:36:43Z run_tests_multi 10.240.0.24:80: [go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2020-01-17T15:36:46Z finish_run_tests_multi after 2.64s; 10.240.0.24:80: [go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2020-01-17T15:36:46Z run_tests_multi 10.240.0.24:80: [go_test:image/png go_test:index/suffixarray]
  2020-01-17T15:36:47Z finish_run_tests_multi after 1.35s; 10.240.0.24:80: [go_test:image/png go_test:index/suffixarray]
  2020-01-17T15:36:47Z run_tests_multi 10.240.0.24:80: [go_test:internal/cpu go_test:internal/fmtsort]
  2020-01-17T15:36:54Z finish_run_tests_multi after 6.16s; 10.240.0.24:80: [go_test:internal/cpu go_test:internal/fmtsort]
  2020-01-17T15:36:54Z run_tests_multi 10.240.0.24:80: [go_test:internal/poll go_test:internal/reflectlite go_test:internal/singleflight go_test:internal/trace]
  2020-01-17T15:36:59Z finish_run_tests_multi after 5.73s; 10.240.0.24:80: [go_test:internal/poll go_test:internal/reflectlite go_test:internal/singleflight go_test:internal/trace]
  2020-01-17T15:36:59Z run_tests_multi 10.240.0.24:80: [go_test:internal/xcoff go_test:io go_test:io/ioutil go_test:log]
  2020-01-17T15:37:01Z finish_run_tests_multi after 1.52s; 10.240.0.24:80: [go_test:internal/xcoff go_test:io go_test:io/ioutil go_test:log]
  2020-01-17T15:37:01Z run_tests_multi 10.240.0.24:80: [go_test:log/syslog go_test:math go_test:math/big go_test:math/bits]
  2020-01-17T15:37:06Z finish_run_tests_multi after 4.81s; 10.240.0.24:80: [go_test:log/syslog go_test:math go_test:math/big go_test:math/bits]
  2020-01-17T15:37:06Z run_tests_multi 10.240.0.24:80: [go_test:math/cmplx go_test:math/rand go_test:mime]
  2020-01-17T15:37:08Z finish_run_tests_multi after 1.52s; 10.240.0.24:80: [go_test:math/cmplx go_test:math/rand go_test:mime]
  2020-01-17T15:37:08Z run_tests_multi 10.240.0.24:80: [go_test:mime/multipart go_test:mime/quotedprintable]
  2020-01-17T15:37:09Z finish_run_tests_multi after 1.32s; 10.240.0.24:80: [go_test:mime/multipart go_test:mime/quotedprintable]
  2020-01-17T15:37:09Z run_test:go_test:net 10.240.0.24:80
  2020-01-17T15:37:21Z finish_run_test:go_test:net after 12.4s; 10.240.0.24:80
  2020-01-17T15:37:21Z run_test:go_test:net/http 10.240.0.24:80
  2020-01-17T15:37:22Z finish_run_test:runtime:cpu124 after 38.9s; 10.240.0.38:80
  2020-01-17T15:37:22Z run_test:testshared 10.240.0.38:80
  2020-01-17T15:37:31Z finish_run_test:cgo_test after 48.4s; 10.240.0.36:80
  2020-01-17T15:37:31Z run_test:race 10.240.0.36:80
  2020-01-17T15:37:34Z finish_run_test:go_test:net/http after 12.6s; 10.240.0.24:80
  2020-01-17T15:37:34Z run_tests_multi 10.240.0.24:80: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2020-01-17T15:37:36Z finish_run_tests_multi after 2.17s; 10.240.0.24:80: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2020-01-17T15:37:36Z run_tests_multi 10.240.0.24:80: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal go_test:net/http/pprof go_test:net/internal/socktest]
  2020-01-17T15:37:41Z finish_run_tests_multi after 4.23s; 10.240.0.24:80: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal go_test:net/http/pprof go_test:net/internal/socktest]
  2020-01-17T15:37:41Z run_tests_multi 10.240.0.24:80: [go_test:net/mail go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto go_test:net/url]
  2020-01-17T15:37:43Z finish_run_tests_multi after 2.61s; 10.240.0.24:80: [go_test:net/mail go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto go_test:net/url]
  2020-01-17T15:37:43Z run_tests_multi 10.240.0.24:80: [go_test:os go_test:os/exec]
  2020-01-17T15:37:47Z finish_run_tests_multi after 3.41s; 10.240.0.24:80: [go_test:os go_test:os/exec]
  2020-01-17T15:37:47Z run_tests_multi 10.240.0.24:80: [go_test:os/signal go_test:os/user go_test:path go_test:path/filepath]
  2020-01-17T15:37:53Z finish_run_tests_multi after 6.37s; 10.240.0.24:80: [go_test:os/signal go_test:os/user go_test:path go_test:path/filepath]
  2020-01-17T15:37:53Z run_tests_multi 10.240.0.24:80: [go_test:plugin go_test:reflect go_test:regexp go_test:regexp/syntax]
  2020-01-17T15:37:58Z finish_run_tests_multi after 4.43s; 10.240.0.24:80: [go_test:plugin go_test:reflect go_test:regexp go_test:regexp/syntax]
  2020-01-17T15:37:58Z run_test:go_test:runtime 10.240.0.24:80
  2020-01-17T15:37:58Z finish_run_test:testshared after 36.4s; 10.240.0.38:80
  2020-01-17T15:37:58Z run_test:test:7_10 10.240.0.38:80
  2020-01-17T15:38:08Z finish_run_test:race after 36.5s; 10.240.0.36:80
  2020-01-17T15:38:08Z run_test:test:0_10 10.240.0.36:80
  2020-01-17T15:38:27Z finish_run_test:go_test:cmd/go after 1m47.5s; 10.240.0.43:80
  2020-01-17T15:38:27Z run_test:testcarchive 10.240.0.43:80
  2020-01-17T15:38:28Z still_waiting_on_test go_test:runtime
  2020-01-17T15:38:31Z finish_run_test:reboot after 1m48.2s; 10.240.0.35:80
  2020-01-17T15:38:31Z run_test:testplugin 10.240.0.35:80
  2020-01-17T15:38:33Z finish_run_test:go_test:runtime after 35.4s; 10.240.0.24:80
  2020-01-17T15:38:33Z run_tests_multi 10.240.0.24:80: [go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math go_test:runtime/internal/sys go_test:runtime/pprof]
  2020-01-17T15:38:36Z finish_run_test:test:0_10 after 28.6s; 10.240.0.36:80
  2020-01-17T15:38:36Z run_test:go_test:cmd/compile/internal/gc 10.240.0.36:80
  2020-01-17T15:38:40Z finish_run_tests_multi after 6.2s; 10.240.0.24:80: [go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math go_test:runtime/internal/sys go_test:runtime/pprof]
  2020-01-17T15:38:40Z run_tests_multi 10.240.0.24:80: [go_test:runtime/pprof/internal/profile go_test:runtime/trace go_test:sort go_test:strconv]
  2020-01-17T15:38:44Z finish_run_tests_multi after 4.61s; 10.240.0.24:80: [go_test:runtime/pprof/internal/profile go_test:runtime/trace go_test:sort go_test:strconv]
  2020-01-17T15:38:44Z run_tests_multi 10.240.0.24:80: [go_test:strings go_test:sync go_test:sync/atomic]
  2020-01-17T15:38:50Z finish_run_tests_multi after 5.36s; 10.240.0.24:80: [go_test:strings go_test:sync go_test:sync/atomic]
  2020-01-17T15:38:50Z run_tests_multi 10.240.0.24:80: [go_test:syscall go_test:testing go_test:testing/iotest]
  2020-01-17T15:38:50Z finish_run_test:testcarchive after 23s; 10.240.0.43:80
  2020-01-17T15:38:50Z run_test:go_test:cmd/compile/internal/ssa 10.240.0.43:80
  2020-01-17T15:38:52Z finish_run_test:testplugin after 21.3s; 10.240.0.35:80
  2020-01-17T15:38:52Z run_test:test:3_10 10.240.0.35:80
  2020-01-17T15:38:54Z finish_run_tests_multi after 3.72s; 10.240.0.24:80: [go_test:syscall go_test:testing go_test:testing/iotest]
  2020-01-17T15:38:54Z run_tests_multi 10.240.0.24:80: [go_test:testing/quick go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse go_test:time]
  2020-01-17T15:38:57Z finish_run_test:go_test:cmd/compile/internal/gc after 20.9s; 10.240.0.36:80
  2020-01-17T15:38:57Z run_test:test:4_10 10.240.0.36:80
  2020-01-17T15:38:59Z finish_run_tests_multi after 5.44s; 10.240.0.24:80: [go_test:testing/quick go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse go_test:time]
  2020-01-17T15:38:59Z run_tests_multi 10.240.0.24:80: [go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex]
  2020-01-17T15:39:03Z finish_run_tests_multi after 4.03s; 10.240.0.24:80: [go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex]
  2020-01-17T15:39:03Z run_test:go_test:cmd/compile 10.240.0.24:80
  2020-01-17T15:39:07Z finish_run_test:go_test:cmd/compile/internal/ssa after 16.5s; 10.240.0.43:80
  2020-01-17T15:39:07Z run_test:test:2_10 10.240.0.43:80
  2020-01-17T15:39:11Z finish_run_test:test:4_10 after 13.8s; 10.240.0.36:80
  2020-01-17T15:39:11Z run_test:test:8_10 10.240.0.36:80
  2020-01-17T15:39:14Z finish_run_test:go_test:cmd/compile after 10.6s; 10.240.0.24:80
  2020-01-17T15:39:14Z run_test:go_test:cmd/compile/internal/logopt 10.240.0.24:80
  2020-01-17T15:39:15Z finish_run_test:go_test:cmd/compile/internal/logopt after 910.1ms; 10.240.0.24:80
  2020-01-17T15:39:15Z run_tests_multi 10.240.0.24:80: [go_test:cmd/compile/internal/syntax go_test:cmd/compile/internal/test go_test:cmd/compile/internal/types go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]
  2020-01-17T15:39:16Z finish_run_test:test:3_10 after 24s; 10.240.0.35:80
  2020-01-17T15:39:16Z run_test:api 10.240.0.35:80
  2020-01-17T15:39:20Z finish_run_test:test:2_10 after 12.5s; 10.240.0.43:80
  2020-01-17T15:39:20Z run_test:test:1_10 10.240.0.43:80
  2020-01-17T15:39:20Z finish_run_tests_multi after 4.62s; 10.240.0.24:80: [go_test:cmd/compile/internal/syntax go_test:cmd/compile/internal/test go_test:cmd/compile/internal/types go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]
  2020-01-17T15:39:20Z run_tests_multi 10.240.0.24:80: [go_test:cmd/go/internal/auth go_test:cmd/go/internal/cache go_test:cmd/go/internal/generate go_test:cmd/go/internal/get go_test:cmd/go/internal/imports go_test:cmd/go/internal/load go_test:cmd/go/internal/lockedfile go_test:cmd/go/internal/lockedfile/internal/filelock]
  2020-01-17T15:39:23Z finish_run_tests_multi after 3.52s; 10.240.0.24:80: [go_test:cmd/go/internal/auth go_test:cmd/go/internal/cache go_test:cmd/go/internal/generate go_test:cmd/go/internal/get go_test:cmd/go/internal/imports go_test:cmd/go/internal/load go_test:cmd/go/internal/lockedfile go_test:cmd/go/internal/lockedfile/internal/filelock]
  2020-01-17T15:39:23Z run_tests_multi 10.240.0.24:80: [go_test:cmd/go/internal/modconv go_test:cmd/go/internal/modfetch go_test:cmd/go/internal/modfetch/codehost go_test:cmd/go/internal/modfetch/zip_sum_test]
  2020-01-17T15:39:24Z finish_run_test:test:8_10 after 13s; 10.240.0.36:80
  2020-01-17T15:39:24Z run_test:test:5_10 10.240.0.36:80
  2020-01-17T15:39:26Z finish_run_tests_multi after 2.54s; 10.240.0.24:80: [go_test:cmd/go/internal/modconv go_test:cmd/go/internal/modfetch go_test:cmd/go/internal/modfetch/codehost go_test:cmd/go/internal/modfetch/zip_sum_test]
  2020-01-17T15:39:26Z run_tests_multi 10.240.0.24:80: [go_test:cmd/go/internal/modload go_test:cmd/go/internal/mvs go_test:cmd/go/internal/par go_test:cmd/go/internal/renameio go_test:cmd/go/internal/search go_test:cmd/go/internal/txtar go_test:cmd/go/internal/web]
  2020-01-17T15:39:28Z finish_run_test:api after 12s; 10.240.0.35:80
  2020-01-17T15:39:28Z run_test:go_test:cmd/vet 10.240.0.35:80
  2020-01-17T15:39:29Z finish_run_tests_multi after 2.5s; 10.240.0.24:80: [go_test:cmd/go/internal/modload go_test:cmd/go/internal/mvs go_test:cmd/go/internal/par go_test:cmd/go/internal/renameio go_test:cmd/go/internal/search go_test:cmd/go/internal/txtar go_test:cmd/go/internal/web]
  2020-01-17T15:39:29Z run_tests_multi 10.240.0.24:80: [go_test:cmd/go/internal/work go_test:cmd/gofmt go_test:cmd/internal/buildid go_test:cmd/internal/dwarf go_test:cmd/internal/edit go_test:cmd/internal/goobj go_test:cmd/internal/obj go_test:cmd/internal/obj/arm64]
  2020-01-17T15:39:31Z finish_run_test:test:1_10 after 11.7s; 10.240.0.43:80
  2020-01-17T15:39:31Z run_test:test:9_10 10.240.0.43:80
  2020-01-17T15:39:32Z finish_run_tests_multi after 3.47s; 10.240.0.24:80: [go_test:cmd/go/internal/work go_test:cmd/gofmt go_test:cmd/internal/buildid go_test:cmd/internal/dwarf go_test:cmd/internal/edit go_test:cmd/internal/goobj go_test:cmd/internal/obj go_test:cmd/internal/obj/arm64]
  2020-01-17T15:39:32Z run_tests_multi 10.240.0.24:80: [go_test:cmd/internal/obj/riscv go_test:cmd/internal/obj/x86 go_test:cmd/internal/objabi go_test:cmd/internal/src go_test:cmd/internal/test2json]
  2020-01-17T15:39:35Z finish_run_tests_multi after 2.21s; 10.240.0.24:80: [go_test:cmd/internal/obj/riscv go_test:cmd/internal/obj/x86 go_test:cmd/internal/objabi go_test:cmd/internal/src go_test:cmd/internal/test2json]
  2020-01-17T15:39:35Z run_test:go_test:cmd/link 10.240.0.24:80
  2020-01-17T15:39:35Z finish_run_test:test:5_10 after 10.5s; 10.240.0.36:80
  2020-01-17T15:39:35Z run_test:test:6_10 10.240.0.36:80
  2020-01-17T15:39:40Z finish_run_test:go_test:cmd/vet after 11.3s; 10.240.0.35:80
  2020-01-17T15:39:40Z run_test:testcshared 10.240.0.35:80
  2020-01-17T15:39:40Z finish_run_test:test:9_10 after 8.87s; 10.240.0.43:80
  2020-01-17T15:39:40Z run_test:doc_progs 10.240.0.43:80
  2020-01-17T15:39:41Z finish_run_test:go_test:cmd/link after 5.86s; 10.240.0.24:80
  2020-01-17T15:39:41Z run_tests_multi 10.240.0.24:80: [go_test:cmd/link/internal/ld go_test:cmd/link/internal/sym go_test:cmd/nm]
  2020-01-17T15:39:44Z finish_run_test:test:6_10 after 8.66s; 10.240.0.36:80
  2020-01-17T15:39:44Z run_test:pie_internal 10.240.0.36:80
  2020-01-17T15:39:46Z finish_run_tests_multi after 5.61s; 10.240.0.24:80: [go_test:cmd/link/internal/ld go_test:cmd/link/internal/sym go_test:cmd/nm]
  2020-01-17T15:39:46Z run_tests_multi 10.240.0.24:80: [go_test:cmd/objdump go_test:cmd/pack go_test:cmd/trace]
  2020-01-17T15:39:47Z finish_run_test:doc_progs after 6.61s; 10.240.0.43:80
  2020-01-17T15:39:47Z run_test:cgo_errors 10.240.0.43:80
  2020-01-17T15:39:50Z finish_run_tests_multi after 3.14s; 10.240.0.24:80: [go_test:cmd/objdump go_test:cmd/pack go_test:cmd/trace]
  2020-01-17T15:39:50Z run_test:osusergo 10.240.0.24:80
  2020-01-17T15:39:50Z finish_run_test:pie_internal after 5.94s; 10.240.0.36:80
  2020-01-17T15:39:50Z run_test:sync_cpu 10.240.0.36:80
  2020-01-17T15:39:50Z finish_run_test:osusergo after 836.6ms; 10.240.0.24:80
  2020-01-17T15:39:50Z run_test:cmd_go_test_terminal 10.240.0.24:80
  2020-01-17T15:39:50Z finish_run_test:testcshared after 10.5s; 10.240.0.35:80
  2020-01-17T15:39:50Z run_test:moved_goroot 10.240.0.35:80
  2020-01-17T15:39:51Z finish_run_test:cmd_go_test_terminal after 508.4ms; 10.240.0.24:80
  2020-01-17T15:39:51Z run_test:nolibgcc:crypto/x509 10.240.0.24:80
  2020-01-17T15:39:52Z finish_run_test:cgo_errors after 4.81s; 10.240.0.43:80
  2020-01-17T15:39:52Z run_test:cgo_stdio 10.240.0.43:80
  2020-01-17T15:39:53Z finish_run_test:nolibgcc:crypto/x509 after 1.27s; 10.240.0.24:80
  2020-01-17T15:39:53Z run_test:nolibgcc:net 10.240.0.24:80
  2020-01-17T15:39:53Z finish_run_test:moved_goroot after 2.67s; 10.240.0.35:80
  2020-01-17T15:39:53Z run_test:pie_internal_cgo 10.240.0.35:80
  2020-01-17T15:39:54Z finish_run_test:nolibgcc:net after 1.17s; 10.240.0.24:80
  2020-01-17T15:39:54Z run_test:nolibgcc:os/user 10.240.0.24:80
  2020-01-17T15:39:54Z finish_run_test:sync_cpu after 4.3s; 10.240.0.36:80
  2020-01-17T15:39:54Z run_test:codewalk 10.240.0.36:80
  2020-01-17T15:39:55Z finish_run_test:cgo_stdio after 2.45s; 10.240.0.43:80
  2020-01-17T15:39:55Z run_test:wiki 10.240.0.43:80
  2020-01-17T15:39:55Z finish_run_test:nolibgcc:os/user after 783.8ms; 10.240.0.24:80
  2020-01-17T15:39:55Z run_test:cgo_life 10.240.0.24:80
  2020-01-17T15:39:57Z finish_run_test:codewalk after 2.24s; 10.240.0.36:80
  2020-01-17T15:39:57Z run_test:cgo_fortran 10.240.0.36:80
  2020-01-17T15:39:57Z finish_run_test:wiki after 2.06s; 10.240.0.43:80
  2020-01-17T15:39:57Z run_test:testsovar 10.240.0.43:80
  2020-01-17T15:39:57Z finish_run_test:cgo_life after 2.01s; 10.240.0.24:80
  2020-01-17T15:39:57Z run_test:swig_stdio 10.240.0.24:80
  2020-01-17T15:39:57Z finish_run_test:pie_internal_cgo after 3.69s; 10.240.0.35:80
  2020-01-17T15:39:57Z run_test:bench_go1 10.240.0.35:80
  2020-01-17T15:39:58Z finish_run_test:swig_stdio after 1.29s; 10.240.0.24:80
  2020-01-17T15:39:58Z run_test:testgodefs 10.240.0.24:80
  2020-01-17T15:39:58Z finish_run_test:cgo_fortran after 1.83s; 10.240.0.36:80
  2020-01-17T15:39:58Z run_test:testso 10.240.0.36:80
  2020-01-17T15:39:58Z finish_run_test:testsovar after 1.67s; 10.240.0.43:80
  2020-01-17T15:39:58Z run_test:testsanitizers/msan 10.240.0.43:80
  2020-01-17T15:39:59Z finish_run_test:bench_go1 after 2.35s; 10.240.0.35:80
  2020-01-17T15:39:59Z run_test:testsigfwd 10.240.0.35:80
  2020-01-17T15:40:00Z finish_run_test:testgodefs after 1.15s; 10.240.0.24:80
  2020-01-17T15:40:00Z run_test:testasan 10.240.0.24:80
  2020-01-17T15:40:00Z finish_run_test:testsanitizers/msan after 1.22s; 10.240.0.43:80
  2020-01-17T15:40:00Z no_new_tests_remain 10.240.0.43:80
  2020-01-17T15:40:00Z closed_helper 10.240.0.43:80
  2020-01-17T15:40:00Z finish_run_test:testso after 1.61s; 10.240.0.36:80
  2020-01-17T15:40:00Z no_new_tests_remain 10.240.0.36:80
  2020-01-17T15:40:00Z closed_helper 10.240.0.36:80
  2020-01-17T15:40:01Z finish_run_test:testasan after 1.07s; 10.240.0.24:80
  2020-01-17T15:40:01Z finish_run_test:testsigfwd after 1.13s; 10.240.0.35:80
  2020-01-17T15:40:01Z no_new_tests_remain 10.240.0.35:80
  2020-01-17T15:40:01Z closed_helper 10.240.0.35:80
  2020-01-17T15:40:31Z still_waiting_on_test test:7_10
  2020-01-17T15:41:01Z still_waiting_on_test test:7_10
  2020-01-17T15:41:31Z still_waiting_on_test test:7_10
  2020-01-17T15:42:01Z still_waiting_on_test test:7_10
  2020-01-17T15:42:31Z still_waiting_on_test test:7_10
  2020-01-17T15:43:01Z still_waiting_on_test test:7_10
  2020-01-17T15:43:31Z still_waiting_on_test test:7_10
  2020-01-17T15:44:01Z still_waiting_on_test test:7_10
  2020-01-17T15:44:31Z still_waiting_on_test test:7_10
  2020-01-17T15:45:01Z still_waiting_on_test test:7_10
  2020-01-17T15:45:31Z still_waiting_on_test test:7_10
  2020-01-17T15:46:01Z still_waiting_on_test test:7_10
  2020-01-17T15:46:31Z still_waiting_on_test test:7_10
  2020-01-17T15:47:01Z still_waiting_on_test test:7_10
  2020-01-17T15:47:31Z still_waiting_on_test test:7_10
  2020-01-17T15:48:01Z still_waiting_on_test test:7_10
  2020-01-17T15:48:31Z still_waiting_on_test test:7_10
  2020-01-17T15:49:01Z still_waiting_on_test test:7_10
  2020-01-17T15:49:31Z still_waiting_on_test test:7_10
  2020-01-17T15:50:01Z still_waiting_on_test test:7_10
  2020-01-17T15:50:31Z still_waiting_on_test test:7_10
  2020-01-17T15:51:01Z still_waiting_on_test test:7_10
  2020-01-17T15:51:31Z still_waiting_on_test test:7_10
   +1.9s (now)

Build log:
linux-amd64 at af9eda2749f3bf51a169b584d98d4bec7482a08a


##### Testing packages.
ok  	archive/tar	0.028s
ok  	archive/zip	0.282s
ok  	bufio	0.085s
ok  	bytes	1.130s
ok  	compress/bzip2	0.132s
ok  	compress/flate	0.501s
ok  	compress/gzip	0.072s
ok  	compress/lzw	0.111s
ok  	compress/zlib	0.509s
ok  	container/heap	0.043s
ok  	container/list	0.052s
ok  	container/ring	0.030s
ok  	context	0.975s
ok  	crypto	0.044s
ok  	crypto/aes	0.056s
ok  	crypto/cipher	0.013s
ok  	crypto/des	0.025s
ok  	crypto/dsa	0.004s
ok  	crypto/ecdsa	0.206s
ok  	crypto/ed25519	0.105s
ok  	crypto/elliptic	0.051s
ok  	crypto/hmac	0.014s
ok  	crypto/internal/subtle	0.002s
ok  	crypto/md5	0.003s
ok  	crypto/rand	0.030s
ok  	crypto/rc4	0.082s
ok  	crypto/rsa	0.121s
ok  	crypto/sha1	0.027s
ok  	crypto/sha256	0.003s
ok  	crypto/sha512	0.004s
ok  	crypto/subtle	0.009s
ok  	crypto/tls	0.562s
ok  	crypto/x509	0.479s
ok  	database/sql	0.552s
ok  	database/sql/driver	0.030s
ok  	debug/dwarf	0.038s
ok  	debug/elf	0.023s
ok  	debug/gosym	0.013s
ok  	debug/macho	0.003s
ok  	debug/pe	0.018s
ok  	debug/plan9obj	0.021s
ok  	encoding/ascii85	0.007s
ok  	encoding/asn1	0.043s
ok  	encoding/base32	0.022s
ok  	encoding/base64	0.025s
ok  	encoding/binary	0.004s
ok  	encoding/csv	0.028s
ok  	encoding/gob	0.019s
ok  	encoding/hex	0.052s
ok  	encoding/json	0.072s
ok  	encoding/pem	0.027s
ok  	encoding/xml	0.023s
ok  	errors	0.018s
ok  	expvar	0.004s
ok  	flag	0.012s
ok  	fmt	0.093s
ok  	go/ast	0.004s
ok  	go/build	0.201s
ok  	go/constant	0.014s
ok  	go/doc	0.092s
ok  	go/format	0.011s
ok  	go/importer	0.126s
ok  	go/internal/gccgoimporter	0.010s
ok  	go/internal/gcimporter	1.125s
ok  	go/internal/srcimporter	2.246s
ok  	go/parser	0.055s
ok  	go/printer	0.407s
ok  	go/scanner	0.043s
ok  	go/token	0.015s
ok  	go/types	0.777s
ok  	hash	0.004s
ok  	hash/adler32	0.018s
ok  	hash/crc32	0.006s
ok  	hash/crc64	0.004s
ok  	hash/fnv	0.030s
ok  	hash/maphash	0.106s
ok  	html	0.003s
ok  	html/template	0.029s
ok  	image	0.092s
ok  	image/color	0.069s
ok  	image/draw	0.121s
ok  	image/gif	0.306s
ok  	image/jpeg	0.188s
ok  	image/png	0.042s
ok  	index/suffixarray	0.181s
ok  	internal/cpu	0.007s
ok  	internal/fmtsort	0.005s
ok  	internal/poll	0.172s
ok  	internal/reflectlite	0.036s
ok  	internal/singleflight	0.026s
ok  	internal/trace	0.088s
ok  	internal/xcoff	0.029s
ok  	io	0.029s
ok  	io/ioutil	0.003s
ok  	log	0.004s
ok  	log/syslog	1.246s
ok  	math	0.007s
ok  	math/big	0.885s
ok  	math/bits	0.009s
ok  	math/cmplx	0.003s
ok  	math/rand	0.196s
ok  	mime	0.007s
ok  	mime/multipart	0.134s
ok  	mime/quotedprintable	0.008s
ok  	net	8.770s
ok  	net/http	7.704s
ok  	net/http/cgi	0.356s
ok  	net/http/cookiejar	0.008s
ok  	net/http/fcgi	0.015s
ok  	net/http/httptest	0.024s
ok  	net/http/httptrace	0.003s
ok  	net/http/httputil	0.083s
ok  	net/http/internal	0.050s
ok  	net/http/pprof	2.032s
ok  	net/internal/socktest	0.003s
ok  	net/mail	0.007s
ok  	net/rpc	0.051s
ok  	net/rpc/jsonrpc	0.051s
ok  	net/smtp	0.017s
ok  	net/textproto	0.005s
ok  	net/url	0.006s
ok  	os	0.638s
ok  	os/exec	0.588s
ok  	os/signal	4.524s
ok  	os/user	0.004s
ok  	path	0.017s
ok  	path/filepath	0.013s
ok  	plugin	0.028s
ok  	reflect	0.106s
ok  	regexp	0.155s
ok  	regexp/syntax	0.511s
ok  	runtime	28.142s
ok  	runtime/debug	0.081s
ok  	runtime/internal/atomic	0.011s
ok  	runtime/internal/math	0.009s
ok  	runtime/internal/sys	0.013s
ok  	runtime/pprof	4.594s
ok  	runtime/pprof/internal/profile	0.006s
ok  	runtime/trace	0.864s
ok  	sort	0.099s
ok  	strconv	0.329s
ok  	strings	1.846s
ok  	sync	0.263s
ok  	sync/atomic	0.035s
ok  	syscall	0.024s
ok  	testing	0.130s
ok  	testing/iotest	0.003s
ok  	testing/quick	0.082s
ok  	text/scanner	0.005s
ok  	text/tabwriter	0.058s
ok  	text/template	0.059s
ok  	text/template/parse	0.017s
ok  	time	1.566s
ok  	unicode	0.015s
ok  	unicode/utf16	0.008s
ok  	unicode/utf8	0.010s
ok  	cmd/addr2line	1.754s
ok  	cmd/api	0.038s
ok  	cmd/asm/internal/asm	0.436s
ok  	cmd/asm/internal/lex	0.005s
ok  	cmd/compile	4.806s
ok  	cmd/compile/internal/gc	11.088s
ok  	cmd/compile/internal/logopt	0.028s
ok  	cmd/compile/internal/ssa	0.389s
ok  	cmd/compile/internal/syntax	0.015s
ok  	cmd/compile/internal/test	0.024s [no tests to run]
ok  	cmd/compile/internal/types	0.022s
ok  	cmd/cover	2.482s
ok  	cmd/doc	0.086s
ok  	cmd/fix	1.711s
ok  	cmd/go	100.879s
ok  	cmd/go/internal/auth	0.017s
ok  	cmd/go/internal/cache	0.084s
ok  	cmd/go/internal/generate	0.018s
ok  	cmd/go/internal/get	0.023s
ok  	cmd/go/internal/imports	0.017s
ok  	cmd/go/internal/load	0.038s
ok  	cmd/go/internal/lockedfile	0.072s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.025s
ok  	cmd/go/internal/modconv	0.015s
ok  	cmd/go/internal/modfetch	0.013s
ok  	cmd/go/internal/modfetch/codehost	0.026s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.035s
ok  	cmd/go/internal/modload	0.012s
ok  	cmd/go/internal/mvs	0.006s
ok  	cmd/go/internal/par	0.026s
ok  	cmd/go/internal/renameio	0.033s
ok  	cmd/go/internal/search	0.028s
ok  	cmd/go/internal/txtar	0.003s
ok  	cmd/go/internal/web	0.004s
ok  	cmd/go/internal/work	0.012s
ok  	cmd/gofmt	0.102s
ok  	cmd/internal/buildid	0.357s
ok  	cmd/internal/dwarf	0.024s
ok  	cmd/internal/edit	0.002s
ok  	cmd/internal/goobj	0.694s
ok  	cmd/internal/obj	0.005s
ok  	cmd/internal/obj/arm64	0.010s
ok  	cmd/internal/obj/riscv	0.032s
ok  	cmd/internal/obj/x86	0.174s
ok  	cmd/internal/objabi	0.029s
ok  	cmd/internal/src	0.021s
ok  	cmd/internal/test2json	0.097s
ok  	cmd/link	4.282s
ok  	cmd/link/internal/ld	2.472s
ok  	cmd/link/internal/sym	0.004s
ok  	cmd/nm	2.997s
ok  	cmd/objdump	1.710s
ok  	cmd/pack	1.456s
ok  	cmd/trace	0.078s
ok  	cmd/vet	8.857s

##### os/user with tag osusergo
ok  	os/user	0.002s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	29.888s

##### cmd/go terminal test
skipping terminal test; stdout/stderr not terminals

##### moved GOROOT
ok  	fmt	0.076s

##### Testing without libgcc.
ok  	crypto/x509	0.319s
ok  	net	0.009s
ok  	os/user	0.003s

##### internal linking of -buildmode=pie
ok  	reflect	1.061s
ok  	os/user	0.256s

##### sync -cpu=10
ok  	sync	0.275s

##### Testing race detector
ok  	runtime/race	12.446s
ok  	flag	0.061s
ok  	net	0.164s
ok  	os	0.170s
ok  	os/exec	0.076s
ok  	encoding/gob	0.067s
ok  	flag	0.058s
ok  	os/exec	0.070s

##### ../misc/cgo/stdio
PASS

##### ../misc/cgo/life
PASS

##### ../misc/cgo/fortran
PASS
ok  	misc/cgo/fortran	0.003s

##### ../misc/swig/stdio
PASS
ok  	misc/swig/stdio	0.003s

##### ../misc/cgo/test
PASS
ok  	misc/cgo/test	6.241s
PASS
ok  	misc/cgo/test	6.344s
PASS
ok  	misc/cgo/test	6.610s
PASS
ok  	misc/cgo/testtls	0.061s
PASS
ok  	misc/cgo/testtls	0.023s
PASS
ok  	misc/cgo/testtls	0.008s
PASS
ok  	misc/cgo/nocgo	0.025s
PASS
ok  	misc/cgo/nocgo	0.012s
PASS
ok  	misc/cgo/nocgo	0.003s
PASS
ok  	misc/cgo/test	9.160s
PASS
ok  	misc/cgo/test	7.019s
PASS
ok  	misc/cgo/test	4.016s
PASS
ok  	misc/cgo/testtls	0.010s
PASS
ok  	misc/cgo/nocgo	0.023s

##### ../misc/cgo/testgodefs
PASS

##### ../misc/cgo/testso
ok  	misc/cgo/testso	0.725s

##### ../misc/cgo/testsovar
ok  	misc/cgo/testsovar	0.825s

##### ../misc/cgo/testcarchive
PASS

##### ../misc/cgo/testcshared
PASS

##### ../misc/cgo/testshared
ok  	misc/cgo/testshared	33.641s

##### ../misc/cgo/testplugin
ok  	misc/cgo/testplugin	20.391s

##### ../misc/cgo/testasan
ok	testasan	71ms

##### ../misc/cgo/testsanitizers
PASS

##### ../misc/cgo/errors
PASS

##### ../misc/cgo/testsigfwd
ok	testsigfwd

##### ../doc/progs
ok	run	5.842s

##### ../doc/articles/wiki
ok  	doc/articles/wiki	0.962s

##### ../doc/codewalk
ok  	command-line-arguments	1.397s

##### ../test/bench/go1

##### ../test

Note the many repeated "still_waiting_on_test test:7_10" events at the end.

/cc @toothrot @cagedmantis
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 17, 2020
@dmitshur dmitshur added this to the Backlog milestone Jan 17, 2020
@bcmills
Copy link
Contributor

bcmills commented Jan 24, 2020

See also #32032.

@toothrot
Copy link
Contributor

toothrot commented Jan 24, 2020 •
edited

Also got stuck in the same spot in js-wasm.

builder: js-wasm rev: 2c418b7c37335c344fc90c8f4868bb95577f84f8 buildlet: http://10.240.0.6 GCE VM: buildlet-js-wasm-rn6557f9f started: 2020-01-24 18:14:34.902956502 +0000 UTC m=+871893.128356397 status: still running

Events:
2020-01-24T18:14:34Z checking_for_snapshot
2020-01-24T18:14:35Z finish_checking_for_snapshot after 14.4ms
2020-01-24T18:14:35Z get_buildlet
2020-01-24T18:15:22Z finish_get_buildlet after 47.1s
2020-01-24T18:15:22Z using_buildlet 10.240.0.6:80
2020-01-24T18:15:22Z write_snapshot_tar
2020-01-24T18:15:27Z finish_write_snapshot_tar after 4.87s
2020-01-24T18:15:27Z make_and_test
2020-01-24T18:15:27Z get_helper helper 3/5
2020-01-24T18:15:27Z get_helper helper 4/5
2020-01-24T18:15:27Z get_helper helper 1/5
2020-01-24T18:15:27Z get_helper helper 2/5
2020-01-24T18:15:27Z get_helper helper 5/5
2020-01-24T18:15:27Z starting_helpers
2020-01-24T18:15:27Z discovering_tests
2020-01-24T18:15:27Z get_test_stats
2020-01-24T18:15:27Z finish_get_test_stats after 0s
2020-01-24T18:15:27Z starting_tests 175 tests
2020-01-24T18:15:27Z run_tests_multi 10.240.0.6:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
2020-01-24T18:15:36Z finish_run_tests_multi after 8.29s; 10.240.0.6:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
2020-01-24T18:15:36Z run_test:go_test:bytes 10.240.0.6:80
2020-01-24T18:16:06Z still_waiting_on_test go_test:bytes
2020-01-24T18:16:15Z finish_run_test:go_test:bytes after 39.7s; 10.240.0.6:80
2020-01-24T18:16:15Z run_tests_multi 10.240.0.6:80: [go_test:bytes/hash go_test:compress/bzip2 go_test:compress/flate]
2020-01-24T18:16:19Z finish_get_helper after 52.6s; helper 3/5
2020-01-24T18:16:19Z empty_helper_ready 10.240.0.49:80
2020-01-24T18:16:19Z got_empty_test_helper http://10.240.0.49 GCE VM: buildlet-js-wasm-rn1754888
2020-01-24T18:16:20Z finish_get_helper after 53.6s; helper 1/5
2020-01-24T18:16:20Z empty_helper_ready 10.240.0.51:80
2020-01-24T18:16:20Z got_empty_test_helper http://10.240.0.51 GCE VM: buildlet-js-wasm-rn72573f1
2020-01-24T18:16:21Z finish_get_helper after 53.5s; helper 4/5
2020-01-24T18:16:21Z empty_helper_ready 10.240.0.52:80
2020-01-24T18:16:21Z got_empty_test_helper http://10.240.0.52 GCE VM: buildlet-js-wasm-rn86f1b99
2020-01-24T18:16:21Z finish_get_helper after 54.1s; helper 2/5
2020-01-24T18:16:21Z empty_helper_ready 10.240.0.46:80
2020-01-24T18:16:21Z got_empty_test_helper http://10.240.0.46 GCE VM: buildlet-js-wasm-rnca43417
2020-01-24T18:16:23Z finish_run_tests_multi after 7.6s; 10.240.0.6:80: [go_test:bytes/hash go_test:compress/bzip2 go_test:compress/flate]
2020-01-24T18:16:23Z run_tests_multi 10.240.0.6:80: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
2020-01-24T18:16:24Z test_helper_set_up 10.240.0.49:80
2020-01-24T18:16:24Z run_test:reboot 10.240.0.49:80
2020-01-24T18:16:25Z finish_get_helper after 57.9s; helper 5/5
2020-01-24T18:16:25Z empty_helper_ready 10.240.0.53:80
2020-01-24T18:16:25Z got_empty_test_helper http://10.240.0.53 GCE VM: buildlet-js-wasm-rn92fc6bd
2020-01-24T18:16:25Z test_helper_set_up 10.240.0.51:80
2020-01-24T18:16:25Z run_test:test:2_10 10.240.0.51:80
2020-01-24T18:16:25Z test_helper_set_up 10.240.0.52:80
2020-01-24T18:16:25Z run_test:test:8_10 10.240.0.52:80
2020-01-24T18:16:26Z test_helper_set_up 10.240.0.46:80
2020-01-24T18:16:26Z run_test:test:1_10 10.240.0.46:80
2020-01-24T18:16:30Z test_helper_set_up 10.240.0.53:80
2020-01-24T18:16:30Z run_test:test:0_10 10.240.0.53:80
2020-01-24T18:16:31Z finish_run_tests_multi after 7.8s; 10.240.0.6:80: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
2020-01-24T18:16:31Z run_tests_multi 10.240.0.6:80: [go_test:container/heap go_test:container/list go_test:container/ring]
2020-01-24T18:16:35Z finish_run_tests_multi after 4.35s; 10.240.0.6:80: [go_test:container/heap go_test:container/list go_test:container/ring]
2020-01-24T18:16:35Z run_tests_multi 10.240.0.6:80: [go_test:context go_test:crypto go_test:crypto/aes]
2020-01-24T18:16:42Z finish_run_tests_multi after 6.18s; 10.240.0.6:80: [go_test:context go_test:crypto go_test:crypto/aes]
2020-01-24T18:16:42Z run_tests_multi 10.240.0.6:80: [go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
2020-01-24T18:16:46Z finish_run_tests_multi after 4.87s; 10.240.0.6:80: [go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
2020-01-24T18:16:46Z run_tests_multi 10.240.0.6:80: [go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/elliptic]
2020-01-24T18:16:52Z finish_run_tests_multi after 5.6s; 10.240.0.6:80: [go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/elliptic]
2020-01-24T18:16:52Z run_tests_multi 10.240.0.6:80: [go_test:crypto/hmac go_test:crypto/md5 go_test:crypto/rand]
2020-01-24T18:16:57Z finish_run_tests_multi after 4.52s; 10.240.0.6:80: [go_test:crypto/hmac go_test:crypto/md5 go_test:crypto/rand]
2020-01-24T18:16:57Z run_tests_multi 10.240.0.6:80: [go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1]
2020-01-24T18:17:02Z finish_run_tests_multi after 5.33s; 10.240.0.6:80: [go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1]
2020-01-24T18:17:02Z run_tests_multi 10.240.0.6:80: [go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle]
2020-01-24T18:17:07Z finish_run_tests_multi after 4.68s; 10.240.0.6:80: [go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle]
2020-01-24T18:17:07Z run_test:go_test:crypto/tls 10.240.0.6:80
2020-01-24T18:17:19Z finish_run_test:go_test:crypto/tls after 11.8s; 10.240.0.6:80
2020-01-24T18:17:19Z run_tests_multi 10.240.0.6:80: [go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver]
2020-01-24T18:17:25Z finish_run_test:test:0_10 after 55.3s; 10.240.0.53:80
2020-01-24T18:17:25Z run_test:test:4_10 10.240.0.53:80
2020-01-24T18:17:27Z finish_run_test:test:8_10 after 1m1.4s; 10.240.0.52:80
2020-01-24T18:17:27Z run_test:test:5_10 10.240.0.52:80
2020-01-24T18:17:28Z finish_run_test:test:1_10 after 1m1.8s; 10.240.0.46:80
2020-01-24T18:17:28Z run_test:test:6_10 10.240.0.46:80
2020-01-24T18:17:29Z finish_run_tests_multi after 10.5s; 10.240.0.6:80: [go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver]
2020-01-24T18:17:29Z run_tests_multi 10.240.0.6:80: [go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym]
2020-01-24T18:17:36Z finish_run_tests_multi after 6.11s; 10.240.0.6:80: [go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym]
2020-01-24T18:17:36Z run_tests_multi 10.240.0.6:80: [go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85]
2020-01-24T18:17:38Z finish_run_test:test:2_10 after 1m12.5s; 10.240.0.51:80
2020-01-24T18:17:38Z run_test:test:7_10 10.240.0.51:80
2020-01-24T18:17:42Z finish_run_tests_multi after 6.49s; 10.240.0.6:80: [go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85]
2020-01-24T18:17:42Z run_tests_multi 10.240.0.6:80: [go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary]
2020-01-24T18:17:49Z finish_run_tests_multi after 6.42s; 10.240.0.6:80: [go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary]
2020-01-24T18:17:49Z run_tests_multi 10.240.0.6:80: [go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex]
2020-01-24T18:17:54Z finish_run_tests_multi after 5.41s; 10.240.0.6:80: [go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex]
2020-01-24T18:17:54Z run_tests_multi 10.240.0.6:80: [go_test:encoding/json go_test:encoding/pem]
2020-01-24T18:18:01Z finish_run_tests_multi after 7.2s; 10.240.0.6:80: [go_test:encoding/json go_test:encoding/pem]
2020-01-24T18:18:01Z run_tests_multi 10.240.0.6:80: [go_test:encoding/xml go_test:errors go_test:expvar]
2020-01-24T18:18:08Z finish_run_tests_multi after 6.58s; 10.240.0.6:80: [go_test:encoding/xml go_test:errors go_test:expvar]
2020-01-24T18:18:08Z run_tests_multi 10.240.0.6:80: [go_test:flag go_test:fmt go_test:go/ast]
2020-01-24T18:18:12Z finish_run_test:test:5_10 after 45.5s; 10.240.0.52:80
2020-01-24T18:18:12Z run_test:test:3_10 10.240.0.52:80
2020-01-24T18:18:15Z finish_run_tests_multi after 6.6s; 10.240.0.6:80: [go_test:flag go_test:fmt go_test:go/ast]
2020-01-24T18:18:15Z run_tests_multi 10.240.0.6:80: [go_test:go/build go_test:go/constant]
2020-01-24T18:18:15Z finish_run_test:test:6_10 after 47.2s; 10.240.0.46:80
2020-01-24T18:18:15Z run_test:test:9_10 10.240.0.46:80
2020-01-24T18:18:20Z finish_run_test:test:7_10 after 42.5s; 10.240.0.51:80
2020-01-24T18:18:20Z run_test:go_test:runtime 10.240.0.51:80
2020-01-24T18:18:22Z finish_run_tests_multi after 6.94s; 10.240.0.6:80: [go_test:go/build go_test:go/constant]
2020-01-24T18:18:22Z run_tests_multi 10.240.0.6:80: [go_test:go/doc go_test:go/format go_test:go/importer]
2020-01-24T18:18:22Z finish_run_test:reboot after 1m58.2s; 10.240.0.49:80
2020-01-24T18:18:22Z run_test:go_test:strings 10.240.0.49:80
2020-01-24T18:18:29Z finish_run_tests_multi after 7.43s; 10.240.0.6:80: [go_test:go/doc go_test:go/format go_test:go/importer]
2020-01-24T18:18:29Z run_tests_multi 10.240.0.6:80: [go_test:go/parser go_test:go/printer go_test:go/scanner]
2020-01-24T18:18:36Z finish_run_tests_multi after 6.16s; 10.240.0.6:80: [go_test:go/parser go_test:go/printer go_test:go/scanner]
2020-01-24T18:18:36Z run_test:go_test:go/token 10.240.0.6:80
2020-01-24T18:18:38Z finish_run_test:go_test:go/token after 2.82s; 10.240.0.6:80
2020-01-24T18:18:38Z run_tests_multi 10.240.0.6:80: [go_test:go/types go_test:hash]
2020-01-24T18:18:46Z finish_run_tests_multi after 7.24s; 10.240.0.6:80: [go_test:go/types go_test:hash]
2020-01-24T18:18:46Z run_tests_multi 10.240.0.6:80: [go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64 go_test:hash/fnv go_test:html]
2020-01-24T18:18:53Z finish_run_tests_multi after 6.95s; 10.240.0.6:80: [go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64 go_test:hash/fnv go_test:html]
2020-01-24T18:18:53Z run_tests_multi 10.240.0.6:80: [go_test:html/template go_test:image go_test:image/color]
2020-01-24T18:18:54Z finish_run_test:test:3_10 after 41.1s; 10.240.0.52:80
2020-01-24T18:18:54Z run_test:go_test:net/http 10.240.0.52:80
2020-01-24T18:18:54Z finish_run_test:test:9_10 after 39.3s; 10.240.0.46:80
2020-01-24T18:18:54Z run_test:go_test:cmd/compile 10.240.0.46:80
2020-01-24T18:18:58Z finish_run_test:go_test:strings after 35.6s; 10.240.0.49:80
2020-01-24T18:18:58Z run_test:go_test:math/big 10.240.0.49:80
2020-01-24T18:18:59Z finish_run_tests_multi after 6.5s; 10.240.0.6:80: [go_test:html/template go_test:image go_test:image/color]
2020-01-24T18:18:59Z run_tests_multi 10.240.0.6:80: [go_test:image/draw go_test:image/gif go_test:image/jpeg]
2020-01-24T18:19:01Z finish_run_test:go_test:runtime after 41.1s; 10.240.0.51:80
2020-01-24T18:19:01Z run_test:go_test:cmd/go 10.240.0.51:80
2020-01-24T18:19:04Z finish_run_test:go_test:math/big after 6.15s; 10.240.0.49:80
2020-01-24T18:19:04Z run_test:go_test:net/http/pprof 10.240.0.49:80
2020-01-24T18:19:06Z finish_run_tests_multi after 6.26s; 10.240.0.6:80: [go_test:image/draw go_test:image/gif go_test:image/jpeg]
2020-01-24T18:19:06Z run_tests_multi 10.240.0.6:80: [go_test:image/png go_test:index/suffixarray]
2020-01-24T18:19:12Z finish_run_tests_multi after 6.09s; 10.240.0.6:80: [go_test:image/png go_test:index/suffixarray]
2020-01-24T18:19:12Z run_tests_multi 10.240.0.6:80: [go_test:internal/cpu go_test:internal/fmtsort]
2020-01-24T18:19:12Z finish_run_test:go_test:net/http/pprof after 7.81s; 10.240.0.49:80
2020-01-24T18:19:12Z run_test:go_test:time 10.240.0.49:80
2020-01-24T18:19:15Z finish_run_test:go_test:cmd/go after 13.2s; 10.240.0.51:80
2020-01-24T18:19:15Z run_test:go_test:reflect 10.240.0.51:80
2020-01-24T18:19:20Z finish_run_tests_multi after 7.72s; 10.240.0.6:80: [go_test:internal/cpu go_test:internal/fmtsort]
2020-01-24T18:19:20Z run_tests_multi 10.240.0.6:80: [go_test:internal/poll go_test:internal/reflectlite]
2020-01-24T18:19:21Z finish_run_test:go_test:time after 8.43s; 10.240.0.49:80
2020-01-24T18:19:21Z run_test:go_test:cmd/link 10.240.0.49:80
2020-01-24T18:19:21Z finish_run_test:go_test:reflect after 6.63s; 10.240.0.51:80
2020-01-24T18:19:21Z run_test:go_test:net/rpc 10.240.0.51:80
2020-01-24T18:19:24Z finish_run_test:go_test:cmd/compile after 29.9s; 10.240.0.46:80
2020-01-24T18:19:24Z run_test:go_test:net/http/httptrace 10.240.0.46:80
2020-01-24T18:19:24Z finish_run_test:go_test:net/http after 30.7s; 10.240.0.52:80
2020-01-24T18:19:24Z run_test:go_test:cmd/vet 10.240.0.52:80
2020-01-24T18:19:26Z finish_run_test:go_test:net/rpc after 4.93s; 10.240.0.51:80
2020-01-24T18:19:26Z run_test:go_test:sync 10.240.0.51:80
2020-01-24T18:19:27Z finish_run_tests_multi after 6.68s; 10.240.0.6:80: [go_test:internal/poll go_test:internal/reflectlite]
2020-01-24T18:19:27Z run_tests_multi 10.240.0.6:80: [go_test:internal/singleflight go_test:internal/trace go_test:internal/xcoff go_test:io]
2020-01-24T18:19:27Z finish_run_test:go_test:cmd/link after 6.49s; 10.240.0.49:80
2020-01-24T18:19:27Z run_test:go_test:strconv 10.240.0.49:80
2020-01-24T18:19:30Z finish_run_test:go_test:net/http/httptrace after 5.53s; 10.240.0.46:80
2020-01-24T18:19:30Z run_test:go_test:net/rpc/jsonrpc 10.240.0.46:80
2020-01-24T18:19:31Z finish_run_test:go_test:cmd/vet after 6.07s; 10.240.0.52:80
2020-01-24T18:19:31Z run_test:go_test:os/exec 10.240.0.52:80
2020-01-24T18:19:32Z finish_run_test:go_test:sync after 5.09s; 10.240.0.51:80
2020-01-24T18:19:32Z run_test:go_test:net/http/httputil 10.240.0.51:80
2020-01-24T18:19:33Z finish_run_test:go_test:strconv after 5.03s; 10.240.0.49:80
2020-01-24T18:19:33Z run_test:go_test:cmd/objdump 10.240.0.49:80
2020-01-24T18:19:33Z finish_run_tests_multi after 6.43s; 10.240.0.6:80: [go_test:internal/singleflight go_test:internal/trace go_test:internal/xcoff go_test:io]
2020-01-24T18:19:33Z run_tests_multi 10.240.0.6:80: [go_test:io/ioutil go_test:log go_test:log/syslog go_test:math]
2020-01-24T18:19:35Z finish_run_test:go_test:net/rpc/jsonrpc after 4.74s; 10.240.0.46:80
2020-01-24T18:19:35Z run_test:go_test:cmd/nm 10.240.0.46:80
2020-01-24T18:19:35Z finish_run_test:go_test:os/exec after 4.13s; 10.240.0.52:80
2020-01-24T18:19:35Z run_test:go_test:os 10.240.0.52:80
2020-01-24T18:19:36Z finish_run_test:go_test:cmd/objdump after 2.99s; 10.240.0.49:80
2020-01-24T18:19:36Z run_test:go_test:net/http/httptest 10.240.0.49:80
2020-01-24T18:19:36Z finish_run_test:go_test:net/http/httputil after 4.29s; 10.240.0.51:80
2020-01-24T18:19:36Z run_test:go_test:net/http/cgi 10.240.0.51:80
2020-01-24T18:19:39Z finish_run_test:go_test:cmd/nm after 3.92s; 10.240.0.46:80
2020-01-24T18:19:39Z run_test:go_test:net/http/cookiejar 10.240.0.46:80
2020-01-24T18:19:39Z finish_run_test:go_test:os after 4.08s; 10.240.0.52:80
2020-01-24T18:19:39Z run_test:go_test:cmd/api 10.240.0.52:80
2020-01-24T18:19:40Z finish_run_test:go_test:net/http/httptest after 3.89s; 10.240.0.49:80
2020-01-24T18:19:40Z run_test:go_test:net/http/fcgi 10.240.0.49:80
2020-01-24T18:19:40Z finish_run_test:go_test:net/http/cgi after 4.07s; 10.240.0.51:80
2020-01-24T18:19:40Z run_test:go_test:syscall/js 10.240.0.51:80
2020-01-24T18:19:41Z finish_run_tests_multi after 7.36s; 10.240.0.6:80: [go_test:io/ioutil go_test:log go_test:log/syslog go_test:math]
2020-01-24T18:19:41Z run_tests_multi 10.240.0.6:80: [go_test:math/bits go_test:math/cmplx go_test:math/rand]
2020-01-24T18:19:42Z finish_run_test:go_test:syscall/js after 1.98s; 10.240.0.51:80
2020-01-24T18:19:42Z run_test:go_test:cmd/doc 10.240.0.51:80
2020-01-24T18:19:43Z finish_run_test:go_test:cmd/api after 3.52s; 10.240.0.52:80
2020-01-24T18:19:43Z run_test:go_test:sort 10.240.0.52:80
2020-01-24T18:19:43Z finish_run_test:go_test:net/http/cookiejar after 3.73s; 10.240.0.46:80
2020-01-24T18:19:43Z run_test:go_test:regexp/syntax 10.240.0.46:80
2020-01-24T18:19:43Z finish_run_test:go_test:net/http/fcgi after 3.6s; 10.240.0.49:80
2020-01-24T18:19:43Z run_test:go_test:net/smtp 10.240.0.49:80
2020-01-24T18:19:46Z finish_run_test:go_test:cmd/doc after 3.29s; 10.240.0.51:80
2020-01-24T18:19:46Z run_test:go_test:mime/multipart 10.240.0.51:80
2020-01-24T18:19:46Z finish_run_test:go_test:regexp/syntax after 3.15s; 10.240.0.46:80
2020-01-24T18:19:46Z run_test:go_test:net 10.240.0.46:80
2020-01-24T18:19:46Z finish_run_test:go_test:sort after 3.41s; 10.240.0.52:80
2020-01-24T18:19:46Z run_test:go_test:cmd/cover 10.240.0.52:80
2020-01-24T18:19:46Z finish_run_tests_multi after 5.85s; 10.240.0.6:80: [go_test:math/bits go_test:math/cmplx go_test:math/rand]
2020-01-24T18:19:46Z run_tests_multi 10.240.0.6:80: [go_test:mime go_test:mime/quotedprintable]
2020-01-24T18:19:47Z finish_run_test:go_test:net/smtp after 3.14s; 10.240.0.49:80
2020-01-24T18:19:47Z run_test:go_test:text/template 10.240.0.49:80
2020-01-24T18:19:49Z finish_run_test:go_test:mime/multipart after 3.02s; 10.240.0.51:80
2020-01-24T18:19:49Z run_test:go_test:path/filepath 10.240.0.51:80
2020-01-24T18:19:49Z finish_run_test:go_test:net after 3s; 10.240.0.46:80
2020-01-24T18:19:49Z run_test:go_test:regexp 10.240.0.46:80
2020-01-24T18:19:49Z finish_run_test:go_test:cmd/cover after 3.02s; 10.240.0.52:80
2020-01-24T18:19:49Z run_test:go_test:testing 10.240.0.52:80
2020-01-24T18:19:50Z finish_run_test:go_test:text/template after 2.96s; 10.240.0.49:80
2020-01-24T18:19:50Z run_test:go_test:cmd/fix 10.240.0.49:80
2020-01-24T18:19:50Z finish_run_tests_multi after 3.45s; 10.240.0.6:80: [go_test:mime go_test:mime/quotedprintable]
2020-01-24T18:19:50Z run_test:go_test:net/http/internal 10.240.0.6:80
2020-01-24T18:19:52Z finish_run_test:go_test:path/filepath after 2.72s; 10.240.0.51:80
2020-01-24T18:19:52Z run_test:go_test:syscall 10.240.0.51:80
2020-01-24T18:19:52Z finish_run_test:go_test:regexp after 2.69s; 10.240.0.46:80
2020-01-24T18:19:52Z run_test:go_test:cmd/gofmt 10.240.0.46:80
2020-01-24T18:19:52Z finish_run_test:go_test:testing after 2.57s; 10.240.0.52:80
2020-01-24T18:19:52Z run_test:go_test:cmd/addr2line 10.240.0.52:80
2020-01-24T18:19:52Z finish_run_test:go_test:net/http/internal after 1.91s; 10.240.0.6:80
2020-01-24T18:19:52Z run_test:go_test:net/mail 10.240.0.6:80
2020-01-24T18:19:52Z finish_run_test:go_test:cmd/fix after 2.57s; 10.240.0.49:80
2020-01-24T18:19:52Z run_test:go_test:runtime/pprof 10.240.0.49:80
2020-01-24T18:19:54Z finish_run_test:go_test:net/mail after 2.15s; 10.240.0.6:80
2020-01-24T18:19:54Z run_test:go_test:net/textproto 10.240.0.6:80
2020-01-24T18:19:54Z finish_run_test:go_test:syscall after 2.6s; 10.240.0.51:80
2020-01-24T18:19:54Z run_test:go_test:net/url 10.240.0.51:80
2020-01-24T18:19:54Z finish_run_test:go_test:cmd/gofmt after 2.57s; 10.240.0.46:80
2020-01-24T18:19:54Z run_test:go_test:runtime/trace 10.240.0.46:80
2020-01-24T18:19:55Z finish_run_test:go_test:runtime/pprof after 2.78s; 10.240.0.49:80
2020-01-24T18:19:55Z run_test:go_test:text/template/parse 10.240.0.49:80
2020-01-24T18:19:56Z finish_run_test:go_test:cmd/addr2line after 3.71s; 10.240.0.52:80
2020-01-24T18:19:56Z run_test:go_test:testing/quick 10.240.0.52:80
2020-01-24T18:19:56Z finish_run_test:go_test:net/textproto after 2.06s; 10.240.0.6:80
2020-01-24T18:19:56Z run_tests_multi 10.240.0.6:80: [go_test:os/signal go_test:os/user]
2020-01-24T18:19:57Z finish_run_test:go_test:net/url after 2.6s; 10.240.0.51:80
2020-01-24T18:19:57Z run_test:nolibgcc:net 10.240.0.51:80
2020-01-24T18:19:57Z finish_run_test:go_test:runtime/trace after 2.57s; 10.240.0.46:80
2020-01-24T18:19:57Z run_test:go_test:cmd/pack 10.240.0.46:80
2020-01-24T18:19:58Z finish_run_test:go_test:text/template/parse after 2.39s; 10.240.0.49:80
2020-01-24T18:19:58Z run_test:go_test:plugin 10.240.0.49:80
2020-01-24T18:19:58Z finish_run_test:go_test:testing/quick after 2.33s; 10.240.0.52:80
2020-01-24T18:19:58Z run_test:go_test:sync/atomic 10.240.0.52:80
2020-01-24T18:19:59Z finish_run_test:go_test:cmd/pack after 2.06s; 10.240.0.46:80
2020-01-24T18:19:59Z run_test:go_test:text/scanner 10.240.0.46:80
2020-01-24T18:20:00Z finish_run_tests_multi after 3.07s; 10.240.0.6:80: [go_test:os/signal go_test:os/user]
2020-01-24T18:20:00Z run_test:go_test:path 10.240.0.6:80
2020-01-24T18:20:00Z finish_run_test:go_test:plugin after 2.03s; 10.240.0.49:80
2020-01-24T18:20:00Z run_test:go_test:unicode 10.240.0.49:80
2020-01-24T18:20:00Z finish_run_test:nolibgcc:net after 2.71s; 10.240.0.51:80
2020-01-24T18:20:00Z run_test:go_test:unicode/utf8 10.240.0.51:80
2020-01-24T18:20:00Z finish_run_test:go_test:sync/atomic after 2.1s; 10.240.0.52:80
2020-01-24T18:20:00Z run_test:go_test:text/tabwriter 10.240.0.52:80
2020-01-24T18:20:01Z finish_run_test:go_test:text/scanner after 2.03s; 10.240.0.46:80
2020-01-24T18:20:01Z run_test:go_test:runtime/debug 10.240.0.46:80
2020-01-24T18:20:02Z finish_run_test:go_test:path after 1.95s; 10.240.0.6:80
2020-01-24T18:20:02Z run_test:go_test:unicode/utf16 10.240.0.6:80
2020-01-24T18:20:02Z finish_run_test:go_test:unicode after 1.98s; 10.240.0.49:80
2020-01-24T18:20:02Z run_test:osusergo 10.240.0.49:80
2020-01-24T18:20:02Z finish_run_test:go_test:unicode/utf8 after 1.99s; 10.240.0.51:80
2020-01-24T18:20:02Z run_test:nolibgcc:os/user 10.240.0.51:80
2020-01-24T18:20:02Z finish_run_test:go_test:text/tabwriter after 1.97s; 10.240.0.52:80
2020-01-24T18:20:02Z run_test:bench_go1 10.240.0.52:80
2020-01-24T18:20:03Z finish_run_test:go_test:runtime/debug after 1.93s; 10.240.0.46:80
2020-01-24T18:20:03Z no_new_tests_remain 10.240.0.46:80
2020-01-24T18:20:03Z closed_helper 10.240.0.46:80
2020-01-24T18:20:04Z finish_run_test:go_test:unicode/utf16 after 1.9s; 10.240.0.6:80
2020-01-24T18:20:04Z finish_run_test:bench_go1 after 1.31s; 10.240.0.52:80
2020-01-24T18:20:04Z no_new_tests_remain 10.240.0.52:80
2020-01-24T18:20:04Z closed_helper 10.240.0.52:80
2020-01-24T18:20:04Z finish_run_test:osusergo after 1.9s; 10.240.0.49:80
2020-01-24T18:20:04Z no_new_tests_remain 10.240.0.49:80
2020-01-24T18:20:04Z closed_helper 10.240.0.49:80
2020-01-24T18:20:04Z finish_run_test:nolibgcc:os/user after 1.93s; 10.240.0.51:80
2020-01-24T18:20:04Z no_new_tests_remain 10.240.0.51:80
2020-01-24T18:20:04Z closed_helper 10.240.0.51:80
2020-01-24T18:20:34Z still_waiting_on_test test:4_10
2020-01-24T18:21:04Z still_waiting_on_test test:4_10
2020-01-24T18:21:34Z still_waiting_on_test test:4_10
2020-01-24T18:22:04Z still_waiting_on_test test:4_10
2020-01-24T18:22:34Z still_waiting_on_test test:4_10
2020-01-24T18:23:04Z still_waiting_on_test test:4_10
2020-01-24T18:23:34Z still_waiting_on_test test:4_10
2020-01-24T18:24:04Z still_waiting_on_test test:4_10
2020-01-24T18:24:34Z still_waiting_on_test test:4_10
2020-01-24T18:25:04Z still_waiting_on_test test:4_10
2020-01-24T18:25:34Z still_waiting_on_test test:4_10
2020-01-24T18:26:04Z still_waiting_on_test test:4_10
2020-01-24T18:26:34Z still_waiting_on_test test:4_10
2020-01-24T18:27:04Z still_waiting_on_test test:4_10
2020-01-24T18:27:34Z still_waiting_on_test test:4_10
2020-01-24T18:28:04Z still_waiting_on_test test:4_10
2020-01-24T18:28:34Z still_waiting_on_test test:4_10
2020-01-24T18:29:04Z still_waiting_on_test test:4_10
2020-01-24T18:29:34Z still_waiting_on_test test:4_10
2020-01-24T18:30:04Z still_waiting_on_test test:4_10
2020-01-24T18:30:34Z still_waiting_on_test test:4_10
2020-01-24T18:31:04Z still_waiting_on_test test:4_10
2020-01-24T18:31:34Z still_waiting_on_test test:4_10
2020-01-24T18:32:04Z still_waiting_on_test test:4_10
2020-01-24T18:32:34Z still_waiting_on_test test:4_10
2020-01-24T18:33:04Z still_waiting_on_test test:4_10
2020-01-24T18:33:34Z still_waiting_on_test test:4_10
2020-01-24T18:34:04Z still_waiting_on_test test:4_10
2020-01-24T18:34:34Z still_waiting_on_test test:4_10
2020-01-24T18:35:04Z still_waiting_on_test test:4_10
2020-01-24T18:35:34Z still_waiting_on_test test:4_10
2020-01-24T18:36:04Z still_waiting_on_test test:4_10
2020-01-24T18:36:34Z still_waiting_on_test test:4_10
2020-01-24T18:37:04Z still_waiting_on_test test:4_10
2020-01-24T18:37:25Z finish_run_test:test:4_10 after 20m0s; err=error copying response: context deadline exceeded; 10.240.0.53:80
2020-01-24T18:37:25Z test_helper_is_broken 10.240.0.53:80
2020-01-24T18:37:25Z closed_helper 10.240.0.53:80
2020-01-24T18:37:29Z run_test:test:4_10 10.240.0.6:80
2020-01-24T18:37:34Z still_waiting_on_test test:4_10
2020-01-24T18:38:04Z still_waiting_on_test test:4_10
2020-01-24T18:38:34Z still_waiting_on_test test:4_10
2020-01-24T18:39:04Z still_waiting_on_test test:4_10
2020-01-24T18:39:34Z still_waiting_on_test test:4_10
2020-01-24T18:40:04Z still_waiting_on_test test:4_10
2020-01-24T18:40:34Z still_waiting_on_test test:4_10
2020-01-24T18:41:04Z still_waiting_on_test test:4_10
2020-01-24T18:41:34Z still_waiting_on_test test:4_10
2020-01-24T18:42:04Z still_waiting_on_test test:4_10
2020-01-24T18:42:34Z still_waiting_on_test test:4_10
2020-01-24T18:43:04Z still_waiting_on_test test:4_10
2020-01-24T18:43:34Z still_waiting_on_test test:4_10
2020-01-24T18:44:04Z still_waiting_on_test test:4_10
2020-01-24T18:44:34Z still_waiting_on_test test:4_10
2020-01-24T18:45:04Z still_waiting_on_test test:4_10
2020-01-24T18:45:34Z still_waiting_on_test test:4_10
2020-01-24T18:46:04Z still_waiting_on_test test:4_10
2020-01-24T18:46:34Z still_waiting_on_test test:4_10
2020-01-24T18:47:04Z still_waiting_on_test test:4_10
2020-01-24T18:47:34Z still_waiting_on_test test:4_10
2020-01-24T18:48:04Z still_waiting_on_test test:4_10
2020-01-24T18:48:34Z still_waiting_on_test test:4_10
2020-01-24T18:49:04Z still_waiting_on_test test:4_10
2020-01-24T18:49:34Z still_waiting_on_test test:4_10
2020-01-24T18:50:04Z still_waiting_on_test test:4_10
2020-01-24T18:50:34Z still_waiting_on_test test:4_10
2020-01-24T18:51:04Z still_waiting_on_test test:4_10
2020-01-24T18:51:34Z still_waiting_on_test test:4_10
2020-01-24T18:52:04Z still_waiting_on_test test:4_10
2020-01-24T18:52:34Z still_waiting_on_test test:4_10
2020-01-24T18:53:04Z still_waiting_on_test test:4_10
+24.2s (now)

Build log:
js-wasm at 2c418b7c37335c344fc90c8f4868bb95577f84f8

Testing packages.

ok archive/tar 4.143s
ok archive/zip 4.970s
ok bufio 4.073s
ok bytes 37.805s
ok bytes/hash 3.685s
ok compress/bzip2 3.377s
ok compress/flate 5.248s
ok compress/gzip 6.357s
ok compress/lzw 3.801s
ok compress/zlib 5.227s
ok container/heap 3.308s
ok container/list 3.263s
ok container/ring 3.189s
ok context 3.987s
ok crypto 3.209s
ok crypto/aes 3.183s
ok crypto/cipher 3.416s
ok crypto/des 3.348s
ok crypto/dsa 3.452s
ok crypto/ecdsa 4.357s
ok crypto/ed25519 3.953s
ok crypto/elliptic 3.569s
ok crypto/hmac 3.174s
ok crypto/md5 3.385s
ok crypto/rand 3.434s
ok crypto/rc4 3.341s
ok crypto/rsa 4.041s
ok crypto/sha1 3.416s
ok crypto/sha256 3.552s
ok crypto/sha512 3.273s
ok crypto/subtle 3.626s
ok crypto/tls 7.768s
ok crypto/x509 7.777s
ok database/sql 5.461s
ok database/sql/driver 2.893s
ok debug/dwarf 3.636s
ok debug/elf 4.115s
ok debug/gosym 3.290s
ok debug/macho 4.392s
ok debug/pe 5.149s
ok debug/plan9obj 4.103s
ok encoding/ascii85 4.331s
ok encoding/asn1 4.611s
ok encoding/base32 4.386s
ok encoding/base64 4.429s
ok encoding/binary 3.662s
ok encoding/csv 2.945s
ok encoding/gob 2.611s
ok encoding/hex 3.173s
ok encoding/json 4.609s
ok encoding/pem 2.801s
ok encoding/xml 3.488s
ok errors 2.891s
ok expvar 4.853s
ok flag 2.935s
ok fmt 3.640s
ok go/ast 3.244s
ok go/build 5.424s
ok go/constant 2.265s
ok go/doc 5.367s
ok go/format 3.862s
ok go/importer 5.548s
ok go/parser 3.771s
ok go/printer 4.602s
ok go/scanner 3.330s
ok go/token 1.424s
ok go/types 3.676s
ok hash 2.301s
ok hash/adler32 4.335s
ok hash/crc32 4.173s
ok hash/crc64 4.159s
ok hash/fnv 4.301s
ok html 1.086s
ok html/template 4.068s
ok image 3.773s
ok image/color 3.494s
ok image/draw 4.225s
ok image/gif 4.827s
ok image/jpeg 4.533s
ok image/png 2.536s
ok index/suffixarray 4.930s
ok internal/cpu 1.106s
ok internal/fmtsort 2.616s
ok internal/poll 2.238s
ok internal/reflectlite 1.530s
ok internal/singleflight 3.881s
ok internal/trace 4.756s
ok internal/xcoff 5.035s
ok io 4.274s
ok io/ioutil 4.096s
ok log 4.138s
ok log/syslog 4.108s [no tests to run]
ok math 1.298s
ok math/big 3.990s
ok math/bits 3.290s
ok math/cmplx 3.235s
ok math/rand 4.358s
ok mime 2.309s
ok mime/quotedprintable 2.189s
ok mime/multipart 2.016s
ok net 1.612s
ok net/http 24.494s
ok net/http/cgi 2.747s
ok net/http/cookiejar 2.537s
ok net/http/fcgi 2.448s
ok net/http/httptest 2.684s
ok net/http/httptrace 2.658s
ok net/http/httputil 2.873s
ok net/http/internal 1.087s
ok net/http/pprof 5.490s
ok net/mail 1.228s
ok net/rpc 3.498s
ok net/rpc/jsonrpc 3.279s
ok net/smtp 2.025s
ok net/textproto 1.168s
ok net/url 1.517s
ok os 1.888s
ok os/exec 2.627s
ok os/signal 1.990s [no tests to run]
ok os/user 2.142s
ok path 1.113s
ok path/filepath 1.551s
ok plugin 1.228s
ok reflect 2.951s
ok regexp 1.493s
ok regexp/syntax 2.012s
ok runtime 34.133s
ok runtime/debug 1.122s
ok runtime/pprof 1.716s
ok runtime/trace 1.604s
ok sort 1.347s
ok strconv 2.454s
ok strings 33.016s
ok sync 1.815s
ok sync/atomic 1.196s
ok syscall 1.099s
ok syscall/js 1.108s
ok testing 1.458s
ok testing/quick 1.448s
ok text/scanner 1.132s
ok text/tabwriter 1.138s
ok text/template 1.715s
ok text/template/parse 1.310s
ok time 5.276s
ok unicode 1.106s
ok unicode/utf16 1.093s
ok unicode/utf8 1.148s
ok cmd/addr2line 1.493s
ok cmd/api 2.443s
ok cmd/compile 6.684s
ok cmd/cover 1.838s
ok cmd/doc 2.227s
ok cmd/fix 1.523s
ok cmd/go 5.562s
ok cmd/gofmt 1.642s
ok cmd/link 1.799s
ok cmd/nm 1.853s
ok cmd/objdump 1.933s
ok cmd/pack 1.187s
ok cmd/vet 2.913s

os/user with tag osusergo

ok os/user 1.087s

Testing without libgcc.

ok net 1.239s [no tests to run]
ok os/user 1.100s

../test/bench/go1
../test

@bcmills bcmills mentioned this issue Feb 25, 2020
Closed
@mdempsky
Copy link
Member

mdempsky commented Jun 18, 2021 •
edited

Edit: The trybot did end up passing.

At https://farmer.golang.org/try?commit=1ac5913e, I'm seeing what seems to be a similar issue with the linux-amd64-unified trybot. It's stuck on testsanitizers instead though.

https://farmer.golang.org/temporarylogs?name=linux-amd64-unified&rev=1ac5913e048627d9f53f58029f946f94f1790dcb&st=0xc00a2ca000

Details 
  builder: linux-amd64-unified
      rev: 1ac5913e048627d9f53f58029f946f94f1790dcb
 buildlet: http://10.240.0.52 GCE VM: buildlet-linux-buster-rnb9bd1c3
  started: 2021-06-18 10:53:50.897680462 +0000 UTC m=+52421.000005289
   status: still running

Events:
  2021-06-18T10:53:50Z ask_maintner_has_ancestor 
  2021-06-18T10:53:50Z finish_ask_maintner_has_ancestor after 900µs
  2021-06-18T10:53:51Z checking_for_snapshot 
  2021-06-18T10:53:51Z finish_checking_for_snapshot after 19.7ms
  2021-06-18T10:53:51Z get_buildlet 
  2021-06-18T10:54:30Z finish_get_buildlet after 39.1s
  2021-06-18T10:54:30Z using_buildlet 10.240.0.52:80
  2021-06-18T10:54:30Z write_version_tar 
  2021-06-18T10:54:30Z get_source go@1ac5913e048627d9f53f58029f946f94f1790dcb
  2021-06-18T10:54:31Z finish_get_source after 1.08s; go@1ac5913e048627d9f53f58029f946f94f1790dcb
  2021-06-18T10:54:31Z write_go_src_tar 
  2021-06-18T10:54:32Z finish_write_go_src_tar after 1.17s
  2021-06-18T10:54:32Z make_and_test 
  2021-06-18T10:54:32Z make src/make.bash
  2021-06-18T10:54:32Z starting_helpers 
  2021-06-18T10:54:32Z get_helper helper 1/4
  2021-06-18T10:54:32Z get_helper helper 3/4
  2021-06-18T10:54:32Z get_helper helper 4/4
  2021-06-18T10:54:32Z get_helper helper 2/4
  2021-06-18T10:55:10Z finish_get_helper after 37.8s; helper 1/4
  2021-06-18T10:55:10Z empty_helper_ready 10.240.0.136:80
  2021-06-18T10:55:11Z finish_get_helper after 38.4s; helper 3/4
  2021-06-18T10:55:11Z empty_helper_ready 10.240.0.133:80
  2021-06-18T10:55:12Z finish_get_helper after 39.8s; helper 4/4
  2021-06-18T10:55:12Z empty_helper_ready 10.240.0.130:80
  2021-06-18T10:55:13Z finish_get_helper after 40.3s; helper 2/4
  2021-06-18T10:55:13Z empty_helper_ready 10.240.0.166:80
  2021-06-18T10:57:53Z finish_make after 3m20.8s; src/make.bash
  2021-06-18T10:57:53Z clean_for_snapshot 
  2021-06-18T10:57:53Z finish_clean_for_snapshot after 800µs
  2021-06-18T10:57:53Z write_snapshot_to_gcs 
  2021-06-18T10:57:53Z fetch_snapshot_reader_from_buildlet 
  2021-06-18T10:57:53Z finish_fetch_snapshot_reader_from_buildlet after 54.9ms
  2021-06-18T10:58:04Z finish_write_snapshot_to_gcs after 10.7s
  2021-06-18T10:58:04Z discovering_tests 
  2021-06-18T10:58:04Z get_test_stats 
  2021-06-18T10:58:04Z finish_get_test_stats after 0s
  2021-06-18T10:58:04Z starting_tests 278 tests
  2021-06-18T10:58:04Z got_empty_test_helper http://10.240.0.166 GCE VM: buildlet-linux-buster-rn21fce0a
  2021-06-18T10:58:04Z got_empty_test_helper http://10.240.0.130 GCE VM: buildlet-linux-buster-rnee94e3c
  2021-06-18T10:58:04Z got_empty_test_helper http://10.240.0.133 GCE VM: buildlet-linux-buster-rnd75988c
  2021-06-18T10:58:04Z run_tests_multi 10.240.0.52:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2021-06-18T10:58:04Z got_empty_test_helper http://10.240.0.136 GCE VM: buildlet-linux-buster-rnea6bda7
  2021-06-18T10:58:09Z finish_run_tests_multi after 4.5s; 10.240.0.52:80: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2021-06-18T10:58:09Z run_tests_multi 10.240.0.52:80: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate go_test:compress/gzip]
  2021-06-18T10:58:09Z test_helper_set_up 10.240.0.136:80
  2021-06-18T10:58:09Z run_test:reboot 10.240.0.136:80
  2021-06-18T10:58:10Z test_helper_set_up 10.240.0.133:80
  2021-06-18T10:58:10Z run_test:go_test:cmd/go 10.240.0.133:80
  2021-06-18T10:58:10Z test_helper_set_up 10.240.0.130:80
  2021-06-18T10:58:10Z run_test:go_test:runtime 10.240.0.130:80
  2021-06-18T10:58:10Z test_helper_set_up 10.240.0.166:80
  2021-06-18T10:58:10Z run_test:cgo_test 10.240.0.166:80
  2021-06-18T10:58:14Z finish_run_tests_multi after 4.85s; 10.240.0.52:80: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate go_test:compress/gzip]
  2021-06-18T10:58:14Z run_tests_multi 10.240.0.52:80: [go_test:compress/lzw go_test:compress/zlib go_test:container/heap go_test:container/list go_test:container/ring]
  2021-06-18T10:58:16Z finish_run_tests_multi after 1.82s; 10.240.0.52:80: [go_test:compress/lzw go_test:compress/zlib go_test:container/heap go_test:container/list go_test:container/ring]
  2021-06-18T10:58:16Z run_tests_multi 10.240.0.52:80: [go_test:context go_test:crypto go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des]
  2021-06-18T10:58:20Z finish_run_tests_multi after 3.8s; 10.240.0.52:80: [go_test:context go_test:crypto go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des]
  2021-06-18T10:58:20Z run_tests_multi 10.240.0.52:80: [go_test:crypto/dsa go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/ed25519/internal/edwards25519]
  2021-06-18T10:58:26Z finish_run_tests_multi after 5.85s; 10.240.0.52:80: [go_test:crypto/dsa go_test:crypto/ecdsa go_test:crypto/ed25519 go_test:crypto/ed25519/internal/edwards25519]
  2021-06-18T10:58:26Z run_tests_multi 10.240.0.52:80: [go_test:crypto/ed25519/internal/edwards25519/field go_test:crypto/elliptic go_test:crypto/elliptic/internal/fiat]
  2021-06-18T10:58:28Z finish_run_tests_multi after 2.78s; 10.240.0.52:80: [go_test:crypto/ed25519/internal/edwards25519/field go_test:crypto/elliptic go_test:crypto/elliptic/internal/fiat]
  2021-06-18T10:58:28Z run_tests_multi 10.240.0.52:80: [go_test:crypto/hmac go_test:crypto/internal/subtle go_test:crypto/md5 go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256]
  2021-06-18T10:58:31Z finish_run_tests_multi after 2.74s; 10.240.0.52:80: [go_test:crypto/hmac go_test:crypto/internal/subtle go_test:crypto/md5 go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256]
  2021-06-18T10:58:31Z run_tests_multi 10.240.0.52:80: [go_test:crypto/sha512 go_test:crypto/subtle go_test:crypto/tls]
  2021-06-18T10:58:37Z finish_run_tests_multi after 6.04s; 10.240.0.52:80: [go_test:crypto/sha512 go_test:crypto/subtle go_test:crypto/tls]
  2021-06-18T10:58:37Z run_tests_multi 10.240.0.52:80: [go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf]
  2021-06-18T10:58:42Z finish_run_tests_multi after 4.85s; 10.240.0.52:80: [go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf]
  2021-06-18T10:58:42Z run_tests_multi 10.240.0.52:80: [go_test:debug/elf go_test:debug/gosym go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:embed/internal/embedtest go_test:encoding/ascii85 go_test:encoding/asn1]
  2021-06-18T10:58:45Z finish_run_tests_multi after 2.79s; 10.240.0.52:80: [go_test:debug/elf go_test:debug/gosym go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:embed/internal/embedtest go_test:encoding/ascii85 go_test:encoding/asn1]
  2021-06-18T10:58:45Z run_tests_multi 10.240.0.52:80: [go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex go_test:encoding/json]
  2021-06-18T10:58:49Z finish_run_tests_multi after 4.24s; 10.240.0.52:80: [go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex go_test:encoding/json]
  2021-06-18T10:58:49Z run_tests_multi 10.240.0.52:80: [go_test:encoding/pem go_test:encoding/xml]
  2021-06-18T10:58:51Z finish_run_tests_multi after 2.03s; 10.240.0.52:80: [go_test:encoding/pem go_test:encoding/xml]
  2021-06-18T10:58:51Z run_tests_multi 10.240.0.52:80: [go_test:errors go_test:expvar go_test:flag go_test:fmt go_test:go/ast]
  2021-06-18T10:58:56Z finish_run_tests_multi after 4.38s; 10.240.0.52:80: [go_test:errors go_test:expvar go_test:flag go_test:fmt go_test:go/ast]
  2021-06-18T10:58:56Z run_tests_multi 10.240.0.52:80: [go_test:go/build go_test:go/build/constraint go_test:go/constant go_test:go/doc go_test:go/format]
  2021-06-18T10:58:59Z finish_run_tests_multi after 3.11s; 10.240.0.52:80: [go_test:go/build go_test:go/build/constraint go_test:go/constant go_test:go/doc go_test:go/format]
  2021-06-18T10:58:59Z run_tests_multi 10.240.0.52:80: [go_test:go/importer go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter]
  2021-06-18T10:59:02Z finish_run_tests_multi after 3.02s; 10.240.0.52:80: [go_test:go/importer go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter]
  2021-06-18T10:59:02Z run_tests_multi 10.240.0.52:80: [go_test:go/internal/srcimporter go_test:go/parser]
  2021-06-18T10:59:06Z finish_run_test:go_test:runtime after 56s; 10.240.0.130:80
  2021-06-18T10:59:06Z run_test:testshared 10.240.0.130:80
  2021-06-18T10:59:08Z finish_run_test:cgo_test after 58.5s; 10.240.0.166:80
  2021-06-18T10:59:08Z run_test:test:4_10 10.240.0.166:80
  2021-06-18T10:59:09Z finish_run_tests_multi after 7.39s; 10.240.0.52:80: [go_test:go/internal/srcimporter go_test:go/parser]
  2021-06-18T10:59:09Z run_tests_multi 10.240.0.52:80: [go_test:go/printer go_test:go/scanner go_test:go/token]
  2021-06-18T10:59:12Z finish_run_tests_multi after 2.46s; 10.240.0.52:80: [go_test:go/printer go_test:go/scanner go_test:go/token]
  2021-06-18T10:59:12Z run_tests_multi 10.240.0.52:80: [go_test:go/types go_test:hash go_test:hash/adler32 go_test:hash/crc32]
  2021-06-18T10:59:16Z finish_run_test:go_test:cmd/go after 1m6.1s; 10.240.0.133:80
  2021-06-18T10:59:16Z run_test:test:7_10 10.240.0.133:80
  2021-06-18T10:59:18Z finish_run_tests_multi after 5.54s; 10.240.0.52:80: [go_test:go/types go_test:hash go_test:hash/adler32 go_test:hash/crc32]
  2021-06-18T10:59:18Z run_tests_multi 10.240.0.52:80: [go_test:hash/crc64 go_test:hash/fnv go_test:hash/maphash go_test:html go_test:html/template]
  2021-06-18T10:59:20Z finish_run_tests_multi after 2.53s; 10.240.0.52:80: [go_test:hash/crc64 go_test:hash/fnv go_test:hash/maphash go_test:html go_test:html/template]
  2021-06-18T10:59:20Z run_tests_multi 10.240.0.52:80: [go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2021-06-18T10:59:23Z finish_run_tests_multi after 2.9s; 10.240.0.52:80: [go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2021-06-18T10:59:23Z run_tests_multi 10.240.0.52:80: [go_test:image/png go_test:index/suffixarray]
  2021-06-18T10:59:25Z finish_run_tests_multi after 1.5s; 10.240.0.52:80: [go_test:image/png go_test:index/suffixarray]
  2021-06-18T10:59:25Z run_test:go_test:internal/abi 10.240.0.52:80
  2021-06-18T10:59:34Z finish_run_test:go_test:internal/abi after 9.41s; 10.240.0.52:80
  2021-06-18T10:59:34Z run_tests_multi 10.240.0.52:80: [go_test:internal/cpu go_test:internal/execabs]
  2021-06-18T10:59:43Z finish_run_tests_multi after 8.79s; 10.240.0.52:80: [go_test:internal/cpu go_test:internal/execabs]
  2021-06-18T10:59:43Z run_tests_multi 10.240.0.52:80: [go_test:internal/fmtsort go_test:internal/itoa go_test:internal/poll go_test:internal/profile]
  2021-06-18T10:59:49Z finish_run_tests_multi after 5.45s; 10.240.0.52:80: [go_test:internal/fmtsort go_test:internal/itoa go_test:internal/poll go_test:internal/profile]
  2021-06-18T10:59:49Z run_tests_multi 10.240.0.52:80: [go_test:internal/reflectlite go_test:internal/singleflight go_test:internal/trace go_test:internal/unsafeheader]
  2021-06-18T10:59:51Z finish_run_test:testshared after 44.9s; 10.240.0.130:80
  2021-06-18T10:59:51Z run_test:runtime:cpu124 10.240.0.130:80
  2021-06-18T10:59:55Z finish_run_tests_multi after 6.63s; 10.240.0.52:80: [go_test:internal/reflectlite go_test:internal/singleflight go_test:internal/trace go_test:internal/unsafeheader]
  2021-06-18T10:59:55Z run_tests_multi 10.240.0.52:80: [go_test:internal/xcoff go_test:io go_test:io/fs]
  2021-06-18T10:59:56Z finish_run_test:test:7_10 after 40s; 10.240.0.133:80
  2021-06-18T10:59:56Z run_test:test:2_10 10.240.0.133:80
  2021-06-18T10:59:56Z finish_run_test:test:4_10 after 47.8s; 10.240.0.166:80
  2021-06-18T10:59:56Z run_test:test:5_10 10.240.0.166:80
  2021-06-18T10:59:59Z finish_run_tests_multi after 3.8s; 10.240.0.52:80: [go_test:internal/xcoff go_test:io go_test:io/fs]
  2021-06-18T10:59:59Z run_tests_multi 10.240.0.52:80: [go_test:io/ioutil go_test:log go_test:log/syslog]
  2021-06-18T11:00:02Z finish_run_tests_multi after 2.5s; 10.240.0.52:80: [go_test:io/ioutil go_test:log go_test:log/syslog]
  2021-06-18T11:00:02Z run_tests_multi 10.240.0.52:80: [go_test:math go_test:math/big go_test:math/bits]
  2021-06-18T11:00:10Z finish_run_tests_multi after 8.06s; 10.240.0.52:80: [go_test:math go_test:math/big go_test:math/bits]
  2021-06-18T11:00:10Z run_tests_multi 10.240.0.52:80: [go_test:math/cmplx go_test:math/rand go_test:mime go_test:mime/multipart go_test:mime/quotedprintable]
  2021-06-18T11:00:12Z finish_run_tests_multi after 2.68s; 10.240.0.52:80: [go_test:math/cmplx go_test:math/rand go_test:mime go_test:mime/multipart go_test:mime/quotedprintable]
  2021-06-18T11:00:12Z run_test:go_test:net 10.240.0.52:80
  2021-06-18T11:00:17Z finish_run_test:runtime:cpu124 after 26s; 10.240.0.130:80
  2021-06-18T11:00:17Z run_test:race 10.240.0.130:80
  2021-06-18T11:00:19Z finish_run_test:test:2_10 after 23.3s; 10.240.0.133:80
  2021-06-18T11:00:19Z run_test:test:0_10 10.240.0.133:80
  2021-06-18T11:00:27Z finish_run_test:go_test:net after 14.8s; 10.240.0.52:80
  2021-06-18T11:00:27Z run_test:go_test:net/http 10.240.0.52:80
  2021-06-18T11:00:30Z finish_run_test:test:5_10 after 33.5s; 10.240.0.166:80
  2021-06-18T11:00:30Z run_test:test:1_10 10.240.0.166:80
  2021-06-18T11:00:37Z finish_run_test:reboot after 2m27.2s; 10.240.0.136:80
  2021-06-18T11:00:37Z run_test:testsanitizers 10.240.0.136:80
  2021-06-18T11:00:42Z finish_run_test:go_test:net/http after 14.7s; 10.240.0.52:80
  2021-06-18T11:00:42Z run_tests_multi 10.240.0.52:80: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2021-06-18T11:00:45Z finish_run_tests_multi after 2.36s; 10.240.0.52:80: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2021-06-18T11:00:45Z run_tests_multi 10.240.0.52:80: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal go_test:net/http/internal/ascii]
  2021-06-18T11:00:46Z finish_run_test:test:0_10 after 26.6s; 10.240.0.133:80
  2021-06-18T11:00:46Z run_test:testplugin 10.240.0.133:80
  2021-06-18T11:00:48Z finish_run_tests_multi after 3.59s; 10.240.0.52:80: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal go_test:net/http/internal/ascii]
  2021-06-18T11:00:48Z run_tests_multi 10.240.0.52:80: [go_test:net/http/pprof go_test:net/internal/socktest go_test:net/mail]
  2021-06-18T11:00:55Z finish_run_tests_multi after 6.59s; 10.240.0.52:80: [go_test:net/http/pprof go_test:net/internal/socktest go_test:net/mail]
  2021-06-18T11:00:55Z run_tests_multi 10.240.0.52:80: [go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto go_test:net/url go_test:os]
  2021-06-18T11:00:56Z finish_run_test:test:1_10 after 26.6s; 10.240.0.166:80
  2021-06-18T11:00:56Z run_test:test:8_10 10.240.0.166:80
  2021-06-18T11:01:01Z finish_run_tests_multi after 5.78s; 10.240.0.52:80: [go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto go_test:net/url go_test:os]
  2021-06-18T11:01:01Z run_tests_multi 10.240.0.52:80: [go_test:os/exec go_test:os/signal go_test:os/user go_test:path]
  2021-06-18T11:01:03Z finish_run_test:race after 45.9s; 10.240.0.130:80
  2021-06-18T11:01:03Z run_test:test:9_10 10.240.0.130:80
  2021-06-18T11:01:05Z finish_run_tests_multi after 4.12s; 10.240.0.52:80: [go_test:os/exec go_test:os/signal go_test:os/user go_test:path]
  2021-06-18T11:01:05Z run_tests_multi 10.240.0.52:80: [go_test:path/filepath go_test:plugin go_test:reflect]
  2021-06-18T11:01:11Z finish_run_tests_multi after 5.92s; 10.240.0.52:80: [go_test:path/filepath go_test:plugin go_test:reflect]
  2021-06-18T11:01:11Z run_tests_multi 10.240.0.52:80: [go_test:regexp go_test:regexp/syntax]
  2021-06-18T11:01:13Z finish_run_tests_multi after 2.05s; 10.240.0.52:80: [go_test:regexp go_test:regexp/syntax]
  2021-06-18T11:01:13Z run_tests_multi 10.240.0.52:80: [go_test:runtime/cgo go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math go_test:runtime/internal/sys go_test:runtime/metrics]
  2021-06-18T11:01:13Z finish_run_test:testplugin after 27.3s; 10.240.0.133:80
  2021-06-18T11:01:13Z run_test:testcarchive 10.240.0.133:80
  2021-06-18T11:01:15Z finish_run_tests_multi after 1.87s; 10.240.0.52:80: [go_test:runtime/cgo go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math go_test:runtime/internal/sys go_test:runtime/metrics]
  2021-06-18T11:01:15Z run_tests_multi 10.240.0.52:80: [go_test:runtime/pprof go_test:runtime/race go_test:runtime/trace]
  2021-06-18T11:01:19Z finish_run_test:test:8_10 after 22.3s; 10.240.0.166:80
  2021-06-18T11:01:19Z run_test:go_test:cmd/compile/internal/test 10.240.0.166:80
  2021-06-18T11:01:21Z finish_run_tests_multi after 6.44s; 10.240.0.52:80: [go_test:runtime/pprof go_test:runtime/race go_test:runtime/trace]
  2021-06-18T11:01:21Z run_tests_multi 10.240.0.52:80: [go_test:sort go_test:strconv go_test:strings]
  2021-06-18T11:01:30Z finish_run_tests_multi after 8.4s; 10.240.0.52:80: [go_test:sort go_test:strconv go_test:strings]
  2021-06-18T11:01:30Z run_tests_multi 10.240.0.52:80: [go_test:sync go_test:sync/atomic]
  2021-06-18T11:01:34Z finish_run_test:testcarchive after 20.7s; 10.240.0.133:80
  2021-06-18T11:01:34Z run_test:test:6_10 10.240.0.133:80
  2021-06-18T11:01:36Z finish_run_tests_multi after 6.48s; 10.240.0.52:80: [go_test:sync go_test:sync/atomic]
  2021-06-18T11:01:36Z run_tests_multi 10.240.0.52:80: [go_test:syscall go_test:testing go_test:testing/fstest go_test:testing/iotest go_test:testing/quick]
  2021-06-18T11:01:39Z finish_run_test:test:9_10 after 36.1s; 10.240.0.130:80
  2021-06-18T11:01:39Z run_test:go_test:cmd/compile/internal/ssa 10.240.0.130:80
  2021-06-18T11:01:39Z finish_run_test:go_test:cmd/compile/internal/test after 20.6s; 10.240.0.166:80
  2021-06-18T11:01:39Z run_test:test:3_10 10.240.0.166:80
  2021-06-18T11:01:43Z finish_run_tests_multi after 6.9s; 10.240.0.52:80: [go_test:syscall go_test:testing go_test:testing/fstest go_test:testing/iotest go_test:testing/quick]
  2021-06-18T11:01:43Z run_tests_multi 10.240.0.52:80: [go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse]
  2021-06-18T11:01:46Z finish_run_tests_multi after 2.23s; 10.240.0.52:80: [go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse]
  2021-06-18T11:01:46Z run_tests_multi 10.240.0.52:80: [go_test:time go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8]
  2021-06-18T11:01:52Z finish_run_tests_multi after 6.68s; 10.240.0.52:80: [go_test:time go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8]
  2021-06-18T11:01:52Z run_tests_multi 10.240.0.52:80: [go_test:cmd/addr2line go_test:cmd/api]
  2021-06-18T11:01:54Z finish_run_test:test:6_10 after 20.3s; 10.240.0.133:80
  2021-06-18T11:01:54Z run_test:tyepparams 10.240.0.133:80
  2021-06-18T11:01:55Z finish_run_test:test:3_10 after 15.5s; 10.240.0.166:80
  2021-06-18T11:01:55Z run_test:go_test:cmd/link 10.240.0.166:80
  2021-06-18T11:01:57Z finish_run_test:go_test:cmd/compile/internal/ssa after 18s; 10.240.0.130:80
  2021-06-18T11:01:57Z run_test:go_test:cmd/vet 10.240.0.130:80
  2021-06-18T11:02:01Z finish_run_tests_multi after 8.74s; 10.240.0.52:80: [go_test:cmd/addr2line go_test:cmd/api]
  2021-06-18T11:02:01Z run_tests_multi 10.240.0.52:80: [go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex go_test:cmd/compile/internal/dwarfgen]
  2021-06-18T11:02:05Z finish_run_test:go_test:cmd/link after 10.2s; 10.240.0.166:80
  2021-06-18T11:02:05Z run_test:testcshared 10.240.0.166:80
  2021-06-18T11:02:06Z finish_run_test:go_test:cmd/vet after 9.18s; 10.240.0.130:80
  2021-06-18T11:02:06Z run_test:pie_internal 10.240.0.130:80
  2021-06-18T11:02:07Z finish_run_test:tyepparams after 12.6s; 10.240.0.133:80
  2021-06-18T11:02:07Z run_test:go_test:cmd/link/internal/ld 10.240.0.133:80
  2021-06-18T11:02:07Z finish_run_tests_multi after 6.05s; 10.240.0.52:80: [go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex go_test:cmd/compile/internal/dwarfgen]
  2021-06-18T11:02:07Z run_tests_multi 10.240.0.52:80: [go_test:cmd/compile/internal/importer go_test:cmd/compile/internal/ir go_test:cmd/compile/internal/logopt go_test:cmd/compile/internal/noder]
  2021-06-18T11:02:12Z finish_run_tests_multi after 5.16s; 10.240.0.52:80: [go_test:cmd/compile/internal/importer go_test:cmd/compile/internal/ir go_test:cmd/compile/internal/logopt go_test:cmd/compile/internal/noder]
  2021-06-18T11:02:12Z run_tests_multi 10.240.0.52:80: [go_test:cmd/compile/internal/ssagen go_test:cmd/compile/internal/syntax]
  2021-06-18T11:02:13Z finish_run_test:pie_internal after 6.43s; 10.240.0.130:80
  2021-06-18T11:02:13Z run_test:cgo_errors 10.240.0.130:80
  2021-06-18T11:02:13Z finish_run_test:testcshared after 7.54s; 10.240.0.166:80
  2021-06-18T11:02:13Z run_test:sync_cpu 10.240.0.166:80
  2021-06-18T11:02:14Z finish_run_test:go_test:cmd/link/internal/ld after 6.79s; 10.240.0.133:80
  2021-06-18T11:02:14Z run_test:go_test:cmd/compile/internal/types2 10.240.0.133:80
  2021-06-18T11:02:15Z finish_run_tests_multi after 2.98s; 10.240.0.52:80: [go_test:cmd/compile/internal/ssagen go_test:cmd/compile/internal/syntax]
  2021-06-18T11:02:15Z run_tests_multi 10.240.0.52:80: [go_test:cmd/compile/internal/typecheck go_test:cmd/compile/internal/types]
  2021-06-18T11:02:18Z finish_run_test:cgo_errors after 5.38s; 10.240.0.130:80
  2021-06-18T11:02:18Z run_test:nolibgcc:net 10.240.0.130:80
  2021-06-18T11:02:18Z finish_run_test:go_test:cmd/compile/internal/types2 after 4.31s; 10.240.0.133:80
  2021-06-18T11:02:18Z run_test:cgo_stdio 10.240.0.133:80
  2021-06-18T11:02:18Z finish_run_tests_multi after 2.77s; 10.240.0.52:80: [go_test:cmd/compile/internal/typecheck go_test:cmd/compile/internal/types]
  2021-06-18T11:02:18Z run_tests_multi 10.240.0.52:80: [go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]
  2021-06-18T11:02:18Z finish_run_test:sync_cpu after 5.4s; 10.240.0.166:80
  2021-06-18T11:02:18Z run_test:go_test:cmd/objdump 10.240.0.166:80
  2021-06-18T11:02:21Z finish_run_test:go_test:cmd/objdump after 2.89s; 10.240.0.166:80
  2021-06-18T11:02:21Z run_test:moved_goroot 10.240.0.166:80
  2021-06-18T11:02:22Z finish_run_test:cgo_stdio after 3.68s; 10.240.0.133:80
  2021-06-18T11:02:22Z run_test:go_test:cmd/trace 10.240.0.133:80
  2021-06-18T11:02:23Z finish_run_tests_multi after 4.39s; 10.240.0.52:80: [go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]
  2021-06-18T11:02:23Z run_tests_multi 10.240.0.52:80: [go_test:cmd/go/internal/auth go_test:cmd/go/internal/cache go_test:cmd/go/internal/fsys go_test:cmd/go/internal/generate go_test:cmd/go/internal/get go_test:cmd/go/internal/imports go_test:cmd/go/internal/load go_test:cmd/go/internal/lockedfile]
  2021-06-18T11:02:23Z finish_run_test:nolibgcc:net after 4.51s; 10.240.0.130:80
  2021-06-18T11:02:23Z run_test:cgo_life 10.240.0.130:80
  2021-06-18T11:02:24Z finish_run_test:go_test:cmd/trace after 1.8s; 10.240.0.133:80
  2021-06-18T11:02:24Z run_test:go_test:cmd/go/internal/test 10.240.0.133:80
  2021-06-18T11:02:24Z finish_run_test:moved_goroot after 2.97s; 10.240.0.166:80
  2021-06-18T11:02:24Z run_test:pie_internal_cgo 10.240.0.166:80
  2021-06-18T11:02:25Z finish_run_test:go_test:cmd/go/internal/test after 1.38s; 10.240.0.133:80
  2021-06-18T11:02:25Z run_test:go_test:cmd/go/internal/work 10.240.0.133:80
  2021-06-18T11:02:25Z finish_run_test:cgo_life after 2.74s; 10.240.0.130:80
  2021-06-18T11:02:25Z run_test:go_test:cmd/internal/obj/x86 10.240.0.130:80
  2021-06-18T11:02:26Z finish_run_test:pie_internal_cgo after 1.53s; 10.240.0.166:80
  2021-06-18T11:02:26Z run_test:go_test:cmd/nm 10.240.0.166:80
  2021-06-18T11:02:26Z finish_run_tests_multi after 3.47s; 10.240.0.52:80: [go_test:cmd/go/internal/auth go_test:cmd/go/internal/cache go_test:cmd/go/internal/fsys go_test:cmd/go/internal/generate go_test:cmd/go/internal/get go_test:cmd/go/internal/imports go_test:cmd/go/internal/load go_test:cmd/go/internal/lockedfile]
  2021-06-18T11:02:26Z run_tests_multi 10.240.0.52:80: [go_test:cmd/go/internal/lockedfile/internal/filelock go_test:cmd/go/internal/modconv go_test:cmd/go/internal/modfetch go_test:cmd/go/internal/modfetch/codehost go_test:cmd/go/internal/modfetch/zip_sum_test go_test:cmd/go/internal/modload go_test:cmd/go/internal/mvs go_test:cmd/go/internal/par]
  2021-06-18T11:02:27Z finish_run_test:go_test:cmd/go/internal/work after 1.93s; 10.240.0.133:80
  2021-06-18T11:02:27Z run_test:go_test:cmd/pack 10.240.0.133:80
  2021-06-18T11:02:28Z finish_run_test:go_test:cmd/internal/obj/x86 after 2.39s; 10.240.0.130:80
  2021-06-18T11:02:28Z run_test:cgo_fortran 10.240.0.130:80
  2021-06-18T11:02:28Z finish_run_test:go_test:cmd/nm after 2.34s; 10.240.0.166:80
  2021-06-18T11:02:28Z run_test:go_test:cmd/internal/moddeps 10.240.0.166:80
  2021-06-18T11:02:29Z finish_run_test:go_test:cmd/pack after 1.94s; 10.240.0.133:80
  2021-06-18T11:02:29Z run_test:testsovar 10.240.0.133:80
  2021-06-18T11:02:30Z finish_run_tests_multi after 3.53s; 10.240.0.52:80: [go_test:cmd/go/internal/lockedfile/internal/filelock go_test:cmd/go/internal/modconv go_test:cmd/go/internal/modfetch go_test:cmd/go/internal/modfetch/codehost go_test:cmd/go/internal/modfetch/zip_sum_test go_test:cmd/go/internal/modload go_test:cmd/go/internal/mvs go_test:cmd/go/internal/par]
  2021-06-18T11:02:30Z run_tests_multi 10.240.0.52:80: [go_test:cmd/go/internal/search go_test:cmd/go/internal/str go_test:cmd/go/internal/txtar go_test:cmd/go/internal/vcs go_test:cmd/go/internal/web]
  2021-06-18T11:02:30Z finish_run_test:cgo_fortran after 1.85s; 10.240.0.130:80
  2021-06-18T11:02:30Z run_test:testso 10.240.0.130:80
  2021-06-18T11:02:30Z finish_run_test:go_test:cmd/internal/moddeps after 1.77s; 10.240.0.166:80
  2021-06-18T11:02:30Z run_test:nolibgcc:os/user 10.240.0.166:80
  2021-06-18T11:02:31Z finish_run_test:testsovar after 1.55s; 10.240.0.133:80
  2021-06-18T11:02:31Z run_test:go_test:cmd/internal/obj/ppc64 10.240.0.133:80
  2021-06-18T11:02:31Z finish_run_test:testso after 1.59s; 10.240.0.130:80
  2021-06-18T11:02:31Z run_test:testasan 10.240.0.130:80
  2021-06-18T11:02:32Z finish_run_tests_multi after 1.82s; 10.240.0.52:80: [go_test:cmd/go/internal/search go_test:cmd/go/internal/str go_test:cmd/go/internal/txtar go_test:cmd/go/internal/vcs go_test:cmd/go/internal/web]
  2021-06-18T11:02:32Z run_tests_multi 10.240.0.52:80: [go_test:cmd/gofmt go_test:cmd/internal/archive go_test:cmd/internal/buildid go_test:cmd/internal/dwarf go_test:cmd/internal/edit go_test:cmd/internal/goobj]
  2021-06-18T11:02:32Z finish_run_test:nolibgcc:os/user after 1.5s; 10.240.0.166:80
  2021-06-18T11:02:32Z run_test:bench_go1 10.240.0.166:80
  2021-06-18T11:02:32Z finish_run_test:go_test:cmd/internal/obj/ppc64 after 1.51s; 10.240.0.133:80
  2021-06-18T11:02:32Z run_test:go_test:cmd/link/internal/loader 10.240.0.133:80
  2021-06-18T11:02:33Z finish_run_test:testasan after 1.37s; 10.240.0.130:80
  2021-06-18T11:02:33Z run_test:testgodefs 10.240.0.130:80
  2021-06-18T11:02:34Z finish_run_test:go_test:cmd/link/internal/loader after 1.26s; 10.240.0.133:80
  2021-06-18T11:02:34Z run_test:go_test:cmd/internal/obj/s390x 10.240.0.133:80
  2021-06-18T11:02:34Z finish_run_tests_multi after 2.19s; 10.240.0.52:80: [go_test:cmd/gofmt go_test:cmd/internal/archive go_test:cmd/internal/buildid go_test:cmd/internal/dwarf go_test:cmd/internal/edit go_test:cmd/internal/goobj]
  2021-06-18T11:02:34Z run_tests_multi 10.240.0.52:80: [go_test:cmd/internal/obj go_test:cmd/internal/obj/riscv]
  2021-06-18T11:02:34Z finish_run_test:bench_go1 after 2.49s; 10.240.0.166:80
  2021-06-18T11:02:34Z run_test:testsigfwd 10.240.0.166:80
  2021-06-18T11:02:34Z finish_run_test:testgodefs after 1.44s; 10.240.0.130:80
  2021-06-18T11:02:34Z run_test:osusergo 10.240.0.130:80
  2021-06-18T11:02:35Z finish_run_test:go_test:cmd/internal/obj/s390x after 1.41s; 10.240.0.133:80
  2021-06-18T11:02:35Z run_test:go_test:cmd/internal/test2json 10.240.0.133:80
  2021-06-18T11:02:35Z finish_run_test:osusergo after 889.6ms; 10.240.0.130:80
  2021-06-18T11:02:35Z run_test:go_test:cmd/internal/src 10.240.0.130:80
  2021-06-18T11:02:35Z finish_run_test:testsigfwd after 1.15s; 10.240.0.166:80
  2021-06-18T11:02:35Z run_test:go_test:cmd/link/internal/benchmark 10.240.0.166:80
  2021-06-18T11:02:36Z finish_run_tests_multi after 1.64s; 10.240.0.52:80: [go_test:cmd/internal/obj go_test:cmd/internal/obj/riscv]
  2021-06-18T11:02:36Z run_tests_multi 10.240.0.52:80: [go_test:cmd/internal/objabi go_test:cmd/internal/pkgpath go_test:cmd/internal/sys]
  2021-06-18T11:02:36Z finish_run_test:go_test:cmd/internal/test2json after 945ms; 10.240.0.133:80
  2021-06-18T11:02:36Z run_test:cmd_go_test_terminal 10.240.0.133:80
  2021-06-18T11:02:36Z finish_run_test:go_test:cmd/internal/src after 979.3ms; 10.240.0.130:80
  2021-06-18T11:02:36Z no_new_tests_remain 10.240.0.130:80
  2021-06-18T11:02:36Z closed_helper 10.240.0.130:80
  2021-06-18T11:02:36Z finish_run_test:go_test:cmd/link/internal/benchmark after 859ms; 10.240.0.166:80
  2021-06-18T11:02:36Z no_new_tests_remain 10.240.0.166:80
  2021-06-18T11:02:36Z closed_helper 10.240.0.166:80
  2021-06-18T11:02:37Z finish_run_test:cmd_go_test_terminal after 555.6ms; 10.240.0.133:80
  2021-06-18T11:02:37Z no_new_tests_remain 10.240.0.133:80
  2021-06-18T11:02:37Z closed_helper 10.240.0.133:80
  2021-06-18T11:02:37Z finish_run_tests_multi after 1.15s; 10.240.0.52:80: [go_test:cmd/internal/objabi go_test:cmd/internal/pkgpath go_test:cmd/internal/sys]
  2021-06-18T11:03:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:03:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:04:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:04:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:05:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:05:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:06:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:06:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:07:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:07:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:08:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:08:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:09:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:09:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:10:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:10:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:11:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:11:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:12:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:12:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:13:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:13:37Z still_waiting_on_test testsanitizers
  2021-06-18T11:14:07Z still_waiting_on_test testsanitizers
  2021-06-18T11:14:37Z still_waiting_on_test testsanitizers
   +1.3s (now)

Build log:
linux-amd64-unified at 1ac5913e048627d9f53f58029f946f94f1790dcb

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-buster-rnb9bd1c3" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=0s" "GO_STAGE0_DL_DELAY=300ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-amd64-unified" "GOROOT_BOOTSTRAP=/go1.4" "GO_DISABLE_OUTBOUND_NETWORK=1" "GOEXPERIMENT=unified" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache"] in dir /workdir/go/src

Building Go cmd/dist using /go1.4. (go1.4 linux/amd64)
Building Go toolchain1 using /go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /workdir/go
Installed commands in /workdir/go/bin

##### Testing packages.
ok  	archive/tar	0.020s
ok  	archive/zip	0.552s
ok  	bufio	0.069s
ok  	bytes	1.234s
ok  	compress/bzip2	0.127s
ok  	compress/flate	0.750s
ok  	compress/gzip	0.064s
ok  	compress/lzw	0.094s
ok  	compress/zlib	0.474s
ok  	container/heap	0.010s
ok  	container/list	0.004s
ok  	container/ring	0.003s
ok  	context	0.011s
ok  	crypto	0.003s
ok  	crypto/aes	0.045s
ok  	crypto/cipher	0.007s
ok  	crypto/des	0.010s
ok  	crypto/dsa	0.025s
ok  	crypto/ecdsa	0.011s
ok  	crypto/ed25519	0.069s
ok  	crypto/ed25519/internal/edwards25519	3.811s
ok  	crypto/ed25519/internal/edwards25519/field	1.605s
ok  	crypto/elliptic	0.100s
ok  	crypto/elliptic/internal/fiat	0.014s
ok  	crypto/hmac	0.033s
ok  	crypto/internal/subtle	0.009s
ok  	crypto/md5	0.016s
ok  	crypto/rand	0.070s
ok  	crypto/rc4	0.071s
ok  	crypto/rsa	0.069s
ok  	crypto/sha1	0.029s
ok  	crypto/sha256	0.018s
ok  	crypto/sha512	0.029s
ok  	crypto/subtle	0.022s
ok  	crypto/tls	0.335s
ok  	crypto/x509	0.672s
ok  	database/sql	0.632s
ok  	database/sql/driver	0.036s
ok  	debug/dwarf	0.011s
ok  	debug/elf	0.036s
ok  	debug/gosym	0.029s
ok  	debug/macho	0.021s
ok  	debug/pe	0.011s
ok  	debug/plan9obj	0.014s
ok  	embed/internal/embedtest	0.014s
ok  	encoding/ascii85	0.021s
ok  	encoding/asn1	0.004s
ok  	encoding/base32	0.051s
ok  	encoding/base64	0.006s
ok  	encoding/binary	0.024s
ok  	encoding/csv	0.043s
ok  	encoding/gob	0.031s
ok  	encoding/hex	0.031s
ok  	encoding/json	0.165s
ok  	encoding/pem	0.012s
ok  	encoding/xml	0.018s
ok  	errors	0.017s
ok  	expvar	0.028s
ok  	flag	0.113s
ok  	fmt	0.114s
ok  	go/ast	0.005s
ok  	go/build	1.000s
ok  	go/build/constraint	0.018s
ok  	go/constant	0.034s
ok  	go/doc	0.031s
ok  	go/format	0.006s
ok  	go/importer	0.155s
ok  	go/internal/gccgoimporter	0.012s
ok  	go/internal/gcimporter	1.140s
ok  	go/internal/srcimporter	5.129s
ok  	go/parser	0.027s
ok  	go/printer	0.274s
ok  	go/scanner	0.018s
ok  	go/token	0.012s
ok  	go/types	1.215s
ok  	hash	0.003s
ok  	hash/adler32	0.013s
ok  	hash/crc32	0.006s
ok  	hash/crc64	0.016s
ok  	hash/fnv	0.003s
ok  	hash/maphash	0.081s
ok  	html	0.003s
ok  	html/template	0.077s
ok  	image	0.101s
ok  	image/color	0.047s
ok  	image/draw	0.078s
ok  	image/gif	0.394s
ok  	image/jpeg	0.149s
ok  	image/png	0.026s
ok  	index/suffixarray	0.149s
ok  	internal/abi	0.025s
ok  	internal/cpu	0.008s
ok  	internal/execabs	0.003s
ok  	internal/fmtsort	0.027s
ok  	internal/itoa	0.039s
ok  	internal/poll	0.206s
ok  	internal/profile	0.006s
ok  	internal/reflectlite	0.039s
ok  	internal/singleflight	0.013s
ok  	internal/trace	0.073s
ok  	internal/unsafeheader	0.045s
ok  	internal/xcoff	0.040s
ok  	io	0.025s
ok  	io/fs	0.018s
ok  	io/ioutil	0.005s
ok  	log	0.004s
ok  	log/syslog	1.213s
ok  	math	0.006s
ok  	math/big	2.130s
ok  	math/bits	0.006s
ok  	math/cmplx	0.010s
ok  	math/rand	0.166s
ok  	mime	0.016s
ok  	mime/multipart	0.206s
ok  	mime/quotedprintable	0.009s
ok  	net	9.991s
ok  	net/http	7.789s
ok  	net/http/cgi	0.257s
ok  	net/http/cookiejar	0.010s
ok  	net/http/fcgi	0.106s
ok  	net/http/httptest	0.025s
ok  	net/http/httptrace	0.003s
ok  	net/http/httputil	0.062s
ok  	net/http/internal	0.023s
ok  	net/http/internal/ascii	0.036s
ok  	net/http/pprof	5.085s
ok  	net/internal/socktest	0.009s
ok  	net/mail	0.005s
ok  	net/rpc	0.061s
ok  	net/rpc/jsonrpc	0.020s
ok  	net/smtp	0.061s
ok  	net/textproto	0.018s
ok  	net/url	0.031s
ok  	os	0.708s
ok  	os/exec	0.752s
ok  	os/signal	1.935s
ok  	os/user	0.016s
ok  	path	0.011s
ok  	path/filepath	0.019s
ok  	plugin	0.008s
ok  	reflect	0.153s
ok  	regexp	0.054s
ok  	regexp/syntax	0.236s
ok  	runtime	43.925s
ok  	runtime/cgo	0.003s
ok  	runtime/debug	0.050s
ok  	runtime/internal/atomic	0.022s
ok  	runtime/internal/math	0.015s
ok  	runtime/internal/sys	0.003s
ok  	runtime/metrics	0.002s
ok  	runtime/pprof	4.666s
ok  	runtime/race	0.402s
ok  	runtime/trace	0.560s
ok  	sort	0.125s
ok  	strconv	0.584s
ok  	strings	0.761s
ok  	sync	0.276s
ok  	sync/atomic	2.171s
ok  	syscall	0.062s
ok  	testing	1.193s
ok  	testing/fstest	0.015s
ok  	testing/iotest	0.017s
ok  	testing/quick	0.056s
ok  	text/scanner	0.019s
ok  	text/tabwriter	0.027s
ok  	text/template	0.036s
ok  	text/template/parse	0.011s
ok  	time	1.709s
ok  	unicode	0.007s
ok  	unicode/utf16	0.002s
ok  	unicode/utf8	0.006s
ok  	cmd/addr2line	2.221s
ok  	cmd/api	6.970s
ok  	cmd/asm/internal/asm	0.523s
ok  	cmd/asm/internal/lex	0.003s
ok  	cmd/compile/internal/dwarfgen	0.231s
ok  	cmd/compile/internal/importer	3.273s
ok  	cmd/compile/internal/ir	0.007s
ok  	cmd/compile/internal/logopt	0.914s
ok  	cmd/compile/internal/noder	0.004s
ok  	cmd/compile/internal/ssa	0.329s
ok  	cmd/compile/internal/ssagen	0.003s
ok  	cmd/compile/internal/syntax	0.029s
ok  	cmd/compile/internal/test	14.933s
ok  	cmd/compile/internal/typecheck	0.271s
ok  	cmd/compile/internal/types	0.003s
ok  	cmd/compile/internal/types2	0.887s
ok  	cmd/cover	2.373s
ok  	cmd/doc	0.165s
ok  	cmd/fix	2.523s
ok  	cmd/go	60.347s
ok  	cmd/go/internal/auth	0.013s
ok  	cmd/go/internal/cache	0.052s
ok  	cmd/go/internal/fsys	0.019s
ok  	cmd/go/internal/generate	0.044s
ok  	cmd/go/internal/get	0.019s
ok  	cmd/go/internal/imports	0.023s
ok  	cmd/go/internal/load	0.010s
ok  	cmd/go/internal/lockedfile	0.064s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.076s
ok  	cmd/go/internal/modconv	0.032s
ok  	cmd/go/internal/modfetch	0.032s
ok  	cmd/go/internal/modfetch/codehost	0.024s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.026s
ok  	cmd/go/internal/modload	0.005s
ok  	cmd/go/internal/mvs	0.035s
ok  	cmd/go/internal/par	0.020s
ok  	cmd/go/internal/search	0.006s
ok  	cmd/go/internal/str	0.023s
ok  	cmd/go/internal/txtar	0.002s
ok  	cmd/go/internal/vcs	0.009s
ok  	cmd/go/internal/web	0.004s
ok  	cmd/go/internal/test	0.010s
ok  	cmd/go/internal/work	0.013s
ok  	cmd/gofmt	0.053s
ok  	cmd/internal/archive	0.762s
ok  	cmd/internal/buildid	0.227s
ok  	cmd/internal/dwarf	0.005s
ok  	cmd/internal/edit	0.006s
ok  	cmd/internal/goobj	0.003s
ok  	cmd/internal/moddeps	0.779s
ok  	cmd/internal/obj	0.016s
ok  	cmd/internal/obj/riscv	0.054s
ok  	cmd/internal/obj/ppc64	0.017s
ok  	cmd/internal/obj/s390x	0.002s
ok  	cmd/internal/obj/x86	0.147s
ok  	cmd/internal/objabi	0.002s
ok  	cmd/internal/pkgpath	0.043s
ok  	cmd/internal/sys	0.008s
ok  	cmd/internal/src	0.002s
ok  	cmd/internal/test2json	0.062s
ok  	cmd/link	8.164s
ok  	cmd/link/internal/benchmark	0.003s
ok  	cmd/link/internal/ld	3.368s
ok  	cmd/link/internal/loader	0.013s
ok  	cmd/nm	1.310s
ok  	cmd/objdump	1.846s
ok  	cmd/pack	1.014s
ok  	cmd/trace	0.242s
ok  	cmd/vet	7.555s

##### os/user with tag osusergo
ok  	os/user	0.002s

##### go/... and cmd/gofmt tests with tag typeparams
ok  	go/ast	0.036s
ok  	go/build	1.478s
ok  	go/build/constraint	0.029s
ok  	go/constant	0.023s
ok  	go/doc	0.090s
ok  	go/format	0.013s
ok  	go/importer	0.196s
ok  	go/internal/gccgoimporter	0.026s
ok  	go/internal/gcimporter	2.696s
ok  	go/internal/srcimporter	6.749s
?   	go/internal/typeparams	[no test files]
ok  	go/parser	0.042s
ok  	go/printer	0.321s
ok  	go/scanner	0.014s
ok  	go/token	0.017s
ok  	go/types	1.286s
ok  	cmd/gofmt	0.085s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	23.866s

##### cmd/go terminal test
skipping terminal test; stdout/stderr not terminals

##### moved GOROOT
ok  	fmt	0.060s

##### Testing without libgcc.
ok  	net	0.009s
ok  	os/user	0.003s

##### internal linking of -buildmode=pie
ok  	reflect	0.143s
ok  	os/user	0.004s

##### sync -cpu=10
ok  	sync	0.264s

##### Testing race detector
ok  	runtime/race	12.417s
ok  	flag	0.080s
ok  	net	0.086s
ok  	os	0.160s
ok  	os/exec	0.085s
ok  	encoding/gob	0.086s
ok  	flag	0.091s
ok  	os/exec	0.088s

##### ../misc/cgo/stdio
PASS

##### ../misc/cgo/life
PASS

##### ../misc/cgo/fortran
PASS
ok  	misc/cgo/fortran	0.003s

##### ../misc/cgo/test
PASS
ok  	misc/cgo/test	6.841s
PASS
ok  	misc/cgo/test	6.747s
PASS
ok  	misc/cgo/test	6.930s
PASS
ok  	misc/cgo/testtls	0.006s
PASS
ok  	misc/cgo/testtls	0.009s
PASS
ok  	misc/cgo/testtls	0.003s
PASS
ok  	misc/cgo/nocgo	0.002s
PASS
ok  	misc/cgo/nocgo	0.007s
PASS
ok  	misc/cgo/nocgo	0.006s
PASS
ok  	misc/cgo/test	12.420s
PASS
ok  	misc/cgo/test	9.580s
PASS
ok  	misc/cgo/test	5.581s
PASS
ok  	misc/cgo/test	5.622s
PASS
ok  	misc/cgo/testtls	0.011s
PASS
ok  	misc/cgo/nocgo	0.013s

##### ../misc/cgo/testgodefs
PASS

##### ../misc/cgo/testso
ok  	misc/cgo/testso	0.709s

##### ../misc/cgo/testsovar
ok  	misc/cgo/testsovar	0.705s

##### ../misc/cgo/testcarchive
PASS

##### ../misc/cgo/testcshared
PASS

##### ../misc/cgo/testshared
ok  	misc/cgo/testshared	42.387s

##### ../misc/cgo/testplugin
ok  	misc/cgo/testplugin	26.397s

##### ../misc/cgo/testasan
ok	testasan	314ms

@dmitshur dmitshur mentioned this issue May 19, 2022
Closed
@dmitshur
Copy link
Contributor Author

I've observed a few more instances of this with testsanitizers. I think the root cause here is the same as issue #42699: some test is stalling, and eventually gets retried and succeeds.

I've filed #52998 for the problem with testsanitizers, since I observed it to happen at least twice in recent weeks myself, and I've seen more recent reports of it happening to others. I've not observed any recent hangs involving test (aka the $GOROOT/test directory tests) so not filing an issue for that, but if someone runs into it again please feel free to.

With the above, I don't think there's anything left to do in this issue, given we understand what happened and have tracking issues for problems still happening today, so closing.

@dmitshur dmitshur closed this as not planned Won't fix, can't repro, duplicate, stale May 19, 2022
@dmitshur dmitshur mentioned this issue May 19, 2022
Closed
@golang golang locked and limited conversation to collaborators May 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@toothrot @mdempsky @dmitshur @bcmills @gopherbot