x/build: difference between post-submit builder and gomote makes repros less likely #42379
Comments
|
One other thing I've noticed which may be related is that it seems like the openbsd-amd64-62 gomote is significantly slower than the openbsd-amd64-62 builder. e.g., compare this post-submit run to my gomote run: Test Output##### Testing packages. ok archive/tar 0.132s ok archive/zip 0.849s ok bufio 0.342s ok bytes 2.026s ok compress/bzip2 0.112s ok compress/flate 4.316s ok compress/gzip 0.657s ok compress/lzw 0.244s ok compress/zlib 1.988s ok container/heap 0.023s ok container/list 0.017s ok container/ring 0.016s ok context 0.153s ok crypto 0.023s ok crypto/aes 0.078s ok crypto/cipher 0.038s ok crypto/des 0.044s ok crypto/dsa 0.031s ok crypto/ecdsa 0.346s ok crypto/ed25519 0.111s ok crypto/elliptic 0.079s ok crypto/hmac 0.024s ok crypto/internal/subtle 0.020s ok crypto/md5 0.020s ok crypto/rand 0.127s ok crypto/rc4 0.055s ok crypto/rsa 0.233s ok crypto/sha1 0.015s ok crypto/sha256 0.016s ok crypto/sha512 0.024s ok crypto/subtle 0.022s ok crypto/tls 2.721s ok crypto/x509 1.747s ok database/sql 1.046s ok database/sql/driver 0.027s ok debug/dwarf 0.045s ok debug/elf 0.066s ok debug/gosym 0.057s ok debug/macho 0.027s ok debug/pe 0.037s ok debug/plan9obj 0.028s ok embed/internal/embedtest 0.033s ok encoding/ascii85 0.058s ok encoding/asn1 0.033s ok encoding/base32 0.130s ok encoding/base64 0.029s ok encoding/binary 0.038s ok encoding/csv 0.029s ok encoding/gob 0.067s ok encoding/hex 0.084s ok encoding/json 0.687s ok encoding/pem 0.048s ok encoding/xml 0.111s ok errors 0.055s ok expvar 0.043s ok flag 0.124s ok fmt 0.171s ok go/ast 0.045s ok go/build 2.028s ok go/constant 0.020s ok go/doc 0.233s ok go/format 0.039s ok go/importer 0.516s ok go/internal/gccgoimporter 0.052s ok go/internal/gcimporter 10.296s ok go/internal/srcimporter 9.709s ok go/parser 0.071s ok go/printer 0.698s ok go/scanner 0.035s ok go/token 0.059s ok go/types 4.011s ok hash 0.019s ok hash/adler32 0.040s ok hash/crc32 0.021s ok hash/crc64 0.022s ok hash/fnv 0.121s ok hash/maphash 0.254s ok html 0.025s ok html/template 0.481s ok image 0.192s ok image/color 0.042s ok image/draw 0.193s ok image/gif 3.448s ok image/jpeg 0.388s ok image/png 0.177s ok index/suffixarray 0.325s ok internal/cpu 0.040s ok internal/fmtsort 0.030s ok internal/poll 0.132s ok internal/profile 0.015s ok internal/reflectlite 0.249s ok internal/singleflight 0.028s ok internal/trace 0.207s ok internal/unsafeheader 0.018s ok internal/xcoff 0.038s ok io 0.359s ok io/fs 0.025s ok io/ioutil 0.030s ok log 0.040s ok log/syslog 1.411s ok math 0.052s ok math/big 2.772s ok math/bits 0.034s ok math/cmplx 0.020s ok math/rand 0.566s ok mime 0.055s ok mime/multipart 0.411s ok mime/quotedprintable 0.057s ok net 29.665s ok net/http 23.805s ok net/http/cgi 1.256s ok net/http/cookiejar 0.112s ok net/http/fcgi 0.040s ok net/http/httptest 0.132s ok net/http/httptrace 0.028s ok net/http/httputil 0.195s ok net/http/internal 0.020s ok net/http/pprof 4.234s ok net/internal/socktest 0.015s ok net/mail 0.029s ok net/rpc 0.068s ok net/rpc/jsonrpc 0.031s ok net/smtp 0.074s ok net/textproto 0.044s ok net/url 0.046s ok os 1.398s ok os/exec 1.937s ok os/signal 2.588s ok os/user 0.026s ok path 0.016s ok path/filepath 0.133s ok plugin 0.019s ok reflect 2.786s ok regexp 0.226s ok regexp/syntax 0.577s Almost every big test takes 50-100%+ longer than the post-submit version. This is a very empirical observation, and I haven't seen the post-submit results from a passing run (since the logs are immediately deleted), so perhaps there is nothing there, but this makes me wonder if the gomote is on a smaller instance or something. Edit: this could also be cause the builder is running fewer tests? -short? |
|
Thanks for reporting. Also CC @golang/release for awareness. The One of the most important differences that comes to mind is that when you have a gomote session running all.bash, all execution happens sequentially. The post-submit builder differs in that it's configured to shard test execution across multiple builder instances of that type: $ goexec 'dashboard.Builders["openbsd-amd64-62"].SplitMakeRun()'
(bool)(true)(See coordinator.go#L2003-L2007 and Which should explain the timing difference you've observed above. I've only had a chance to look briefly so far, I'll see if I can look more later on. |
dmitshur
added
the
NeedsInvestigation
I've looked up the logs for some passing post-submit builder runs, for reference:
They all look similar, like so: |
|
I've investigated this and I believe I'm able to reproduce the problem of #42237 on the Investigation LogI limited my scope to the In other words, the Here's a relevant bit from an event log of a recent failed build (full log here): Notably, it tells me the This gave me the idea to try reproducing the failure by doing just Reproduction Steps
Intermittent Slow TestsI added Running just those tests individually didn't reproduce the problem: So it seems there's some sort of interaction between all the runtime tests when run one after another. I hope this helps @prattmic be able to also reproduce the problem and use it to investigate #42237 more easily. Please let me know if it still doesn't reproduce for you for some reason. |
|
As far as I can tell, the root cause here turned out to be that the problem was reproducible only when running |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Bug under investigation not in latest releast
What operating system and processor architecture are you using (
go env)?openbsd-amd64-62 builder
What did you do?
Attempt to reproduce failures from #42237. These failures are occurring on nearly 100% of post-submit openbsd-amd64-62 builds on build.golang.org.
My attempt to repro (run from
/src):What did you expect to see?
Test failures similar to the post-submit builders in almost all runs.
What did you see instead?
Failures almost never occur. Running all.bash takes ~20min, so I haven't run it tons of times, but I'd say I've probably gotten 1 or 2 failures out of around 10 runs. I've managed to get a few more repros running
go test runtime, but I'm not sure the failure rate is any higher; that test just runs faster.The text was updated successfully, but these errors were encountered: