New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: plan9-arm builder failing frequently with a variety of errors #49338
Comments
For most of plan9-arm failures, the cause is filesystem flakiness on the builder machines (a cluster of 3 x raspberry pi 4). I've just reconfigured the build script to stop using ramfs (which has a locking bug) for /tmp, and stop using the ad hoc directory cache (which still seems to have some flaws). This should bypass most of the failures but will make the test suite a lot slower to run (by go.17 it had already regressed to about 2 hours even with these filesystem tweaks). There are some other flakes of the "shouldn't be possible" variety, which are are rare and perplexing and impossible to replicate. They are occurring in general go runtime code, not specific to plan 9. There's a slight whiff of possible memory cache coherence trouble in some of these: It would be good to have a server-class ARM platform capable of running plan 9 as a more reliable builder. I don't know of one. |
Change https://golang.org/cl/362975 mentions this issue: |
The plan9 builders have relatively slow filesystems, and the website repo is relatively filesystem-intensive — many of its tests walk and process large trees of website content. Instead of trying to precisely tune the x/website tests to pass on the plan9 builders, let's skip this repo on them. For golang/go#49338. Change-Id: Ieaf50fb6955a6b39012a23ff72e61c1c31e4ce7c Reviewed-on: https://go-review.googlesource.com/c/build/+/362975 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Many of the failure modes seem to have cleared up around 11 Nov.; it's not clear to me what fixed them. One of the failures visible on the dashboard today is #49653; the other is the |
Change https://golang.org/cl/369018 mentions this issue: |
To prevent interleaving of output when 'go build' compiles several packages in parallel, the output mutex in the Builder struct must be locked around any calls to Builder.Print which could generate arbitrary amounts of text (ie more than is guaranteed to be written atomically to a pipe). Fixes #49987 For #49338 Change-Id: I7947df57667deeff3f03f231824298d823f8a943 Reviewed-on: https://go-review.googlesource.com/c/go/+/369018 Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Trust: Russ Cox <rsc@golang.org>
Change https://golang.org/cl/380414 mentions this issue: |
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
A sampling from the past week or so. Undiagnosed failures in bold.
2022-01-27T05:30:27-be5769c-a991d9d/plan9-arm
That's stuck running 2022-01-27T05:30:27-bbe1937-a991d9d/plan9-arm (#50857)
2022-01-24T16:42:11-2cc1836-19d819d/plan9-arm (#50775, mitigated)
That's running a 2022-01-21T21:27:57-e7c9de2-9eba5ff/plan9-arm (#46520, not specific to |
Change https://go.dev/cl/408697 mentions this issue: |
For golang/go#52653. Updates golang/go#49338. Change-Id: Ib23be56bd79bce8b3168cb40064916a2a91dbb88 Reviewed-on: https://go-review.googlesource.com/c/build/+/408697 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Alex Rakoczy <alex@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
This comment was marked as resolved.
This comment was marked as resolved.
Found new dashboard test flakes for:
2022-12-05 22:01 plan9-arm go@185e1a7b os/exec.TestExtraFilesRace (log)
|
Found new dashboard test flakes for:
2022-12-06 20:08 plan9-arm net@1e63c2f0 go@a4a86c7b x/net/http2.TestTransportUsesGoAwayDebugError_Body (log)
|
Found new dashboard test flakes for:
2022-12-22 15:34 plan9-arm benchmarks@03c43f28 go@f721fa3b (log)
|
Found new dashboard test flakes for:
2022-12-01 20:58 plan9-arm go@cc1771e2 net/http.TestOmitHTTP2Vet (log)
2022-12-12 22:32 plan9-arm crypto@23edec0b go@357ea858 (log)
2022-12-12 23:10 plan9-arm go@b16e94d1 (log)
2022-12-13 21:00 plan9-arm benchmarks@555ee2b5 go@357ea858 (log)
2022-12-15 00:54 plan9-arm tools@5899b6a2 go@ea14d1b6 (log)2022-12-23 17:16 plan9-arm go@38cfb3be net.TestPipe (log)
2022-12-29 17:47 plan9-arm go@642fd5f7 net/http.TestServerReadTimeout (log)
|
Found new dashboard test flakes for:
2023-01-20 21:10 plan9-arm benchmarks@33cb3306 go@27b6ace2 (log)
|
Found new dashboard test flakes for:
2023-01-05 16:42 plan9-arm net@f8411da7 go@cc82867f x/net/http2/h2c.TestPropagation (log)
2023-01-19 20:43 plan9-arm go@f89badf6 (log)
2023-01-19 20:46 plan9-arm go@e08642ca (log)2023-01-20 02:42 plan9-arm tools@a7f033af go@c641d48a x/tools/gopls/internal/lsp/filecache.TestConcurrency (log)
2023-01-20 17:52 plan9-arm go@1011ccad os.TestDirFS (log)
2023-01-20 19:38 plan9-arm go@f07910bd os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestWriteAtInAppendMode (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestOpenFileLimit (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFSRootDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReaddirSmallSeek (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadFile (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddirnames (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReadDir (log)
2023-01-23 18:50 plan9-arm go@b90971b2 os.TestDirFS (log)
|
Found new dashboard test flakes for:
2023-01-24 19:29 plan9-arm go@083d94f6 (log)
|
Found new dashboard test flakes for:
2023-01-20 09:27 plan9-arm go@1bef0707 os.TestDirFS (log)
2023-01-25 16:38 plan9-arm go@1d3088ef os.TestFileReadDir (log)
2023-01-25 18:50 plan9-arm go@3c7884a2 (log)2023-01-26 01:06 plan9-arm go@89de906a os.TestDirFS (log)
2023-01-26 19:45 plan9-arm go@627f1286 os.TestFileReadDir (log)
2023-01-26 21:33 plan9-arm net@296f09aa go@dd44d981 x/net/http2/h2c.TestPropagation (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestWriteAtInAppendMode (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestOpenFileLimit (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadFile (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFSRootDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFS (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReaddirSmallSeek (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddirnames (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os/exec (log)
2023-01-27 03:32 plan9-arm go@4e3abee2 os.TestDirFS (log)
2023-01-30 16:32 plan9-arm go@87860873 net/http.TestServerReadTimeout (log)
2023-01-30 20:09 plan9-arm go@45bee5e7 os.TestReadFile (log)
2023-01-31 19:43 plan9-arm go@b89a840d (log)
|
Found new dashboard test flakes for:
2023-02-02 23:42 plan9-arm tools@81111180 go@a9648761 x/tools/cmd/bundle.TestBundle (log)
|
Found new dashboard test flakes for:
2023-02-07 20:59 plan9-arm go@1bd0405b os.TestDirFS (log)
|
Found new dashboard test flakes for:
2024-03-06 19:15 plan9-arm go@d463de61 cmd/link (log)
|
Found new dashboard test flakes for:
2024-03-08 01:59 plan9-arm go@b57a544f (log)
|
Found new dashboard test flakes for:
2024-03-08 22:35 plan9-arm go@61d6817c (log)
|
Found new dashboard test flakes for:
2024-03-08 22:14 plan9-arm go@73811232 (log)
|
Found new dashboard test flakes for:
2024-03-08 21:09 plan9-arm go@2171e628 cmd/vet.TestTags (log)
|
Found new dashboard test flakes for:
2024-03-13 18:24 plan9-arm go@4f07bb39 net/http.TestUnsupportedTransferEncodingsReturn501 (log)
|
Found new dashboard test flakes for:
2024-03-13 21:36 plan9-arm go@508bb17e cmd/compile/internal/types2.TestStdTest (log)
2024-03-13 21:36 plan9-arm go@508bb17e cmd/compile/internal/types2.TestStdFixed (log)
|
Found new dashboard test flakes for:
2024-03-14 16:30 plan9-arm go@90a2dcf1 go/types.TestSelf (log)
|
Found new dashboard test flakes for:
2024-03-15 17:12 plan9-arm go@6f121623 (log)
|
Found new dashboard test flakes for:
2024-03-12 20:49 plan9-arm sys@95f07eca go@bedda245 (log)
|
Found new dashboard test flakes for:
2024-03-04 18:29 plan9-arm crypto@70672239 go@b822f098 x/crypto/ssh/agent.TestServerResponseTooLarge (log)
|
Found new dashboard test flakes for:
2024-03-16 02:25 plan9-arm go@68d3a9e4 net/http.TestChunkedNoContent (log)
|
Found new dashboard test flakes for:
2024-03-18 19:36 plan9-arm go@3c78ace2 cmd/link/internal/ld.TestMemProfileCheck (log)
2024-03-18 19:36 plan9-arm go@3c78ace2 cmd/link/internal/ld.TestDeadcode (log)
|
Found new dashboard test flakes for:
2024-03-19 20:02 plan9-arm go@99522de1 net/http (log)
|
Found new dashboard test flakes for:
2024-03-12 20:05 plan9-arm oauth2@3c9c1f6d go@5f5b20c4 x/oauth2/jwt.test [build] (log)
2024-03-22 16:33 plan9-arm go@1c864333 cmd/link.TestStrictDup (log)
|
Found new dashboard test flakes for:
2024-03-25 15:26 plan9-arm go@25aa45af (log)
|
Found new dashboard test flakes for:
2024-03-29 14:36 plan9-arm go@d0051be8 go/types.TestCheck (log)
2024-03-29 14:36 plan9-arm go@d0051be8 go/types.TestSelf (log)
2024-03-29 16:41 plan9-arm go@8f59b28f cmd/go_test [build] (log)2024-03-29 16:41 plan9-arm go@8f59b28f [cmd/go_test] [build] (log)
|
Found new dashboard test flakes for:
2024-03-29 21:53 plan9-arm go@29fcd156 (log)
|
Found new dashboard test flakes for:
2024-03-29 21:55 plan9-arm go@dcf046f0 (log)
|
Found new dashboard test flakes for:
2024-03-12 20:49 plan9-arm sys@95f07eca go@cd294f55 internal/cpu [build] (log)
2024-03-12 20:49 plan9-arm sys@95f07eca go@cd294f55 internal/goexperiment [build] (log)
2024-03-12 20:49 plan9-arm sys@95f07eca go@cd294f55 internal/abi [build] (log)
|
Found new dashboard test flakes for:
2024-04-02 17:12 plan9-arm go@973befe7 net/http (log)
|
Found new dashboard test flakes for:
2024-04-02 21:51 plan9-arm go@3d61f248 crypto/x509 [build] (log)2024-04-02 21:51 plan9-arm go@3d61f248 [crypto/x509] [build] (log)
2024-04-02 21:51 plan9-arm go@3d61f248 crypto/rsa_test [build] (log)
|
Found new dashboard test flakes for:
2024-04-03 21:36 plan9-arm go@885fdfc0 cmd/go/internal/lockedfile.TestSpuriousEDEADLK (log)
|
Found new dashboard test flakes for:
2024-04-04 16:59 plan9-arm crypto@d042a396 go@b24ec88b (log) |
Found new dashboard test flakes for:
2024-04-04 16:59 plan9-arm crypto@d042a396 go@5ec7395a (log) |
Found new dashboard test flakes for:
2024-04-05 22:18 plan9-arm go@5d18f889 net/http (log)
|
Found new dashboard test flakes for:
2024-04-09 16:41 plan9-arm go@26a1f4ae testing/internal/testdeps [build] (log)
2024-04-09 17:42 plan9-arm go@82e929e4 net/http.TestCaseSensitiveMethod (log)
|
Found new dashboard test flakes for:
2024-04-10 20:23 plan9-arm go@c23579f0 (log)
|
Found new dashboard test flakes for:
2024-04-11 00:32 plan9-arm go@b6778c52 net/http.TestTLSServerRejectHTTPRequests (log)
|
Found new dashboard test flakes for:
2024-04-11 16:36 plan9-arm go@45703b50 (log)
|
The one remaining
plan9
builder is failing a significant fraction of build attempts.Many of the failures follow specific known patterns (notably #49337, #46526, and #41952). However, many do not.
@millerresearch, @0intro, @fhs: is there someone who can investigate these failures and bring the
plan9
port back up to par? With the0intro
builders missing (#49327, #49328), there is no longer anyplan9
builder consistently passing tests.greplogs --dashboard -md -l -e \\Aplan9. -E . --since=2021-10-01
2021-11-04T01:44:54-d9bb5f6/plan9-arm
2021-11-03T23:48:28-283373d/plan9-arm
2021-11-03T21:45:33-4a4e1f2-d3f5dd5/plan9-arm
2021-11-03T21:45:33-2c98350-d3f5dd5/plan9-arm
2021-11-03T19:32:33-74f99d0/plan9-arm
2021-11-03T18:37:22-cfd016d/plan9-arm
2021-11-03T15:59:11-5f0e5d2/plan9-arm
2021-11-03T09:23:34-6458b2e/plan9-arm
2021-11-03T05:00:00-519c0a2/plan9-arm
2021-11-03T00:49:50-a4b2c57/plan9-arm
2021-11-03T00:21:24-8f0ca7d/plan9-arm
2021-11-03T00:07:38-49a0063/plan9-arm
2021-11-03T00:06:52-32d2752/plan9-arm
2021-11-02T23:44:58-a0f373c/plan9-arm
2021-11-02T23:05:17-714668c-eb63cbd/plan9-arm
2021-11-02T21:18:39-60fd3ed/plan9-arm
2021-11-02T20:59:34-79024cf/plan9-arm
2021-11-02T19:56:02-c45c32b/plan9-arm
2021-11-02T19:37:42-1011e26/plan9-arm
2021-11-02T18:35:29-433ba58/plan9-arm
2021-11-02T18:35:22-0b37bd9/plan9-arm
2021-11-02T18:31:53-79c5240/plan9-arm
2021-11-02T18:24:18-f7a95d2/plan9-arm
2021-11-02T15:33:01-4d06874-da7173a/plan9-arm
2021-11-02T13:43:24-ee08195-4d7bf41/plan9-arm
2021-11-02T13:43:24-4ff95d6-4d7bf41/plan9-arm
2021-11-02T06:25:39-4ff95d6-088bb4b/plan9-arm
2021-11-02T06:21:44-3ee426a/plan9-arm
2021-11-02T06:20:21-c1ea339/plan9-arm
2021-11-02T06:19:33-4633b2d/plan9-arm
2021-11-02T05:35:11-6f327f7/plan9-arm
2021-11-02T03:55:19-4ff95d6-6f1e9a9/plan9-arm
2021-11-02T03:54:24-3c61cb3/plan9-arm
2021-11-02T03:53:56-b0472aa/plan9-arm
2021-11-02T03:52:33-9268565/plan9-arm
2021-11-01T22:55:50-02e5913/plan9-arm
2021-11-01T19:42:30-4ff95d6-5f1bcd1/plan9-arm
2021-11-01T18:53:06-b5a5b7b/plan9-arm
2021-11-01T16:31:02-03971e3-2bcf1c0/plan9-arm
2021-11-01T15:55:25-e2e910e/plan9-arm
2021-11-01T15:55:20-d2b5121/plan9-arm
2021-11-01T14:45:28-82f902a/plan9-arm
2021-11-01T13:12:37-4056934/plan9-arm
2021-11-01T13:08:16-732db40/plan9-arm
2021-11-01T02:47:30-fde4cc2/plan9-arm
2021-10-31T18:39:05-89c5270/plan9-arm
2021-10-31T18:13:09-fd09e88/plan9-arm
2021-10-31T17:52:41-3fa9ee4/plan9-arm
2021-10-30T18:30:34-6113dac/plan9-arm
2021-10-30T16:46:47-d1dceaf/plan9-arm
2021-10-30T12:32:06-351c04c-5d6d9f5/plan9-arm
2021-10-29T23:25:18-5d6d9f5/plan9-arm
2021-10-29T22:29:31-c812b97/plan9-arm
2021-10-29T22:27:26-3571ab5/plan9-arm
2021-10-29T19:35:47-5ec139f/plan9-arm
2021-10-29T18:56:29-6afdf01/plan9-arm
2021-10-29T18:35:29-9da6415/plan9-arm
2021-10-29T18:23:03-413672f/plan9-arm
2021-10-29T17:44:52-353d5b6/plan9-arm
2021-10-29T04:21:07-103d89b-f288526/plan9-arm
2021-10-29T02:16:47-2c7cdec/plan9-arm
2021-10-29T02:11:42-33c392f/plan9-arm
2021-10-29T00:04:41-d6a9af8-af05d8b/plan9-arm
2021-10-28T20:50:34-d0751b1/plan9-arm
2021-10-28T20:50:34-103d89b-d0751b1/plan9-arm
2021-10-28T20:43:39-b954024/plan9-arm
2021-10-28T20:43:02-e741e2f/plan9-arm
2021-10-28T18:48:25-e7eb6f6-f229e70/plan9-arm
2021-10-28T18:10:59-ed475fe/plan9-arm
2021-10-28T18:10:50-2bc8ed8/plan9-arm
2021-10-28T17:57:57-2666262/plan9-arm
2021-10-28T17:52:59-3124968/plan9-arm
2021-10-28T02:35:22-056dfe6/plan9-arm
2021-10-28T01:15:26-b2fe2eb/plan9-arm
2021-10-28T00:11:24-a91d0b6/plan9-arm
2021-10-27T22:17:35-79ff663/plan9-arm
2021-10-27T21:37:54-749f6e9/plan9-arm
2021-10-27T21:35:48-a3c4ac0/plan9-arm
2021-10-27T21:34:20-4fefd43/plan9-arm
2021-10-27T20:25:06-bbc0595/plan9-arm
2021-10-27T20:24:34-56dcf97/plan9-arm
2021-10-27T20:03:17-68bd512/plan9-arm
2021-10-27T19:34:49-ffd2284/plan9-arm
2021-10-27T19:18:20-514ebae/plan9-arm
2021-10-27T18:24:50-259735f/plan9-amd64-0intro
2021-10-27T18:24:50-259735f/plan9-arm
2021-10-27T16:59:43-c0ac39c/plan9-arm
2021-10-27T13:12:49-cfb5321/plan9-arm
2021-10-27T08:50:27-e1e2965-bdefb77/plan9-arm
2021-10-27T08:50:27-bdefb77/plan9-arm
2021-10-26T23:12:17-13eccaa/plan9-arm
2021-10-26T22:05:53-80be4a4/plan9-386-0intro
2021-10-26T19:04:53-9626607-1dc77a3/plan9-arm
2021-10-26T19:04:53-03971e3-1dc77a3/plan9-arm
2021-10-26T18:33:39-244f92e-76cef81/plan9-386-0intro
2021-10-26T17:10:45-d418f37-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-amd64-0intro
2021-10-26T16:00:26-94870a3-1b2362b/plan9-arm
2021-10-26T14:05:47-a2b8c18/plan9-arm
2021-10-26T11:58:05-1e2820a/plan9-arm
2021-10-26T05:05:24-23fdd7f/plan9-arm
2021-10-26T01:22:47-f916b54-adfb85b/plan9-arm
2021-10-26T01:22:47-22652b4-adfb85b/plan9-arm
2021-10-26T01:07:41-41f7407/plan9-arm
2021-10-25T23:28:03-f916b54-e9eb66d/plan9-arm
2021-10-25T23:28:03-e9eb66d/plan9-arm
2021-10-25T21:03:33-60c3069/plan9-arm
2021-10-25T18:46:45-f47335e/plan9-arm
2021-10-25T17:02:16-52b10ab/plan9-arm
2021-10-25T16:07:11-fc8b4ca-c6e82e5/plan9-arm
2021-10-23T20:23:35-c96bc14-8dbf3e9/plan9-386-0intro
2021-10-23T20:23:35-8dbf3e9/plan9-arm
2021-10-23T16:18:19-d418f37-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-affba50-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-16318df/plan9-arm
2021-10-23T16:18:15-12e4404/plan9-arm
2021-10-23T16:18:09-c526cf8/plan9-arm
2021-10-23T16:18:04-2d205eb/plan9-arm
2021-10-23T16:17:58-3cd28ba/plan9-arm
2021-10-23T12:44:47-b0f7eb6/plan9-arm
2021-10-22T21:48:34-ae4d67c/plan9-arm
2021-10-22T21:12:07-76dd01f/plan9-arm
2021-10-22T14:05:31-ff8115d/plan9-arm
2021-10-21T17:31:42-392bb06/plan9-arm
2021-10-21T17:23:58-02a3666/plan9-arm
2021-10-21T13:05:07-8b9c0d1/plan9-arm
2021-10-21T02:35:47-9b675d0-fca46d0/plan9-arm
2021-10-20T20:39:19-d419a80/plan9-arm
2021-10-20T20:38:59-75b73d6/plan9-arm
2021-10-20T20:38:47-3ec8d4b/plan9-arm
2021-10-20T19:30:13-cbf1d01-3ff39c5/plan9-arm
2021-10-20T19:30:13-3ff39c5/plan9-arm
2021-10-20T19:29:12-4320949/plan9-arm
2021-10-20T18:48:51-cbf1d01-4e565f7/plan9-arm
2021-10-20T05:28:45-b091189/plan9-arm
2021-10-19T23:27:41-4ea6123-2be5b84/plan9-arm
2021-10-19T22:47:35-9fa8551/plan9-arm
2021-10-19T18:56:08-07e5527/plan9-arm
2021-10-19T18:43:12-d944984/plan9-arm
2021-10-19T18:43:08-1d63052/plan9-arm
2021-10-19T18:22:59-6c3cd5d/plan9-arm
2021-10-19T18:18:37-9820602/plan9-arm
2021-10-19T17:12:53-404f84d/plan9-arm
2021-10-19T17:01:42-3a07ab7/plan9-arm
2021-10-19T16:19:27-98f6e03-f92a358/plan9-arm
2021-10-19T08:44:22-1e49210/plan9-arm
2021-10-19T07:45:46-281050f-ee92daa/plan9-arm
2021-10-18T22:56:07-eba0e86/plan9-arm
2021-10-18T22:55:05-7397178/plan9-arm
2021-10-18T19:46:27-98f6e03-394a1ad/plan9-arm
2021-10-18T16:32:38-543a513/plan9-arm
2021-10-18T15:38:40-74acbaf/plan9-arm
2021-10-16T16:27:40-c96bc14-640a49b/plan9-386-0intro
2021-10-16T16:27:40-98f6e03-640a49b/plan9-arm
2021-10-16T16:27:40-640a49b/plan9-arm
2021-10-16T16:27:40-4f30a5c-640a49b/plan9-386-0intro
2021-10-16T14:50:21-c96bc14-680caf1/plan9-386-0intro
2021-10-16T14:50:21-98f6e03-680caf1/plan9-386-0intro
2021-10-16T14:50:21-4f30a5c-680caf1/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-c96bc14-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-4f30a5c-ed1c8db/plan9-amd64-0intro
2021-10-15T21:44:07-4a7975e/plan9-386-0intro
2021-10-15T20:29:48-85cbdda/plan9-arm
2021-10-15T20:29:36-a17b2e8/plan9-386-0intro
2021-10-15T19:54:44-22951fb/plan9-arm
2021-10-15T19:03:33-98f6e03-cfe6763/plan9-arm
2021-10-15T18:07:49-98f6e03-8331f25/plan9-arm
2021-10-15T02:45:22-288d0b3-3da0ff8/plan9-arm
2021-10-15T02:02:50-ce04ca3-3da0ff8/plan9-arm
2021-10-15T02:02:50-3da0ff8/plan9-386-0intro
2021-10-15T00:16:20-ce04ca3-9e8ed86/plan9-arm
2021-10-14T23:32:09-3a269dc-9e8ed86/plan9-arm
2021-10-14T22:15:24-281050f-b59467e/plan9-arm
2021-10-14T18:44:37-a37bebc/plan9-arm
2021-10-14T18:44:29-a8c5a99/plan9-arm
2021-10-14T15:17:03-fbdf830/plan9-arm
2021-10-14T13:09:28-88e80df-24e798e/plan9-arm
2021-10-14T13:09:28-24e798e/plan9-arm
2021-10-14T12:38:31-88e80df-2feb2cc/plan9-arm
2021-10-14T07:18:59-1349c6e/plan9-arm
2021-10-14T04:18:44-acac939-9e4dc6f/plan9-arm
2021-10-14T01:51:22-276fb27/plan9-arm
2021-10-13T22:21:08-ac2ed98-b90d258/plan9-arm
2021-10-13T20:31:25-b90d258/plan9-arm
2021-10-13T18:15:48-40f82f8/plan9-amd64-0intro
2021-10-13T18:15:48-40f82f8/plan9-arm
2021-10-13T18:15:48-281050f-40f82f8/plan9-arm
2021-10-13T17:02:43-b5904f3/plan9-arm
2021-10-13T16:39:23-3986e5c/plan9-arm
2021-10-13T16:36:59-69041c7/plan9-arm
2021-10-13T15:11:16-0454d73/plan9-arm
2021-10-13T03:12:54-24389d4-4fb2e1c/plan9-arm
2021-10-12T22:02:39-732f6fa/plan9-arm
2021-10-12T21:30:45-0b930fb-3283d1a/plan9-arm
2021-10-12T21:15:50-3d051ba/plan9-arm
2021-10-12T20:53:52-ac6d706/plan9-arm
2021-10-12T20:20:41-d032b2b/plan9-arm
2021-10-12T18:30:24-281050f-6e0adde/plan9-arm
2021-10-12T12:24:09-c8768b9-4679670/plan9-arm
2021-10-12T06:55:50-c83725b-d887d3b/plan9-arm
2021-10-11T20:46:14-7023535/plan9-arm
2021-10-11T20:46:14-18fa840-7023535/plan9-arm
2021-10-11T19:20:12-18fa840-65ffee6/plan9-arm
2021-10-11T15:48:29-2478b6f/plan9-arm
2021-10-11T12:53:11-3b03d30-34f7b1f/plan9-386-0intro
2021-10-11T12:53:11-34f7b1f/plan9-arm
2021-10-11T12:42:06-59b2f51/plan9-arm
2021-10-10T15:11:26-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-d2d21d9/plan9-amd64-0intro
2021-10-10T15:11:26-d2d21d9/plan9-arm
2021-10-10T15:11:26-3b03d30-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-1af23bd-d2d21d9/plan9-386-0intro
2021-10-10T13:37:52-1af23bd-b99abf3/plan9-arm
2021-10-09T23:29:26-b99abf3/plan9-386-0intro
2021-10-09T23:29:26-b99abf3/plan9-arm
2021-10-08T22:21:41-ee04797-74abcab/plan9-arm
2021-10-08T19:41:16-c5188f2-7fcf9a1/plan9-arm
2021-10-08T19:41:16-7fcf9a1/plan9-386-0intro
2021-10-08T19:30:58-f430bda/plan9-arm
2021-10-08T19:19:09-b9e1e1b/plan9-386-0intro
2021-10-08T18:07:25-d480b5c/plan9-386-0intro
2021-10-08T17:58:41-16a3cef/plan9-arm
2021-10-08T17:32:31-d7ba1d2/plan9-386-0intro
2021-10-08T17:32:25-a7d3a0e/plan9-arm
2021-10-08T15:27:20-b3a4b2d/plan9-arm
2021-10-08T14:08:12-5b9206f/plan9-386-0intro
2021-10-08T14:08:12-5b9206f/plan9-arm
2021-10-07T22:19:58-1ceb723/plan9-arm
2021-10-07T21:39:17-6b4cf2b/plan9-arm
2021-10-07T20:37:43-d477ef3-ef2ebbe/plan9-arm
2021-10-07T19:56:19-4ab3c10/plan9-arm
2021-10-07T18:58:33-7286502/plan9-arm
2021-10-07T18:09:53-4607ebc/plan9-arm
2021-10-07T16:38:24-be571a3/plan9-arm
2021-10-07T14:37:45-ebeab63/plan9-arm
2021-10-07T14:37:36-375a1fb/plan9-arm
2021-10-07T02:30:47-2c98350-39bbf08/plan9-arm
2021-10-06T23:26:57-c8dd89e/plan9-arm
2021-10-06T23:26:57-86761ae-c8dd89e/plan9-arm
2021-10-06T21:17:08-2e107b4/plan9-arm
2021-10-06T21:06:48-197b328/plan9-arm
2021-10-06T15:53:04-4d8db00/plan9-arm
2021-10-05T23:58:29-ac60900/plan9-arm
2021-10-05T23:36:43-e82ed0c/plan9-arm
2021-10-05T22:42:11-96fface/plan9-arm
2021-10-05T20:52:58-0d6561b/plan9-arm
2021-10-05T20:35:54-990c9c6/plan9-arm
2021-10-05T19:15:03-ae83301/plan9-arm
2021-10-05T18:31:14-5d946f1/plan9-arm
2021-10-05T17:10:45-2dc2755-5140ad1/plan9-arm
2021-10-05T15:47:57-017ffcd/plan9-arm
2021-10-05T15:47:07-cfd74f7/plan9-arm
2021-10-05T01:30:56-d4b1ae0-123393a/plan9-386-0intro
2021-10-04T22:59:44-78c2529/plan9-arm
2021-10-04T22:59:13-199ec42/plan9-arm
2021-10-04T22:54:58-7c79e8e/plan9-arm
2021-10-04T22:00:53-d9952ff/plan9-386-0intro
2021-10-04T22:00:19-739328c/plan9-arm
2021-10-04T21:10:16-db89b5a-f0db7ea/plan9-arm
2021-10-04T18:15:09-ccaa907-9f8d558/plan9-arm
2021-10-04T17:17:11-cedda3a-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-ccaa907-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-cc5e3de/plan9-amd64-0intro
2021-10-04T15:50:43-9432320/plan9-386-0intro
2021-10-04T15:50:43-9432320/plan9-amd64-0intro
2021-10-02T16:05:55-0ebff1a-a7fe161/plan9-arm
2021-10-02T13:15:42-0ebff1a-7bf7bbc/plan9-arm
2021-10-01T23:51:16-c129af9/plan9-arm
2021-10-01T21:26:43-0ebff1a-0d65c27/plan9-arm
2021-10-01T19:30:53-13c407c-0d65c27/plan9-arm
2021-10-01T18:18:46-0d65c27/plan9-arm
2021-10-01T17:18:20-5279e53/plan9-arm
2021-10-01T16:45:09-58ad7eb-f19b2d5/plan9-arm
2021-10-01T16:17:08-f19b2d5/plan9-arm
2021-10-01T16:01:09-558b9da-8d494b0/plan9-arm
2021-10-01T15:59:38-e5ad363/plan9-arm
2021-10-01T14:06:36-96d3ba8/plan9-arm
The text was updated successfully, but these errors were encountered: