x/build/dashboard: darwin-arm-mg912baios and darwin-arm64-mn4m2zdaios builders consistently failing #31497
Labels
Builders
x/build issues (builders, bots, dashboards)
FrozenDueToAge
mobile
Android, iOS, and x/mobile
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Testing
An issue that has been verified to require only test changes, not just a test failure.
Milestone
Comments
bcmills
added
Testing
|
Yes, I've pinged @steeve. I believe he's on it. |
|
I am. I'll try to completely factory reset the arm64 phone. arm32 should be ok I think |
|
Any update? |
|
Yes, the builders seems ok now. |
|
Looks like |
|
|
|
@bcmills yes, the 32 bits device is having issues, I'm trying to get another device to test. |
|
I bisected the problem in I have opened libimobiledevice/ideviceinstaller#119 to track it. In the mean time I'm going to revert that commit on the build machine. |
|
I have downgraded |
|
The builder is back online for the two archs, and now they are failing for good reasons |
It looks like the current failures are not sending output back to the runner. Is there some change that needs to be made to extract it from the device on failure? Are there issues filed for the current failures? |
|
I am not aware regarding issues for the current failures? Probably because of lack of output. |
|
I don't know of any way than manually running the tests. See |
|
Thanks, I'll do it tomorrow! |
|
FWIW, you might find #34501 (comment) useful. You can now use TRY= to target a particular set of builders. |
|
I couldn't reproduce the failures on an iPhone 5S running iOS 10. I ran: (GO_BUILDER_NAME set runs tests that normally doesn't run). Perhaps the devices are somehow broken, although it is unlikely. I noticed that the builder test times quite a bit higher than the times for my old 5S. Compare https://build.golang.org/log/c6e5a939431f25560b0f1b6837fe465f7c1cb0e5 with $ GO_BUILDER_NAME=darwin-arm64-mn4m2zdaios GOARCH=arm64 CGO_ENABLED=1 CC_FOR_TARGET=$(pwd)/../misc/ios/clangwrap.sh ./all.bash Building Go cmd/dist using /Users/elias/go1.10.1. Building Go toolchain1 using /Users/elias/go1.10.1. Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1. Building Go toolchain2 using go_bootstrap and Go toolchain1. Building Go toolchain3 using go_bootstrap and Go toolchain2. Building packages and commands for host, darwin/amd64. Building packages and commands for target, darwin/arm64. |
|
Looking at just the names of the failing tests, my hypothesis is that they are the memory-intensive ones, and that the tests are being killed by some sort of OOM signal. That would also explain the lack of output: Go tests buffer their output until the test completes, so if the test binary is killed before it completes, it may well produce no output. |
|
If that hypothesis is correct, then it seems to me that the ideal fix is to have the memory-intensive tests in the affected packages query (somehow‽) for the process's memory limit and invoke |
|
Do these devices already scale the test parallelism down to 1? If not, that might be another good starting point. |
|
If the were with memory or parallelism, why can't I reproduce the problem on the modest iPhone 5S? I still believe the devices need a factory reset or replacement. @steeve ? |
|
I am giving a a look at the device builder. Click to expand |
|
I don't see any actual test failures in the verbose output. Is the |
|
@bcmills I'm editing the log in place just in case, you might want to refresh |
|
For some reason I can't run the |
|
2020-02-20T05:04:39-49f8d45/darwin-arm64-mn4m2zdaios |
|
@steeve this is far outside the four week failure window specified in our porting policy, which is grounds for removal. Are you able to devote the needed time to maintain this port? Is there someone else who should take over? |
|
Since we've already said in the 1.14 release notes:
This may be the time for us to remove the 32 bit darwin ports. Apple started requiring 64 bit in iOS 11 (released September 2017). Is there a compelling reason for us to continue to support darwin/arm, @steeve @eliasnaur ? |
|
Per #37611, we will be removing As there hasn’t been a reply yet about either builder, it’s likely that we’ll remove both from the dashboard as all they’re providing is noise at the moment. |
|
Change https://golang.org/cl/222720 mentions this issue: |
|
Hey guys Unfortunately the builds are failing and my attempts at finding the root cause was unsucessful. In any case, the Corellium builder is way more reliable than what we have at the moment. If somebody is willing to help me debug it tho, I can take half a day and do so. |
The reason was to keep some WatchOS versions working. |
Also see #34751 where it was discussed in more detail. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Starting around https://build.golang.org/log/776c688e90aaa17f09782242647391a368eeabc7 (Apr. 9), the two
darwin-arm*builders have been failing with lots of messages of the formIt seems like that may be due to a change in the builder configuration, but I'm not really sure.
@eliasnaur, could you look into it (or loop in someone who can)?
The text was updated successfully, but these errors were encountered: