runtime: exceeded thread wakeup limitation on iOS #29284
Comments
|
Please do try to figure out a way to reproduce the error; fixing the problem will be much more likely if you do. |
|
I found a thread with a similar issue: https://forum.unity.com/threads/the-weird-ios8-unity3d-hiccup.273186/. It seems that Unity apps don't actually crash, only freezes while the exception report is generated. |
|
Hey I want to work on this issue since I "translated" my app to go - and now I face problems here. I found this thread at https://developer.apple.com/forums/thread/124180 and user eskimo has some insights about what to look for.
At first I tried to reproduce the issue in a minified example by doing thousands of Now I'm kind of out of ideas what to look out for there, I'm trying to think of other design patterns that I use in my main project that could potentially fall into that "thread unblock or interrupt" territory but the @hajimehoshi did make any further findings back then? @eliasnaur has something like that ever been brought up again? Any info that could help me showcase a minimal example of the issue to debug or fix? |
|
I have not seen this issue recently. |
|
@hajimehoshi I do 🙈 Is the project where you have experienced the problem a few years ago open source by any chance or would you open-source it (of course only if it has no further value for you that would be destroyed by open sourcing)? Maybe that would make it easier to find the reason for the problem |
|
I'm afraid it is not an open source... |
|
Oh too bad but don't worry about it, if at any time you remember something you stumbled across when debugging please let me know. And maybe Elias knows something too haha |
|
It's the channels. My example project is quite messy, I am going minify my example project and let you know. So happy I found a way to reproduce the error :) |
@eliasnaur I have just made a project that reliably produces that error. https://github.com/g3devv/channelwakeups It would be very exciting to hear from you if you have an idea how we could approach getting this resolved in gomobile |
|
This is unlikely to be a gomobile issue, and more likely to be an iOS limitation that affects Go runtime scheduling. If you have a reliable reproducer, the Go runtime team may be able to help. I can come up with one clarifying question: is this still an issue on the very latest iOS version? CC @golang/runtime . |
|
Dear Elias, thank you for your response! I completely agree that this very well may not be an issue with gomobile intrinsically, however I fear that some "specialties" that Go comes with make this issue more likely to come up in a Go app compared to native apps or apps developed with other frameworks. What I mean by that (and I'm a complete novice when it comes to golang so maybe it's a bad example) is that for example I once read that Go is super fast at context switching and for that reason does it at a higher frequency than other language runtimes do. So maybe, if we are able to find a singular "feature" to blame, it would be possible to pass a build flag (or compile flag or what it's called) to optionally disable (or throttle) such a feature; in a way that the generated framework is less likely at hitting these iOS restrictions. Over the weekend, I was able to come up with a more concise example (5 lines of code) to reproduce the issue, which should make it easier to pin down the underlying cause. The error rate with the new code is still a bit temperamental (either because it's not hitting the nail OR iOS doesn't emit the crash info on each run). Once I was able to confirm my new demo is better, I will ping you again if that's okay with you. About the iOS version, my production app showcased that issue on 15.2.1 and I currently test my demo project on 14.3 but I will update the phone tomorrow just to make sure. |
|
You're welcome to ping me, but I'm unlikely to be able to do much more for this issue.
It's very unlikely an explicit flag will be accepted by the Go team, especially for a workaround for what seems to be a misfeature of a secondary port. What may be possible is that the Go runtime could somehow automatically limit thread switching when |
|
Without knowing more context about the application, it is hard to provide specific advice, but one thing you could try is lowering GOMAXPROCS. Why? GOMAXPROCS represents the parallelism available to the Go scheduler. The scheduler will attempt to maximize utilization of these resources (work conservation). In practice, this means (simplified) that there will be GOMAXPROCS OS threads running Go code. If there are fewer than GOMAXPROCS runnable goroutines, then extra threads will sleep. Then when new work is available, we will wake up one of those extra threads to run the work (this will count as a wakeup). If GOMAXPROCS is usually less than the number of runnable goroutines, then threads won't sleep and thus won't be woken either. Of course, this will also hurt the scheduling latency of the application, since there are not enough CPU resources for all of the work. This is not the only possible source of excessive wakeups, but it is certainly the top one that comes to my mind. |
prattmic
changed the title
gopherbot
added
the
compiler/runtime
|
Ah, I see you provided a reproducer in https://github.com/g3devv/channelwakeups/blob/main/goproj/goprojlib/goprojlib.go. I don't think that there is anything we can do here. If I am reading correctly, RunChannelConciseExample wakes every 4ms from sleep, which is 250 wakeups[1] per second. Even ignoring the rest of the program, this already far exceeds a 100 wakeup/s limit. In a real application, the use of an unbuffered channel could exacerbate the issue because the producer side must sleep if the consumer isn't waiting for the next value. Adding a buffer would allow both sides to execute concurrently. [1] Disclaimer, I don't know exactly what iOS considers a "wakeup", but I would certainly consider wake from sleep to be a wakeup. |
|
This is kind of unfortunate, and we're not sure what to do about it. We could enforce coarse-grained wakeups on iOS, so that there isn't more than one wakeup every 10 ms (either spinning or waiting to wake up another thread). The immediate workaround is to just have user application enforce this, but that can be difficult. It's surprising that iOS crashes the process instead of just refusing to wake up the process, though. That's a pretty hard hammer for what seems like a fixable problem. |
|
CC @golang/ios |
|
Assigning to @changkun as the port maintainer, but feel free to respond and unassign. |
|
We're seeing this issue on macOS too, when running an app that links against a framework built with go. It's extremely difficult to reproduce, and is only occurring on certain devices (of course, never our own). The process is killed, and a diagnostic report is written to /Library/Logs/DiagnosticReports staying "Action taken: none", yet the process does appear to either crash or be killed by the OS. Here's an excerpt from a diagnostic report when this happens: |
|
Also to note, the framework is not built using gomobile, it's built directly with |
|
Some other folks appearing to run into a similar issue: |
|
@iamcalledrob Thanks for the references. For posterity, the reproducer from https://groups.google.com/g/golang-nuts/c/yywUWpRA1EE is: func main() {
time.Sleep(10*time.Second) // added sleep to make sure it run enough to show in mac activities monitor
for count < 10000 {
count += 1
netConn, err := dialer.Dial("tcp", ":8080")
if err == nil {
netConn.Close()
}
}
time.Sleep(10*time.Second) // added sleep to make sure it run enough to show in mac activities monitor
}@iamcalledrob does this program reproduce the problem for you as well? |
|
@iamcalledrob what version of Go are you running? Can you paste |
FWIW, on https://developer.apple.com/forums/thread/124180?answerId=742350022#742350022 and following posts, it seems that this report is not supposed to actually kill the process, though, like you, the user seems to report that their process is dying. |
$ go envI'm able to (unreliably) reproduce an "excessive wakeups" diagnostic report in a plain go application: package main
func main() {
for i := 0; i < 10000; i++ {
go func() {
for {
<-time.After(10 * time.Millisecond)
}
}()
}
select {}
}Note: I say unreliably, because macOS seems to be applying some magical logic to what it does when it encounters "excessive wakeups". Sometimes it writes a DiagnosticReport, sometimes it doesn't. Never when you want it to, of course. And it appears that sometimes it kills the process, and sometimes it doesn't. In my observations, it suppresses subsequent reports from the same process name. And it never writes reports when the app is "Run" from Xcode. It does seem like there's some special condition where the OS does choose to kill the process, since the crashes I (and others) are experiencing occur immediately after this diagnostic report is written, with no additional crash logs. System log caused by excessive wakeups: When it chooses to write a diagnostic report, there are entries relating to (Note: I failed to copy+paste the exact log line when I was getting these, but it's similar to this) Here's a .diag Diagnostic Report from starting many timers (snippet above)If I could understand the logic being used for when the high wakes trigger spindump (or not) and when the process is killed (or not) then this would feel like something that might be addressable. Perhaps the answer lies in the darwin source somewhere? |
|
Thanks for the additional details. At a high level, I am still unsure what, if anything, we should do about this at the Go runtime level. The program that you shared is requesting 1 million goroutine wakeups per second (10k goroutines waking 100 times/s). So it doesn't seem fundamentally incorrect for the OS to warn about this. The program in #29284 (comment) doesn't obviously contain sleeps, but 45k wakeups during 10k dials implies each dial sleeps at least 4.5 times on average. Perhaps we could do something about that? |
|
Right, I totally get that the "reliable repro" is The challenge we are facing is that this "excessive wakeups -> crash" problem is occurring seemingly randomly in non-contrived production code. Code that isn't starting thousands of timers :) I'm curious if there may be paths occasionally taken by the Go runtime that might be leading to an unexpectedly high number of wakeups? We do a lot of network and disk IO, and these crashes are happening "in the wild" during these operations. |
|
With the last iOS update I've seen virtually zero crashes to my app anymore. Maybe the crashes we've experienced came from another reason which added to the wakeup limitations or Apple has made changes on their end, or the thread wakeups have became less for some reason. But what I can tell is that it's not really an issue for us any more, it seemed to be a lot more critical when I first noticed it but now it's mostly fine ig :) |
We found my gomobile iOS application crash with the below message:
I guess Go runtime switched threads so many times that iOS complained, but I am not sure. It is hard to create a minimal code to reproduce this problem, but I'd like to report this (potential) problem in gomobile.
The text was updated successfully, but these errors were encountered: