Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime, x/mobile: Severe performance regression since 1.18+ on iOS, when combined with Xcode #57651

Open
rayvbr opened this issue Jan 6, 2023 · 19 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@rayvbr
Copy link

rayvbr commented Jan 6, 2023

What version of Go are you using (go version)?

1.18 and higher

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

ios/arm64

What did you do?

We’ve been noticing for a while now that starting with Go 1.18, performance for iOS apps that contain libraries written in Go has severely regressed, to the point where such apps have become unusable under certain conditions. Under those conditions, an otherwise empty app that is calling an .xcframework generated from Go code using gomobile/bind, is experiencing severe fps drops (going as low as 5fps) even though CPU load is close to 0.

Interestingly, the problem:

  • Does not affect Go 1.17 and lower
  • Affects all Go version since 1.18 (including git head)
  • Is only present when the iOS device is connected to Xcode (either through USB or Wifi)
  • Is not present when the iOS device is connected to Xcode but the Xcode profiler is running.
  • Is not present when the iOS device is NOT connected to Xcode

See attached for a very simple Go project, which is then compiled for iOS with gomobile/bind and wrapped in a Swift wrapper project. For convenience and ease of reproducibility, we’ve including both the full Swift Xcode project as well as pre-compiled .xcframework files based on Go 1.16.15 and 1.18.9 (in addition to the full Go source code). The actual Go project does nothing more than simulate some load by calculating a few sha265 hashes and then sleeping.

To reproduce: simply run the project in XCode, and observe the fps counter.

Note that the drop in performance doesn't just affect the Go code, but the full app containing the library, even code that does not depend in any way on the code being executed by the Go runtime.

Perhaps this is in some way connected to the stack frame pointers/new register-based calling convention introduced for ios/arm64 in Go 1.18?

What did you expect to see?

  • Performance to be roughly similar regardless of using Go 1.17 or 1.18+.
  • Performance to be roughly similar regardless of being connected to Xcode or not

What did you see instead?

Performance being fine with Go 1.17 and lower, while being very bad on Go 1.18 and higher. While CPU load is close to 0.

Project to reproduce

go-ios-performance-regression-project.zip

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jan 6, 2023
@heschi
Copy link
Contributor

heschi commented Jan 6, 2023

cc @hyangah

@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 6, 2023
@heschi heschi added this to the Backlog milestone Jan 6, 2023
@cherrymui
Copy link
Member

  • Is only present when the iOS device is connected to Xcode (either through USB or Wifi)
  • Is not present when the iOS device is connected to Xcode but the Xcode profiler is running.
  • Is not present when the iOS device is NOT connected to Xcode

This is interesting. It would be good to know what is the difference Xcode and Xcode profiler cause.

Perhaps this is in some way connected to the stack frame pointers/new register-based calling convention introduced for ios/arm64 in Go 1.17?

The frame pointer has been enabled on iOS before Go 1.17.
The register calling convention change for ARM64 is in Go 1.18, not 1.17.
So it seems these may not be the cause?

@heschi
Copy link
Contributor

heschi commented Jan 9, 2023

Oops, I should've cc'ed @golang/ios .

@rayvbr
Copy link
Author

rayvbr commented Jan 10, 2023

This is interesting. It would be good to know what is the difference Xcode and Xcode profiler cause

Difficult to say for sure. But one thing we were able to say for sure is that when running XCode Profiler, the console/terminal output is not available. So perhaps that is a useful pointer. Another thing we notice is that when connecting the iOS device to XCode over Wifi, the issue is significantly worse than when connecting over USB (although even USB mode is significantly worse than not connecting to XCode at all)

The frame pointer has been enabled on iOS before Go 1.17.
The register calling convention change for ARM64 is in Go 1.18, not 1.17.
So it seems these may not be the cause?

Sorry, I seem to have made a mistake when creating the issue. The issue starts ocurring from Go 1.18 onwards, not 1.17 onwards. I have just updated the ticket title and and description accordingly

@rayvbr rayvbr changed the title runtime, x/mobile: Severe performance regression since 1.17+ on iOS, when combined with Xcode runtime, x/mobile: Severe performance regression since 1.18+ on iOS, when combined with Xcode Jan 10, 2023
@cherrymui
Copy link
Member

Thanks. For 1.18, could you try building the code with GOEXPERIMENT=noregabi, to see if it is the register calling convention? Thanks. (Note that the GOEXPERIMENT only affects 1.18, not later versions of Go.)

@rayvbr
Copy link
Author

rayvbr commented Jan 12, 2023

@cherrymui we just tried by making a Go 1.18.10 build with GOEXPERIMENT=noregabi but it did not solve the performance regression. So I guess that's good news in the sense that if it would've been related to the register-based calling conventions it would not be easy to fix the root cause.

@mknyszek
Copy link
Contributor

Which 1.17 release works OK? One potential culprit (if the XCode is doing something address-space proportional or... something) is https://go.dev/cl/344401. That CL was backported to 1.17 and 1.16 though, so it depends on which minor release you're using. I suppose you could try rolling that CL back in 1.18 and see what happens?

@mknyszek mknyszek self-assigned this Jan 18, 2023
@rayvbr
Copy link
Author

rayvbr commented Jan 18, 2023

Thanks for checking @mknyszek, but I'm afraid that 1.16.15 and 1.17.13 (both of which contain the CL in question) are working fine. Given that we are the original reporters of #46860 that lead to https://go.dev/cl/344401, we have been running 1.16.15 in production for quite some time now without any problems.

@mknyszek
Copy link
Contributor

Oops! Haha, it's been a while. 😅

Though, I'm afraid we don't have any other leads. I think the most useful thing at this point would be to try and bisect between 1.18 and 1.17, roughly 5da2010 to bcb89fc I think?

@rayvbr
Copy link
Author

rayvbr commented Jan 19, 2023

Haha, no worries, I didn't expect you too to notice, just a funny coincidence :)

Anyway, we'll get started on the bisect and report back the results.

@rayvbr
Copy link
Author

rayvbr commented Jan 20, 2023

@mknyszek some good news. My colleague @mees-vdb (🥇) spent the day doing a bisect and we seem to have found the culprit. Fortunately it seems to essentially be a single-line: a4b2c57

In all honesty, I have no idea how this commit can have such a severe performance impact, but it seems it does. Perhaps @ianlancetaylor can shed some light on this?

@cherrymui
Copy link
Member

Thanks. Interesting. If this is only for running when connected to XCode, I think you can set the environment variable GODEBUG=asyncpreemptoff=1 to disable async preemption. In the past I remember LLDB had hard time handling signals (seems better now?). Maybe that is related.

@rayvbr
Copy link
Author

rayvbr commented Jan 23, 2023

I can confirm that setting GODEBUG=asyncpreemptoff=1 at runtime solves the issue.

I don't understand enough of the underlying problem that a4b2c57 is solving to understand if there could be other ways of solving this though, ways that would not result in the issues with XCode without having to set GODEBUG.

@cherrymui
Copy link
Member

cherrymui commented Jan 23, 2023

That change made sure the signal handler for asynchronous preemption signal is installed in c-archive and c-shared modes, so asynchronous preemption is enabled in those modes. Previously it was not enabled, due to neglect, not intentional. This changed fixed the issue. (iOS Go programs almost always use those build modes.)

I think the slowdown is due to the slowness of signal handling in XCode/LLDB. As this only affect running under XCode (i.e. debugging), I think setting GODEBUG is actually the best option.

Which version of XCode are you using?

@rayvbr
Copy link
Author

rayvbr commented Jan 24, 2023

Which version of XCode are you using?

We tested on a various XCode versions, the oldest one being ~1.5 years old, and the newest one being the latest 14.2.

I think the slowdown is due to the slowness of signal handling in XCode/LLDB. As this only affect running under XCode (i.e. debugging), I think setting GODEBUG is actually the best option.

I'm a bit out of my depth here, but if the problem is Xcode/LLDB-related, isn't it odd that only threads created by the Go runtime suffer from it?

@mknyszek
Copy link
Contributor

I'm a bit out of my depth here, but if the problem is Xcode/LLDB-related, isn't it odd that only threads created by the Go runtime suffer from it?

The Go runtime just happens to send OS signals much more often than other programs due to the design of asynchronous preemption (what GODEBUG=asyncpreemptoff=1 is disabling). The Go runtime only sends these signals to Go threads because it's trying to preempt running goroutines that have been taking a while to stop running (for one reason or another). It's still not that many (and it depends on the application; lots of tight loops without preemption points in user code makes this more likely), but if the debugger's emulation/delivery of signals is really slow, it can look like the whole program is slowing down a lot.

@ignoramous
Copy link

Anyone know if this issue affects Android?

@cherrymui
Copy link
Member

Given that it is specific to Xcode and LLDB, I don't think it affects Android (at least not in the same way).

@arjenveenhuizen
Copy link

Given that it is specific to Xcode and LLDB, I don't think it affects Android (at least not in the same way).

We (colleague of @rayvbr) have not observed this performance issue on Android. Indeed, this appears to be iOS-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
Development

No branches or pull requests

7 participants