Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/mobile: add bitcode support #22395

Closed
ashutosh-android opened this issue Oct 23, 2017 · 55 comments
Closed

x/mobile: add bitcode support #22395

ashutosh-android opened this issue Oct 23, 2017 · 55 comments
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile
Milestone

Comments

@ashutosh-android
Copy link

I am able to successfully generate and use .framework file using gomobile bind command, however generated lib doesn't have bitcode enabled
Is there a way to enable bitcode in generated binary or are there are any plan to provide support in near future ?

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64
gomobile version +44a54e9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

MacOS

What did you do?

Generated .framework lib from go source code using 'gomobile bind'

What did you expect to see?

bitcode should be enabled in generated framework

What did you see instead?

generated framework doesn't have bitcode enabled

@gbbr gbbr changed the title 'gomobile bind' generated framework file doesn't have bitcode support x/mobile: 'gomobile bind' generated framework file doesn't have bitcode support Oct 23, 2017
@gopherbot gopherbot added this to the Unreleased milestone Oct 23, 2017
@gopherbot gopherbot added the mobile Android, iOS, and x/mobile label Oct 23, 2017
@ashutosh-android
Copy link
Author

Can you please update on this. This is a blocker item for one of my project. It will be great if I get some visibility on release plan for this requirement. Thanks.

@timcooijmans
Copy link
Contributor

timcooijmans commented Oct 27, 2017

I'm having the same problem: Currently the Go compiler doesn't support compiling to BitCode intermediate representation. I think something has to be done for Gomobile to have a future. It's expected that Apple will mandate BitCode for iOS-apps in the near future.

However there seems to be some work going on on Google-side: https://go.googlesource.com/gollvm/

You could also try https://dragonegg.llvm.org/ but I have no idea if this is going to work.

@ashutosh-android
Copy link
Author

Any update on this ?

@eliasnaur
Copy link
Contributor

I don't think there is any update other than gollvm is progressing. May I ask why this issue is blocking your project?

@ashutosh-android
Copy link
Author

@eliasnaur thanks for looking into this query.
I am working on a mobile project where we will be distributing our service as library project. We are writing alot of business logic in go code so that we can reuse them for both android and iOS using go-mobile. Since framework generated by go-mobile doesn't have bitcode enabled so any developer who will be integrating our solution will not be able to enable bitcode for there application. Considering how apple is strongly recommending developers to enable bitcode it will impact adoption our product in a big way.

I hope this answers your question. Let me know if you have any suggestions/query. Thanks again.

@matti777
Copy link

Just ran into this. Yes, bitcode support would be a must. Any updates?

@jpop32
Copy link

jpop32 commented Apr 26, 2018

Yes, lack of bitcode support on iOS is a showstopper for library projects, and a big liability for others. We can safely assume that 'strong recommendation' to use bitcode is going to turn into 'requirement' sooner or later.

@adam-p
Copy link
Contributor

adam-p commented Apr 26, 2018

Since it hasn't been mentioned here yet: Apple TV apps must support Bitcode. Not optional. Search for "bitcode" here: https://developer.apple.com/tvos/submit/

This has caused us problems when trying to support a client wanting to use our library on Apple TV.

@zboralski
Copy link

Just ran into this.

@drewpitchford
Copy link

I'll throw my hat in this ring too. We have iOS/Android sdks that use a go-mobile framework for some common encryption/decryption. Lack of bitcode is a major issue for us. I am attempting to replace the go-mobile framework in the iOS project with a js framework that does the same stuff, but it'd be nice to be able to keep the go framework

@pavlo
Copy link

pavlo commented May 24, 2018

It is a showstopper for us as well

@biscottigelato
Copy link

WWDC is coming soon, hopefully Bitcode will not become a requirement... The lack of any response from anyone in the know about how one an get Bitcode going for golang/gomobile is definitely concerning.

@ianlancetaylor
Copy link
Contributor

As far as I know the only way to get the required Bitcode is to run LLVM. There is an active project to add Go support to LLVM; you can follow it at https://go.googlesource.com/gollvm/ . Until that project is complete and working, there will be no way to get Bitcode.

That is essentially what @eliasnaur said on March 16. While I understand that the pace of these things is frustrating, I don't think it's entirely fair to say that there has been no response. Go is an open source project; if you want this to happen faster, try building GoLLVM, report bugs, and send in fixes.

@ashutosh-android
Copy link
Author

Hi,

Checking to see if there is any new development on this ?

Thanks,
Ashutosh

@ncastagnoli
Copy link

Hi,

To add to the need, if that's important.

I'm working on a project that started cloud based, and elected to write the code in "go." It turns out we also need to run the code on the client, both iOS and Android. But, our partners are demanding bit-code compliance. We are at a decision point to decide whether to rewrite the entire set of code so we can have a single code base. Curious about the thinking regarding bit code compliance for "go."

Is it a project? If so, what is the time-frame? Go is perfect for our application, incidentally.

@ashutosh-android
Copy link
Author

Hi,

Checking again on this. Is there any roadmap for this support ?

Thanks,
Ashutosh

@eliasnaur
Copy link
Contributor

Please refrain from asking about updates on this issue; fixing it is a lot of work and only benefits niche platforms for Go: tvOS and watchOS. @ianlancetaylor suggested you try gollvm; have you done that? If so, what problems did you run into?

@eliasnaur eliasnaur changed the title x/mobile: 'gomobile bind' generated framework file doesn't have bitcode support x/mobile: add bitcode support Dec 18, 2018
@ianlancetaylor
Copy link
Contributor

@matti777
Copy link

Ahem, what do you mean tvOS and watchOS? What about iOS? :) Not sure if that is a 'niche' environment for gomobile bind..

@eliasnaur
Copy link
Contributor

Only watchOS and tvOS require bitcode; iOS does not.

@drewpitchford
Copy link

drewpitchford commented Dec 18, 2018 via email

@eliasnaur
Copy link
Contributor

Whatever the likelyhood, let's worry about it when it happens and not waste precious development efforts on hypotheticals. Who knows, perhaps the very existence of non-bitcode gomobile apps will delay Apple's decision.

@matti777
Copy link

iOS "requires" bitcode in the way that if you are making a library with golang / gomobile bind, the library you produce will force the iOS app developer to disable bitcode in their app; and we iOS devs really do not like that. :) Also of course there is a point to having bitcode support in the first place and that is now lost. third, who knows when iSteve will decide bitcode support will become mandatory ..

@ianlancetaylor
Copy link
Contributor

Everyone agrees that it would be good if we could compile Go into bitcode. As mentioned above, the only way that is ever going to happen is if you use the GoLLVM compiler. So if you want Go compiled into bitcode, please help with GoLLVM.

@tmm1
Copy link
Contributor

tmm1 commented Mar 13, 2019

The Kotlin/Native team has found a work-around to full bitcode support: they add a __LLVM,__asm marker (via JetBrains/kotlin-native#2186) which tells Xcode the object was created with a bitcode-aware compiler, and that the object was generated from assembly and thus has no bitcode equivalent. (More details about the bitcode format on https://jonasdevlieghere.com/embedded-bitcode/).

See discussion on the kotlin-native repo: JetBrains/kotlin-native#1202 (comment)

The technique was discovered on an earlier PR, which attempted to emit bitcode but was unable to do so reliably without using Apple's version of LLVM: JetBrains/kotlin-native#1564 (comment)

Would it be possible to add a similar empty __LLVM,__asm section to the generated gomobile .framework?

@randall77
Copy link
Contributor

It would not be hard to add an empty section to the binary.

I worry that this is essentially a way to hack around Apple's policy. As such, they could revoke this exception at any time and then we're back to square one.

gopherbot pushed a commit that referenced this issue Mar 19, 2019
Updates #22395

Change-Id: I6c207934b32d38374875f756c4f8c6dfe38d8cb0
Reviewed-on: https://go-review.googlesource.com/c/go/+/168318
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit to golang/mobile that referenced this issue Mar 19, 2019
Add -fembed-bitcode to the host compiler and linker to instruct it
to add bitcode to object files.

Updates golang/go#22395

Change-Id: Ie0297079f27c9b5b8ea55e0a36067dc768342986
Reviewed-on: https://go-review.googlesource.com/c/mobile/+/168062
Run-TryBot: Elias Naur <mail@eliasnaur.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@tmm1
Copy link
Contributor

tmm1 commented Mar 19, 2019

@eliasnaur I'm curious what impact your whitelist change in b4f3b8a has?

@eliasnaur
Copy link
Contributor

Very little; it allows the tvOS and watchOS flags in #cgo directives. I noticed the missing flags while working on this.

@tmm1
Copy link
Contributor

tmm1 commented Mar 19, 2019

Very little; it allows the tvOS and watchOS flags in #cgo directives. I noticed the missing flags while working on this.

Aha, okay. It was working for me already via the CGO_CFLAGS env vars so I was curious what the change enabled.

Maybe a GODARWIN would be simpler.

Another idea would be to add some new flags to the go linker, so the user could do something like -ldflags="-machoplatform=ios -machominver=11.0.0"

If we can settle on a good approach here, I'm happy to start putting together some CLs.

Another thing to keep in mind is that the LC_VERSION_MIN_* have been replaced with LC_BUILD_VERSION for the next macOS release, and that this new load command will likely be required to build marzipan apps which work on both iOS and macOS. However since all existing macOS/iOS/tvOS/watchOS releases still use the older variant, we will need to keep supporting that for the foreseeable future.

For reference, here are the two load command definitions:

    struct version_min_command {
      uint32_t cmd;       // LC_VERSION_MIN_MACOSX or
                          // LC_VERSION_MIN_IPHONEOS
      uint32_t cmdsize;   // sizeof(struct version_min_command)
      uint32_t version;   // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t sdk;       // X.Y.Z is encoded in nibbles xxxx.yy.zz
    };

    struct build_tool_version {
      uint32_t tool;      // enum for the tool
      uint32_t version;   // version of the tool
    };

    struct build_version_command {
      uint32_t cmd;       // LC_BUILD_VERSION
      uint32_t cmdsize;   // sizeof(struct build_version_command) +
                          // ntools * sizeof(struct build_tool_version)
      uint32_t platform;  // platform
      uint32_t minos;     // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t sdk;       // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t ntools;    // number of tool entries following this
    };

One advantage to the linker flag approach is that we could add more flags like -machosdkver etc as the need arises. Whereas GODARWIN could only support at-most platform and minver (via GODARWIN=ios or GODARWIN=ios11.0.0).

@gopherbot
Copy link

Change https://golang.org/cl/168321 mentions this issue: cmd/link/internal/ld: copy LC_VERSION_MIN_* commands to Go object file

@gopherbot
Copy link

Change https://golang.org/cl/168320 mentions this issue: cmd/link: enable bitcode build for iOS/tvOS/watchOS

@eliasnaur
Copy link
Contributor

You can check out my current progress at the two CLs posted above. I got LC_VERSION_MIN_* and LC_BUILD_VERSION copying to work, and only need tests.

tmm1 pushed a commit to fancybits/go that referenced this issue Mar 20, 2019
Updates golang#22395

Change-Id: I6c207934b32d38374875f756c4f8c6dfe38d8cb0
Reviewed-on: https://go-review.googlesource.com/c/go/+/168318
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/168459 mentions this issue: cmd/link/internal/ld: extract Mach-O load command parsing

gopherbot pushed a commit that referenced this issue Mar 21, 2019
We're going to need the ability to extract the LC_VERSION_MIN_* and
LC_BUILD_VERSION load commands. This CL adds peekMachoPlatform to do
that and in the process simplifies machoCombineDwarf.

While here, disable DWARF combining for Apple platforms other than
macOS (watchOS, tvOS, bridgeOS), not just iOS.

Updates #22395

Change-Id: I4862b0f15ccc87b7be1a6532b4d37b47c8f7f243
Reviewed-on: https://go-review.googlesource.com/c/go/+/168459
Reviewed-by: Ian Lance Taylor <iant@golang.org>
tmm1 pushed a commit to fancybits/go that referenced this issue Mar 21, 2019
The Go toolchain cannot output bitcode, but there is a trick where
object code can be marked with an __asm section, persuading the
Apple toolchain to include our object code in bitcode builds.

This enables Go builds with bitcode enabled; the next CL adds
the necessary plumbing for building on tvOS and watchOS.

Thanks to Aman Gupta for the trick.

Test is added two CLs from here.

Fixes golang#22395 (at least until Apple tightens bitcode requirements.)

Change-Id: Ic1c1448c4d46222bb3dd097b1f4df80848051e5f
Reviewed-on: https://go-review.googlesource.com/c/go/+/168320
Reviewed-by: Cherry Zhang <cherryyz@google.com>
(cherry picked from commit ba96564)
tmm1 pushed a commit to fancybits/go that referenced this issue Mar 21, 2019
We're going to need the ability to extract the LC_VERSION_MIN_* and
LC_BUILD_VERSION load commands. This CL adds peekMachoPlatform to do
that and in the process simplifies machoCombineDwarf.

While here, disable DWARF combining for Apple platforms other than
macOS (watchOS, tvOS, bridgeOS), not just iOS.

Updates golang#22395

Change-Id: I4862b0f15ccc87b7be1a6532b4d37b47c8f7f243
Reviewed-on: https://go-review.googlesource.com/c/go/+/168459
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 0fe1986)
gopherbot pushed a commit that referenced this issue Mar 21, 2019
To build for watchOS and tvOS the Apple toolchain requires a Mach-O
load command that matches the platform for all object files in a build.
The go.o object file produced by the Go linker contains no such command.

The loader commands are mutually exclusive so we need to pick the
right one. Fortunately, cgo must be enabled for watchOS and tvOS to
be useful, so we can copy the first loader command we find in the object
files produced by the host compiler.

Add a test that builds a small program for tvOS to test both this CL
and the previous CL that added bitcode support.

Updates #22395

Change-Id: I7a47d19be9d80f0459dc358c600cddd9f236c444
Reviewed-on: https://go-review.googlesource.com/c/go/+/168321
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
tmm1 pushed a commit to fancybits/go that referenced this issue Mar 21, 2019
To build for watchOS and tvOS the Apple toolchain requires a Mach-O
load command that matches the platform for all object files in a build.
The go.o object file produced by the Go linker contains no such command.

The loader commands are mutually exclusive so we need to pick the
right one. Fortunately, cgo must be enabled for watchOS and tvOS to
be useful, so we can copy the first loader command we find in the object
files produced by the host compiler.

Add a test that builds a small program for tvOS to test both this CL
and the previous CL that added bitcode support.

Updates golang#22395

Change-Id: I7a47d19be9d80f0459dc358c600cddd9f236c444
Reviewed-on: https://go-review.googlesource.com/c/go/+/168321
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
(cherry picked from commit 277609f)
@liyimeng
Copy link

How this should be used? Just update my gomobile package? I did so, but Xcode still fail and complains the framework is not bitcode enabled.

@liyimeng
Copy link

go.o does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@tmm1
Copy link
Contributor

tmm1 commented Mar 21, 2019

You would need to compile golang from master and use that.

@liyimeng
Copy link

aha, thanks! I though run
go get -u golang.org/x/mobile/cmd/... is enough. Let me give it another try.

@tmm1
Copy link
Contributor

tmm1 commented Mar 25, 2019

I've been using the new golang patches on master for a few days and have been able to run my golang code in the simulator and on tvOS/iOS devices. (Note that I am still using go directly to generate lib.a as outlined above instead of leveraging gomobile).

Now I'm trying to submit an app to the store and running into some weird issues. When submitting to the App Store, I get an email saying processing was stopped:

App Store Connect: Processing stopped for MyApp
While processing your iOS app errors occurred in the app thinning process, and your app couldn’t be thinned. If your app contains bitcode, bitcode processing may have failed. Because of these errors, this build of your app will not be able to be submitted for review or placed on the App Store. For information that may help resolve this issue, see Tech Note 2432.

I tried the debugging steps linked in the Tech Note to "rebuild from bitcode" in Xcode, and am getting a strange error about dSYMs missing in IDEDistributionPipeline.log:

ipatool failed with an exception: #<CmdSpec::NonZeroExcitException: Command exited with pid 71395 exit 1:
/Applications/Xcode.app/Contents/Developer/usr/bin/symbols -failIfMissingDsym -symbolsPackageDir /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp
Stdout:
    ERROR: Did not find dSYM for MyApp
    /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp [arm64, 0.001634 seconds]:

Stderr:
    error: Could not load symbol information for 22D519D5-0C78-3905-8681-93FE91325191.  Symbols file will contain no data.

>
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:248:in `run'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2473:in `block in CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `each'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2628:in `ProcessIPA'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:3445:in `<main>'

I noticed this:

https://github.com/golang/mobile/blob/6621de06e1e9d52c38a8c542770bdbeac9ecfef6/cmd/gomobile/build_iosapp.go#L74-L75

So I tried again with -ldflags=-w, and that worked! I assumed that wasn't necessary anymore after bcdbd58 but I guess it is still required.

I am now able to submit builds up to App Store:

App Store Connect: Version 2019 (3.25.619) for MyApp has completed processing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile
Projects
None yet
Development

No branches or pull requests