Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/mobile/cmd/gomobile: multiple packages binding #12245

Closed
hyangah opened this issue Aug 21, 2015 · 18 comments
Closed

x/mobile/cmd/gomobile: multiple packages binding #12245

hyangah opened this issue Aug 21, 2015 · 18 comments
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile
Milestone

Comments

@hyangah
Copy link
Contributor

hyangah commented Aug 21, 2015

I need to gobind multiple packages for an app.

I considered running gomobile for each package and import multiple .so files, but that will fail due to various reasons:

  • gobind framework for java assumes a single 'gojni.so' file
  • duplicate code for go runtime in multiple .so files; conflicting symbols, storage, ...

Option 1:

gomobile bind -target=android -o=myapp.aar pkg1 pkg2 pkg3

The command will generate separate java classes for those packages, and gather all asset files under one asset directory. Conflicting package names and asset file names are error.

Option 1.a

gomobile bind -target=android -o=myapp.aar -pkg=com.example.foo,org.example.bar,net.example.baz pkg pkg2 pkg3

Option 1.b

gomobile bind -target=android -o=myapp.aar pkg1:com.example.foo pkg2:org.example.bar pkg3 ...

Option 2:

gomobile bind -target=android -o=myapp.aar -list=something.json

Make gomobile take an input file of a certain format that lists all packages for gobind.

   /* something.json */
   {
        Name: "pkg1",
        Android: "com.example.foo",
        IOS: "MyFoo"
   }

Option 3:

Fake main package file - so when gomobile bind sees the 'main' package, it runs gobind on all the imported packages, and process the command that specifies the custom package name for java and prefix for obj-c.

gomobile bind -target=android -o myapp.aar myapp.go

   /* myapp.go */
   package main
   import (
          _ "pkg1"  // android:com.example.foo ios:MyFoo
          ...
   )
   func main() {
   }

/cc @crawshaw @rakyll

@hyangah hyangah added this to the Unreleased milestone Aug 21, 2015
@hyangah hyangah self-assigned this Aug 21, 2015
@rakyll
Copy link
Contributor

rakyll commented Aug 24, 2015

There are two distinct problems in the scope of this issue. Although they are relatively connected, we need to address them individually.

Problems:

  1. Name congestions among multiple packages.
  2. Being able to name Java packages, using custom prefixes for headers on iOS.

The solution to the first problem is relatively easy. If there are name congestions, reject to generate bindings or name the packages.

An example:

 $ gomobile bind -target=android a/sensor b/sensor

It may generate two classes ASensor.java and BSensor.java. We can simply prefix the class name in the most minimal way it will be distinguishable from the other package. Given the fact that, Go import path is unique within the GOPATH, we will eventually find a unique name.

If the user is not happy with the auto-assigned names, they should be able to override the namespace themselves.

I don't have a strong preference how we should represent the mapping but the information shouldn't be sitting in a separate file. Would you like to carry the discussion to golang-dev@ to ask feedback from a wider audience?

@hyangah
Copy link
Contributor Author

hyangah commented Aug 26, 2015

I will not worry too much about the collision in the package names at this moment. I assume the user has full control over what the package name will be. So, for two sensor packages, the user can name one in sensorA, and another in sensorB. For binding 3rd party packages, the user can wrap them with more meaningful names.

I think what I will end up implementing is

  $ gomobile bind -target=android -pkgpath=com.golang.app a.b.c x.y.z

This will creates two classes C.java and Z.java under the same package com.golang.app. All classes will be under the same package path (com.golang.app). If pkgpath is not given, go.c.C and go.z.Z.

For objective-c,

  $ gomobile bind -target=ios -prefix=App a.b.c x.y.z

This will create AppC.{h,m} and AppZ.{h.m}, and AppC and AppZ will be used as prefixes of generated functions and types. If -prefix is omitted, GoC and GoZ.

@crawshaw
Copy link
Member

SGTM

@gopherbot
Copy link

CL https://golang.org/cl/13969 mentions this issue.

hyangah added a commit to golang/mobile that referenced this issue Aug 28, 2015
Introduce options -javapkg and -prefix for gobind command.

The following generates java class Testpkg with package name com.example.

gobind -lang=java -javapkg=com.example testpkg

The following generates objective-c files where function and type names
are prefixed with ExampleTestpkg.

gobind -lang=objc -prefix=Example testpkg

As discussed in golang/go#9660 and golang/go#12245.

Gomobile support is not yet implemented.

Change-Id: Ib9e39997ce915580a5a2e25643c0c28373f27ee1
Reviewed-on: https://go-review.googlesource.com/13969
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@sridharv
Copy link

sridharv commented Oct 1, 2015

For binding 3rd party packages, the user can wrap them with more meaningful names

Please reconsider this decision. One of the strengths with the Go ecosystem is that it is incredibly easy to share code. If I build a useful library and every developer will now have to manually write a wrapper the barrier to adoption is significantly higher. For example, I would love to just be able to go get a library to handle networking (checking network type WiFi/3G/etc) or location and just have it work. Having to now write a bunch of code to essentially re-implement the API imposes an unnecessary burden.

All of these problems are very easily solved if you just provide a separate command in gomobile to build. Then I can do the following easily:

$ gobind pkg1
$ gobind pkg2
$ gobind pkg3

Now manually create my main

package main

import (
    _ "pkg1"
    _ "pkg2"
    _ "pkg3"
)

func main() {
    panic("Does nothing!")
}

And build

$ gomobile build -target=ios -buildmode=objc-framework

This is consistent with how the go tool works and makes sharing code very easy. Even better, I can use go generate and it all fits beautifully into the existing go workflow. As an added bonus the source is now available for me to debug.

EDIT: I forgot to add, that the above approach has the advantage that it makes incremental builds possible. Having gobind being run automatically by Intellij/Android Studio/XCode is very helpful since it avoids build errors. Right now the framework / aar is rebuilt on every iteration.

@hyangah
Copy link
Contributor Author

hyangah commented Oct 1, 2015

@sridharv Can you explain what the gobind does? Currently it does only code generation (.java, .h, .m, .go) but your mention of incremental builds makes me think the gobind in your proposal does something more than that.

  • what's the output of the gobind? (source code? compiled object or class files? archive?)
  • where should the gobind output go (so, the following gomobile build command finds the output of previous gobind commands and IDEs build task)? Every IDE has different assumptions on source code layout, or default build framework.

Use of buildmode flag in gomobile build is interesting, and I want to hear more about how it connects the main package and the product of previous gobind+IDE runs.

Besides, I want to know how to handle the case when exported functions of pkg1 and pkg2 both depend on some types of pkg3.

package pkg1
func Read(io.Reader)

package pkg2
func Work(io.Reader)

Our plan was to export only one type for io.Reader in this case, so both Read and Work use the same type.

I don't understand what prevents the use of go generate currently.

@sridharv
Copy link

sridharv commented Oct 2, 2015

Firstly, thank you for responding so quickly. Secondly, I should apologize for saying the problem is "very easily solved". It definitely comes off as glib, which I didn't realize when writing it. In addition I don't think I explained some of the usage scenarios I had in mind properly. Now onto your questions.

Besides, I want to know how to handle the case when exported functions of pkg1 and pkg2 both depend on some types of pkg3.

That's a very good point. What I described earlier doesn't address this properly since there is room for user error when running gobind manually for each package, which is never a good thing. Having said that, a single step for code generation makes the "separate bind and build" irrelevant for this specific issue. Now that I think about it, I was primarily motivated by some other problems I am currently facing with having the code gen and build be part of one step. I'll answer your questions regarding that in a separate comment and create a separate issue for the questions I have.

Proposal

There are two parts to this. One addresses the problem of manually wrapping libraries and the other, sharing cross platform code.

Binding multiple libraries in the face of package conflicts

Create a main file as you suggested in Option 3. Instead of adding comments per package just list all the packages. For packages that have name collisions use the standard Go mechanism of resolving name collisions.

package main

import (
    _ "pkg1"
    otherpkg1 "some/other/pkg1"
    _ "pkg2"
)

// Avoid the unused import warning
var _ = otherpkg1.SomeFunction

func main() {
    panic("placebo main")
}

Now run gomobile bind as you decided on earlier, but instead of listing all packages just point to this main package.

Java:

$ gomobile bind -target=android -pkgpath=com.golang.app path.to.main.package

Objective-C:

$ gomobile bind -target=ios -prefix=App path.to.main.package

This makes it consistent with how go resolves the same package namespace collision issue and keeps things simple as with your earlier decision. Things can be made even simpler if you disallow having any other exported members in the main package so you don't have to worry about how to bind those.

Sharing libraries with supporting Java or ObjC code

This is a separate issue, but it is affected by the choice of prefix/package path.

Consider the following scenario. I am writing an SDK app with go code shared between iOS and Android. I want to do the following in my shared go code:

  • Know which directory to store my database files in.
  • Know whether the network is on WiFi or 3G so I can decide whether or not to download a file.
  • Run a download in the background.

All the above have standardized solutions in Android and iOS respectively. If I wanted a common interface to the above I could write a go library that accepts an interface and have Java/ObjC implement that interface. The go code then calls that interface to get the directory, network state, etc. This is the kind of code that can easily live in a third party library and make development easy for everyone. However, in order to do this, the library author needs to know what Java interface / ObjC protocol to implement in their Java / ObjC supporting code.

One solution is to allow package authors to specify their own Java/ObjCpackage prefix.

//Package network implements cross-platform support for network state on iOS and Android
package network // java:com.foo.network ios:GoNet

Packages that force a prefix are disallowed from having third party package references in their exported code. This ensures that the problem you described does not occur.

Now I can do the following:

$ go get gomobile/support # (or something similar)

And write go code like so:

import (
    "gomobile/support/network"
    "gomobile/support/storage"
)

func DoSomething() {
    switch network.Type() {
    case network.WiFi:
    // do something
    }
}

func DoSomethingElse() {
    // ... compute url
    location := filepath.Join(storage.UserDataDir(), "somefile")
    network.DownloadInBackground(url, location)
}

The network and storage packages would handle their platform specific code by using their SDK provided Java/ObjC code.

Distributing the supporting code

For this, one way would be to ask library authors to use the native code distribution options for their language. They would use Maven for Java and Cocoapods/Carthage for iOS. They just have to be careful not to package the generated binding code. Another would be allow them to place these files in a directory in their go tree. These files are then automatically copied by gomobile bind when it is generating the bindings for these packages. This makes things easier since a user of the library would just go get a library and be able to use it without having to worry about the right version of the supporting native code.

The gobind tool could follow the convention that any files found in a directory named java under a go package are copied to the generated java package. For ObjC a directory named objc is treated similarly. It would be simpler to have the Java and ObjC files live in the same directory as the go package, but this causes problems when building for android since the go tool complains about ObjC files being present without cgo being used.

I'll put together a small prototype describing the above scenarios so it is clearer.

@sridharv
Copy link

sridharv commented Oct 2, 2015

I'm placing my answers to your questions about code gen + build here.

what's the output of the gobind? (source code? compiled object or class files? archive?)

The outputs are still source files.

where should the gobind output go (so, the following gomobile build command finds the output of previous gobind commands and IDEs build task)? Every IDE has different assumptions on source code layout, or default build framework.

This is an important question and one I didn't address in my comment. I don't think IDEs need to automatically know about the location of the source. The only thing that needs to know is the gomobile tool so that it can pick up these files for the build step. This can be achieved with a command line flag.

Use of buildmode flag in gomobile build is interesting, and I want to hear more about how it connects the main package and the product of previous gobind+IDE runs

It can pick up the location of the files using a command line flag. The IDE will will pick the output of gomobile build in the same way it picks up the output of gomobile bind. The IDE can run gomobile build automatically (XCode Run Script Phase/Gradle plugin) whenever it builds the SDK app. However, the more I think about it this can be done in gomobile bind as well.

I don't understand what prevents the use of go generate currently.

Nothing whatsoever :). However, my understanding is that go generate is intended to be used for pre-build steps such as code generation. Library authors can then commit generated source to their library source tree. It feels like a misuse of go generate to use it to generate a .framework or .aar. Those operations feel like they should be done by go build/gomobile build/Makefile/build.gradle.

EDIT: I created #12819 to track my main concerns about code generation in gobind. Most of what I would like can be achieved without separating the two phases.

@sridharv
Copy link

sridharv commented Oct 2, 2015

I put together a prototype for reading the list of packages from a main file and specifying a package name override. It only works for gobind and is not fully tested right now. Depending on my time constraints I'll port it to gomobile and finish testing tomorrow.

sridharv/gomobile-java@710bb3c

@sridharv
Copy link

I currently need this feature for some work I'm doing. I'm happy to implement what was decided earlier on Aug 27 if you haven't had the time to look at it yet. Please let me know if this will be useful to you.

@sridharv
Copy link

I had time on Sunday and ended up implementing this (what was decided on Aug 27) for my personal use (using some of the earlier prototype code I'd written). The change is on the following branch: https://github.com/sridharv/mobile/tree/multibind2

The current status:

  • Tested and working on Android.
  • Referring to exported types from another package supported.
  • Common generation code for gobind and gomobile bind.
  • Custom prefixes/package paths not tested.
  • Unit tests present for ObjC but not tested in an actual app.

If this will be useful I'll write up the approach I took, test for iOS and send the CL for review. I'm happy to make any changes needed. If it's not useful I'll use it for my work now and cut over to the official version when it lands :)

@hyangah
Copy link
Contributor Author

hyangah commented Nov 16, 2015

@sridharv Thanks for working on this, and sorry for not responding quickly.

I am unfortunately lost where the discussion went; too many things are going on in this issue. Let's limit this issue to be what's mentioned in the title. Multiple package support.

By 'what's decided earlier on Aug 27', do you mean my posting on Aug 26? Then, yes, feel free to send a CL for that. I expect the first one should be just changing the current gomobile to support multiple packages. gobind already supports binding of multiple packages. The types should be limited to each package.

For supporting types from other packages, there is a separate issue (#12570) so update that one.

@crawshaw is now changing the way to extract exported types, so please wait for us to finish https://go-review.googlesource.com/#/c/16913/ and sync.

For other stuff you mentioned in your proposals -

  1. Binding multiple libraries in the face of package conflicts: let's get back to this when we find the value of this approach compared to what you will implement for now.
  2. Sharing libraries with supporting Java or ObjC code: I don't think I fully understand this. But let's discuss this in the golang-nuts forum, not in this issue. (please cc @crawshaw and @hyangah)

Thanks!

@sridharv
Copy link

@hyangah Thanks for the quick response!

I'll split this into two CLs. One for what you decided on Aug 26 (it shows up as 27 in my timezone) and another for the external types portion. Given how I've implemented this change it might be worth it to wait for https://go-review.googlesource.com/#/c/16913/ for both changes, but I'll defer that decision to the review. Expect this in 2-3 days time (the week has started so I'm restricted to the evenings for this).

For the other stuff, I'll send an email to golang-nuts for (2) and we can revisit (1) later as you suggested.

@gopherbot
Copy link

CL https://golang.org/cl/17010 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/17023 mentions this issue.

hyangah pushed a commit to golang/mobile that referenced this issue Nov 24, 2015
As discussed in golang/go#12245

Usage: gomobile bind [options] a.b.c x.y.z

For java gobind and gomobile  will generate go.c.C.java and go.z.Z.java.
If -javapkg=com.example is specified they will generate
com.example.C.java and com.example.Z.java.

Tested on Darwin.

Change-Id: Ia8e57c8fec7967131d55de71cc705d9e736ccca0
Reviewed-on: https://go-review.googlesource.com/17023
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@sridharv
Copy link

sridharv commented Dec 1, 2015

I will send the 2nd CL (iOS) late this week and the 3rd CL (Documentation) next week. I had hoped to do this last week, but had time constraints.

@gopherbot
Copy link

CL https://golang.org/cl/17475 mentions this issue.

hyangah pushed a commit to golang/mobile that referenced this issue Dec 9, 2015
As discussed in golang/go#12245

Usage: gomobile bind [options] a.b.c x.y.z

For ObjC, gomobile bind will generate GoC.{h,m} and GoZ.{h,m}. If
-prefix=App is specified it will generate AppC.{h,m} and AppZ.{h,m}.

Tested on Darwin.

Change-Id: I6af8539a0fb7ed6256f3773efc514eff436014b4
Reviewed-on: https://go-review.googlesource.com/17475
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@gopherbot gopherbot added the mobile Android, iOS, and x/mobile label Jul 20, 2017
@eliasnaur
Copy link
Contributor

I believe this has been working for quite a while now.

@golang golang locked and limited conversation to collaborators Mar 8, 2019
@rsc rsc unassigned hyangah Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile
Projects
None yet
Development

No branches or pull requests

6 participants