Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/mobile/cmd/gomobile: re-implement marshal/unmarshal for android binary resources #13109

Closed
dskinner opened this issue Oct 30, 2015 · 13 comments

Comments

@dskinner
Copy link
Member

Currently, there exists an implementation of encoding the binary xml format for android which has the following issues:

New Implementation

Binary resource structs will support unmarshalling the binary output of aapt against a targeted version of the sdk build-tools for compatibility. Implementations of marshalling for each struct must produce the exact input sent to unmarshalling. This allows tests to validate each struct representation of the binary format as follows:

  • unmarshal the output of aapt
  • marshal the struct representation
  • perform byte-to-byte comparison with aapt output per chunk header and body

This process should strive to make structs idiomatic to make parsing xml text into structs trivial.

Once the struct representation is validated, tests for parsing xml text into structs can become self-referential as the following holds true:

  • the unmarshalled input of aapt output is the only valid target
  • the unmarshalled input of xml text may be compared to the unmarshalled input of aapt output to identify errors, e.g. text-trims, wrong flags, sorting, etc

This provides validation, byte-for-byte, for producing binary xml resources against the targetted build-tools and allows pin-pointing implementation issues down to the chunk header of the binary format.

While the new implementation would support both marshal and unmarshal of all android binary resource types, the scope of this functionality would be as follows:

  • cmd gomobile supports marshalling of binary xml resources
  • a new go:generate command would support unmarshalling of binary resources for code generation of type safety information for use by cmd gomobile in the marshalling of binary xml resources
  • tests would make use of unmarshal and marshal for validating the implementation

Tests would be accompanied by a testdata/ directory containing manifests to identify corner cases of the implementation and the binary versions of each manifest, generated with the targeted build-tools version.

Regarding a new go:generate command to produce type safety information by cmd gomobile, this would use the new structs to decode the resource table from the android.jar of the targeted minimum api of gomobile, providing the necessary information for use by cmd gomobile.

It should be noted that this proposal was put together by piecemeal development to determine viability and much of the details of this proposal are already proven to be working by tests though no CL has yet to be sent as the work is incomplete.

@rakyll
Copy link
Contributor

rakyll commented Nov 2, 2015

/cc @hyangah @crawshaw

@crawshaw
Copy link
Member

crawshaw commented Nov 2, 2015

@dskinner what you're describing here sounds like reasonable improvements to the tool. Could you point me at some of your in-progress code so I could take a look?

I would like, ideally for us to target just one version of the manifest. Maybe it should be a later version than the one we target now, so we can support some extra features? I believe our practical lower API bound is 15, not 9, already.

Ideally users who want more sophisticated manifests can eventually do everything they need with `gomobile bind``, and use the .aar files in Android Studio. That still needs some extra work: we haven't made x/mobile/app and x/mobile/gl work with the bind mode, but I believe it's much easier than trying to make gomobile build do all of the work of Android Studio.

@dskinner
Copy link
Member Author

dskinner commented Nov 2, 2015

@crawshaw I submitted the following CL to show current progress: https://go-review.googlesource.com/16553

I've been holding off until some sort of "ok" was given, so I'll look at picking up work of this again.

Notably, right now it supports unmarshal/marshal of aapt output that matches byte for byte. I'm now looking to simplify the string pool struct to be more idiomatic but this is not yet complete.

This also supports reading portions of the resource table from an android.jar, including the table's string pool (which happens to be utf-8) and the beginning of the package resource type.

I failed to mention any current deficiencies with this proposal, here's a short list of what I have/know at the moment:

  • marshalling will generate a lot of garbage due to implementation details to conform to testing
  • I'm aware of a quirk with high/low bytes for string length where high word is 0 (wrt utf8 strings). The case I saw this for was a byte length of 147. My current guess is it uses a second byte if length overflows an int8 and not a uint8 but haven't looked further. If such quirks can't be identified, then tests wouldn't complete successfully.

I think a minimum api of 15 is ideal, notably to break away from the old menu action and styling before 4.x, e.g. see ... button that does nothing with a pure go app that defines minimum api-9. The display of old menu button may sound trivial compared to "supporting more features", but I don't really anticipate the need for a pure go app to require many features, just the basics.

Also, here's a reference for current api device statistics in the wild: http://developer.android.com/about/dashboards/index.html

I also think the cmd should only target one api version as you mentioned. What I've considered in this case is that specifying a minimum or target api within the manifest would produce an error and abort. This information would instead be inserted by gomobile cmd.

I also agree about not making gomobile try and do everything. More complex interactions using the gradle plugin and bind is definitely the place to go so it can scale to all features normally supported in a java app.

I'm also in favor of either a whitelist or blacklist (whichever is more practical) of what's supported in a manifest for a pure go app. But, this needs some thought put into it.

@crawshaw
Copy link
Member

crawshaw commented Nov 3, 2015

I think your CL is heading in the right direction. I've started reviewing it, and will hopefully have some more time tonight.

I'm not worried about the marshaling garbage generated. It's small compared to the I/O, and worst case can be done concurrently with some other tasks.

@dskinner
Copy link
Member Author

I've just submitted https://go-review.googlesource.com/19040

I believe this tackles some of the last difficult problems with the implementation, with only one pending with regards to how sorting is handled by attributes of the same data type (which may be of no consequence).

At this point, the only thing left to do is rewrite many of the MarshalBinary methods to actually generate a correct chunk header (as type Pool already does) based on the data it actually contains and fill in a few other fields missing data. Once this is done, things should be working.

There are a number of TODOs I'll be reviewing in the source and I may refactor a few things. Aside from this, I have the following concerns with moving forward.

  • How will packed.arsc.gz be distributed? (this is the stripped down resources from android.jar used for validation of attributes and values, currently 63KB)
  • Do we want to add a temporary flag to cmd gomobile for using this method to handle manifests? This may allow feedback from any current users before migrating.
  • should any current functionality related to android that exists elsewhere make its way into the package? In bind tool, see minAndroidAPI, androidAPIPath(), which are relevant to binres functioning as intended.
  • What current issues warrant the need for exports in the package? Right now I suspect providing an app icon may depend on generating a resource table in the apk, pending further investigation. All that's necessary to have exported for encoding manifest is binres.UnmarshalXML and binres.XML. Or, do we want to leave some slack since this is an internal package?

@dskinner
Copy link
Member Author

@hyangah or @crawshaw, can you comment on the distribution of packed.arsc.gz? Is this something gomobile init should take care of downloading? If so, I'm not quite sure what I should do to assist with that infrastructure-wise. Putting the data in a generated file as []byte seems impractical as the size of such a file is 373KB.

@hyangah
Copy link
Contributor

hyangah commented Feb 29, 2016

Sorry for the delay.

It sounds reasonable to me to make gomobile init take care of downloading. x/mobile/cmd/gomobile/release.go is the code that prepares archives that are downloaded by the gomobile init command. It also generates fetchHashes found in hashes.go. To make the release
process easier, please put the code for building packed.arsc.gz in the release.go. Once done, we can make the file available for download.

Re: use of minAndroidAPI and androidAPIPath of 'gomobile bind' is used only to pick the android.jar file needed to run javac and androidAPIPath just searches the ANDROID_HOME directory to find any target platform that satisfies minAndroidAPI requirement. gomobile bind itself doesn't deal with binary resources directly, so I doubt the binres package is relevant.

Re: temporary flag. I'd like to avoid any flag if possible. We can try to communicate with current users to test it (via golang-dev or golang-nuts or forum) if there is a chance of breaking the current usage. Or, like useStrippedNDK (in init.go), we can keep the old code for a while during migration and guide users to utilize that if necessary.

@dskinner
Copy link
Member Author

sounds good, I'll update https://go-review.googlesource.com/#/c/20030/ with changes for release.

@hyangah
Copy link
Contributor

hyangah commented Feb 29, 2016

just double checking - is the file 373kb or 63kb?

@dskinner
Copy link
Member Author

dskinner commented Mar 1, 2016

@hyangah
Copy link
Contributor

hyangah commented Mar 1, 2016

I just ran go-bindata to pack the gz file and the generated source is under ~230KB. Compared to golang.org/x/tools/godoc/static/static.go (~930KB) or golang.org/x/text/*/tables.go (~4.7MB), it's still small. Why can't we just embed it as a []byte in a generated file? Probably I am missing something.

@dskinner
Copy link
Member Author

dskinner commented Mar 1, 2016

we could certainly do that. I took a naive approach of simply doing a fmt.Sprintf("var arsc = %#v", bin) with a go generate cmd and produced a file 373KB in size which is what I mentioned above. This includes a bunch of extraneous 0xs.

I just simply don't know what is acceptable as @crawshaw initially expressed concern about the arsc file size of 5.6MB but didn't mention an ideal size. I stripped a bunch of unused portions from the table and gzip'd it bringing the size down to 63KB. I believe for the foreseeable future the parts stripped will not be required given the current goals.

If embedding the byte array in source is acceptable at that size (230KB with go-bindata) then I'd definitely prefer doing that.

@gopherbot
Copy link

CL https://golang.org/cl/20030 mentions this issue.

@golang golang locked and limited conversation to collaborators Mar 13, 2017
imWildCat pushed a commit to imWildCat/go-mobile that referenced this issue Apr 10, 2021
Current output byte-for-byte of pkg binres is close,
but not exact, to output of aapt.
The current exceptions to this are as follows:
 * sort order of certain attributes
 * typed value of minSdkVersion
These differences do not appear to affect the encoded
manifest from working correctly. Further details on
the byte differences can be seen in TestEncode.

Fixes golang/go#13109

Change-Id: Ibfb7731143f0e2baeeb7dd5b04aa649566606a53
Reviewed-on: https://go-review.googlesource.com/20030
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
imWildCat pushed a commit to imWildCat/go-mobile that referenced this issue Apr 11, 2021
Current output byte-for-byte of pkg binres is close,
but not exact, to output of aapt.
The current exceptions to this are as follows:
 * sort order of certain attributes
 * typed value of minSdkVersion
These differences do not appear to affect the encoded
manifest from working correctly. Further details on
the byte differences can be seen in TestEncode.

Fixes golang/go#13109

Change-Id: Ibfb7731143f0e2baeeb7dd5b04aa649566606a53
Reviewed-on: https://go-review.googlesource.com/20030
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants