-
Notifications
You must be signed in to change notification settings - Fork 18k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: x/mobile/cmd/gomobile: re-implement marshal/unmarshal for android binary resources #13109
Comments
@dskinner what you're describing here sounds like reasonable improvements to the tool. Could you point me at some of your in-progress code so I could take a look? I would like, ideally for us to target just one version of the manifest. Maybe it should be a later version than the one we target now, so we can support some extra features? I believe our practical lower API bound is 15, not 9, already. Ideally users who want more sophisticated manifests can eventually do everything they need with `gomobile bind``, and use the .aar files in Android Studio. That still needs some extra work: we haven't made x/mobile/app and x/mobile/gl work with the bind mode, but I believe it's much easier than trying to make gomobile build do all of the work of Android Studio. |
@crawshaw I submitted the following CL to show current progress: https://go-review.googlesource.com/16553 I've been holding off until some sort of "ok" was given, so I'll look at picking up work of this again. Notably, right now it supports unmarshal/marshal of aapt output that matches byte for byte. I'm now looking to simplify the string pool struct to be more idiomatic but this is not yet complete. This also supports reading portions of the resource table from an android.jar, including the table's string pool (which happens to be utf-8) and the beginning of the package resource type. I failed to mention any current deficiencies with this proposal, here's a short list of what I have/know at the moment:
I think a minimum api of 15 is ideal, notably to break away from the old menu action and styling before 4.x, e.g. see Also, here's a reference for current api device statistics in the wild: http://developer.android.com/about/dashboards/index.html I also think the cmd should only target one api version as you mentioned. What I've considered in this case is that specifying a minimum or target api within the manifest would produce an error and abort. This information would instead be inserted by gomobile cmd. I also agree about not making gomobile try and do everything. More complex interactions using the gradle plugin and I'm also in favor of either a whitelist or blacklist (whichever is more practical) of what's supported in a manifest for a pure go app. But, this needs some thought put into it. |
I think your CL is heading in the right direction. I've started reviewing it, and will hopefully have some more time tonight. I'm not worried about the marshaling garbage generated. It's small compared to the I/O, and worst case can be done concurrently with some other tasks. |
I've just submitted https://go-review.googlesource.com/19040 I believe this tackles some of the last difficult problems with the implementation, with only one pending with regards to how sorting is handled by attributes of the same data type (which may be of no consequence). At this point, the only thing left to do is rewrite many of the MarshalBinary methods to actually generate a correct chunk header (as type Pool already does) based on the data it actually contains and fill in a few other fields missing data. Once this is done, things should be working. There are a number of TODOs I'll be reviewing in the source and I may refactor a few things. Aside from this, I have the following concerns with moving forward.
|
@hyangah or @crawshaw, can you comment on the distribution of packed.arsc.gz? Is this something |
Sorry for the delay. It sounds reasonable to me to make gomobile init take care of downloading. x/mobile/cmd/gomobile/release.go is the code that prepares archives that are downloaded by the gomobile init command. It also generates fetchHashes found in hashes.go. To make the release Re: use of minAndroidAPI and androidAPIPath of 'gomobile bind' is used only to pick the android.jar file needed to run javac and androidAPIPath just searches the ANDROID_HOME directory to find any target platform that satisfies minAndroidAPI requirement. gomobile bind itself doesn't deal with binary resources directly, so I doubt the binres package is relevant. Re: temporary flag. I'd like to avoid any flag if possible. We can try to communicate with current users to test it (via golang-dev or golang-nuts or forum) if there is a chance of breaking the current usage. Or, like useStrippedNDK (in init.go), we can keep the old code for a while during migration and guide users to utilize that if necessary. |
sounds good, I'll update https://go-review.googlesource.com/#/c/20030/ with changes for release. |
just double checking - is the file 373kb or 63kb? |
63kb, it's already in the repo here: https://github.com/golang/mobile/blob/master/internal/binres/data/packed.arsc.gz |
I just ran go-bindata to pack the gz file and the generated source is under ~230KB. Compared to golang.org/x/tools/godoc/static/static.go (~930KB) or golang.org/x/text/*/tables.go (~4.7MB), it's still small. Why can't we just embed it as a []byte in a generated file? Probably I am missing something. |
we could certainly do that. I took a naive approach of simply doing a I just simply don't know what is acceptable as @crawshaw initially expressed concern about the arsc file size of 5.6MB but didn't mention an ideal size. I stripped a bunch of unused portions from the table and gzip'd it bringing the size down to 63KB. I believe for the foreseeable future the parts stripped will not be required given the current goals. If embedding the byte array in source is acceptable at that size (230KB with go-bindata) then I'd definitely prefer doing that. |
CL https://golang.org/cl/20030 mentions this issue. |
Current output byte-for-byte of pkg binres is close, but not exact, to output of aapt. The current exceptions to this are as follows: * sort order of certain attributes * typed value of minSdkVersion These differences do not appear to affect the encoded manifest from working correctly. Further details on the byte differences can be seen in TestEncode. Fixes golang/go#13109 Change-Id: Ibfb7731143f0e2baeeb7dd5b04aa649566606a53 Reviewed-on: https://go-review.googlesource.com/20030 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Current output byte-for-byte of pkg binres is close, but not exact, to output of aapt. The current exceptions to this are as follows: * sort order of certain attributes * typed value of minSdkVersion These differences do not appear to affect the encoded manifest from working correctly. Further details on the byte differences can be seen in TestEncode. Fixes golang/go#13109 Change-Id: Ibfb7731143f0e2baeeb7dd5b04aa649566606a53 Reviewed-on: https://go-review.googlesource.com/20030 Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Currently, there exists an implementation of encoding the binary xml format for android which has the following issues:
New Implementation
Binary resource structs will support unmarshalling the binary output of aapt against a targeted version of the sdk build-tools for compatibility. Implementations of marshalling for each struct must produce the exact input sent to unmarshalling. This allows tests to validate each struct representation of the binary format as follows:
This process should strive to make structs idiomatic to make parsing xml text into structs trivial.
Once the struct representation is validated, tests for parsing xml text into structs can become self-referential as the following holds true:
This provides validation, byte-for-byte, for producing binary xml resources against the targetted build-tools and allows pin-pointing implementation issues down to the chunk header of the binary format.
While the new implementation would support both marshal and unmarshal of all android binary resource types, the scope of this functionality would be as follows:
Tests would be accompanied by a
testdata/
directory containing manifests to identify corner cases of the implementation and the binary versions of each manifest, generated with the targeted build-tools version.Regarding a new go:generate command to produce type safety information by cmd gomobile, this would use the new structs to decode the resource table from the android.jar of the targeted minimum api of gomobile, providing the necessary information for use by cmd gomobile.
It should be noted that this proposal was put together by piecemeal development to determine viability and much of the details of this proposal are already proven to be working by tests though no CL has yet to be sent as the work is incomplete.
The text was updated successfully, but these errors were encountered: