Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/mobile: binding doens't seem to reuse build cache for modules #37902

Closed
karalabe opened this issue Mar 17, 2020 · 14 comments · May be fixed by golang/mobile#58
Closed

x/mobile: binding doens't seem to reuse build cache for modules #37902

karalabe opened this issue Mar 17, 2020 · 14 comments · May be fixed by golang/mobile#58
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@karalabe
Copy link
Contributor

Apologies for not adhering to the default issue template, the issue is not a "bug" per se and not an issue with Go itself, so all those version and env infos would be useless.

gomobile added module support recently, which works spectacularly. One thing, however, that I did notice, is that if I call gomobile bind on a package that has heavy dependencies, those get rebuilt from scratch for every build, making the build very painful.

E.g. Binding https://github.com/coronanet/go-coronanet will take 30 mins as it needs to build a dependent package that contains a huge C library (Tor). However, trying to call bind again with zero modifications will again take 30 mins. This is in contrast with building simply with Go locally, which will take 5 mins for the first run, but will be ~instantaneous afterwards (only link time is left).

My best guess until now is that gomobile bind creates its own temporary working directory /tmp/gomobile-work-xxxxxxxx and thus all code operations are short circuited into that folder, away from the system caches. This results in all object files produced by one build run to be deleted at the end.

Issue opened per @hajimehoshi 's request in #27234 (comment).

Continuing a thread from the above issue:

Aha, gobind, which is invoked by gomobile, generates a source code from Go packages for binding, and they are not cached indeed. Did you mean 'cache' of them?

No, I'm not referring to caching the generated files, rather the module dependencies of them, which should most definitely be cacheable.
@gopherbot gopherbot added this to the Unreleased milestone Mar 17, 2020
@gopherbot gopherbot added the mobile Android, iOS, and x/mobile label Mar 17, 2020
This was referenced Mar 17, 2020
Closed
Open
@hajimehoshi
Copy link
Member

CC @hyangah

@OneOfOne OneOfOne mentioned this issue Mar 20, 2020
Closed
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 23, 2020
@MariusVanDerWijden
Copy link

The temporary directory is created in the following line: https://github.com/golang/mobile/blob/0df4eb2385467a487d418c6358313e9e838256ae/cmd/gomobile/env.go#L61
Maybe you could add a flag that allows to change this line to an existing directory e.g.
gomobile bind -dep /home/user/mobile-deps

@sunny-lan
Copy link

The solution mentioned by @MariusVanDerWijden seems to work. Is this going to be added anytime soon?

@aeddi aeddi mentioned this issue Nov 26, 2020
Open
@hajimehoshi
Copy link
Member

hajimehoshi commented Apr 19, 2021 •
edited

My understanding is that Go's build cache is created somewhere, wherever you build your application. Then the temporary directory should not matter. Is that correct? I believe gomobile does not delete Go's cache explicitly.

If so, is the culprit building Java or C files?

@hajimehoshi
Copy link
Member

@karalabe I'm sorry for the delay. Is this problem happening in Android or iOS, or both?

@spoukke
Copy link

spoukke commented Apr 25, 2021

Hi @hajimehoshi

I'm having the same issue here. Indeed, I specify the workdir using the flag -cache when I use gomobile bind in Github Actions. I put this directory in cache with actions/cache@v2.1.4 as long as the generated .aar file for android, and the .framework file for ios. However, the build starts over each time.

@fortuna
Copy link

fortuna commented Jan 20, 2022 •
edited

Any chance we can get the solution in https://go-review.googlesource.com/c/mobile/+/273406 merged?

PR on Github: golang/mobile#58 (though the discussion is on googlesource)

@modasi
Copy link

modasi commented Apr 25, 2022

you can try this :
https://github.com/modasi/gomobile-cache-mod

@hajimehoshi
Copy link
Member

This is not related to the cache, but I hope https://go-review.googlesource.com/c/mobile/+/426274 would mitigate the slowness of Android building. I'll try iOS later.

@hajimehoshi
Copy link
Member

hajimehoshi commented Aug 28, 2022 •
edited

@modasi What change did you add to gomobile?

EDIT: I think this is it (initializing tmpdir)

	if buildN {
		tmpdir = "$WORK"
		cleanupFn = func() {}
	} else if buildWork {
		tmpdir = "/tmp/gomobilework"
		fmt.Fprintln(xout, "WORK="+tmpdir)

	} else {
		tmpdir, err = ioutil.TempDir("", "gomobile-work-")
		if err != nil {
			return nil, err
		}
	}

@hajimehoshi
Copy link
Member

hajimehoshi commented Aug 28, 2022 •
edited

gomobile added module support recently, which works spectacularly. One thing, however, that I did notice, is that if I call gomobile bind on a package that has heavy dependencies, those get rebuilt from scratch for every build, making the build very painful.

Did this issue exist before Go modules support was introduced? If not, I was wondering how supporting Go modules triggerred this...

@gopherbot
Copy link

Change https://go.dev/cl/442195 mentions this issue: cmd/gomobile: concurrent build for iOS archive files

gopherbot pushed a commit to golang/mobile that referenced this issue Oct 20, 2022
@hajimehoshi
cmd/gomobile: concurrent build for iOS archive files
b36e624 
This change makes building archive files for iOS concurrent for each
architecture and each platform. The strategy is basically the same as
my previous CL for Android: https://go.dev/cl/426274.

This change also specifies GOMODCACHE explicitly when executing Go
commands so that the existing cache is always used. The default
GOMODCACHE is $GOPATH/pkg/mod, and this path varies when a temporary
GOPATH is specified, which results in cold cache.

Before this change (on my MacBook Pro 2020):

$ time go run ./cmd/gomobile/ bind -target ios ./example/bind/hello/

real    0m23.274s
user    0m15.751s
sys     0m10.469s

After this change:

$ time go run ./cmd/gomobile/ bind -target ios ./example/bind/hello/

real    0m8.059s
user    0m13.763s
sys     0m9.004s

Updates golang/go#37902
Updates golang/go#54770

Change-Id: Iaeb077b58c22ab63d28f78972a0af76660883a05
Reviewed-on: https://go-review.googlesource.com/c/mobile/+/442195
Reviewed-by: Changkun Ou <mail@changkun.de>
Run-TryBot: Hajime Hoshi <hajimehoshi@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@hajimehoshi
Copy link
Member

Can anyone test the latest gomobile?

@hajimehoshi
Copy link
Member

Let me close this. Feel free to reopen this when you find this unresolved.

@hajimehoshi hajimehoshi mentioned this issue Jul 29, 2023
Closed
@golang golang locked and limited conversation to collaborators Oct 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge mobile Android, iOS, and x/mobile NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

Successfully merging a pull request may close this issue.

cmd/gomobile: add a flag to manually specify a work dir aeddi/mobile
9 participants
@hajimehoshi @fortuna @karalabe @cagedmantis @modasi @gopherbot @sunny-lan @MariusVanDerWijden @spoukke