cc @eliasnaur

This seems like a lot of work to support .syso files. Why do you need them? Is it to avoid the need for SDKs and system compilers/linkers? If so, it seems to me the effort is better spent on that. Shameless plug: #38917.

Kind of an odd idea, but could fat .syso files solve the problem? Not sure whether those are even a thing, but if it were possible then that would allow us to transfer the problem to the loader.
The loader can distinguish amd64 vs arm64. Can it distinguish the other axes, like simulator vs. non-simulator?

This seems like a lot of work to support .syso files. Why do you need them? Is it to avoid the need for SDKs and system compilers/linkers? If so, it seems to me the effort is better spent on that. Shameless plug: #38917.

A lot of work for a contribution for a CL to address this, or a lot of work for a 3rd party developer who might make use of this?

• On the former, I'm happy to make the contribution; I don't think it'd be anywhere near as complex as cmd/link: fails to link package having a .syso file when using external linker #33139 was. At a high-level, the proposal is very simple: support selection of syso files at build time based on user-defined build tags. The proposal does not suggest adding any pre-defined build tags into gc.
• On the latter, it's not much work: we use a Makefile to generate each platform object using a cross compiler. The resulting syso files are committed to the project using Git-LFS and are thereafter "transparent" to Go builds, subject to the limitations that motivated this issue.

We use syso files for two purposes:

1. To have a "go gettable" package that statically links a library that is called from cgo. See this public example. This is probably what you had in mind w.r.t. proposal: cmd/cgo: add support for pre-compiled cgo packages #38917.
2. To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Kind of an odd idea, but could fat .syso files solve the problem? Not sure whether those are even a thing, but if it were possible then that would allow us to transfer the problem to the loader.
The loader can distinguish amd64 vs arm64. Can it distinguish the other axes, like simulator vs. non-simulator?

That's actually a great suggestion. I just tried it and unfortunately hit this with lipo:

fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: lib_device.o and lib_sim.o have the same architectures (arm64) and can't be in the same fat output file

Clang's linker can handle "fat objects", but each fat object can only host dissimilar architectures, even if each component is built for a different platform. This becomes pretty clear when you look at the Mach-O fat header format: each slice is keyed on a CPU and sub-CPU architecture magic, and has no reference to a platform magic.

That means we cannot, for example, have a fat object that targets both iOS (arm64 device) and the iOS-simulator (arm64 Apple Silicon Mac); or iOS (arm64 device) and tvOS (arm64 device) and/or the respective simulators (arm64 Apple Silicon Mac).

To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Currently there isn't anything stop you from doing this. But this is not really "supported".

To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Currently there isn't anything stop you from doing this. But this is not really "supported".

@cherrymui, you wrote something similar back in July last year. As I replied back then, the Go Wiki documents this pattern since at least 2014 [1]. I spent a lot of time to contribute a CL to "support" that late last year, and I'm offering to support it again here.

Separate to the motivation for the proposal, I'd be grateful for any comments or support for the concrete use of #tags in a syso filename suffix. With some blessing, or at least no objections, I'd be happy to start working on a CL.

[1] See wiki commit 5bc444, which was "import from Google Code": it was probably written much earlier than that.

I know the Wiki page mentions it, as a "trick". But I wouldn't say it is "supported".

I agree that we probably don't want to intentionally break it, and probably will keep it working if it doesn't take much effort. But I don't think this kind of use is a good motivation for new proposals.

I'd prefer to avoid an off-topic discussion about the semantics of what "supported" means.

This proposal is rather minor: it adds an optional feature, limited in scope to just syso files, so that it keeps an existing compiler feature working for new architectures that are being adopted more broadly (i.e. #38485).

The community effort here will mainly be review of a CL I am volunteering to develop. I don't expect the CL to be overly complex.

Are there any objections to the concrete tag naming scheme for the filename suffix?

I think it might be a good idea to have some mechanism to match build tags for syso files (although I don't necessarily agree with the motivation).

I think the proposal needs to be a little clearer about the matching semantics. Is the tag after the # character must be present, so a build with no tags specified doesn't match any?

If we do this, I think supporting multiple tags is fine.

I think the proposal needs to be a little clearer about the matching semantics. Is the tag after the # character must be present, so a build with no tags specified doesn't match any?

I would propose:

• No build tags: same semantics as present: it matches on GOOS and/or GOARCH if present, or if not, is always included.
• More than one build tag: AND semantics.
  • This would be the same as having multiple // +build lines in a Go source file. That would also be consistent with the existing GOOS/GOARCH semantics; e.g. *_linux_amd64.syso means GOOS=linux AND GOARCH=amd64.
  • Where a user really wants an OR condition of two or more tags, they could add the syso file multiple times, each being a symlink with a single #tag suffix to the real syso file.
  • I think this latter situation will be rare: we're more likely to see #iphonesimulator AND #debug, or #iphonesimulator AND #release.

I'm happy to update the proposal to make this more clear.

This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case.

It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:

//go:build iphone
package p
import _ "p/iphone_syso"

and:

//go:build tv
package p
import _ "p/tv_syso"

and so on. Then the p/iphone_syso being imported can be an empty package (with a 'package iphone_syso' and nothing else in a single Go file) along with a syso.

On iPhone, p/tv_syso won't be imported and the syso won't go to the linker. But p/iphone_syso will, and that syso will go to the linker?

Thoughts?

Yeah, this is what I suggested in #38485 (comment)

This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case.

It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:
[snipped]

Thank you for reviewing the proposal, @rsc.

My outline could have been more clear on your workaround, also suggested by @cherrymui: it was referred to the "sub-package" approach without further detail. I've updated the proposal at the top of this thread, and also included more clarity on the treatment of multiple build tags.

Compared to the workaround available today, both @cherrymui and I agree that it is not pretty. As outlined in the update, it requires two additional Go source files and one new sub-package directory for every unique syso file.

I have several packages supporting macOS (amd64 + soon arm64), iOS (arm64), iOS-simulator (amd64 + soon arm64), Android (arm64), Linux (amd64, arm64), and Windows (amd64, 386). I hope to support tvOS and the tvOS simulator in time. For each platform object, I have Release and Debug builds. The macOS, iOS (tvOS), iOS(tvOS)-simulator, Android, and Linux syso files can't be uniquely selected for a build without the workaround which makes the package a mess, while the Windows syso files remain in the package root. (This happens because GOOS uses non-unique aliases for macOS, iOS/tvOS, Android, and Linux.) This proposal would unify and simplify that layout significantly.

I appreciate the flexibility that +build and now go:build lines offer, and use the former extensively in my packages. I would however argue that basic tag selection in both syso and Go source filenames would have been a welcome feature from the get-go. For all Go source files where I use +build lines, I also add a decorative filename suffix to make it clear in my IDE that such files are built conditionally. I appreciate that others might not see it that way, which is why I limited this proposal to just syso files where I believe there is no good alternative.

I'm about to start looking at a CL for this, and I expect the change to be small. I hope you're able to keep an open mind.

Change https://golang.org/cl/269280 mentions this issue: go/build: allow build tags to select syso files

Using subpackages is not pretty, but using build tags in the file name starting with "#" is also not pretty. Subpackages, while verbose, do not require any new features and are thus clear to most Go programmers. Build tags starting with "#" will always be a very obscure feature that seems likely to trip people up. Given that very very few people will ever want to use this feature, and given that it can be supported without adding any new features, I personally think there is a strong argument for not adding any new features here.

Using subpackages is not pretty, but using build tags in the file name starting with "#" is also not pretty.

The lesser of the two evils, and a much better developer experience, is surely just naming one file lib_#macosx_amd64.syso over the alternative of adding a bunch of extra files and another folder just to work around a trivial limitation of the existing build constraint file selection.

The filename lib_#macosx_amd64.syso doesn't look ugly to me? Anyone else want to chime in? Would a different token be acceptable?

Given that very very few people will ever want to use this feature, and given that it can be supported without adding any new features, I personally think there is a strong argument for not adding any new features here.

I think more people would use this functionality if:

1. It becomes more widely known and/or documented.
2. More developers start using Go for interactive apps, where you need to link into non-Go native code much more often, and where having a "go gettable" package while statically linking that code into the executable is desirable.
3. The previous bullet will start becoming more commonplace as some of the Go UI projects start maturing. That kind of work has directly led to this proposal.

The feature is rather trivial: visible only in a syso filename and nowhere else, and the implementation is also minimal as you can see in the simple CL I submitted 1/2-hour ago.

Thanks for taking the time to read the proposal, @ianlancetaylor. I hope you'll reconsider with the above arguments in mind, and have a quick look at the associated CL.

The feature is rather trivial: visible only in a syso filename and nowhere else

Today, the *_GOOS_GOARCH naming pattern for .syso files is the same pattern used for build constraints for ordinary Go source files: there is nothing particularly special about .syso files in that regard, and anyone who has worked with platform-dependent .go files should be able to map that understanding fairly easily onto the .syso files.

I think it would be a mistake to add a separate naming convention that applies only to .syso files.

On the flip-side, users are already confused enough about the implicit build constraints for ordinary Go source files. If this new naming scheme did not apply only to .syso files, I think that would be even more confusing.

Given how rarely .syso files are used, I think that on balance it would add more complexity than it removes. The subpackage approach is already possible today and doesn't seem that bad to me, and we can always revisit in a few years if for some reason .syso usage grows dramatically.

Today, the *_GOOS_GOARCH naming pattern for .syso files is the same pattern used for build constraints for ordinary Go source files: there is nothing particularly special about .syso files in that regard, and anyone who has worked with platform-dependent .go files should be able to map that understanding fairly easily onto the .syso files.

Thanks for chiming in. I agree with this, and the proposal doesn't change the *_GOOS_GOARCH naming pattern for those having learned using it with .go files. It still carries across. This proposal is additive.

I think it would be a mistake to add a separate naming convention that applies only to .syso files.

If consistency is important to you, I'd be happy to see the proposed *_#tag naming pattern extended to .go files. (It even further simplifies the CL.) I avoided proposing that because I expected an even louder objection than what I have seen in this thread.

On the flip-side, users are already confused enough about the implicit build constraints for ordinary Go source files. If this new naming scheme did not apply only to .syso files, I think that would be even more confusing.

I agree that developers can be easily confused about the explicit +build constraint syntax: I was pleased to see @rsc's work on improving that with the new go:build line. I smiled when he said that people can't remember the syntax for too long because I can relate!

But the implicit build constraints in the filename are straightforward because they are so rigid and simplistic: you just summarized it with a simple string pattern in your post! Can you point out instances where developers get confused with this?

Given how rarely .syso files are used, I think that on balance it would add more complexity than it removes. The subpackage approach is already possible today and doesn't seem that bad to me, and we can always revisit in a few years if for some reason .syso usage grows dramatically.

The prospect of living with this messy workaround for "a few years" is unappealing. We already have syso files in over 15 packages and are just getting started.

Based on the discussion so far, what interest (if any!) is there for:

1. Unifying the build tag filename patterns for both .go and .syso files?
2. A filename pattern that is not considered "ugly"?
3. A new build line, like //go:syso [filename] to avoid having to use a sub-package, and with no change to the file naming pattern?
   • This would still require one extra .go source file to specify the above syntax and a //go:build line to constrain selection to the desired platform.
   • It may also require the filename to not end in .syso so it doesn't conflict with existing matching rules.
4. A variation on (3) like //go:syso [go:build expression] [filename] that means you can place it in any existing .go file?
   • Example: //go:syso macosx && amd64 lib_macosx_amd64.o, where the file lib_macosx_amd64.o sits in the same directory as the corresponding .go file.
   • I would place it near existing Go code that uses the resources from the syso file.

We have several Go packages that targets multiple platforms, including macOS, iOS, tvOS, iOS-simulator, tvOS-simulator, windows, linux, etc. The package has one or more syso file dependencies.

Syso files can be selected for a build by a GOOS/GOARCH filename suffix but not build tags, unlike Go source files that use // +build or //go:build lines. If we ignore tvOS for the moment, some time ago we could build a syso file for darwin/amd64 and include that in both macOS and iOS-simulator builds just fine. With recent Xcode 12, that approach no longer works:

ld: building for iOS-simulator, but linking in object file built for macOS, file '/var/folders/h7/r_43shxn71v0pq_gx72f57ph0000gn/T/go-link-172505736/000035.o' for architecture x86_64

The situation worsens when we support tvOS (+ the tvOS-simulator), and the new ARM-based Apple Silicon Macs because we can no longer use a GOOS/GOARCH filename build constraint to uniquely select the appropriate syso variant.

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform.

@jpap It seems to me that you are arguing in favor of a change that 1) very few people will want to use; 2) you can already do today in a different, albeit more awkward, way.

A good argument here would be pointing to several different popular packages, by different people, that would use this feature if it were available.

The argument that more people are going to want to use this feature in the future doesn't seem particularly convincing to me. It's particularly unconvincing if the reason that people want to use this feature is to bypass cgo safety in the name of speed. If that is the problem that people are trying to address, then we should address that, not encourage increasingly complex use of a workaround.

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform.

Yes -- the Go linker could filter all of the syso files with GOOS=darwin and extract only the ones for a given platform, but in order to do that, it needs to know what the build target platform is: for example, is it iOS-device, or iOS-simulator (Apple Silicon Mac)?

In the code you've cited, it takes the first object file involved in the build, extracts the platform, and if it is macOS, inserts the required LC_VERSION_MIN_MACOSX load command, that would otherwise break lldb if missing. That code implicitly assumes that all of the object files are built for the same target platform. In this case, what do you do when the linker is presented with a collection of heterogeneous objects? Declare the target platform as the one having a majority of objects presented? You would likely need some kind of target platform argument to make it explicit. In the proposal here, that explicit argument is the user-defined build tag so the linker is presented only with homogenous objects. You could condition the Go linker on such a build tag, but I don't see anyone embracing that kind of coupling.

Even if you were to use some magic in the Go linker to solve the above; you still have the problem with Android and Linux. When you build for Android, the GOOS=linux alias will select all *_linux_arm64.syso files, for example. ELF doesn't have an explicit platform header, so the linker won't know what these objects target, and the link will succeed so long as all import symbols are found. Not a problem if the functions in the object are purely compute (e.g. an image processing algorithm, data compression, or hashing), because there might be no imports. But if the object calls a platform API, you're in trouble because of the syso file selection issue.

The fundamental problem here is that GOOS was thought to be tied to a kernel, and not a "platform" or "OS". I understand the reasons for doing so -- it simplified cross-platform support and brought Go more quickly to a bunch of new platforms; the "Go way" was also to avoid platform APIs as much as possible and hit the kernel interface directly. And it wouldn't be a problem here, either, if we had the build tag escape hatch for these syso files. (I recognize the workaround; but it is a mess.)

Elias, did you have have an opinion on the proposal itself? I appreciate the thoughts going into other workarounds, but would love to know if someone else who is knee-deep in non-Go code and platform interop can see the benefits proposed here.

A good argument here would be pointing to several different popular packages, by different people, that would use this feature if it were available.

Thanks for painting a path forward. I can't help being a little disappointed that it isn't to discuss the technical merits of the proposal, or some of the proposed alternatives, but to judge it solely on its apparent popularity, especially given how minor this feature is and how small the CL turned out to be. :)

I agree that syso usage is not an everyday use-case for the majority of Go users. That is because of the demographic: the Go Survey clearly shows that the vast majority of Go use-cases are API/RPC services and CLIs. Those using Go for Desktop and Mobile apps are in the tiny minority, but that is where syso shines.

This situation is chicken-and-egg: Go developers don't write GUI apps because Go lacks great UI packages. That will change as one of more Go UI projects matures. I feel it is a bit disingenuous to ask for references from "several different popular packages, by different people" when the number of people working in this space is miniscule.

It's great to have Elias in this thread, and I've put it to him in my last post. His Gio project may not benefit as much from improved syso support, as his goal is to avoid interfacing with the (UI-component part of the) platform as much as possible, by rendering a completely custom UI into a blank canvas. When you write GUI apps that interface into platform SDKs and platform UI components, you tend to rely on cgo much more frequently, and leveraging syso leads to a much better developer experience.

The argument that more people are going to want to use this feature in the future doesn't seem particularly convincing to me. It's particularly unconvincing if the reason that people want to use this feature is to bypass cgo safety in the name of speed. If that is the problem that people are trying to address, then we should address that, not encourage increasingly complex use of a workaround.

The stain of "let's use syso to bypass cgo" is unwarranted here. Syso files are the only way to host a "Go gettable" package that uses or wraps a nontrivial non-Go library that gets automatically and statically compiled into the final Go executable while using cgo as the glue. I've already linked to a public example in a previous post. I think this approach isn't popular because it isn't well documented. A lot of projects that provide cgo wrappers for a non-Go library typically assume that library is pre-installed on the system. It is a much better developer and end-user experience, and more "Go like", to have the library statically linked to the executable.

I have successfully used the statically compiled syso approach for interop with many 3rd party non-Go libraries:

• Wrapping a video codec library to support image coding with H.265.
• Wrapping a lossless arithmetic compression library, that has no native-Go implementation, to support interop with a popular file format.
• Wrapping libpng for higher performance PNG and more control, compared to the image/png package.
• Wrapping mozjpeg for higher performance and better compression over the image/jpeg package.
• Wrapping littleCMS to work with ICC profiles and perform image color management.
• Wrapping a disassembly library (linked above) to support static analysis of x86_64 text.
• Wrapping pdfium to render PDFs into images.

I have also used the approach to statically link many 1st party non-Go libraries that interface with platform SDKs. I am not going to list those here. All but one of these packages are currently private; that might change in the future, but the work being done here is commercially focused.

All of these packages use cgo.

especially given how minor this feature is and how small the CL turned out to be. :)

I want to clarify that this is not a strong argument. Every feature must be documented, maintained, and tested. Small features are not better from a long-term maintenance perspective. They are worse, because they are less used, less tested, and less understood, but they must be fixed when they break.

Thanks for the comments about cgo. I wonder if this is an argument for having better interaction between go build and code that is not written in Go, so that we can build that code rather than having to ship .syso files.

especially given how minor this feature is and how small the CL turned out to be. :)

I want to clarify that this is not a strong argument. Every feature must be documented, maintained, and tested. Small features are not better from a long-term maintenance perspective. They are worse, because they are less used, less tested, and less understood, but they must be fixed when they break.

I agree on the importance of documentation, testing, and maintenance in a large, long term project. One could argue that my CL included the first two things: it included tests, and added both godoc and go help buildconstraint usage text. I additionally performed some manual smoke tests before pushing the CL. Maintenance is a long-term burden on all contributors: in submitting that CL, I was essentially performing that task on the "syso" feature today. If the CL warrants further work in those areas, add it to your review, and I'd happily address it.

Broad generalizations are helpful but there are always special cases. What you say about small features may be true in general, but in this case, having never previously looked at the relevant code, I was able to understand and produce the CL, including testing, within an hour. While some of the Go codebase is very complex for newcomers, perhaps some parts brittle to breakage when unrelated changes are made, after working on it I feel this CL is far from both of these things. :)

Thanks for the comments about cgo. I wonder if this is an argument for having better interaction between go build and code that is not written in Go, so that we can build that code rather than having to ship .syso files.

Based on my experience, I would recommend bolstering syso support or something like that, where you are ingesting precompiled objects or archives. The reason for this is that for many nontrivial non-Go projects, the build process is complex and sometimes bespoke. They might use CMake, Autoconf, GYP and Ninja, etc. On platforms like Android, you not only have the C-JNI glue, but corresponding Java sources as well.

In all of these cases, I am not suggesting syso objects be shipped in isolation; in my projects I also ship a Makefile or script to fetch the non-Go project, cross compile it where possible, and then (re)produce the objects that are checked into the repository with Git-LFS.

One of the great things about Go is not only the ability to target across platforms, but also the ability to cross-compile easily. Doing so with cgo is possible, but requires more care. Doing so with cgo + a non-Go project build system is even harder. Cross-compiling some of the 3rd party libraries I mentioned in my last post was not always an easy task. In some sense that is what Elias is looking for in #38917 -- if you are linking prebuilt objects, there's no need to worry about SDK headers that you are licensed to only use on a certain platform. ;)

It is much easier to compartmentalize 3rd party non-Go code than work directly with its source files, especially when that code was not originally intended for consumption by Go.

In recent months I have also spent time working out how to pragmatically support cross-compilation of cross-platform apps that make extensive use of a FFI, for example, calling Objective-C APIs on Apple's platforms. I think I'm converging on something that works really well, but some of it hinges on prebuilt objects for calls into platform SDKs to make cross-compilation a breeze. Syso files are at the core of that, which is another reason I am so keen to see this proposal through now and not in "a few years". (Again, this is all using cgo as the glue.)

I have a separate viable path to do all of that without this proposal, while still maintaining a good developer experience, but unfortunately that path involves much more work on my end and a departure from the standard Go project structure. It would require a new build tool that sits above go build, that scans a package's imports recursively, filters prebuilt objects ("syso" files) in each package directory, while feeding the paths of those filtered objects to the external linker via -extldflags. Since cmd/go already does this with the "syso" feature, under the limitations at the top of this thread, it makes sense to spend that hour creating a CL instead of a week or more effort on a new non-standard tool and live on a non-standard project layout.

If there's any interest in discussing some of my earlier suggestions, I'd be all for it. Otherwise I fear we're not making the best use of our time here. For that, I apologize, no hard feelings, and thanks for the discussion. I can't wait for you to see what I've been working on. :)

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform.

Yes -- the Go linker could filter all of the syso files with GOOS=darwin and extract only the ones for a given platform, but in order to do that, it needs to know what the build target platform is: for example, is it iOS-device, or iOS-simulator (Apple Silicon Mac)?

In the code you've cited, it takes the first object file involved in the build, extracts the platform, and if it is macOS, inserts the required LC_VERSION_MIN_MACOSX load command, that would otherwise break lldb if missing. That code implicitly assumes that all of the object files are built for the same target platform. In this case, what do you do when the linker is presented with a collection of heterogeneous objects? Declare the target platform as the one having a majority of objects presented? You would likely need some kind of target platform argument to make it explicit. In the proposal here, that explicit argument is the user-defined build tag so the linker is presented only with homogenous objects. You could condition the Go linker on such a build tag, but I don't see anyone embracing that kind of coupling.

Build tags in filenames is a blunt instrument. I believe there are lighter weight options available. Off the top of my head:

• Improve the LC_VERSION_MIN_ detection. For example, the cgo tool or the Go linker could query $CC for the platform, and then ignore incompatible syso objects.
• If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

Even if you were to use some magic in the Go linker to solve the above; you still have the problem with Android and Linux. When you build for Android, the GOOS=linux alias will select all *_linux_arm64.syso files, for example. ELF doesn't have an explicit platform header, so the linker won't know what these objects target, and the link will succeed so long as all import symbols are found. Not a problem if the functions in the object are purely compute (e.g. an image processing algorithm, data compression, or hashing), because there might be no imports. But if the object calls a platform API, you're in trouble because of the syso file selection issue.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

The fundamental problem here is that GOOS was thought to be tied to a kernel, and not a "platform" or "OS". I understand the reasons for doing so -- it simplified cross-platform support and brought Go more quickly to a bunch of new platforms; the "Go way" was also to avoid platform APIs as much as possible and hit the kernel interface directly. And it wouldn't be a problem here, either, if we had the build tag escape hatch for these syso files. (I recognize the workaround; but it is a mess.)

Elias, did you have have an opinion on the proposal itself? I appreciate the thoughts going into other workarounds, but would love to know if someone else who is knee-deep in non-Go code and platform interop can see the benefits proposed here.

I would love easy linking of static libraries. I'll probably need it for integrating something like Harfbuzz in Gio.

However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

Appreciate your reply, Elias. Some comments inline below.

Build tags in filenames is a blunt instrument. I believe there are lighter weight options available. Off the top of my head:

• Improve the LC_VERSION_MIN_ detection. For example, the cgo tool or the Go linker could query $CC for the platform, and then ignore incompatible syso objects.

You could invoke the compiler to produce a dummy object, then extract the platform from it. The extra round-trip would also elongate build times.

• If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

That is possible, however how can you deal with Debug vs. Release object selection? Build tags offer a lot more flexibility because their meaning is user-defined, and not hard-coded into a flag.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

What information would you like to know about my setup? Based on an explicit linker flag, and the inability to discern Android ELF from "Desktop Linux" ELF, how can we filter syso files in this case?

However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

I look forward to your suggestions, and any feedback you might have on my other alternative proposals.

• If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

That is possible, however how can you deal with Debug vs. Release object selection? Build tags offer a lot more flexibility because their meaning is user-defined, and not hard-coded into a flag.

Having separate debug vs release builds is unusual in Go, so I'm not sure how far the Go tools should be stretched to accommodate that variant axis. The ugly workaround seems good enough here.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

What information would you like to know about my setup? Based on an explicit linker flag, and the inability to discern Android ELF from "Desktop Linux" ELF, how can we filter syso files in this case?

"How to filter syso files" smells like an XY problem to me. Why do you need different syso files for Android vs Linux? Can the static libraries be built in a way they don't refer to Android-specific APIs?

However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

I look forward to your suggestions, and any feedback you might have on my other alternative proposals.

If we decide filtering syso files is required, then I like your magic comments better. Perhaps add them to #cgo directive, because it understands build constraints already:

// #cgo android,amd64 syso: blah_android_amd64.syso

Having separate debug vs release builds is unusual in Go, so I'm not sure how far the Go tools should be stretched to accommodate that variant axis. The ugly workaround seems good enough here.

Go doesn't suffer from the debug-vs-release dichotomy because it doesn't perform extensive optimizations, favoring faster build times (and a simpler compiler?). There's also good reason for keeping debugging information in the binary for panic stack traces at runtime. These things are fine if you're deploying a backend service on your own/rented network; but for user interactive apps that run on foreign machines it's a bit different:

• Debug features to introspect/control the UI state remotely becomes important for builds that run on an attached device (e.g. iPhone or Android phone connected over USB). You don't want some kind of UI debug server still embedded and active in an app released to an end-user. There are some aspects to this that you absolutely must not keep in the executable, for example calls to mmap to create code pages that flip between {w, rx}-states at runtime to support hot code reload. Leaving them in would certainly leave you vulnerable to being rejected from the Apple's App Store, for instance.

• Statically linked non-Go code (using cgo!!) built using -Os is a good idea for energy efficiency on mobile devices and smaller executables in general. But debugging those builds with lldb becomes a pain, so an -O0 debug build becomes desirable. Crash reporting of such code is largely a solved problem with exception handling "crash reporters" and remote symbolication. (Having a similar thing on the Go side is a topic for another GitHub issue.)

I am really looking forward to gollvm maturing so we can get the best of both worlds: fast compilation during development with gc, and a more extensive optimization for release builds with gollvm, which will also produce LLVM bitcode that is encouraged (read: perhaps required in future) by Apple on their App Store submissions.

"How to filter syso files" smells like an XY problem to me. Why do you need different syso files for Android vs Linux? Can the static libraries be built in a way they don't refer to Android-specific APIs?

We need native code that uses the C(++) ABI on:

• Android to hit the JNI to interact with the ART, and/or to hit the NDK.
• Linux to hit a UI toolkit like GTK, Tk, Qt, etc.

That native code is most easily written in C(++), interfaced to Go via cgo (+ SWIG, or an extern C wrapper).

If you use {.c, .cc} files in your Go project and/or self-contained cgo, then yes, you don't need platform-specific syso files anymore, but now cross compilation becomes more difficult, especially from macOS and Windows. That, again, relates to your shameless plug. When that native code sits in a syso file, cross-compiling of the Go project becomes a breeze, and we're now back on-topic with this proposal.

I see X == "need native code that hits platform-specific APIs" and Y == "need syso".

If we decide filtering syso files is required, then I like your magic comments better. Perhaps add them to #cgo directive, because it understands build constraints already:

// #cgo android,amd64 syso: blah_android_amd64.syso

I would caution against associating syso with cgo here: they are independent features, even though they might work well together under many circumstances.

FWIW, I appreciate that it's a little more work in this use case, but sometimes that's preferable to complicating the general case for something that most people will never see. It becomes one more thing people have to understand when they do see it. If you know about regular build tags and subdirectories (which most people do), then you know what you're looking at when you see the subdirectory solution. On the other hand, if you see a file named lib_#appletvsimulator_amd64.syso, that's surprising and uninterpretable without going off to learn a new rule.

Based on the discussion above, this seems like a likely decline.

Given the current state of this proposal, I've instead implemented a custom build tool with a nonstandard Go project layout in favor of a better DX over the proposed workaround.

Thank you for the consideration and detailed discussion by all.

A rather unfortunate side effect of using multiple packages is that you'll need to instruct the linker to look up symbols at runtime. So, this program that previously wouldn't compile now crashes: https://play.golang.org/p/zLi2iNfwiZO

This is a problem I've run into when creating .syso files for x86-64 Android and Linux. Since the android GOOS also matches the linux GOOS, the compiler attempts to use both lib_android_amd64.syso and lib_linux_amd64.syso when GOOS=android. (And for $reasons we can't use the Linux object files on Android.)

@rsc I'd like to reopen this issue (or make another if you'd prefer, I'm not a fan of the "#" syntax). .syso files are a little esoteric, but incredibly useful.

As far as I can tell you can address that issue using the approach described at #42477 (comment). If not, why not?

@ianlancetaylor yes, multiple packages works. But using a tennis racket as a canoe paddle also works. :)

In particular, cmd/cgo does not pull in other packages, so linking fails unless you instruct the linker to ignore undefined symbols. This could cause the program to explode at runtime instead of compile time. (See the playground link above.)

Or, you have to manually link against the .syso files in #cgo directives, which mostly defeats the purpose of using .syso files.

I think we discussed all the tradeoffs above, and came to a decision. If multiple packages works, then I don't see a need to reconsider. I understand that this is more work, but it is possible, and it's for an unusual case. Thanks.