In general developers will not be required to make their code work on every secondary port.

Does this mean that we can only consider the first port when adding new features in the future, such as regabi.

And will we move a lot of secondary port related code out of the master branch, so that our code can work without regard to these ports.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@erifan

Does this mean that we can only consider the first port when adding new features in the future, such as regabi.

I think it depends on what you mean by "consider." I think an implementation for anything that covers all GOARCH values but differs for each one must as least consider whether and how it can be implemented for each GOARCH. We shouldn't add features that can never work for some GOARCH. And the initial implementation should not break any other GOARCH. But it's OK to implement something like regabi one GOARCH at a time, as indeed is being done for regabi.

And will we move a lot of secondary port related code out of the master branch, so that our code can work without regard to these ports.

I am not proposing that at all. That would be what I call an "out of tree port." I think it would be great to support that, but that is very hard, and it is not this proposal.

Thanks @ianlancetaylor , I see. This sentence "This is not intended to be a big change to the current process. However, it is intended to be a change" is very accurate.

@ianlancetaylor

The core problem with porting Go to other platforms, and making out of tree ports now is that the go runtime, standard library and compiler are not very modular when it comes to operating system and architecture support. Now, packes have files for all platforms confounded are in the same package using build tags. I think this should be refactored and split up into packages per operating system and/or architecture.

One should be able to write a platform support for the Go compiler, runtime and std lib just by writing a module which the compiler , etc. could then import.

If course this would involve a serious refactoring of Go, but I think it will be for the better, and encourage third party ports.

@beoran Thanks. I agree with all of that. But this is not that issue. The changes proposed here should not have to wait for the changes that you suggest. The changes you suggest will take a long time to design and implement, and nobody is working on them.

@ianlancetaylor Well I wanted to bring it up but I agree it is a separate issue in the end.

@beoran , we are making baby steps toward better modularization in the runtime by moving the per-OS/arch syscall layer into runtime/internal/syscall and reorganizing it to have significantly less per-arch code. We're part way through migrating the Linux arches (we've migrated the raw syscall functions for all Linux arches, but not yet the Go interface). It's a small step, but should help some.

I think the discussion helped a lot with addressing most of people's concerns, but I'm still a bit surprised about how little discussion is happening here. Does anyone have any remaining objections that we should discuss?

Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.

Microsoft dev here. We would like to help promoting windows/arm64 to be a first-class port. If it is just a problem of having reliable hardware, we can discuss providing reliable windows/arm64 hosts running in our own infrastructure. Would that be enough? If not, what else would be missing?

CC @golang/port-maintainers

CC @golang/release See comment by @qmuntal above regarding reliable windows/arm64 hardware. That should probably be discussed on a separate issue. Thanks.

..but I'm still a bit surprised about how little discussion is happening here.

I'm not affiliated with any company (just a FreeBSD user). Trying not to offend anyone here but there is not much in a way of open items up for discussion.
The core Google Go team doesn't wish to maintain the secondary ports and would like to have it done by other parties.
Unfortinetly to me the conditions above look like part of a contract or SLA you'd sign with another company providing development services instead of volunteer members of the open source community.

Additionally maintaining the official (because they are also used to build the Go release binaries) enviornments for freebsd/386, freebsd/amd64 outside of Google is extreamly frustrating.
It entails waiting for a member of the release team to run a bunch of bash scripts under a Linux VM to create and deploy new GCE images every time a new OS version is updated. This can sometimes take more than a year due to their limited avilability.
As for freebsd/arm, I don't plan to invest in new hardware after already upgrading twice. Maybe I'll be able to get a spare rpi3b to pass the build - but I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

@aclements That's a great first step, and i am all in favor of that.
@paulzhol It sounds like maintaining a secondary port is pretty labor intensive and /or inconvenient. Maybe that too is something that should be improved?

@paulzhol

The core Google Go team doesn't wish to maintain the secondary ports and would like to have it done by other parties.
Unfortinetly to me the conditions above look like part of a contract or SLA you'd sign with another company providing development services instead of volunteer members of the open source community.

You're not wrong. But I think there is another perspective. For a project like Go that has millions of users and that aims to provide a very high level of stability and reliability, it's not fair to our users to say "here is a port that may work, we don't know." We want to say either "this port works to the best of our knowledge and ability" or "you are on your own, good luck." To millions of Go users, we are in fact "another company providing development services." And to treat those users well, we have to carry that attitude through all core Go development. So, yes, we depend on volunteer members of the open source community, but we have to be clear about what they, and the core Go team, can and can't promise to Go's users. Again, you're not wrong, but that doesn't mean that nothing about the current porting policy should change.

I do think that there are things up for discussion, like: will this wind up hurting Go's users and the Go ecosystem? Obviously, I don't think it will, or I wouldn't have proposed it, but I could certainly be making a mistake.

Additionally maintaining the official (because they are also used to build the Go release binaries) enviornments for freebsd/386, freebsd/amd64 outside of Google is extreamly frustrating.

I'm sorry to hear that. I don't know much about what is required, or what would have to change, but I really hope that we can make things better somehow. Separately, I don't mean to be facile but I'm not sure that this proposal affects that one way or the other.

I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

Can you say more about that? Thanks.

@ianlancetaylor If this proposal makes it harder for ports to be available then this will damage the community.

Maybe I am staying the obvious, but would it be possible for Google to invest a bit more in this project and hire a few more porting engineer and provide some more hardware for them? That seems like a more optimal solution.

While this is an organizational issue, the underlying problem also seems to be technical. It seems to me that investing some work in improving the tools for ports would already help to alleviate the work needed to maintain a port. So it seems worth while to also discuss this here.

Maybe I am staying the obvious, but would it be possible for Google to invest a bit more in this project and hire a few more porting engineer and provide some more hardware for them? That seems like a more optimal solution.

Unfortunately I can't see that happening.

While this is an organizational issue, the underlying problem also seems to be technical. It seems to me that investing some work in improving the tools for ports would already help to alleviate the work needed to maintain a port. So it seems worth while to also discuss this here.

I think it's worth understanding how we can make it easier to maintain a port.

I'm not sure it makes a difference one way or another to this proposal. What do you think could or should change in the proposal?

@ianlancetaylor With regards to this proposal:

1. Seeing the current difficulty in maintaining a secondary port. I think the main Go project should provide as many resources as possible.

2. The main Go project should be a bit more lenient on these secondary ports. While it is true that ideally, all ports should be of equal quality, in practice this doesn't seem to be possible right now. The people who need such secondary ports are likely to accept that they have to do more testing themselves. The value of the secondary ports is in them existing, and allowing users on such systems to keep using Go.

Seeing the current difficulty in maintaining a secondary port. I think the main Go project should provide as many resources as possible.

I agree that that sounds good, but I don't know what it really means in practice. For example, I think we would all like it if Google would hire a full time person to do nothing but work on ports. But nothing like that is going to happen. On the other hand, Google is already prepared to donate significant Google Cloud Platform resources to ports, but of course that doesn't really help for ports to GOARCH values that GCP does not support.

One of the things I'm trying to do with this proposal is get more specific. What can we really promise beyond "we'll do our best?" What can people count on us to do beyond "do as much as possible?" What is possible?

The main Go project should be a bit more lenient on these secondary ports.

I'm already trying to do that in this proposal. The proposal is explicitly more lenient than the current porting policy, which says, for example, that a port will be removed if it is broken for four weeks.

2. The main Go project should be a bit more lenient on these secondary ports. While it is true that ideally, all ports should be of equal quality, in practice this doesn't seem to be possible right now. The people who need such secondary ports are likely to accept that they have to do more testing themselves. The value of the secondary ports is in them existing, and allowing users on such systems to keep using Go.

Fully agree here. I, user of one of the systems which would fall into the secondary port category, would greatly benefit of having a "best effort" approach for it rather than a "save yourself" approach because some projects just don't have the required resources to keep up with such effort, which will be (most likely) a big one.

On the other hand, if there are no new hires in the horizon to support the current burden that means that the burnout within the Go team will increase and that surely helps no one either, hence, some changes in the compromise towards those so-called secondary ports will be required. So, it seems things aren't going to get any easier in this front either ...

I'm generally OK with this proposal, one thing that is somewhat bothering me is

The default set of trybots will change to only cover first class ports.

Does this mean that there will be no usual pre-commit checks performed on secondary port builders? That could easily lead to a situation when innocuously looking commit breaks a secondary port and maintainers will have to fix it after the fact. I'm not sure this will be manageable by volunteers.

Does this mean that there will be no usual pre-commit checks performed on secondary port builders? That could easily lead to a situation when innocuously looking commit breaks a secondary port and maintainers will have to fix it after the fact. I'm not sure this will be manageable by volunteers.

That is a fair point. There is some discussion of introducing a submit queue, which would address this problem: submitting a CL would not submit it directly to the repo, but would instead run it through more comprehensive tests and then submit if those tests pass.

Perhaps we should drop or delay that part of the proposal until the submit queue is created.

I'm OK with this proposal.
Just a nit: There are ports are maintained by company/full-time developers like arm, ppc, loong, windows.
Could we show this information on go.dev/dl and "community support" like freebsd, plan9, riscv ?

@beoran

It sounds like maintaining a secondary port is pretty labor intensive and /or inconvenient. Maybe that too is something that should be improved?

Yes, I mainly find myself spinning up and maintaining multiple setups to test my code and verify cgo -godefs output on all the supported FreeBSD GOARCHes before submitting changes. It sometimes can take longer than the actual development.
Recently I've learned of TRY=freebsd-386,freebsd-amd64,freebsd-arm64,freebsd-arm. It with mentioned submit queue would help, but it still not a substitute for actual CI machines. Maybe allowing full gomote access could address that.

@ianlancetaylor

I'm sorry to hear that. I don't know much about what is required, or what would have to change, but I really hope that we can make things better somehow. Separately, I don't mean to be facile but I'm not sure that this proposal affects that one way or the other.

The proposal starts off with dropping official releases for freebsd/386, freebsd/amd64. I reasoned that means the port maintainers will have access to some GCE project/credit so we could provide more frequent builder image releases based on the upstream project's GCE images instead of the qemu-inside-Linux being done now and maintained by the release team. I may have misunderstood.

I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

Can you say more about that? Thanks.

These are much more constrained environments. To limit sdcard wear, the root filesystem is usually mounted read-only (at least for my builder). There's much less available RAM (in total and per-core) which is shared between the running Go toolchain and the page cache for both build artifacts and go list walks.
On my builder I use spinning rust over iSCSI as scratch workspace and swap - this adds additional load on the networking stack causing network tests to be flaky.
Additionally the slow IO coupled with the generally much slower CPU cores introduces behaviors which are dependent on the kernel scheduler and its preemption of user threads in the middle of profiling tests (making those flaky as well).
There is a way to address it I think, by splitting the bootstrap+compilation and just running of actual tests on the builder. I think it is/was done on the IOS builder?
But maybe it should be the norm for secondary builders on embedded hardware (tier 3 if you will). Preferably without the requirement of each port maintainer to have to roll their own version of it.

@paulzhol Thanks for providing concrete pain points and suggestions. I think it is important they are brought up here.

@ianlancetaylor It kind of worries me that Google does not seem to want to give more resources to the Go team. I hope it is not a bad sign of things to come. Dropping official support for several platforms is not going to help improve the future of Go language, and that is what I fear this proposal may lead to.

As we can see from some replies above, the burden for porting Go is also technical, I would say, even due to technical debt. I stated this before on occasion, but I feel that there should be release of Go, say, next year, with no new features at all, which focuses apart from bug fixes on fixing long standing issues, ease of porting, and lessening this technical debt. After such a release, then with the Go compiler and runtime in an optimal shape, would be a better time to start devolving the secondary ports and advance this proposal.

as far as I can see, these ports already meet the criteria specified for broken ports (hence currently being marked as having issues on the build page) - is that not the case?

Is anybody actively working on them? Do they have maintainers who intend to address the issues?

I do not know the answer to that, however the issues linked are:

• x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756 (last update 29th January 2022)
• cmd/compile,runtime: frequent test timeouts on solaris-amd64-oraclerel #51443 (last update 7th March 2022)
• os: (*Process).Wait sometimes hangs on netbsd #50138 (last update 21st April 2022)
• all: frequent test timeouts on Android builders #42212 (last update 28th April 2022)

Which means there have been no updates within the last 90 days.

I don't want to be draconian about it, but if nobody is maintaining them, and they don't work, then perhaps they should be marked as broken.

Right, and I'm not sure that it is being draconian, rather just adhering to the proposed policies.

I guess put another way, what is the relationship between a builder/port marked with "has known issue" and a "broken port" (are they the same thing)?

Let's be clear, though: a few failing tests shouldn't lead to a port being broken. In particular, it's always OK to skip the tests on that port (there should be an open issue for any such cases). As the policy written above says: "In general, a port can be considered broken if its builder has failed multiple times in a development cycle with a failure mode that does not occur for first class ports, and that failure mode is not believed to have been fixed or suppressed by a change in either a Go repository or the builder's configuration, and maintainers are not actively working on a solution." Note in particular that it's fine to suppress a failure mode. Of course that's not going to work if the port simply isn't working at all, but it's sufficient if there are some test failures.

Sure, although there is also a fine balance between the extreme of disabling a large number of tests to make the port "non-broken" versus having some reasonable level of stability and a small number of tests being disabled (with corresponding issues). That said, in most of the cases above, these seem to be long standing issues with (as far as I can tell) no clear or immediate resolution.

I can see where you would only want to run the default trybots when making a change that touches only general purpose files that may or may not affect secondary ports. But in the case where there are changes to files that are specific to a secondary port, it seems like those CLs should run trybots for those targets. Otherwise changes could be merged which don't even compile and break the build for that port.

@laboger That sounds like an interesting idea. Thanks. I filed #54375.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

I've updated https://go.dev/wiki/PortingPolicy with the changes described in this proposal.

@ianlancetaylor, is there anything remaining to do for this proposal? (Can it be closed as complete?)

What remains here is completion of #53862, and adding the notion of a broken port to cmd/dist (along with a way to build a broken port).

Filed #56679 to track broken ports.

That completes the work for this proposal. Thanks to all the commenters.

@ianlancetaylor can you please clarify:

1. What is a process for becoming a port maintainer?
2. How can maintainers publish a binary for port? (x/website: add linux/riscv64 downloads #59113)

I'm currently working on bringing riscv64 to Kubernetes (and friends), some dependencies rely on Docker Official Image packaging for golang, which gets binaries from https://dl.google.com/go and I'm not sure that maintainers will accept unofficial binaries.

Thank you!

Speaking as one of the maintainers of that image, we do support non official builds (there's a case statement in our build that will either download or build from the official sources); the problem with riscv64 is support in mainstream distros (still edge only in Alpine, still ports only in Debian, for example).

@ernado

What is a process for becoming a port maintainer?

See https://go.dev/wiki/PortingPolicy. Basically, at least for an existing port, file an issue and get the current port maintainers to agree.

Our hope is to be able to publish binaries for all supported ports. See #53862.