New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: add LUCI openbsd-ppc64 builder #63480
Comments
Change https://go.dev/cl/534976 mentions this issue: |
Thanks. Here's the resulting certificate: openbsd-ppc64-n2vi-1697128325.cert.txt. I've mailed CLs to define your new builder in LUCI and will comment once that's done. |
Thank you; I confirm that using the cert I get a plausible looking luci_machine_tokend/token.json. |
Since the list of BUILDER_TYPES is nearly sorted, keep that up, and sort (using 'Sort Lines' in $EDITOR) two of Linux run mods. For golang/go#63480. For golang/go#63481. For golang/go#63482. Change-Id: Icef633ab7a0d53b5807c2ab4a076d74c291dc0ea Reviewed-on: https://go-review.googlesource.com/c/build/+/534976 TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Carlos Amedee <carlos@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
I have not read the code yet to diagnose this; leaving assigned to me.
|
I don't see anything in the code or logs here that help me diagnose. It just looks like the server didn't like the token.json that had been refreshed just a minute before. Maybe someone there can check server-side luci logs? Unable to reassign to dmitshur; hope someone there sees this. |
Thanks for the update. I recall there was a similar looking error in #61666 (comment). We'll take a look. |
In case it helps... I set both -token-file-path on the bootstrapswarm command line and also LUCI_MACHINE_TOKEN in the environment. The logs don't indicate any trouble reading the token.json file, though they're not very explicit. I appreciate that there have been serious security flaws in the past from too-detailed error messages. But I'd venture that it is safe for luci to say more than "403". I recognize I'm a guinea pig for the Go LUCI stuff, so happy to give you a login on t.n2vi.com if you would find it easier to debug directly or hop on a video call with screen sharing. Finally, I recognize I'm a newcomer to Go Builders. So it could well be user error here. |
Thanks for your patience as we work through this and smooth out the builder onboarding process.
To confirm, are both of them set to the same value, which is the file path location of the token.json file? If you don't mind experimenting on your side, you can check if anything is different if you leave LUCI_MACHINE_TOKEN unset and instead rely on the default location for your OS ( We'll keep looking into this on our side. Though next week we might be somewhat occupied by a team event, so please expect some delays. Thanks again. |
Yes, both are set to the same value /home/luci/luci_machine_tokend/token.json. (My OS doesn't have /var/lib and anyway not a fan of leaving cleartext credentials in obscure corners of the filesystem.) This morning I've retried the same invocation of bootstrapswarm as before and don't get the 403 Client Error. So maybe there was just a transient issue. Happy to set this effort on the shelf for a week or two; enjoy the team event! |
CC @golang/release. |
Over the last week I tried swarm a few more times with no problems, so whatever issue I saw before indeed seems transient. I never saw swarm do any actual work, presumably because some server-side table is still pointing to my machine as in the old-builder state rather than new-builder. Fine by me. I'll have limited ability to work on it from November 8 - 20, but happy to work on it during the next few days if you're waiting on me. |
The builder is currently in a "Quarantined—Had 6 consecutive BOT_DIED tasks" state. @n2vi Can you please restart the swarming bot on your side and see if that's enough to get it out of that state? We've applied changes on our side (e.g., CL 546715) that should help avoid this repeating, but it's possible more work will be needed. Let's see what happens after you restart it next time. Thanks. |
Restarted. There is a message from swarming_bot urllib3 that they only support openssl but this is compiled with libressl. From the GitHub issue cited, I think this is merely a warning but if you're otherwise stuck I can investigate deeper. Or if there is something else I can do to help, just ask. FWIW, on my other openbsd-ppc64 machine, I've succeeded in compiling go1.22-devel from tip without change. Both machines are running what OpenBSD calls -current, i.e. compiled from source as of a few days ago. |
Thanks. I saw it came back up in idle state. I gave it some work, and I see it failed with:
That's useful information and on us to fix. Specifically, we need to set things up for CIPD packages to be built for the openbsd/ppc64 platform. (That will involve somewhat similar to crrev.com/c/5086069, with the caveat it can't be done using e.g. Go 1.21.0 since it doesn't support openbsd/ppc64 yet, whereas Go 1.22.0 will work.) We'll update this issue once that's done. |
Any news on this? t.n2vi.net aka host-openbsd-ppc64-n2vi had been getting kernel panics from the gopher buildlet stream so, amidst power outages and other troubles, I brought software up to date in an effort to debug. Go 1.22 is running fine here. Currently the buildlet fails to compile and while I work on that I thought I'd check in parallel on LUCI progress. If we can just cut over to the new system maybe I don't need to debug the old one? |
Thanks for checking in. I've looked into this and as things stand now, there is expected additional time after the public Go 1.22.0 release, before it's available for the CIPD package building pipeline to use. This delay might decrease in the future, and we might be able to work around it, but not yet. We'll update this issue after the CIPD packages are ready. Thanks. |
@dmitshur any update on this given the hard deadline for May 17th? |
@n2vi Can you try again? |
I'm traveling, but will try remotely soon. Just to confirm: you'd like both the old builder and the new swarming running in parallel for now, correct? |
Killed off the old processes running as swarm, rebuilt golang.org/x/build/cmd/bootstrapswarm@latest, and tried to restart but got an error message about flag -token-file-path provided but not defined. In the morning I'll start digging into what has changed in the API since October. |
As long as you still have |
Thanks. Restarted without error messages. |
Following the instructions at Dashboard builders:
hostname openbsd-ppc64-n2vi
CSR is attached after renaming since Github doesn't seem to allow attaching with the name openbsd-ppc64-n2vi.csr you asked for.
The text was updated successfully, but these errors were encountered: