Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: bsdthread_register error on macOS 12 #49425

Closed
DeanWronowski opened this issue Nov 7, 2021 · 48 comments
Closed

runtime: bsdthread_register error on macOS 12 #49425

DeanWronowski opened this issue Nov 7, 2021 · 48 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@DeanWronowski
Copy link

DeanWronowski commented Nov 7, 2021

Everything has been working perfectly for a while until I decided to upgrade Mac OS to Monterey.

What version of Go are you using (go version)?

go version go1.17.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes. I have tried updating go to the latest version, as well as updating to the latest of xcode and command tools. Also ran brew upgrade to make sure everything is updated. Also tried diffrent shells to see if that makes any difference.
I am also running the latest version of docker.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/deanwronowski/Library/Caches/go-build"
GOENV="/Users/deanwronowski/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/deanwronowski/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/deanwronowski/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.17.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.17.2/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/1p/q77885x9157c_yvzss7zx1480000gn/T/go-build2753144081=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When running the following command to connect to a server running docker it produces the error. It does not matter if the server is hosted on AWS EC2 or Digital Ocean droplets.

It seems this issue is not related just to Docker, as I am also getting this error when running a command in the terminal inside of VS.

eval "$(docker-machine env machinegoeshere)"

What did you expect to see?

Usually nothing displays which indicates its connected to the server.

What did you see instead?

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x20594e0, 0x21)
	/usr/local/go/src/runtime/panic.go:619 +0x81 fp=0x7ff7bfeff878 sp=0x7ff7bfeff858 pc=0x1029751
runtime.goenvs()
	/usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff8a8 sp=0x7ff7bfeff878 pc=0x10272d3
runtime.schedinit()
	/usr/local/go/src/runtime/proc.go:496 +0xa4 fp=0x7ff7bfeff900 sp=0x7ff7bfeff8a8 pc=0x102c014
runtime.rt0_go(0x7ff7bfeff930, 0x3, 0x7ff7bfeff930, 0x1000000, 0x3, 0x7ff7bfeffab0, 0x7ff7bfeffabf, 0x7ff7bfeffac3, 0x0, 0x7ff7bfeffacc, ...)
	/usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff908 sp=0x7ff7bfeff900 pc=0x1052c64

I have seen reports starting to appear with similar issues as well

I noticed this error popped up a few years ago, I wonder if its just a case of waiting for everyone to recompile their code again, although surprised this has not been done yet?

@zchee
Copy link
Contributor

zchee commented Nov 7, 2021

cc @cherrymui

@cjaentsch

This comment has been minimized.

@gn-mtrlitza

This comment has been minimized.

@seankhliao
Copy link
Member

a reminder of our no +1 policy
please use the voting at the top if you have nothing to add

@cherrymui cherrymui changed the title fatal error: runtime: bsdthread_register error runtime: bsdthread_register error on macOS 12 Nov 8, 2021
@cherrymui
Copy link
Member

@DeanWronowski Thanks for the report. What binaries are failing? (The docker command, docker-machine, or anything, it wasn't very clear in the original post) What versions of Go the binaries are built with? You may be able to find that out by running go version <binary>. Thanks.

@koryhutchison
Copy link

I'm seeing the same thing with Kubernetes. I was on version 1.18.0 of kubectl and that uses golang 1.13.9. I know that's a really old version, but everything was working fine until I upgraded to Monterey and now I'm seeing this same error.

@DeanWronowski
Copy link
Author

@cherrymui - Sorry, my apologies.

The docker command seems to be working fine, as well as docker-compose build namegoeshere.

However, when running the following commands the error is presented
eval "$(docker-machine env -u)"
docker-machine - even without any parameters
docker-machine ls

When running the command go version docker-machine inside of /usr/local/bin/ it says it cannot find the go version.

Hope that helps

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 8, 2021
@cagedmantis cagedmantis added this to the Backlog milestone Nov 8, 2021
@cherrymui
Copy link
Member

cherrymui commented Nov 8, 2021

@koryhutchison what version of Go that kubectl was built with?

As far as I can tell, the "bsdthread_register error" message last appears in Go 1.10 and is removed since Go 1.11.

@cherrymui
Copy link
Member

@DeanWronowski thanks, and no worries. For newer versions of Go I'd think go version would work, so I guess your docker-machine binary would be build with an old version of Go. Would it be possible to update/rebuild it with a newer version of Go? Thanks.

@DeanWronowski
Copy link
Author

DeanWronowski commented Nov 9, 2021

Just upgraded my iMac machine (separate machine to the original one) to Monterey and ran the command docker-machine which worked perfectly fine. This machine did not have go binary installed though. After installing Go via brew install go it still works fine. Not sure why it works on one machine and not the other.

They are both Intel Core i7

These are the architectures for both machines:

  • Darwin Deans-MBP-2.lan 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64
  • Darwin Deans-iMac.lan 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

The only noticeable differences is that I have Avast Business Antivirus installed on the Macbook Pro machine. However, I tried disabling it and still didn't make any difference. Must be some sort of memory protection on threads happening maybe?

@cherrymui - The original source code for docker-machine (https://github.com/docker/machine) is archived and recommends you to use Docker Desktop for Mac, which I have installed. However, not sure they have released the source code for it. I could try recompiling the docker-machine and see what happens?

Also noticed a discussion happening over at - rancher/rancher#35346 about the same issue. I have included the link in the original post as well

@mayconritzmann
Copy link

I have the same problem using terraform on version 0.11.7 and with kubectl on 1.10.13

@DeanWronowski
Copy link
Author

Docker released a new update for Docker Desktop - https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-420 - but hasn't made any difference.

@koryhutchison
Copy link

@cherrymui I went ahead and upgraded my kubectl version to v1.20.0 and I still get the error. But you can see from the output from the kubectl version --client command that it's using go1.15.5:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}

Also I verified it in the Kubernetes CHANGELOG.

@cherrymui
Copy link
Member

Sorry, I'm not familiar with docker and what can or cannot be built from source. Just as an experiment, you could try building docker-machine from the archived source with a new version of Go and see if the issue occurs.

It still looks like it fails with a binary built with an early (or unknown) version of Go. Does anyone see the issue with a newly built binary, preferably built from source? Thanks.

@cherrymui
Copy link
Member

cherrymui commented Nov 9, 2021

@koryhutchison how do you run kubectl version --client command? Doesn't kubectl fail at startup? If it is built with Go 1.15.5, go version /path/to/kubectl would likely work and print the Go version. Could you try that? Thanks.

@koryhutchison
Copy link

koryhutchison commented Nov 9, 2021

No so kubectl works until I try to kubectl get pods. So like, any time it tries to make a network request. When I run go version /path/to/kubectl I get zsh: command not found: go. I don't know if macOS is supposed to have go installed by default, but yeah I don't have it on my machine. But I've never had to install it to get kubectl to work.

Thank you for your quick responses!

@cherrymui
Copy link
Member

No so kubectl works until I try to kubectl get pods

That looks like kubectl probably spawns a subprocess and that subprocess fails. Would it be possible to find out what that subprocess is?

Yeah, kubectl should work without installing Go. If you don't mind, you could install Go from https://golang.org/dl/ . Or if your kubectl binary is downloaded from the internet, could you provide the URL for that binary? Thanks.

@koryhutchison
Copy link

koryhutchison commented Nov 9, 2021

I just made another discovery. It looks like the problem is with aws-iam-authenticator. At least for me. Because I'm getting this when I run aws-iam-authenticator help:

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x2439ce8, 0x21)
	/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff648 sp=0x7ff7bfeff628 pc=0x1029781
runtime.goenvs()
	/usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff678 sp=0x7ff7bfeff648 pc=0x1027303
runtime.schedinit()
	/usr/local/go/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff6e0 sp=0x7ff7bfeff678 pc=0x102c076
runtime.rt0_go(0x7ff7bfeff718, 0x2, 0x7ff7bfeff718, 0x0, 0x1000000, 0x2, 0x7ff7bfeff898, 0x7ff7bfeff8ae, 0x0, 0x7ff7bfeff8b3, ...)
	/usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff6e8 sp=0x7ff7bfeff6e0 pc=0x10527e4

And kubectl uses aws-iam-authenticator to authenticate with the Kubernetes cluster. So I think it's the culprit and not kubectl. As of right now I haven't found the go version for that. But the link to AWS's binary is in that link above.

I'm also seeing this issue there: kubernetes-sigs/aws-iam-authenticator#409

@koryhutchison
Copy link

Update: turns out that the problem is solved if aws-iam-authenticator gets installed via homebrew. Not exactly sure why, but my guess is they have an updated version in homebrew. So as for me, everything is working now. They must be using an older version of go or something. 🤷

@cherrymui
Copy link
Member

@koryhutchison Yeah, I think you're right. The newly installed aws-iam-authenticator is probably built with a new version of Go. Thanks!

@DeanWronowski
Copy link
Author

With regards to docker-machine not working. I found a fork repo from Gitlab that has kept the docker-machine project active, with builds using more up to date Go versions. This has fixed my issue.

The link to the builds are here https://gitlab-docker-machine-downloads.s3.amazonaws.com/main/docker-machine-Darwin-x86_64

@cherrymui
Copy link
Member

cherrymui commented Nov 9, 2021

It appears that this issue only occurs with binaries built with old unsupported versions of Go. Closing. See https://golang.org/wiki/MacOS12BSDThreadRegisterIssue

If anyone has a reproduce with a program built with Go 1.16 or later, feel free to reopen. Thanks.

@tel-kho
Copy link

tel-kho commented Nov 29, 2021

Hello,

I add this message to say that the same error occured on my side when trying to use the "gitlab-runner" CLI on my macOS Monterey 12.0.1.
After reading all messages, I tried to update it with brew as suggested and I no longer have the error.

Thank you all

@Rovel
Copy link

Rovel commented Nov 29, 2021

If anyone still have this issue, I installed the last official version of the deprecated project, they had a last release in Sep 02, 2019, with the Gitlab fork some machines with older hosts were not working.
Install one liner:

curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine && \
  chmod +x /usr/local/bin/docker-machine

@sameer-dighe
Copy link

sameer-dighe commented Dec 21, 2021

Hi all, I am also facing the same issue

I have upgraded to macos monteray and I am just running simple git command so getting this below error

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x115021a, 0x21)
        /home/travis/.gimme/versions/go1.10.2.linux.amd64/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff0d8 sp=0x7ff7bfeff0b8 pc=0x1027a21
runtime.goenvs()
        /home/travis/.gimme/versions/go1.10.2.linux.amd64/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff108 sp=0x7ff7bfeff0d8 pc=0x10255a3
runtime.schedinit()
        /home/travis/.gimme/versions/go1.10.2.linux.amd64/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff170 sp=0x7ff7bfeff108 pc=0x102a316
runtime.rt0_go(0x7ff7bfeff1a0, 0x3, 0x7ff7bfeff1a0, 0x1000000, 0x3, 0x7ff7bfeff408, 0x7ff7bfeff433, 0x7ff7bfeff447, 0x0, 0x7ff7bfeff44f, ...)
        /home/travis/.gimme/versions/go1.10.2.linux.amd64/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff178 sp=0x7ff7bfeff170 pc=0x104f2c4


Can someone please help?

@zchee
Copy link
Contributor

zchee commented Dec 21, 2021

@dmytro-martyniuk
Copy link

dmytro-martyniuk commented Jan 13, 2022

https://gitlab-docker-machine-downloads.s3.amazonaws.com/main/docker-machine-Darwin-x86_64

@DeanWronowski Could you provide a bit more details how you installed custom docker machine.
I use command:

base=https://gitlab-docker-machine-downloads.s3.amazonaws.com/main/docker-machine-Darwin-x86_64 \ && curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/usr/local/bin/docker-machine \ && chmod +x /usr/local/bin/docker-machine

but during checking docker-machine ls I'm getting:
/usr/local/bin/docker-machine: line 1: syntax error near unexpected token newline /usr/local/bin/docker-machine: line 1: <?xml version="1.0" encoding="UTF-8"?>

@justme-sports
Copy link

Am still getting this error while trying to use git commit, can some one please help to see how I can resolve it
git version: 2.35.0
go version: go1.17.6 darwin/amd64
Screen Shot 2022-01-20 at 11 27 52

@cherrymui
Copy link
Member

@justme-sports git by itself is not a Go program. You probably have git commit hook installed, for example, gofmt, or some checker. You'll need to find out what that is (such as by looking at .git/hooks) and rebuild that program with a newer version of Go.

@justme-sports
Copy link

justme-sports commented Jan 31, 2022

@justme-sports git by itself is not a Go program. You probably have git commit hook installed, for example, gofmt, or some checker. You'll need to find out what that is (such as by looking at .git/hooks) and rebuild that program with a newer version of Go.

@cherrymui Thanks so much for the information, really grateful. I do see gofmt in my /usr/local/bin/ folder.
Can you please help me: When we say rebuild the program with newer version of Go, how should i do it ?

Appreciate all your help so much!

@cherrymui
Copy link
Member

@justme-sports you can copy GOROOT/bin/gofmt to /usr/local/bin/, where GOROOT is the top-level directory of your Go 1.17.6 installation.

@justme-sports
Copy link

justme-sports commented Feb 1, 2022

@cherrymui Still facing the same issue!! is there a way we can screen share to look at this ?

@azec-pdx
Copy link

azec-pdx commented Feb 7, 2022

I can confirm having this issue after upgrade to macOS Monterey Version 12.2 (21D49) with https://github.com/shyiko/kubesec . It's latest version of binary compiled for macOS back in 2019 was built with go 1.9.1 ....

$ kubesec --version

atal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x156bf2a, 0x21)
	/home/shyiko/Development/golang/go1.9.1.linux-amd64/src/runtime/panic.go:605 +0x95 fp=0x7ff7bfeff060 sp=0x7ff7bfeff040 pc=0x1029ad5
runtime.goenvs()
	/home/shyiko/Development/golang/go1.9.1.linux-amd64/src/runtime/os_darwin.go:108 +0x83 fp=0x7ff7bfeff090 sp=0x7ff7bfeff060 pc=0x1027373
runtime.schedinit()
	/home/shyiko/Development/golang/go1.9.1.linux-amd64/src/runtime/proc.go:482 +0xa1 fp=0x7ff7bfeff0d0 sp=0x7ff7bfeff090 pc=0x102c481
runtime.rt0_go(0x7ff7bfeff108, 0x2, 0x7ff7bfeff108, 0x0, 0x1000000, 0x2, 0x7ff7bfeff378, 0x7ff7bfeff380, 0x0, 0x7ff7bfeff38a, ...)
	/home/shyiko/Development/golang/go1.9.1.linux-amd64/src/runtime/asm_amd64.s:175 +0x1eb fp=0x7ff7bfeff0d8 sp=0x7ff7bfeff0d0 pc=0x1053d4b

@serhiiromaniuk
Copy link

Just updated with Software Updates to macOS Monterey 12.2.1 and each binary became broken. Tested with either go@1.17 or go@1.13 using brew.

@justme-sports
Copy link

justme-sports commented Mar 2, 2022

I had to get the system reformat unfortunately due to this issue !! @serhiiromaniuk and @azec-pdx

@serhiiromaniuk
Copy link

@justme-sports it's totally not a fix, reformatting or deleting some stuff

@serhiiromaniuk
Copy link

So, as @zchee mentioned before, https://github.com/golang/go/wiki/MacOS12BSDThreadRegisterIssue
we should rebuild our software tools

@myang-clgx
Copy link

@justme-sports Did you resolve the issue with your git? or you reformat your system. I have the same issue and I removed the GO but the reference is still there and folder does not exist.

@justme-sports
Copy link

@justme-sports Did you resolve the issue with your git? or you reformat your system. I have the same issue and I removed the GO but the reference is still there and folder does not exist.

Unfortunately as much as I hate the reformat route, I had to go that route - when i faced this issue there was not much guidance on what could have been done to solve it !!

@dbaltor
Copy link

dbaltor commented Jun 11, 2022

I've started getting the same error with git 2.30.1 after upgrading to MacOS Monterey 12.3.1. The following error only happened to git commit and no other git commands:

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x1364ecc, 0x21)
	/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff2c8 sp=0x7ff7bfeff2a8 pc=0x1028e51
runtime.goenvs()
	/usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff2f8 sp=0x7ff7bfeff2c8 pc=0x10269d3
runtime.schedinit()
	/usr/local/go/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff360 sp=0x7ff7bfeff2f8 pc=0x102b746
runtime.rt0_go(0x7ff7bfeff398, 0x4, 0x7ff7bfeff398, 0x0, 0x1000000, 0x4, 0x7ff7bfeff5e0, 0x7ff7bfeff5ef, 0x7ff7bfeff5f4, 0x7ff7bfeff5fb, ...)
	/usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff368 sp=0x7ff7bfeff360 pc=0x1051804

Following the tips above about 1) this being caused by an old version of some go program and 2) git isn't a go program itself therefore there must be some hook in place,
I found an old hook in my core.hookspath global git configuration and managed to fix this error removing it via
git config --global --unset core.hookspath

@ajostergaard
Copy link

One of our team had this problem recently, turns out the version of kubectl we were downloading was not the one being executed (despite being first in PATH) but rather an older version of kubectl which was part of Docker CE, installation of the latest Docker resolved the issue. Though still leaves the mystery as to why the downloaded version was ignored despite being first in the PATH.

@arashkaffamanesh
Copy link

in my case I had this line for kops completion in my zshrc file, which caused the problem:
source <(kops completion zsh)
removed it and the problem is vanished.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests