x/crypto/ssh: "ssh-rsa-cert-v01@openssh.com" does not work for sshd OpenSSH 7.2-7.7 #58371

masp · 2023-02-07T00:09:43Z

What version of Go are you using (`go version`)?

$ go version
go version go1.20 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
> go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/masp/Library/Caches/go-build"
GOENV="/Users/masp/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/masp/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/masp/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="..."
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zy/mh971c6s3q31kqyyfzrh_y79ztk14w/T/go-build2848549523=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I have an OpenSSH server version 7.4p1.

sshd -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

I have the below program which works with golang.org/x/crypto/ssh in Go version 1.17 but not the latest tagged version 0.5.0.

main.goCode

package main

import (
	"log"
	"os"
	"time"

	"golang.org/x/crypto/ssh"
)

func main() {
	pubkeybytes, err := os.ReadFile("/Users/masp/.ssh/id_rsa-cert.pub")
	if err != nil {
		log.Fatalf("could not read keyfile: %v", err)
	}
	privkeybytes, err := os.ReadFile("/Users/masp/.ssh/id_rsa")
	if err != nil {
		log.Fatalf("could not read keyfile: %v", err)
	}
	pubkey, _, _, _, err := ssh.ParseAuthorizedKey(pubkeybytes)
	if err != nil {
		log.Fatalf("could not parse public key: %v", err)
	}
	privkey, err := ssh.ParsePrivateKey(privkeybytes)
	if err != nil {
		log.Fatalf("could not parse private key: %v", err)
	}
	signer, err := ssh.NewCertSigner(pubkey.(*ssh.Certificate), privkey)
	if err != nil {
		log.Fatalf("could not create signer: %v", err)
	}
	config := &ssh.ClientConfig{
		User:            "masp",
		Auth:            []ssh.AuthMethod{ssh.PublicKeys(signer)},
		Timeout:         time.Second * 10,
		HostKeyCallback: ssh.InsecureIgnoreHostKey(),
	}

	device := "openssh-7.4p1-test-host:22"
	_, err = ssh.Dial("tcp", device, config)
	if err != nil {
		log.Fatalf("dial error: %v", err)
	}
	log.Printf("OK")
}

What did you expect to see?

I expect it to dial successfully as it did in previous versions 0.4.0.

What did you see instead?

> go run main.go
2023/02/06 17:47:43 dial error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Debugging and Root Cause

After investigating to try and understand what was the issue, I found the following behavior.

OpenSSH 7.2: Supports rsa-sha2-256/512 alternatives, and shares those two with the client
OpenSSH 7.8: Adds support for the certificate versions of rsa-sha2-256 which are rsa-sha2-256-cert-v01@openssh.com and rsa-sha2-512-cert-v01@openssh.com. Important to note that the server does not broadcast these.

In the latest code, the server's SSH_MSG_EXT_INFO contains only rsa-sha2-256 without the certificate variants, which is why the code correctly adds those to the list of supported for those versions.

However, this breaks the fallback mechanisms if the client is using ssh-rsa-cert-v01@openssh.com, because it incorrectly identifies the server as supporting SHA2 certificates, when in reality it only supports SHA2 keys. Because of this, the client tries to get the key verified with the server, and the server rejects it which causes the permission denied error. What should happen is the client still tries the SHA1 certificate if the SHA2 one fails.

Because the library does not support any kind of configuring of signatures, I reproduced it with OpenSSH client:

> ssh -V
OpenSSH_9.2p1, OpenSSL 1.1.1s  1 Nov 2022

> ssh -Q sig
ssh-ed25519
sk-ssh-ed25519@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-dss
ssh-rsa
<--- NO ssh-rsa support!
rsa-sha2-256
rsa-sha2-512

Output

> ssh openssh-7.4p1-test-host -v -F /dev/null
debug1: send_pubkey_test: no mutual signature algorithm
masp@openssh-7.4p1-test-host: Permission denied (publickey)

Manually adding support for certificate:

ssh openssh-7.4p1-test-host -o "pubkeyacceptedalgorithms=+ssh-rsa,ssh-rsa-cert-v01@openssh.com" hostname
openssh-7.4p1-test-host

        privkey, err := ssh.ParsePrivateKey(privkeybytes)
	if err != nil {
		log.Fatalf("could not parse private key: %v", err)
	}
	mas, err := ssh.NewSignerWithAlgorithms(privkey.(ssh.AlgorithmSigner), []string{ssh.KeyAlgoRSA})
	if err != nil {
		log.Fatalf("could not create multi algo signer: %v", err)
	}
	signer, err := ssh.NewCertSigner(pubkey.(*ssh.Certificate), mas)
	if err != nil {
		log.Fatalf("could not create signer: %v", err)
	}

or

        signer, err := ssh.NewCertSigner(pubkey.(*ssh.Certificate), privkey)
	if err != nil {
		log.Fatalf("could not create signer: %v", err)
	}
	mas, err := ssh.NewSignerWithAlgorithms(signer.(ssh.AlgorithmSigner), []string{ssh.KeyAlgoRSA})
	if err != nil {
		log.Fatalf("could not create multi algorithm signer: %v", err)
	}
	signer = mas

both works. This is very similar to OpenSSH now, it works if you set the algorithm

masp · 2023-07-10T23:16:59Z

Looks promising, much appreciated! I'll give it a try once I get access to the box again.

FiloSottile · 2023-07-11T13:23:02Z

@drakkan using NewSignerWithAlgorithms to disable SHA-2 is a valid workaround, but it's a bit unfortunate to make it spread in the ecosystem.

How does OpenSSH itself connect to these misbehaving OpenSSH 7.2-7.7? (This is a bug in OpenSSH 7.2-7.7, right? If advertising the underlying algorithm, the server must support the certificate one?)

Is an automatic fallback (assuming SHA-1 is not disabled) an option?

gopherbot · 2023-07-16T12:31:01Z

Change https://go.dev/cl/510155 mentions this issue: ssh: fix certificate authentication with OpenSSH 7.2-7.7

drakkan · 2023-07-16T12:35:17Z

@masp, in the above CL I implemented an automatic fallback.
Please don't use my previous suggestion, SHA-1 based algorithms should be used only as last resort because they are insecure.
The automatic fallback uses ssh-rsa only if it is not disabled and if the server refuses more secure algorithms. Sorry for the confusion. Thank you

@FiloSottile these versions of OpenSSH accept sha2 algorithms for public key authentication, only for certificates they are rejected. I hope this is the last edge case introduced with server-sig-algs support

achal1012 · 2023-08-10T17:39:24Z

Hi @masp how did you repro this with openssh client:

Can you help me understand what did you change to get the client that can repro this issue?

Because the library does not support any kind of configuring of signatures, I reproduced it with OpenSSH client:
> ssh -V
OpenSSH_9.2p1, OpenSSL 1.1.1s  1 Nov 2022

> ssh -Q sig
ssh-ed25519
sk-ssh-ed25519@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-dss
ssh-rsa
<--- NO ssh-rsa support!
rsa-sha2-256
rsa-sha2-512

I want to understand how openssh client solved this issue, and I think if I understand how you repro this issue I could understand how they fixed it.

Thanks for your time.

masp · 2023-08-15T02:30:09Z

Thanks @drakkan looks like that will solve the issue! Appreciate the fix!

Hi @achal1012, it is certainly confusing, looking back at my notes I am trying to remember what I did as well. @drakkan can probably give a more in depth reason why it's failing, but I'll try my best to remember.

I tested with the following:

OpenSSH Client: OpenSSH_9.2p1. I used a ssh-rsa-cert-v01@openssh.com certificate. In this version, the client does not support sha1 certificates by default, so it appropriately says there is no common algorithm. If I add sha1 certificate support manually, it works, because even though the server advertises rsa-sha2-256 it doesn't assume that the certificate support exists as well. AFAIK the OpenSSH server never advertises that sha2 certificate, so the client has to ask for sha1 signing as well in case the server doesn't support sha2 certificates even if it advertises sha2 public keys.

OpenSSH Server: openssh-7.4p1 The server advertises itself as supporting rsa-sha2-256 but no support for sha2 signing for rsa certificates. Must be between 7.2 and 7.7.

Closing the issue since the newest CL seems to fix it.

drakkan · 2023-09-24T13:40:04Z

Please re-open. The patch that fixes this issue has not yet been merged

gopherbot added this to the Unreleased milestone Feb 7, 2023

dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 7, 2023

masp closed this as completed Aug 15, 2023

seankhliao reopened this Sep 24, 2023

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 23, 2023

gopherbot closed this as completed in golang/crypto@1c17e20 Nov 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/crypto/ssh: "ssh-rsa-cert-v01@openssh.com" does not work for sshd OpenSSH 7.2-7.7 #58371

x/crypto/ssh: "ssh-rsa-cert-v01@openssh.com" does not work for sshd OpenSSH 7.2-7.7 #58371

masp commented Feb 7, 2023 •

edited

dr2chase commented Feb 7, 2023

drakkan commented Jul 9, 2023 •

edited

masp commented Jul 10, 2023

FiloSottile commented Jul 11, 2023

gopherbot commented Jul 16, 2023

drakkan commented Jul 16, 2023 •

edited

achal1012 commented Aug 10, 2023 •

edited

masp commented Aug 15, 2023

drakkan commented Sep 24, 2023

x/crypto/ssh: "ssh-rsa-cert-v01@openssh.com" does not work for sshd OpenSSH 7.2-7.7 #58371

x/crypto/ssh: "ssh-rsa-cert-v01@openssh.com" does not work for sshd OpenSSH 7.2-7.7 #58371

Comments

masp commented Feb 7, 2023 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Debugging and Root Cause

Related

dr2chase commented Feb 7, 2023

drakkan commented Jul 9, 2023 • edited

masp commented Jul 10, 2023

FiloSottile commented Jul 11, 2023

gopherbot commented Jul 16, 2023

drakkan commented Jul 16, 2023 • edited

achal1012 commented Aug 10, 2023 • edited

masp commented Aug 15, 2023

drakkan commented Sep 24, 2023

masp commented Feb 7, 2023 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

drakkan commented Jul 9, 2023 •

edited

drakkan commented Jul 16, 2023 •

edited

achal1012 commented Aug 10, 2023 •

edited