crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 #65633

VictorLowther · 2024-02-09T14:45:42Z

Go version

go 1.22.0 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/victor/.cache/go-build'
GOENV='/home/victor/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/victor/go/pkg/mod'
GONOPROXY='gitlab.com/rackn'
GONOSUMDB='gitlab.com/rackn'
GOOS='linux'
GOPATH='/home/victor/go'
GOPRIVATE='gitlab.com/rackn'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2733416375=/tmp/go-build -gno-record-gcc-switches'

What did you do?

A project of mine directly embeds the *x509.Certificate struct in a larger struct, which is encoded and decoded using encoding/gob. The struct added fields that use a new x509.OID structure, which contains a single non-exported field.

Test program is here on the Go playground. Running it under Go 1.22 displays the error, and running it under Go 1.21 does not.

What did you see happen?

go run main.go
2024/02/09 08:42:03 Failed to gob encode enpty x.509 cert: gob: type x509.OID has no exported fields

What did you expect to see?

go run main.go
2024/02/09 08:41:47 Encoded empty x.509 cert

Adding MarshalBinary and UnmarshalBinary methods to *x509.OID allows encoding to succeed. Patch against 1.22.0:

diff --git a/src/crypto/x509/oid.go b/src/crypto/x509/oid.go
index 5359af624b..0e2b872539 100644
--- a/src/crypto/x509/oid.go
+++ b/src/crypto/x509/oid.go
@@ -24,6 +24,15 @@ type OID struct {
        der []byte
 }

+func (o *OID) MarshalBinary() ([]byte, error) {
+       return append([]byte{}, o.der...), nil
+}
+
+func (o *OID) UnmarshalBinary(b []byte) error {
+       o.der = append(o.der[:0], b...)
+       return nil
+}
+
 func newOIDFromDER(der []byte) (OID, bool) {
        if len(der) == 0 || der[len(der)-1]&0x80 != 0 {
                return OID{}, false

The text was updated successfully, but these errors were encountered:

mateusz834 · 2024-02-09T15:59:36Z

CC @rolandshoemaker

Akaame · 2024-03-04T14:25:54Z

Has there been any progress on this matter?

VictorLowther · 2024-03-06T17:38:30Z

As far as I can tell, no. I didn'r see a fix go into 1.22.1

rolandshoemaker · 2024-03-06T18:30:05Z

While I'm not strictly opposed to fixing this, we've never explicitly claimed that Certificate is compatible with encoding/gob, so I don't think we will consider this a breaking change, and I'm not particularly sure how much I want to encourage using encoding/gob here.

Is there a particular reason to use encoding/gob over the DER encoding of a certificate (which is available via Certificate.Raw), which is likely to be significantly smaller than a gob encoded struct (which will, itself, contain the raw DER encoding of the certificate, since it is a part of the struct)?

If its just because it's embedded in another struct, this problem can be worked around by defining your own type which wraps x509.Certificate (i.e. type gobCert *x509.Certificate) and implements MarshalBinary/UnmarshalBinary methods which encode it (i.e. by just returning the Raw field, and calling x509.ParseCertificate respectively). This may actually end up being more performant because of the specialized nature of the x509 parser. i.e. https://go.dev/play/p/exGyeSWuFAL.

VictorLowther · 2024-03-06T18:51:13Z

The struct in question is embedded in the replicated state machine for a Raft cluster, making dealing with marshalling changes on the fly like that challenging -- changing how the struct is marshalled like that would involve manual intervention and downtime for our customers, whereas encoding/gob will transparently just work across changes like this as long as the OID field implements MarshalBinary/UnmarshalBinary with some possibly minor fixups to arrange for the Policies field to be filled out if it isn't already.

And while y'all never claimed that Certificate is compatible with encoding/gob, Hyrum's Law still applies and y'all made a change that broke encoding/gob that could have been avoided by making the OID a []byte directly instead of doing the single private struct field thing, and making it the only thing in an x509.Certificate that is not exported.

rsc · 2024-03-11T17:00:12Z

There are ways to work around this, yes, but if gob worked before on these types we should probably keep it working one way or another. We have run into similar issues in the past and in general treat those wire changes as subject to compatibility. Unfortunately, the fix seems to be to add new methods, which as a matter of policy we don't do in point releases. So the fix (new methods) would have to wait for Go 1.23. Unless we have evidence of widespread impact, that may be the best approach.

tycho · 2024-03-11T17:02:54Z

How widespread is widespread enough? I found this issue because it was causing builds of Mattermost and its plugins to fail to work properly. To work around it I had to modify the build to avoid using the system-installed Go and download a Go 1.21 binary.

rsc · 2024-03-11T17:04:56Z

Filed #66249. Thanks for that extra impact information @tycho. That's more impact than I realized.

rsc · 2024-03-12T16:54:40Z

@gopherbot please backport go1.22

gopherbot · 2024-03-12T16:55:23Z

Backport issue(s) opened: #66273 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2024-03-12T16:59:11Z

Change https://go.dev/cl/571095 mentions this issue: encoding/gob: make x509.Certificate marshalable again

rsc · 2024-03-12T17:14:13Z

I spoke to @robpike about this yesterday. We think the simplest, least invasive change for Go 1.22.x would be to change gob to look for the x509.Certificate type and have it skip over the Policies field entirely. That seems like it would fix Mattermost and other affected users, and then we can proceed with new methods on the normal timeline and then remove the gob change. The pending CL 571095 does this.

gopherbot · 2024-03-14T16:10:09Z

Change https://go.dev/cl/571715 mentions this issue: [release-branch.go1.22] encoding/gob: make x509.Certificate marshalable again

…le again The OID type is not exported data like most of the other x509 structs. Using it in x509.Certificate made Certificate not gob-compatible anymore, which breaks real-world code. As a temporary fix, make gob ignore that field, making it work as well as it did in Go 1.21. For Go 1.23, we anticipate adding a proper fix and removing the gob workaround. See #65633 and #66249 for more details. For #66249. For #65633. Fixes #66273. Change-Id: Idd1431d15063b3009e15d0565cd3120b9fa13f61 Reviewed-on: https://go-review.googlesource.com/c/go/+/571095 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> Reviewed-on: https://go-review.googlesource.com/c/go/+/571715 Reviewed-by: David Chase <drchase@google.com>

mateusz834 added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 9, 2024

odeke-em changed the title ~~crypto/x509: x509.Certificate no longer encodable using encoding/gob~~ crypto/x509: Certificate no longer encodable using encoding/gob Feb 14, 2024

odeke-em changed the title ~~crypto/x509: Certificate no longer encodable using encoding/gob~~ crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 Feb 14, 2024

rsc mentioned this issue Mar 11, 2024

crypto/x509: make OID have text, binary marshal methods #66249

Open

gopherbot mentioned this issue Mar 12, 2024

crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 [1.22 backport] #66273

Closed

gopherbot closed this as completed in 376be64 Mar 14, 2024

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 20, 2024

dmitshur added this to the Go1.23 milestone Mar 20, 2024

streamer45 mentioned this issue Mar 21, 2024

Update 9.4< and after are breaking calls mattermost/mattermost-plugin-calls#664

Closed

This was referenced Mar 21, 2024

Bug: Update 9.4 and after are breaking focalboard mattermost/focalboard#4994

Closed

Update 9.4 and after are breaking focalboard mattermost/mattermost#26425

Closed

[BUG]Meeting creation is not working mattermost-community/mattermost-plugin-jitsi#243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 #65633

crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 #65633

VictorLowther commented Feb 9, 2024

mateusz834 commented Feb 9, 2024

Akaame commented Mar 4, 2024

VictorLowther commented Mar 6, 2024

rolandshoemaker commented Mar 6, 2024

VictorLowther commented Mar 6, 2024

rsc commented Mar 11, 2024

tycho commented Mar 11, 2024

rsc commented Mar 11, 2024

rsc commented Mar 12, 2024

gopherbot commented Mar 12, 2024

gopherbot commented Mar 12, 2024

rsc commented Mar 12, 2024

gopherbot commented Mar 14, 2024

crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 #65633

crypto/x509: Certificate no longer encodable using encoding/gob in Go1.22 #65633

Comments

VictorLowther commented Feb 9, 2024

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

mateusz834 commented Feb 9, 2024

Akaame commented Mar 4, 2024

VictorLowther commented Mar 6, 2024

rolandshoemaker commented Mar 6, 2024

VictorLowther commented Mar 6, 2024

rsc commented Mar 11, 2024

tycho commented Mar 11, 2024

rsc commented Mar 11, 2024

rsc commented Mar 12, 2024

gopherbot commented Mar 12, 2024

gopherbot commented Mar 12, 2024

rsc commented Mar 12, 2024

gopherbot commented Mar 14, 2024

Output of `go env` in your module/workspace: