Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: add Raw to ClientHelloInfo #32936

Open
phuslu opened this issue Jul 4, 2019 · 24 comments
Open

crypto/tls: add Raw to ClientHelloInfo #32936

phuslu opened this issue Jul 4, 2019 · 24 comments
Labels
FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@phuslu
Copy link

phuslu commented Jul 4, 2019

Similar aws/s2n-tls#607, Having access to raw ClientHello can be useful for fingerprinting clients [1] for further analysis.
Plus, With raw ClientHello message, we could also implements SNI Proxy in tls.Config.GetConfigForClient [2] , e.g. tlsrouter [3] more easily.
In openssl this can be done by setting up callback through SSL_CTX_set_msg_callback.
Would be nice to have similar ability for golang crypto.

[1] https://github.com/salesforce/ja3
[2] https://golang.org/pkg/crypto/tls/#Config
[3] https://github.com/google/tcpproxy/tree/master/cmd/tlsrouter

@bcmills bcmills added FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 8, 2019
@bcmills bcmills added this to the Unplanned milestone Jul 8, 2019
@bcmills
Copy link
Contributor

bcmills commented Jul 8, 2019

CC @FiloSottile

@deancn

This comment was marked as duplicate.

@mysticaltech
Copy link

That is really needed!

@deancn
Copy link

deancn commented Aug 12, 2022

Possibly add a raw byte for TLS?

Just reference:
https://github.com/bfenetworks/bfe/blob/develop/bfe_tls/handshake_messages.go#L27

@elindsey
Copy link

JA3 fingerprinting in particular has proven to be quite useful, and we've been running a patched stdlib for a few years in order to support it.

I personally see less need to eg. more easily support SNI proxying - tlsrouter's support is already straightforward, and I'm not sure that's a common enough use case that it'd make sense to adjust the stdlib to better accommodate. Exposing the raw ClientHello would also bring up a discussion about how we want to handle the client random, if it should be included or sanitized.

For fingerprinting use cases, the only thing missing is exposing extensions. That's a small change to the API surface, in line with existing ClientHelloInfo fields, and addresses a concrete need. I'd be happy to contribute a patch if there was agreement in that direction.

cc @FiloSottile this seems to be in your wheelhouse. Do you have any thoughts on the request or how we can move this towards a decision?

@komuw
Copy link
Contributor

komuw commented Jan 15, 2023

I ran into a need for this because I wanted to implement ja3 fingerprinting on my app.

Note that ja3 is now kind of going mainstream with AWS[1], cloudflare[2], fastly[3], among others, supporting it.

Also, it seems like caddy[4] ran into a need for something like this.

  1. https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-cloudfront-supports-ja3-fingerprint-headers/
  2. https://developers.cloudflare.com/bots/concepts/ja3-fingerprint/
  3. https://docs.rs/fastly/latest/fastly/struct.Request.html#method.get_tls_ja3_md5
  4. Detect HTTPS interception caddyserver/caddy#1430

PS: @elindsey , is your patch somewhere public that someone like me can have a look?

@phuslu
Copy link
Author

phuslu commented Jan 16, 2023

As supplementary material, I also implement a high performance nginx module in https://github.com/phuslu/nginx-ssl-fingerprint

@mysticaltech
Copy link

This is extremely useful in controlling bot abuses on a website!

@gopherbot
Copy link

Change https://go.dev/cl/471396 mentions this issue: crypto/tls: expose extensions presented by client to GetCertificate

@komuw
Copy link
Contributor

komuw commented Mar 4, 2023

if this is done, I think we should also update http.Request.TLS *tls.ConnectionState. This is because the initial ask in this issue was ClientHello can be useful for fingerprinting clients, as such I believe this is best utilised in the request-response lifecyle.
http.Request.TLS is already documented as

// allows HTTP servers and other software to record
// information about the TLS connection on which the request
// was received. 

@bpowers
Copy link
Contributor

bpowers commented Mar 4, 2023

I think the problem with extending *tls.ConnectionState is that (a) it would either entail retaining the entire ClientHello, which can be up to 64k, or (b) require blessing a specific TLS fingerprinting technique. While the JA3 format is the most widely used today, its not the only format or dimension that one could imagine. With the change I posted above, folks who need fingerprinting can wrap http.Server such that their handler gets the fingerprint in each request's Context, while folks who don't need it pay no performance or memory penalty

@komuw
Copy link
Contributor

komuw commented Mar 4, 2023 via email

@bpowers
Copy link
Contributor

bpowers commented Mar 4, 2023

@komuw I wrote an example, along with a test showing that its working: https://github.com/bpowers/go-fingerprint-example/blob/main/fingerprinting_server.go

The approach requires wrapping the underlying net.Conn, and then storing a shared struct on both the wrapped conn (Which is accessible from GetCertificate, which is called with the *tls.ClientHelloInfo) and the context returned from ConnContext, which is used to derive every request's context

@1366613
Copy link

1366613 commented Mar 7, 2023

Any progress on this?

I've developed a WAF for my own private use and just because the stdlib doesn't support having access to the raw TLS handshake, I had to go so deep intercepting all TLS handshakes with gopacket and correlating it with the HTTPS requests using some dirty tricks.

@phuslu
Copy link
Author

phuslu commented Jul 16, 2023

Finally I managed to extract clienthello raw bytes by MirrorHeaderConn trick.

func (ln TCPListener) Accept() (c net.Conn, err error) {
	c, err = ln.Listener.Accept()
	if err != nil {
		return
	}
	if ln.MirrorHeader {
		c = &MirrorHeaderConn{Conn: c, Header: nil}
	}
	if ln.TLSConfig != nil {
		c = tls.Server(c, ln.TLSConfig)
	}
	return
}

type MirrorHeaderConn struct {
	net.Conn
	Header []byte
}

func (c *MirrorHeaderConn) Read(b []byte) (n int, err error) {
	n, err = c.Conn.Read(b)
	if c.Header == nil && n > 0 && err == nil {
		c.Header = make([]byte, n)
		copy(c.Header, b[:n])
	}

	return
}

For more details please see phuslu/liner@594e552 and phuslu/liner@9f6ef2d

@gospider007
Copy link

Packaging based on net. Conn, achieving ja3 fingerprint, http2 fingerprint, and ja4 fingerprint recognition
https://github.com/gospider007/fp#quick-start-with-gin

@elindsey
Copy link

@gospider007 your code appears to be making the assumption that the full ClientHello will be present in the first call to Read().

@gospider007
Copy link

@gospider007 your code appears to be making the assumption that the full ClientHello will be present in the first call to Read().

Thank you for the reminder. It has now been fixed

@murph12F
Copy link

is this been implemented yet? i can see here https://go-review.googlesource.com/c/go/+/471396 that it should have been added but dont seem to be in the documentation. I ve seen that somebody been using a patched stdlib, is that public? thx

@elindsey
Copy link

The extensions change has not been merged, and it seems unlikely given the (very reasonable) upstream policy of not exposing API surface if it only supports finger printing use cases. The two options are to patch stdlib or to separately store and re-parse the client handshake bytes (dealing with the extra overhead that entails).

@murph12F
Copy link

and can i patch the stdlib? would i have to just add the same changes of the change i ve sent before? sorry never patched stdlibs

@phuslu
Copy link
Author

phuslu commented Oct 26, 2023

@murph12F here're an example of "patched stdlib" golang https://github.com/phuslu/go

@joeshaw
Copy link
Contributor

joeshaw commented Oct 27, 2023

You may be interested in the https://github.com/AGWA/tlshacks package, which adds net.Conn and net.Listener implementations that capture the ClientHello and parse it. It also contains functions for generating a JA3 fingerprint from it.

@1366613
Copy link

1366613 commented Mar 2, 2024

A fully working example of a pure Go method to process TLS client hello data:

package main

import (
	"crypto/tls"
	"encoding/json"
	"github.com/projectdiscovery/sslcert"
	"log"
	"net/http"
	"src.agwa.name/go-listener"
	"src.agwa.name/tlshacks"
)

func handler(w http.ResponseWriter, req *http.Request) {
	if req.URL.Path != "/" {
		http.NotFound(w, req)
		return
	}

	clientHello := req.Context().Value(tlshacks.ClientHelloKey).([]byte)
	info := tlshacks.UnmarshalClientHello(clientHello)

	w.Header().Set("Content-Type", "application/json")
	encoder := json.NewEncoder(w)
	encoder.SetIndent("", "    ")
	encoder.Encode(info)
}

func main() {
	tlsOptions := sslcert.DefaultOptions
	hostname := "localhost"
	tlsOptions.Host = hostname

	tlsConf, err := sslcert.NewTLSConfig(tlsOptions)
	if err != nil {
		log.Panic(err.Error())
	}

	httpServer := &http.Server{
		Handler:     http.HandlerFunc(handler),
		ConnContext: tlshacks.ConnContext,
	}

	streamListener, err := listener.Open("tcp:443")
	if err != nil {
		log.Fatal(err)
	}
	defer streamListener.Close()

	tlsListener := tls.NewListener(tlshacks.NewListener(streamListener), tlsConf)
	log.Fatal(httpServer.Serve(tlsListener))
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests