New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crypto/tls: add Raw to ClientHelloInfo #32936
Comments
CC @FiloSottile |
This comment was marked as duplicate.
This comment was marked as duplicate.
That is really needed! |
Possibly add a raw byte for TLS? Just reference: |
JA3 fingerprinting in particular has proven to be quite useful, and we've been running a patched stdlib for a few years in order to support it. I personally see less need to eg. more easily support SNI proxying - tlsrouter's support is already straightforward, and I'm not sure that's a common enough use case that it'd make sense to adjust the stdlib to better accommodate. Exposing the raw ClientHello would also bring up a discussion about how we want to handle the client random, if it should be included or sanitized. For fingerprinting use cases, the only thing missing is exposing extensions. That's a small change to the API surface, in line with existing ClientHelloInfo fields, and addresses a concrete need. I'd be happy to contribute a patch if there was agreement in that direction. cc @FiloSottile this seems to be in your wheelhouse. Do you have any thoughts on the request or how we can move this towards a decision? |
I ran into a need for this because I wanted to implement ja3 fingerprinting on my app. Note that ja3 is now kind of going mainstream with AWS[1], cloudflare[2], fastly[3], among others, supporting it. Also, it seems like caddy[4] ran into a need for something like this.
PS: @elindsey , is your patch somewhere public that someone like me can have a look? |
As supplementary material, I also implement a high performance nginx module in https://github.com/phuslu/nginx-ssl-fingerprint |
This is extremely useful in controlling bot abuses on a website! |
Change https://go.dev/cl/471396 mentions this issue: |
if this is done, I think we should also update
|
I think the problem with extending |
On Sat, 4 Mar 2023 at 20:22, Bobby Powers ***@***.***> wrote:
I think the problem with extending *tls.ConnectionState is that (a) it
would either entail retaining the entire ClientHello, which can be up to
64k, or (b) require blessing a specific TLS fingerprinting technique. While
the JA3 format is the most widely used today, its not the only format or
dimension that one could imagine.
I’m in agreement.
With the change I posted above, folks who need fingerprinting can wrap
http.Server such that their handler gets the fingerprint in each
request's Context, while folks who don't need it pay no performance or
memory penalty
I think my imagination is failing me here. http.Server.Basecontext &
http.Server.ConnContext are the ones that i can think of whose context will
end up in the request. But those two methods do not have access to
clientHello. Do you mind elaborating a little bit of what you have in mind?
Thanks.
… —
Reply to this email directly, view it on GitHub
<#32936 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHMWUJ23QA645OMA726HVLW2N24HANCNFSM4H5WPGZQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
@komuw I wrote an example, along with a test showing that its working: https://github.com/bpowers/go-fingerprint-example/blob/main/fingerprinting_server.go The approach requires wrapping the underlying |
Any progress on this? I've developed a WAF for my own private use and just because the stdlib doesn't support having access to the raw TLS handshake, I had to go so deep intercepting all TLS handshakes with gopacket and correlating it with the HTTPS requests using some dirty tricks. |
Finally I managed to extract clienthello raw bytes by func (ln TCPListener) Accept() (c net.Conn, err error) {
c, err = ln.Listener.Accept()
if err != nil {
return
}
if ln.MirrorHeader {
c = &MirrorHeaderConn{Conn: c, Header: nil}
}
if ln.TLSConfig != nil {
c = tls.Server(c, ln.TLSConfig)
}
return
}
type MirrorHeaderConn struct {
net.Conn
Header []byte
}
func (c *MirrorHeaderConn) Read(b []byte) (n int, err error) {
n, err = c.Conn.Read(b)
if c.Header == nil && n > 0 && err == nil {
c.Header = make([]byte, n)
copy(c.Header, b[:n])
}
return
} For more details please see phuslu/liner@594e552 and phuslu/liner@9f6ef2d |
Packaging based on net. Conn, achieving ja3 fingerprint, http2 fingerprint, and ja4 fingerprint recognition |
@gospider007 your code appears to be making the assumption that the full ClientHello will be present in the first call to Read(). |
Thank you for the reminder. It has now been fixed |
is this been implemented yet? i can see here https://go-review.googlesource.com/c/go/+/471396 that it should have been added but dont seem to be in the documentation. I ve seen that somebody been using a patched stdlib, is that public? thx |
The extensions change has not been merged, and it seems unlikely given the (very reasonable) upstream policy of not exposing API surface if it only supports finger printing use cases. The two options are to patch stdlib or to separately store and re-parse the client handshake bytes (dealing with the extra overhead that entails). |
and can i patch the stdlib? would i have to just add the same changes of the change i ve sent before? sorry never patched stdlibs |
@murph12F here're an example of "patched stdlib" golang https://github.com/phuslu/go |
You may be interested in the https://github.com/AGWA/tlshacks package, which adds |
A fully working example of a pure Go method to process TLS client hello data: package main
import (
"crypto/tls"
"encoding/json"
"github.com/projectdiscovery/sslcert"
"log"
"net/http"
"src.agwa.name/go-listener"
"src.agwa.name/tlshacks"
)
func handler(w http.ResponseWriter, req *http.Request) {
if req.URL.Path != "/" {
http.NotFound(w, req)
return
}
clientHello := req.Context().Value(tlshacks.ClientHelloKey).([]byte)
info := tlshacks.UnmarshalClientHello(clientHello)
w.Header().Set("Content-Type", "application/json")
encoder := json.NewEncoder(w)
encoder.SetIndent("", " ")
encoder.Encode(info)
}
func main() {
tlsOptions := sslcert.DefaultOptions
hostname := "localhost"
tlsOptions.Host = hostname
tlsConf, err := sslcert.NewTLSConfig(tlsOptions)
if err != nil {
log.Panic(err.Error())
}
httpServer := &http.Server{
Handler: http.HandlerFunc(handler),
ConnContext: tlshacks.ConnContext,
}
streamListener, err := listener.Open("tcp:443")
if err != nil {
log.Fatal(err)
}
defer streamListener.Close()
tlsListener := tls.NewListener(tlshacks.NewListener(streamListener), tlsConf)
log.Fatal(httpServer.Serve(tlsListener))
} |
Similar aws/s2n-tls#607, Having access to raw ClientHello can be useful for fingerprinting clients [1] for further analysis.
Plus, With raw ClientHello message, we could also implements SNI Proxy in tls.Config.GetConfigForClient [2] , e.g. tlsrouter [3] more easily.
In openssl this can be done by setting up callback through SSL_CTX_set_msg_callback.
Would be nice to have similar ability for golang crypto.
[1] https://github.com/salesforce/ja3
[2] https://golang.org/pkg/crypto/tls/#Config
[3] https://github.com/google/tcpproxy/tree/master/cmd/tlsrouter
The text was updated successfully, but these errors were encountered: