/cc @bradfitz @rsc

Also /cc @FiloSottile for crypto/tls.

This is not supposed to happen, and it could be a bug on the server side as well.

Can you give us some details on what the server is? Any logs from it would also help.

Also, to ensure this has nothing to do with net/http, can you check that tls.Dial also fails?

I will write a local test to confirm with dial, I have remote server details just not on me. Will update shortly. Thanks guys.

The remote system is a Java based web.api, I can get more details from the author later. Here is what I do know off the top of my head. I switch to go version go1.13.7 windows/amd64 about 5 days ago and began dev testing my software today where I identified a problem that had me tearing my hair out. The previous version of my app (Golang 1.12.4) is currently hitting the system in question without this issue. That Java Web Api isn't really in my domain of knowledge - but I think it is a Spring framework, and if I had to guess, they are using OpenJDK 11.

I am now concerned after reading this that it might not be a cut and dry Golang problem:
https://webtide.com/openjdk-11-and-tls-1-3-issues/

There weren't any logs on external system indicating an error and does currently work against with the 1.12.4 version. Client side, we just received a remote error: tls: handshake failure on our end on any attempt to call the HTTPS endpoints to the system in question. HTTP was unaffected (no shocker there). I suspected Tls v1.3 as it was recently introduced and confirmed my TLSClientConfig change resumed communication as previously indicated.

@FiloSottile
I was successfully able to ping pong with Tcp Dialer and a self-signed cert for both server and client with InsecureSkipVerify false. I don't know if this is a good representation of things based on using loopback localhost - everything else should be similar conditions.

package main

import (
	"bufio"
	"crypto/tls"
	"crypto/x509"
	"io/ioutil"
	"log"
	"net"
)

func main() {

	readyToReceive := make(chan struct{}, 1)
	go startServer(readyToReceive)
	<-readyToReceive

	runClient()
}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer(readyToReceive chan struct{}) {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}
	ln, err := tls.Listen("tcp", ":443", config)
	if err != nil {
		log.Println(err)
		return
	}
	defer ln.Close()

	readyToReceive <- struct{}{}
	for {
		conn, err := ln.Accept()
		if err != nil {
			log.Println(err)
			continue
		}
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()
	r := bufio.NewReader(conn)
	for {
		msg, err := r.ReadString('\n')
		if err != nil {
			log.Println(err)
			return
		}

		println(msg)

		n, err := conn.Write([]byte("pong\n"))
		if err != nil {
			log.Println(n, err)
			return
		}
	}
}

func runClient() {
	caCert, err := ioutil.ReadFile("server.crt")
	if err != nil {
		log.Fatal(err)
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	log.SetFlags(log.Lshortfile)

	conf := &tls.Config{
		//InsecureSkipVerify: true,
		RootCAs:    caCertPool,
		MinVersion: tls.VersionTLS10,
		MaxVersion: tls.VersionTLS13,
	}

	//conn, err := tls.Dial("tcp", "127.0.0.1:443", conf)
	conn, err := tls.Dial("tcp", "localhost:443", conf)
	if err != nil {
		log.Println(err)
		return
	}
	defer conn.Close()

	n, err := conn.Write([]byte("ping\n"))
	if err != nil {
		log.Println(n, err)
		return
	}

	buf := make([]byte, 100)
	n, err = conn.Read(buf)
	if err != nil {
		log.Println(n, err)
		return
	}

	println(string(buf[:n]))
}

If I had to speculate - I think the JDK based system is returning an unrecognizable type of error - it really doesn't support TLS1.3 and throws a weird exception internally - for instance. I think internally Golang attributes this response to a more generic type of error and doesn't attempt to fallback and retry with the next protocol down.

Just a hunch.

Starting to play around with NewServeMux and Tls1.2 as the only TLS protocol allowed and what Postman was configured for - I am getting the following:
server.go:3056: http: TLS handshake error from [::1]:61003: EOF

So this is Golang TLS 1.2 (server) and Postman TLS 1.2 (client)

package main

import (
	"crypto/tls"
	"log"
	"net/http"
)

func main() {

	startServer()

}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer() {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
		w.Write([]byte("This is an example server.\n"))
	})
	srv := &http.Server{
		Addr:         ":443",
		Handler:      mux,
		TLSConfig:    config,
		TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
	}
	log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key"))
}

HttpClient w/ Server

Unable to reproduce it either locally, or golang to golang. No errors other than the above EOF ^ in the example above.

I tested with total mismatch (only v1.2 server, and only v1.3 client) and received the correct error I was expecting from the Golang handshake.

Get https://localhost:443/: remote error: tls: protocol version not supported

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/url"
	"time"
)

func main() {

	readyToReceive := make(chan struct{}, 1)
	go startServer(readyToReceive)
	<-readyToReceive

	startClient()
}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer(readyToReceive chan struct{}) {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.Write([]byte("This is an example server."))
	})
	srv := &http.Server{
		Addr:         ":443",
		Handler:      mux,
		TLSConfig:    config,
		TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
	}
	readyToReceive <- struct{}{}
	log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key"))
}

func startClient() {
	client := &http.Client{
		Transport: BuildHTTPTransport(nil),
	}

	req, err := http.NewRequest("GET", "https://localhost:443/", nil)
	if err != nil {
		fmt.Println(err)
		return
	}

	for {
		resp, err := client.Do(req)
		if err != nil {
			fmt.Println(err)
			return
		}
		defer resp.Body.Close()

		if resp.StatusCode == http.StatusOK {
			bodyBytes, err := ioutil.ReadAll(resp.Body)
			if err != nil {
				log.Fatal(err)
			}

			fmt.Printf("Client - Response Body: %s\r\n", string(bodyBytes))
		} else {
			fmt.Printf("Client - Response Status: %d\r\n", resp.StatusCode)
		}

		time.Sleep(1 * time.Second)
	}
}

func BuildHTTPTransport(proxy func(*http.Request) (*url.URL, error)) *http.Transport {

	caCert, err := ioutil.ReadFile("server.crt")
	if err != nil {
		log.Fatal(err)
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	transport := &http.Transport{
		Proxy: proxy, // Can Be Nil
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
			DualStack: true,
		}).DialContext,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   100, // Default was 2
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: time.Second,
	}

	transport.TLSClientConfig = &tls.Config{
		RootCAs:    caCertPool,
		MinVersion: tls.VersionTLS10,
		MaxVersion: tls.VersionTLS13,
	}

	return transport
}

After doing some more research, I think Java JDK (v11) is a - or perhaps THE- key missing ingredient from my testing.

This was found in another thread that was working in reverse (Golang Servers with Java Clients). Odds are, our Java Web.Api side maybe out of date. I will try and track down what version we are using.

Related to #35722
https://hg.openjdk.java.net/jdk/jdk/rev/b9d1ce20dd4b

It does sound like another OpenJDK issue. Can you update the server and check if it goes away?

Otherwise, it would be useful to get PCAPs of the connection.

(A correctly implemented TLS 1.2 server will automatically negotiate down if the client supports at least a shared protocol version.)

@FiloSottile I have started that conversation internally. I would like to ask about the EOF ^ I got above on TLS.

#37808 (comment)

That just means the client closed the connection. Without client side logs or details it's hard to tell, but it might just be that it did not like the certificate. Anyway it's a separate issue so if you think it might be a Go bug, please open a new issue.

Fair point, I am going to close this issue - thanks for reviewing it with me Filo. I have updated the title so its more readily searchable in issues - I know I didn't see the other related tickets because of that. If you disprove you can rename back.

Still interested in hearing if this is resolved by an update, if not we might need to work with OpenJDK to report the bug.

I wasn't able to update the client from JDK 11, but I did manage to write a little something with JDK 11 web.api with only TLS1.2 enabled and did confirm the issue matched what happens in production for me.

With the world ending, I have been a little busy, so I apologize for the delay.

Take your time, no need to apologize, and take care.

I don't know if it is related but I am having a similar issue. I am not a Go developer so my apologies in advance. However, I am playing about with a project to try to learn go and the tls negotiation seems to be failing sometimes. However, if I proxy the requests through Burp, it works fine - presumably because burp then handles the tls negotiation with the server?

I captured both TLS "Client Hello" messages with wireshark, without a proxy on the left and with on the right. I noticed that without the proxy, the TLS version is reported as 1.0 and 1.2 in the same packet. Is this expected behaviour?

@Jab2870 That's intended behavior, yeah, the record layer is a legacy field completely stripped of any semantic meaning. We just send the most compatible value.

If you can share a PCAP of a failed handshake, we can look at it and try to understand what's happening.

(Is the issue happening between a Go client and a JDK11 server? If not, please open a separate issue with details about the server.)

@FiloSottile Thanks for your quick response and explanation.

@houseofcat Closing for timeout, but let me know if you get more info.