Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: server will drop tcp connections after a certain number of concurrent connections are made #61924

Closed
matthewdavidrodgers opened this issue Aug 10, 2023 · 5 comments
Labels
arch-arm64 NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@matthewdavidrodgers
Copy link

What version of Go are you using (go version)?

Tested on two machines:

arm:

$ go version
go version go1.20.6 darwin/arm64

x86:

$ go version
go version go1.20.6 linux/amd64

Does this issue reproduce with the latest release?

yes. tested on go version devel go1.22-8ecdc4e Thu Aug 10 14:38:22 2023 +0000 darwin/arm64 as well

What operating system and processor architecture are you using (go env)?

go env Output
arm:
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/matthewrodgers/Library/Caches/go-build"
GOENV="/Users/matthewrodgers/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/matthewrodgers/go/pkg/mod"
GONOPROXY="code.cfops.it"
GONOSUMDB="code.cfops.it"
GOOS="darwin"
GOPATH="/Users/matthewrodgers/go"
GOPRIVATE="code.cfops.it"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.20.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.6/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/d9/b6xts_292jl1xm48py971tm40000gp/T/go-build4134932022=/tmp/go-build -gno-record-gcc-switches -fno-common"
x86:
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matthewrodgers/.cache/go-build"
GOENV="/home/matthewrodgers/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matthewrodgers/go/pkg/mod"
GONOPROXY="code.cfops.it"
GONOSUMDB="code.cfops.it"
GOOS="linux"
GOPATH="/home/matthewrodgers/go"
GOPRIVATE="code.cfops.it"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3084080192=/tmp/go-build -gno-record-gcc-switches"

What did you do?

ran this test:

package main

import (
	"fmt"
	"io"
	"net"
	"net/http"
	"testing"
)

type testserv struct {
	t *testing.T
}

func (ss *testserv) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	_, err := io.WriteString(w, "OK")
	if err != nil {
		ss.t.Fatal("could not start server: ", err)
	}
}

func TestConcurrentConnections(t *testing.T) {
	for _, conns := range []int{100, 1000} {
		t.Run(fmt.Sprintf("%d connections", conns), func(t *testing.T) {
			completed := make(chan struct{}, conns)
			failed := make(chan error)

			listener, err := net.Listen("tcp", "127.0.0.1:0")
			if err != nil {
				t.Fatal("could not list to localhost: ", err)
			}
			defer listener.Close()

			go http.Serve(listener, &testserv{t: t})

			// make concurrent requests to the server
			for i := 0; i < conns; i++ {
				go func() {
					_, err := http.Get("http://" + listener.Addr().String())
					// if the request failed, report back to main goroutine
					if err != nil {
						failed <- err
					}
					completed <- struct{}{}
				}()
			}

			completedCount := 0
			for {
				select {
				case err := <-failed:
					t.Fatal(fmt.Sprintf("completed %d connections but got err: %v", completedCount, err))
				case <-completed:
					completedCount += 1
				}

				if completedCount == conns {
					break
				}
			}
		})
	}
}

What did you expect to see?

both test cases, opening 100 concurrent connections to a server, and opening 1000 concurrent connections to a server, succeed

What did you see instead?

on x64, both test cases pass as expected
on arm, only the 100 concurrent connections case succeeds - the 1000 concurrent connections case fails with:
completed 0 connections but got err: Get "http://127.0.0.1:56934": read tcp 127.0.0.1:56943->127.0.0.1:56934: read: connection reset by peer, with completed connections varying slightly. from my testing, something around 130 concurrent connections triggers this behavior

@thanm
Copy link
Contributor

thanm commented Aug 10, 2023

I think this may be a darwin problem as opposed to a linux problem; when I run your test on linux/arm64 I don't see any issues. Perhaps macos is imposing some sort of resource limit in this case?

@mknyszek mknyszek changed the title arm: a go server will drop tcp connections after a certain number of concurrent connections are made net/http: server will drop tcp connections after a certain number of concurrent connections are made Aug 10, 2023
@mknyszek mknyszek added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-arm64 labels Aug 10, 2023
@mknyszek mknyszek added this to the Backlog milestone Aug 10, 2023
@mknyszek
Copy link
Contributor

CC @neild

@panjf2000
Copy link
Member

I think this has something to do with #54529

Try this solution, it ought to work.

@thanm
Copy link
Contributor

thanm commented Aug 11, 2023

Thanks @panjf2000 that is helpful. I am going to close this as a duplicate of #54529.

@thanm
Copy link
Contributor

thanm commented Aug 11, 2023

Duplicate of #54529.

@thanm thanm closed this as completed Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm64 NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests

4 participants