Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: PUT: DefaultClient significantly slower than DefaultTransport when network is slow #26702

Open
subbu05 opened this issue Jul 30, 2018 · 11 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Milestone

Comments

@subbu05
Copy link

subbu05 commented Jul 30, 2018

What version of Go are you using (go version)?
go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?
yes

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN="/Users/test/go/bin"
GOCACHE="/Users/test/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/test/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/76/trlnwqwj0bs5003x850kzpg00000gn/T/go-build781190647=/tmp/go-build -gno-record-gcc-switches -fno-common"

Scenario:
I have a network with Download speed :400kbit/s Upload:400kbit/s RTT:200ms
I have http client developed in Golang.
Wanted to transfer 1MB file to a web server.

What did you expect to see?
File upload must almost be same to curl or other web clients.

What did you see instead?
Curl took ~28s
Go client took ~42s

More Information on environment, tools:
How does Go react when there are issues with network.
To mimic network delay - https://github.com/sitespeedio/throttle

web server : https://github.com/sashgorokhov/docker-nginx-webdav

Source Code

package main

import (
  
   "io/ioutil"
   "time"
   "fmt"
   "net/http"
   "strings"
  
)

func main() {

   client := &http.Client{}
   url := "http://localhost:9080/file.txt"
   rb, _ := ioutil.ReadFile("file1mb.txt")
   request, err := http.NewRequest("PUT", url, strings.NewReader(string(rb)))
   //request.ContentLength = int64(len(string(rb)))
   sTime := time.Now()
   response, err := client.Do(request)
   eTime := time.Since(sTime)
   fmt.Println(eTime.String())
   if err != nil {
      fmt.Println(err)
   } else {
      fmt.Println(request.Header)
      defer response.Body.Close()
      fmt.Println(response.Status)
    }
}

Tried with and without Content-Length header(chunked encoding) in the http request.

My concern is for 1mb file I see a difference what if file size is 100MB+?

@subbu05
Copy link
Author

subbu05 commented Jul 30, 2018

For a 30MB file we saw difference of about 3minutes+ with Go net/http when compared to Curl.
Condition/Environment: When network is slow.

@meirf meirf changed the title File upload using http PUT is more slower than to CURL when network is slow net/http: File PUT is slower than CURL when network is slow Jul 31, 2018
@agnivade
Copy link
Contributor

Is this reproducible with client and server being on the same machine ? Would like to investigate this on my laptop.

@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 31, 2018
@agnivade agnivade added this to the Unplanned milestone Jul 31, 2018
@subbu05
Copy link
Author

subbu05 commented Jul 31, 2018

@agnivade Yes this is continuously reproducible, provided we delay the n/w.

@agnivade
Copy link
Contributor

Oh, I see this is for http. Could you also kindly check for http2 client and see if you get similar results ?

This would help us pin-point the issue faster.

@subbu05
Copy link
Author

subbu05 commented Jul 31, 2018

I tested with http2 transport and there is a small difference between Go code and Curl.
Network Condition: Down:400kbit/s Up:400kbit/s RTT:200ms
Ex: 1MB file - Avg upload speed Curl ~25s Go ~29s
10MB - Avg upload speed Curl ~7m:30s Go ~7m:56s

@meirf
Copy link
Contributor

meirf commented Aug 1, 2018

@subbu05 have you had a chance to run the same code using http.DefaultTransport which is the core of the Client? That could help us narrow down where the bottleneck is.

@agnivade
Copy link
Contributor

agnivade commented Aug 1, 2018

/cc @bradfitz

@subbu05
Copy link
Author

subbu05 commented Aug 1, 2018

@meirf I tested with http.DefaultTransport and I see curl and golang client file upload are all most ~same rate.

@meirf meirf changed the title net/http: File PUT is slower than CURL when network is slow net/http: PUT: DefaultClient significantly slower than DefaultTransport when network is slow Aug 2, 2018
@meirf
Copy link
Contributor

meirf commented Aug 4, 2018

@subbu05 using your exact code as the client and the below Go code as the server, I am not seeing any difference in performance vs curl. Can you please show us the exact command you are using to run https://github.com/sashgorokhov/docker-nginx-webdav?

Go server:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"log"
)

func main() {
	http.HandleFunc("/file.txt", uploadHandler)
	http.ListenAndServe(":9080", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	log.Printf("uploadHandler start")
	file, err := os.Create("./result")
	if err != nil {
		panic(err)
	}
	n, err := io.Copy(file, r.Body)
	if err != nil {
		panic(err)
	}

	w.Write([]byte(fmt.Sprintf("%d bytes are recieved.\n", n)))
	log.Printf("uploadHandler end")
}

curl -X PUT -H "Content-Type: application/octet-stream" --data-binary '@file1mb.txt' http://127.0.0.1:9080/file.txt

@subbu05
Copy link
Author

subbu05 commented Aug 6, 2018

@meirf
Steps:
0. Install throttle - https://github.com/sitespeedio/throttle

  1. Started the nginx container - docker run --name webdav -p 9080:80 -v /media:/media -d sashgorokhov/webdav
  2. to mimic the latency in the network run the command throttle --profile 3gslow
  3. run curl command - time curl -X PUT -T file.txt "http://localhost:9080/file1.txt"
  4. run go client with same file.txt to the url http://localhost:9080/file2.txt (capture the time)

Observed:
30Mb file upload - I am seeing 3 minutes+ difference between curl and go.

@bradfitz
Copy link
Contributor

@meirf, the title of this bug no longer makes sense, since DefaultClient uses DefaultTransport. I don't see how they could have different speeds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Projects
None yet
Development

No branches or pull requests

4 participants