@janakawicks see #30118.

I'm not sure this is a bug, without the content length header the client is using chunked encoding to transfer the body of the file. I note that your client is using POST in your first example, but the output you showed is for a PUT request. Possibly you looked at the wrong request.

@davecheney I think you are right, chunk transfer had introduce that 400 (i.e. 1024 chunk size) at the beginning of data. setting to 'identity' got transferred without chunking,

uploadRequest.TransferEncoding = []string{"identity"}

About the POST, I've tried both PUT and POST to see any difference and I've copied the POST code instead PUT code (updated now).

Yeah, it makes sense that io reader may be not able to set the content-length ahead of time.

For S3 upload chunking was not the issue but they needed content-length.

Only a small standard issue, when setting transfer-encoding to "identity" it does not send the Transfer-Encoding header at all. i.e

PUT / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

]#���...

According to the rfc https://tools.ietf.org/html/rfc7230#section-3.3.2 you MUST send the Content-Length when no Transfer-Encoding header is sent & request method defines a meaning for an enclosed payload body (POST and PUT, I think)

Actually, that way AWS S3 also violating the standard because "A sender MUST NOT send a Content-Length header field in any message that contains a Transfer-Encoding header field."

@fraenkel @odeke-em

This isn't a bug since there is no support for any special type of uploading.
Some things to consider if support were ever to be added.
How do we know the reader is at the beginning of the file? What if someone wanted to have control on uploads for partial / resumeable support? It becomes harder and harder to handle the variety of cases.

Thank you @janakawicks for filing this issue and for the ping @agnivade!
In deed, as @fraenkel and @davecheney have noted, this isn't a bug

In regards to

Only a small standard issue, when setting transfer-encoding to "identity" it does not send the Transfer-Encoding header at all. i.e

PUT / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

]#���...
According to the rfc https://tools.ietf.org/html/rfc7230#section-3.3.2 you MUST send the Content-Length when no Transfer-Encoding header is sent & request method defines a meaning for an enclosed payload body (POST and PUT, I think)

As per that RFC, sending "identity" transfer encoding was removed as per https://tools.ietf.org/html/rfc7230#section-3.3

and even in the "Transfer codings" listing in section 4 https://tools.ietf.org/html/rfc7230#section-4

"identity" doesn't exist
so I'd say Go's net/http isn't really violating anything as per the spec.

I'd advise that perhaps you take a look at the official AWS S3 SDK at https://docs.aws.amazon.com/sdk-for-go/api/, it handles all this stuff for you and I believe that it even requires that for a PUT request, that the body be of type io.ReadSeeker as per https://godoc.org/github.com/aws/aws-sdk-go/service/s3#PutObjectInput.Body

One other thing, you can examine the entirety of your requests and responses by using Go's net/http/httputil package without having to make actual requests so this can serve like your "postman" during development https://golang.org/pkg/net/http/httputil/#pkg-examples,
for example https://play.golang.org/p/WMAENUmJ9b4 or inlined below

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httputil"
	"strings"
)

func main() {
	pr, pwc := io.Pipe()
	go func() {
		defer pwc.Close()
		for i := 0; i < 10; i++ {
			io.Copy(pwc, strings.NewReader(strings.Repeat("a", 20)))
		}
	}()

	req, _ := http.NewRequest("POST", "https://example.org/foo", pr)
	req.TransferEncoding = []string{"chunked"}
	blob, _ := httputil.DumpRequestOut(req, true)
	fmt.Printf("%s\n", blob)
}

produces

POST /foo HTTP/1.1
Host: example.org
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Accept-Encoding: gzip

c8
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
0

Hope this helps and please don't hesitate to reach out in case of any questions or ideas.