Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity" on docker engine REST API #40735

Closed
h8liu opened this issue Aug 12, 2020 · 13 comments
Closed

net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity" on docker engine REST API #40735

h8liu opened this issue Aug 12, 2020 · 13 comments
Labels
FrozenDueToAge

Comments

@h8liu
Copy link

h8liu commented Aug 12, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/h8liu/Library/Caches/go-build"
GOENV="/Users/h8liu/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/h8liu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/h8liu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/vf/2lnfy_h578s0mc2911f90qc00000gn/T/go-build362761685=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/url"
	"strings"
)

func main() {
	// Before running this needs to:
	// - Have docker 19.03.12 installed
	// - Run "docker create --name test alpine sleep 10000" to create the container
	// - Run "docker start test" to start it.
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	d := new(net.Dialer)
	x := &http.Transport{
		DialContext: func(ctx context.Context, net, addr string) (net.Conn, error) {
			return d.DialContext(ctx, "unix", "/var/run/docker.sock")
		},
	}

	c := &http.Client{Transport: x}

	u := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/containers/test/exec",
	}
	var req = struct {
		Cmd          []string
		AttachStdout bool
		AttachStderr bool
	}{
		Cmd:          []string{"echo", "hello"},
		AttachStderr: true,
		AttachStdout: true,
	}
	body, err := json.Marshal(req)
	if err != nil {
		return err
	}
	header := make(http.Header)
	header.Add("Content-Type", "application/json")
	httpReq := &http.Request{
		Method: "POST",
		URL:    u,
		Header: header,
		Body:   ioutil.NopCloser(bytes.NewBuffer(body)),
	}
	resp, err := c.Do(httpReq)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	bodyBytes, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return err
	}
	var createResp struct {
		ID string `json:"Id"`
	}
	if err := json.Unmarshal(bodyBytes, &createResp); err != nil {
		return err
	}

	u2 := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/exec/" + createResp.ID + "/start",
	}

	httpReq2 := &http.Request{
		Method: "POST",
		URL:    u2,
		Header: header,
		Body:   ioutil.NopCloser(strings.NewReader("{}")),
	}
	resp2, err := c.Do(httpReq2)
	if err != nil {
		return err
	}

	defer resp2.Body.Close()
	bodyBytes2, err := ioutil.ReadAll(resp2.Body)
	if err != nil {
		return err
	}
	fmt.Println(string(bodyBytes2))

	return nil
}

What did you expect to see?

See hello.

What did you see instead?

2020/08/12 10:37:28 Post "http://uds/v1.40/exec/bea64d1e75682a89d1e75d59da45fde21977aa8103b523a5b6e9d8a7eba85ab4/start": net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity"
exit status 1

Docker version

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Some additional notes:

  • The issue does not reproduce on go1.14.7 run (because the transfer encoding enforcement was not there I think).
  • The issue (weirdly) does not reproduce on a Linux machine of mine. Maybe the docker server is on a different version, and handles requests differently?
@h8liu
Copy link
Author

h8liu commented Aug 12, 2020

after some learning, this seems to be expected behavior..

@h8liu h8liu closed this as completed Aug 12, 2020
@rkuprov
Copy link

rkuprov commented Aug 14, 2020

@h8liu how did you solve it? I'm having a similar issue.

@h8liu
Copy link
Author

h8liu commented Aug 15, 2020

I did not solve it. I mitigated it by running all stuff on a linux machine. The issue seems to be OSX only. I am guessing that the OSX docker desktop app has some kind of reverse proxy for the socket that is doing stuff incorrectly, but I don't know for sure.

@h8liu
Copy link
Author

h8liu commented Aug 15, 2020

I closed it because I don't think golang client library is doing anything wrong, although this seems like a regression.

@kostyay kostyay mentioned this issue Aug 18, 2020
Closed
@sazzer
Copy link

sazzer commented Aug 18, 2020

It's worth mentioning - since I came here from the testcontainers-go issue - that the Docker daemon on macOS works perfectly well with the appropriate TestContainers libraries for Go 1.14.7, with Rust, with Node and with Java, as well as just the standard Docker tooling. It only doesn't work with Go 1.15.

I'm not sure if the Go libraries for communicating with Docker work differently than in all those other languages - given that Docker itself is written in Go, that is possible - but when something is broken in only one version of one language which has itself only just been released, it very much feels like a newly introduced bug.

@iulian-gm
Copy link

I also stumbled upon this via testcontainers-go (testcontainers/testcontainers-go#232). Having the exact same issue with Go 1.15 only.

@davecheney
Copy link
Contributor

@sazzer @iulian-gm as you have noted that In your opinion this issue is not fixed would you please raise a new issue, fill out the template and provide a way for someone to reproduce the issues you are experiencing. Thank you

@tankbusta
Copy link

tankbusta commented Aug 20, 2020 •
edited

I'm seeing the same thing - can this issue just be reopened? It follows the template and provides a way to reproduce

Edit: It might have been caused by the work to fix #38867 but I couldn't find any points in Docker that explicitly set the transfer encoding to identity (although I'm not too familiar with transfer-encoding/identity). Debugging the example provided by @h8liu does indeed show "identity" being returned by the server

Screen Shot 2020-08-20 at 09 59 21

@davecheney
Copy link
Contributor

Thank you to those who have commented.

Re new issue vs continuing one that was closed without conclusion. It is my experience that the likelihood of an issue being closed successfully is inversely proportionate to its length.

Focused issues with clear instructions to reproduce the problem are often closed quickly.

To that end, please default to raising new issues. If there is a cost in terms of deduplicating issues that is one myself and the other issue gardeners are happy to invest in to see issues diagnosed and resolved quickly.

@h8liu h8liu reopened this Aug 21, 2020
@h8liu
Copy link
Author

h8liu commented Aug 21, 2020

reopened per requests @tankbusta

the head of this thread has a working example program that can reproduce the issue on OSX, but it does not reproduce on Linux (with very close docker versions that are compiled with the same golang version).

So I think it is not that the docker API server socket is broken, but there is an adopter layer specifically used in OSX's docker app, that is forwarding the http requests between the http client and server incorrectly.

At the very least, I think golang's http clients in go1.15 works well with former golang http servers, which is why I think it is not golang's "fault", although it is certainly a regression (breaking change), but for a good security reason.

I am also filing an issue to docker/for-mac: https://github.com/docker/for-mac/issues

@h8liu h8liu mentioned this issue Aug 21, 2020
Closed
2 tasks
@h8liu
Copy link
Author

h8liu commented Aug 21, 2020

imho, I don't think golang team needs to do anything. I am just leaving this issue open so that people can more easily discover this. the root cause of this issue very likely lies in docker's macos app.

@davecheney
Copy link
Contributor

@h8liu i would prefer if this issue were closed and discussion happened elsewhere until it was determined that there was a bug in the net/http package.

To be clear, if its a bug in the http package, it'll be fixed. If its related to how another project is using the http package, then that discussion belongs elsewhere.

@h8liu
Copy link
Author

h8liu commented Aug 21, 2020

okay, you are the boss. :)

@h8liu h8liu closed this as completed Aug 21, 2020
@stephenc stephenc mentioned this issue Sep 25, 2020
Closed
@l50 l50 mentioned this issue Sep 28, 2020
Closed
jgehrcke added a commit to opstrace/opstrace that referenced this issue Jan 15, 2021
@jgehrcke
ci: container: golang 1.14 instead of 1.15
2d6477f 
This runtime IIUC is what after all
runs the Go unit tests invoked by
makefile target `go-unit-tests`.

For #257 try out if "creating reaper failed"
goes away with Go 1.14.

https: //github.com/testcontainers/testcontainers-go/issues/232
https: //github.com/golang/go/issues/40735
Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>
jgehrcke added a commit to opstrace/opstrace that referenced this issue Jan 18, 2021
@jgehrcke
ci: container: golang 1.14 instead of 1.15
b427e45 
This runtime IIUC is what after all
runs the Go unit tests invoked by
makefile target `go-unit-tests`.

For #257 try out if "creating reaper failed"
goes away with Go 1.14.

https: //github.com/testcontainers/testcontainers-go/issues/232
https: //github.com/golang/go/issues/40735
Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>
jgehrcke added a commit to opstrace/opstrace that referenced this issue Jan 20, 2021
@jgehrcke
ci: container: golang 1.14 instead of 1.15
c2ab0ca 
This runtime IIUC is what after all
runs the Go unit tests invoked by
makefile target `go-unit-tests`.

For #257 try out if "creating reaper failed"
goes away with Go 1.14.

https: //github.com/testcontainers/testcontainers-go/issues/232
https: //github.com/golang/go/issues/40735
Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>
jgehrcke added a commit to opstrace/opstrace that referenced this issue Jan 20, 2021
@jgehrcke
ci: container: golang 1.14 instead of 1.15
104d060 
This runtime IIUC is what after all
runs the Go unit tests invoked by
makefile target `go-unit-tests`.

For #257 try out if "creating reaper failed"
goes away with Go 1.14.

https: //github.com/testcontainers/testcontainers-go/issues/232
https: //github.com/golang/go/issues/40735
Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>
jgehrcke added a commit to opstrace/opstrace that referenced this issue Jan 20, 2021
@jgehrcke
ci: container: golang 1.14 instead of 1.15
2f861ba 
This runtime IIUC is what after all
runs the Go unit tests invoked by
makefile target `go-unit-tests`.

For #257 try out if "creating reaper failed"
goes away with Go 1.14.

https: //github.com/testcontainers/testcontainers-go/issues/232
https: //github.com/golang/go/issues/40735
Signed-off-by: Dr. Jan-Philip Gehrcke <jp@opstrace.com>
@m4ns0ur m4ns0ur mentioned this issue Feb 3, 2021
Open
@golang golang locked and limited conversation to collaborators Aug 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@davecheney @sazzer @tankbusta @h8liu @gopherbot @rkuprov @iulian-gm