Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) #14001

Closed
clnperez opened this issue Jan 18, 2016 · 6 comments
Closed

net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) #14001

clnperez opened this issue Jan 18, 2016 · 6 comments
Labels
FrozenDueToAge
Milestone
Go1.6

Comments

@clnperez
Copy link 

time="2016-01-18T16:31:37.532208479Z" level=debug msg="POST /v1.22/build?buildargs=%7B%7D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&shmsize=0&t=testbuildaddnotfound&ulimits=null" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x7a9ad1]

goroutine 2807 [running]:
net/http.startCloseNotifyBackgroundRead(0xc820e1c080, 0xc8217cc180)
        /usr/local/go/src/net/http/server.go:1560 +0x21
net/http.(*response).CloseNotify(0xc820fe8600, 0x0)
        /usr/local/go/src/net/http/server.go:1553 +0x2a5
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild.func4(0xc82150e3c0, 0x7f50a0304f58, 0xc820fe8600, 0xc820b195f0)
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:247 +0x39
created by github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:251 +0x14ca

I've traced this error back to commit 99fb191

The docker image itself is running Ubuntu trusty:
docker run -it --privileged docker-go1.6beta2 uname -a
Linux 4b2dfe4ce10b 4.2.8-200.fc22.x86_64 #1 SMP Tue Dec 15 16:50:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here's how to get the docker env set up to recreate (all in docker images).
0) Install docker

  1. git clone https://github.com/docker/docker.git && cd docker
  2. edit Dockerfile:
    -ENV GO_VERSION 1.5.3
    +ENV GO_VERSION 1.6beta2
  3. docker build -t docker .
  4. docker run -it --privileged -v pwd:/go/src/github.com/docker/docker/ ./hack/make.sh binary test-integration-cli

You'll see all the docker tests start to fail with "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

The -v flag will volume-mount your local directory into the container, so then you can look at logs locally after the container exits. Check less bundles/latest/test-integration-cli/docker.log for the stack-trace I pasted above.

Because of how the tests are run it's not trivial to tell which of the tests causes the panic that kills the docker daemon, but I'm working on that.
@ianlancetaylor ianlancetaylor changed the title "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) Jan 18, 2016
@ianlancetaylor ianlancetaylor added this to the Go1.6 milestone Jan 18, 2016
@ianlancetaylor
Copy link
Contributor

CC @bradfitz

@laboger
Copy link
Contributor

laboger commented Jan 18, 2016

fails on x86 and ppc64le

@bradfitz
Copy link
Contributor

Docker wasn't using CloseNotify from the right goroutine. Patch for Docker:

diff --git a/api/server/server.go b/api/server/server.go
index a1c57a8..e6a2957 100644
--- a/api/server/server.go
+++ b/api/server/server.go
@@ -1091,10 +1091,11 @@ func postBuild(eng *engine.Engine, version version.Version, w http.ResponseWrite
        if closeNotifier, ok := w.(http.CloseNotifier); ok {
                finished := make(chan struct{})
                defer close(finished)
+               clientGone := closeNotifier.CloseNotify()
                go func() {
                        select {
                        case <-finished:
-                       case <-closeNotifier.CloseNotify():
+                       case <-clientGone:
                                log.Infof("Client disconnected, cancelling job: %v", job)
                                job.Cancel()
                        }

If it worked before, it was just luck. Or it was racy. We've improved the CloseNotifier docs a bunch in this cycle.

But I'll make Go panic with a more useful error message so you and others can diagnose this earlier and more easily in the future.

@gopherbot
Copy link

CL https://golang.org/cl/18708 mentions this issue.

@clnperez
Copy link
Author

Thanks. It looks like that's the only spot in the docker code that's an issue, so I can submit a PR to them for this.

clnperez added a commit to clnperez/moby that referenced this issue Jan 18, 2016
@clnperez
Move closeNotify to fix panic with newer golang
2df5daf 
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
@clnperez clnperez mentioned this issue Jan 18, 2016
Merged
@thaJeztah thaJeztah mentioned this issue Jan 19, 2016
Closed
@cpuguy83
Copy link

<3 Thanks. Probably haven't seen it before b/c we haven't done a release with GOMAXPROCS set higher than 1.

tiborvass pushed a commit to tiborvass/docker that referenced this issue Jan 26, 2016
@clnperez
Move closeNotify to fix panic with newer golang
35ca330 
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
aditirajagopal pushed a commit to aditirajagopal/docker that referenced this issue Feb 8, 2016
@clnperez
Move closeNotify to fix panic with newer golang
523156e 
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
@rwcarlsen rwcarlsen mentioned this issue Feb 8, 2016
Merged
@golang golang locked and limited conversation to collaborators Jan 18, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.6
Development

No branches or pull requests
6 participants
@bradfitz @cpuguy83 @ianlancetaylor @clnperez @gopherbot @laboger