Skip to content

runtime: segmentation violation in net/http.ListenAndServe #5861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gopherbot opened this issue Jul 11, 2013 · 8 comments
Closed

runtime: segmentation violation in net/http.ListenAndServe #5861

gopherbot opened this issue Jul 11, 2013 · 8 comments
Labels
FrozenDueToAge
Milestone
Go1.2

Comments

@gopherbot
Copy link
Contributor

by zarcardfly:

Before filing a bug, please check whether it has been fixed since the
latest release. Search the issue tracker and check that you're running the
latest version of Go:

Run "go version" and compare against
http://golang.org/doc/devel/release.html  If a newer version of Go exists,
install it and retry what you did to reproduce the problem.

Thanks.

What steps will reproduce the problem?
If possible, include a link to a program on play.golang.org.
1. We encounter a problem in production environments that our go program crashed with 
stack trace.
2. The full stack trace is show below:
SIGSEGV: segmentation violation
PC=0x4121b3


goroutine 1 [IO wait]:
net.runtime_pollWait(0x7ff6b5a99f00, 0x72, 0x0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001cb080, 0xb, 0xc2000b93f0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001cb000, 0x734eb0, 0x0, 0xc2000b93f0, 0xb, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc20031df90, 0x4b15a6, 0x7ff6c7c7daf8, 0x4b15a6)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc20031df90, 0xc2000d8540, 0xc2005ed8b0, 0xc20027e510, 0x0,
...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc2001c6e10, 0xc2000e3740, 0xc20031df90, 0x0, 0x0, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc2001c6e10, 0xc2001c6e10, 0xc2001c48a0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1532 +0x9e
net/http.ListenAndServe(0xc2000ccf40, 0x13, 0xc2001c48a0, 0xc2001c4630, 0x0, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1597 +0x65
qbox.us/mc.Run(0xc2000ccf40, 0x13, 0xc2000b5c10, 0xd, 0xc2001c44e0, ...)
    /home/jenkins/qbox/nio/mc/src/qbox.us/mc/mc_svr.go:252 +0x721
main.main()
    /home/jenkins/qbox/service/service/src/qboxmc/qboxmc.go:67 +0x757

goroutine 2 [syscall]:

goroutine 7 [chan receive]:
qbox.us/servend/digest_auth.(*Service).timedTask(0xc2000d8120, 0x15d3ef79800)
    /home/jenkins/qbox/base/biz/src/qbox.us/servend/digest_auth/digest_auth_servend.go:230 +0x3c
created by qbox.us/servend/digest_auth.(*Service).Init
    /home/jenkins/qbox/base/biz/src/qbox.us/servend/digest_auth/digest_auth_servend.go:223 +0x1ea

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7ff6b5a99e60, 0x72, 0x0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001ce080, 0xb, 0xc2000b93f0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).accept(0xc2001ce000, 0x734eb0, 0x0, 0xc2000b93f0, 0xb, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/fd_unix.go:385 +0x2c1
net.(*TCPListener).AcceptTCP(0xc2001cd030, 0x18, 0xc20010c810, 0x4b1397)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/tcpsock_posix.go:229 +0x45
net.(*TCPListener).Accept(0xc2001cd030, 0x0, 0x0, 0x0, 0xc2000e83f0, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/tcpsock_posix.go:239 +0x25
net/http.(*Server).Serve(0xc2001cc000, 0xc2000e3740, 0xc2001cd030, 0x0, 0x0, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1542 +0x85
net/http.(*Server).ListenAndServe(0xc2001cc000, 0xc2001cc000, 0x0)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1532 +0x9e
net/http.ListenAndServe(0xc2000b5c10, 0xd, 0xc2001c48a0, 0xc2001c4810, 0x86b040, ...)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1597 +0x65
qbox.us/mc.func·001()
    /home/jenkins/qbox/nio/mc/src/qbox.us/mc/mc_svr.go:246 +0x64
created by qbox.us/mc.Run
    /home/jenkins/qbox/nio/mc/src/qbox.us/mc/mc_svr.go:249 +0x6e2

goroutine 180863207 [running]:
net/http.(*conn).serve(0xc20027e510)
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1017
created by net/http.(*Server).Serve
    /home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/net/http/server.go:1564 +0x266
rax     0x0
rbx     0x7ff6b74fe000
rcx     0x30040b8a5
rdx     0x7ff6b74fe000
rdi     0x7ff6b74fe000
rsi     0x0
rbp     0x7ff6b74fe000
rsp     0x7ff6aeffcc18
r8      0x0
r9      0x0
r10     0x0
r11     0x246
r12     0x0
r13     0x9
r14     0x2cde1fea
r15     0x69f4af21
rip     0x4121b3
rflags  0x10206
cs      0x33
fs      0x0
gs      0x0
3. The code is show below:
/home/jenkins/qbox/nio/mc/src/qbox.us/mc/mc_svr.go:252         return
http.ListenAndServe(addr, mux)
/home/jenkins/qbox/base/biz/src/qbox.us/servend/digest_auth/digest_auth_servend.go:230  
              <-time.After(time.Duration(duration))
/home/jenkins/qbox/nio/mc/src/qbox.us/mc/mc_svr.go:246                         if err :=
http.ListenAndServe(gateAddr, gateMux); err != nil {
4. In fact, I'm not sure if it was panic in net/http.ListenAndServe.

What is the expected output?
Never panic (and most of the time, it did).

What do you see instead?
The program panics accidental.

Which compiler are you using (5g, 6g, 8g, gccgo)?
6g

Which operating system are you using?
x86_64 GNU/Linux

Which version are you using?  (run 'go version')
go version go1.1 linux/amd64

Please provide any additional information below.
1. The program recieves about 120 Http Requests Per Second.
2. The program processes requests within 0.5ms at most time. It receives the request,
make another http request to another server, and do some light weight logic work.
3. It's the second time we encountered the problem(The stack trace of first time was
lost, sorry for that).
4. I wonder if there is a Tutorials for reading golang stack trace. I don't know how to
determine which goroutine cause the panic, the first one in stack trace or the 'running'
one. I hope someone can give me some tips if possible, thanks in advance.
5. I'd be very much interested to know why the panic happened.
6. If someone need more details, contact me with zarcardfly@gmail.com.
@rsc
Copy link
Contributor

rsc commented Jul 12, 2013

Comment 1:

If you still have the binary, could you please run gdb on it and execute
the command "x/10i 0x4121b3"? Maybe also "list *0x4121b3". That value is
the rip printed by the SIGSEGV. It is possible that goroutine 10 was the
one that faulted, since it's the only one listed as running, but then again
it might not have been.
I don't know of a tutorial for reading go stack traces. The running
goroutine is supposed to be first but was not at the time you copied tip
(nor in Go 1.1 I fear). I've fixed that at current tip, although current
tip has other problems and I wouldn't trust it for production use. (In fact
I wouldn't trust most tips. I'd stick with 1.1. But I appreciate the report
nonetheless.)
Russ

@gopherbot
Copy link
Contributor Author

Comment 2 by zarcardfly:

Thanks for your reply. I did as you recommended, the result is showed below:
(gdb) list *0x4121b3
0x4121b3 is in runtime.MHeap_Free
(/home/jenkins/jenkins_bin/go-tip2013-06-02/src/pkg/runtime/mheap.c:300).
295 void
296 runtime·MHeap_Free(MHeap *h, MSpan *s, int32 acct)
297 {
298  runtime·lock(h);
299  runtime·purgecachedstats(m->mcache);
300  mstats.heap_inuse -= s->npages<<PageShift;
301  if(acct) {
302  mstats.heap_alloc -= s->npages<<PageShift;
303  mstats.heap_objects--;
304  }
(gdb) x/10i 0x4121b3
   0x4121b3 <runtime.MHeap_Free+67>:  mov    0x18(%rcx),%rax
   0x4121b7 <runtime.MHeap_Free+71>:  shl    $0xc,%rax
   0x4121bb <runtime.MHeap_Free+75>:  sub    %rax,0x878c28
   0x4121c3 <runtime.MHeap_Free+83>:  cmpl   $0x0,0x28(%rsp)
   0x4121c8 <runtime.MHeap_Free+88>:  je     0x4121e2 <runtime.MHeap_Free+114>
   0x4121ca <runtime.MHeap_Free+90>:  mov    0x18(%rcx),%rax
   0x4121ce <runtime.MHeap_Free+94>:  shl    $0xc,%rax
   0x4121d2 <runtime.MHeap_Free+98>:  sub    %rax,0x878c10
   0x4121da <runtime.MHeap_Free+106>: decq   0x878c38
   0x4121e2 <runtime.MHeap_Free+114>: mov    0x18(%rsp),%rax
It seemed like the address store in 's' has been polluted.

@remyoudompheng
Copy link
Contributor

Comment 3:

Did you run your program under the race detector? Why are your directories named
go-tip2013-06-02 ?

@gopherbot
Copy link
Contributor Author

Comment 4 by zarcardfly:

1. I nerver run my program under the race detector. I have difficulty in doing it under
production environment. But will a data race cause segmentation violation as with the
situation above?
2. I named directories as go-tip2013-06-02 just because I downloaded the source code at
2013-06-02.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 5:

This looks like a garbage collector bug. Are you still seeing this?

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 6:

Labels changed: added go1.2.

@gopherbot
Copy link
Contributor Author

Comment 7 by zarcardfly:

It never happened again. It seems hard to reproduce the crash.  I'll reply the comment
if it happen again.

@rsc
Copy link
Contributor

rsc commented Aug 1, 2013

Comment 8:

Please open a new bug (and mention this one) if you see it again.

Status changed to Retracted.

@rsc rsc added this to the Go1.2 milestone Apr 14, 2015
@rsc rsc removed the go1.2 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.2
Development

No branches or pull requests
3 participants
@rsc @remyoudompheng @gopherbot