panic: runtime error: slice bounds out of range in bufio -> net/textproto #1725
Labels
Comments
Sorry, I'm not familiar with the 386 corruption reports. Can you point me to a specific issue number or email thread? I have also reproduced the issue on linux 386. Would a bisection be helpful or do you already have a general idea of what might have caused it? Thanks, Tarmigan |
OK I have instrumented src/pkg/bufio/bufio.go with the following (sorry cut-n-paste may
lose formatting):
diff --git a/src/pkg/bufio/bufio.go b/src/pkg/bufio/bufio.go
index eaae8bb..ad168f1 100644
--- a/src/pkg/bufio/bufio.go
+++ b/src/pkg/bufio/bufio.go
@@ -9,6 +9,7 @@ package bufio
import (
"bytes"
+ "fmt"
"io"
"os"
"strconv"
@@ -86,6 +87,8 @@ func NewReader(rd io.Reader) *Reader {
// fill reads a new chunk into the buffer.
func (b *Reader) fill() {
+ t := b.w
+
// Slide existing data to beginning.
if b.r > 0 {
copy(b.buf, b.buf[b.r:b.w])
@@ -99,6 +102,9 @@ func (b *Reader) fill() {
if e != nil {
b.err = e
}
+ if b.w > 4096 {
+ fmt.Println("WTF", len(b.buf), b.r, b.w, n, t)
+ }
}
// Peek returns the next n bytes without advancing the reader. The bytes stop
@@ -249,6 +255,11 @@ func (b *Reader) Buffered() int { return b.w - b.r }
// ReadSlice returns err != nil if and only if line does not end in delim.
func (b *Reader) ReadSlice(delim byte) (line []byte, err os.Error) {
// Look in buffer.
+ if b.r-1 > len(b.buf) || b.w-1 > len(b.buf) {
+ fmt.Println(len(b.buf), b.r, b.w)
+ return line, os.NewError("Bufio problem")
+ }
+
if i := bytes.IndexByte(b.buf[b.r:b.w], delim); i >= 0 {
line1 := b.buf[b.r : b.r+i+1]
b.r += i + 1
My program no longer panics, and when I run my program I see lines like
WTF 4096 1871 8192 4096 1871
4096 1871 8192
WTF 4096 2978 8192 4096 2978
4096 2978 8192
so something is going wrong in func (b *Reader) fill(). Looks like funny things happen
on b.rd.Read(buf[START:]) when START == len(buf).
But I'm also not sure how b.r != 0. The "slide" should be setting b.r = 0 and b.w -=
b.r. Is someone else messing with b.r behind our back?
[...]
OK, I've now changed to this patch (edited context slightly by hand to allow you to read
the flow of fill() better):
diff --git a/src/pkg/bufio/bufio.go b/src/pkg/bufio/bufio.go
index eaae8bb..06c4764 100644
--- a/src/pkg/bufio/bufio.go
+++ b/src/pkg/bufio/bufio.go
@@ -9,6 +9,7 @@ package bufio
import (
"bytes"
+ "fmt"
"io"
"os"
"strconv"
@@ -86,12 +87,16 @@ func NewReader(rd io.Reader) *Reader {
// fill reads a new chunk into the buffer.
func (b *Reader) fill() {
+ w := b.w
+ r := b.r
// Slide existing data to beginning.
if b.r > 0 {
copy(b.buf, b.buf[b.r:b.w])
b.w -= b.r
b.r = 0
}
+ w2 := b.w
+ r2 := b.r
// Read new data.
n, e := b.rd.Read(b.buf[b.w:])
b.w += n
if e != nil {
b.err = e
}
+ if b.w > 4096 {
+ fmt.Println("WTF", len(b.buf), cap(b.buf), n, b.r, b.w, r, w, r2, w2)
+ }
}
// Peek returns the next n bytes without advancing the reader. The bytes stop
@@ -249,6 +257,11 @@ func (b *Reader) Buffered() int { return b.w - b.r }
// ReadSlice returns err != nil if and only if line does not end in delim.
func (b *Reader) ReadSlice(delim byte) (line []byte, err os.Error) {
// Look in buffer.
+ if b.r-1 > len(b.buf) || b.w-1 > len(b.buf) {
+ fmt.Println(len(b.buf), b.r, b.w)
+ return line, os.NewError("Bufio problem")
+ }
+
if i := bytes.IndexByte(b.buf[b.r:b.w], delim); i >= 0 {
I get this as the output:
WTF 4096 4096 4096 1772 8192 1772 1772 0 0
4096 1772 8192
WTF 4096 4096 4096 1301 8192 1301 1301 0 0
4096 1301 8192
Mapping that last one by hand:
len(b.buf) == cap(b.buf) == n == 4096 which looks OK
r2 == w2 == 0 so the slide seems have happened OK
r == w == 1301 OK maybe it's a clue that they are the same.
b.r == 1301, but, but it was just 0!
b.w == 8192 but, but 0+4092 != 8192
I'm stumped, but maybe the disassembly has the answer.
Thanks,
Tarmigan
|
OK, here's the disassembly for fill() without my patches. I haven't looked at it yet. --- prog list "*Reader·fill" --- 0131 (bufio.go:88) TEXT *Reader·fill+0(SB),$120-8 0132 (bufio.go:88) MOVQ b+0(FP),AX 0133 (bufio.go:90) JMP ,135 0134 (bufio.go:90) JMP ,179 0135 (bufio.go:90) MOVL 32(AX),BX 0136 (bufio.go:90) CMPL BX,$0 0137 (bufio.go:90) JLE ,134 0138 (bufio.go:91) MOVQ (AX),BP 0139 (bufio.go:91) MOVQ BP,autotmp_0000+-40(SP) 0140 (bufio.go:91) MOVL 8(AX),BP 0141 (bufio.go:91) MOVL BP,autotmp_0000+-32(SP) 0142 (bufio.go:91) MOVL 12(AX),BP 0143 (bufio.go:91) MOVL BP,autotmp_0000+-28(SP) 0144 (bufio.go:91) MOVL 32(AX),BX 0145 (bufio.go:91) MOVLQSX BX,BX 0146 (bufio.go:91) MOVL 36(AX),BP 0147 (bufio.go:91) MOVLQSX BP,BP 0148 (bufio.go:91) MOVLQZX autotmp_0000+-28(SP),R8 0149 (bufio.go:91) CMPQ BP,R8 0150 (bufio.go:91) JLS ,152 0151 (bufio.go:91) CALL ,runtime.panicslice+0(SB) 0152 (bufio.go:91) CMPQ BX,BP 0153 (bufio.go:91) JHI ,151 0154 (bufio.go:91) SUBL BX,BP 0155 (bufio.go:91) MOVL BP,autotmp_0000+-32(SP) 0156 (bufio.go:91) MOVL autotmp_0000+-28(SP),BP 0157 (bufio.go:91) SUBL BX,BP 0158 (bufio.go:91) MOVL BP,autotmp_0000+-28(SP) 0159 (bufio.go:91) ADDQ autotmp_0000+-40(SP),BX 0160 (bufio.go:91) MOVQ BX,autotmp_0000+-40(SP) 0161 (bufio.go:91) LEAQ 16(SP),BX 0162 (bufio.go:91) MOVQ autotmp_0000+-40(SP),BP 0163 (bufio.go:91) MOVQ BP,(BX) 0164 (bufio.go:91) MOVL autotmp_0000+-32(SP),BP 0165 (bufio.go:91) MOVL BP,8(BX) 0166 (bufio.go:91) MOVL autotmp_0000+-28(SP),BP 0167 (bufio.go:91) MOVL BP,12(BX) 0168 (bufio.go:91) MOVQ AX,SI 0169 (bufio.go:91) LEAQ (SP),DI 0170 (bufio.go:91) MOVSQ , 0171 (bufio.go:91) MOVSQ , 0172 (bufio.go:91) MOVL $1,32(SP) 0173 (bufio.go:91) CALL ,runtime.slicecopy+0(SB) 0174 (bufio.go:91) MOVQ b+0(FP),AX 0175 (bufio.go:92) MOVL 32(AX),BP 0176 (bufio.go:92) SUBL BP,36(AX) 0177 (bufio.go:93) MOVL $0,32(AX) 0178 (bufio.go:90) JMP ,179 0179 (bufio.go:97) MOVQ AX,BX 0180 (bufio.go:97) ADDQ $16,BX 0181 (bufio.go:97) MOVQ (BX),BP 0182 (bufio.go:97) MOVQ BP,autotmp_0001+-56(SP) 0183 (bufio.go:97) MOVQ 8(BX),BP 0184 (bufio.go:97) MOVQ BP,autotmp_0001+-48(SP) 0185 (bufio.go:97) MOVQ (AX),BP 0186 (bufio.go:97) MOVQ BP,autotmp_0002+-72(SP) 0187 (bufio.go:97) MOVL 8(AX),BP 0188 (bufio.go:97) MOVL BP,autotmp_0002+-64(SP) 0189 (bufio.go:97) MOVL 12(AX),BP 0190 (bufio.go:97) MOVL BP,autotmp_0002+-60(SP) 0191 (bufio.go:97) MOVL 36(AX),BX 0192 (bufio.go:97) MOVLQSX BX,BX 0193 (bufio.go:97) MOVLQZX autotmp_0002+-64(SP),BP 0194 (bufio.go:97) CMPQ BX,BP 0195 (bufio.go:97) JLS ,197 0196 (bufio.go:97) CALL ,runtime.panicslice+0(SB) 0197 (bufio.go:97) MOVL autotmp_0002+-64(SP),BP 0198 (bufio.go:97) SUBL BX,BP 0199 (bufio.go:97) MOVL BP,autotmp_0002+-64(SP) 0200 (bufio.go:97) MOVL autotmp_0002+-60(SP),BP 0201 (bufio.go:97) SUBL BX,BP 0202 (bufio.go:97) MOVL BP,autotmp_0002+-60(SP) 0203 (bufio.go:97) ADDQ autotmp_0002+-72(SP),BX 0204 (bufio.go:97) MOVQ BX,autotmp_0002+-72(SP) 0205 (bufio.go:97) LEAQ 8(SP),BX 0206 (bufio.go:97) MOVQ autotmp_0002+-72(SP),BP 0207 (bufio.go:97) MOVQ BP,(BX) 0208 (bufio.go:97) MOVL autotmp_0002+-64(SP),BP 0209 (bufio.go:97) MOVL BP,8(BX) 0210 (bufio.go:97) MOVL autotmp_0002+-60(SP),BP 0211 (bufio.go:97) MOVL BP,12(BX) 0212 (bufio.go:97) LEAQ autotmp_0001+-56(SP),BX 0213 (bufio.go:97) MOVQ 8(BX),BP 0214 (bufio.go:97) MOVQ BP,(SP) 0215 (bufio.go:97) MOVQ (BX),BX 0216 (bufio.go:97) MOVQ 32(BX),BX 0217 (bufio.go:97) CALL ,BX 0218 (bufio.go:97) MOVQ b+0(FP),CX 0219 (bufio.go:97) MOVL 24(SP),AX 0220 (bufio.go:97) LEAQ 32(SP),BX 0221 (bufio.go:97) MOVQ (BX),BP 0222 (bufio.go:97) MOVQ BP,e+-24(SP) 0223 (bufio.go:97) MOVQ 8(BX),BP 0224 (bufio.go:97) MOVQ BP,e+-16(SP) 0225 (bufio.go:98) ADDL AX,36(CX) 0226 (bufio.go:99) JMP ,228 0227 (bufio.go:99) JMP ,238 0228 (bufio.go:99) LEAQ e+-24(SP),BX 0229 (bufio.go:99) CMPQ (BX),$0 0230 (bufio.go:99) JEQ ,227 0231 (bufio.go:100) MOVQ CX,BX 0232 (bufio.go:100) ADDQ $40,BX 0233 (bufio.go:100) MOVQ e+-24(SP),BP 0234 (bufio.go:100) MOVQ BP,(BX) 0235 (bufio.go:100) MOVQ e+-16(SP),BP 0236 (bufio.go:100) MOVQ BP,8(BX) 0237 (bufio.go:99) JMP ,238 0238 (bufio.go:102) RET , |
And so no one else has to dig to find the line numbers, here's a numbered version
(without my patches)
87 // fill reads a new chunk into the buffer.
88 func (b *Reader) fill() {
89 // Slide existing data to beginning.
90 if b.r > 0 {
91 copy(b.buf, b.buf[b.r:b.w])
92 b.w -= b.r
93 b.r = 0
94 }
95
96 // Read new data.
97 n, e := b.rd.Read(b.buf[b.w:])
98 b.w += n
99 if e != nil {
100 b.err = e
101 }
102 }
|
OK, here's the 386 disassembly: --- prog list "*Reader·fill" --- 0153 (bufio.go:88) TEXT *Reader·fill+0(SB),$200-4 0154 (bufio.go:88) MOVL b+0(FP),CX 0155 (bufio.go:90) JMP ,157 0156 (bufio.go:90) JMP ,218 0157 (bufio.go:90) MOVL CX,AX 0158 (bufio.go:90) ADDL $20,CX 0159 (bufio.go:90) MOVL (CX),BP 0160 (bufio.go:90) CMPL BP,$0 0161 (bufio.go:90) JLE ,156 0162 (bufio.go:91) MOVL AX,SI 0163 (bufio.go:91) LEAL autotmp_0032+-32(SP),DI 0164 (bufio.go:91) CLD , 0165 (bufio.go:91) MOVSL , 0166 (bufio.go:91) MOVSL , 0167 (bufio.go:91) MOVSL , 0168 (bufio.go:91) MOVL AX,DX 0169 (bufio.go:91) ADDL $20,DX 0170 (bufio.go:91) MOVL (DX),CX 0171 (bufio.go:91) MOVL AX,BP 0172 (bufio.go:91) ADDL $24,BP 0173 (bufio.go:91) MOVL (BP),DX 0174 (bufio.go:91) CMPL DX,autotmp_0032+-24(SP) 0175 (bufio.go:91) JLS ,177 0176 (bufio.go:91) CALL ,runtime.panicslice+0(SB) 0177 (bufio.go:91) CMPL CX,DX 0178 (bufio.go:91) JHI ,176 0179 (bufio.go:91) SUBL CX,DX 0180 (bufio.go:91) MOVL DX,autotmp_0032+-28(SP) 0181 (bufio.go:91) MOVL autotmp_0032+-24(SP),DX 0182 (bufio.go:91) SUBL CX,DX 0183 (bufio.go:91) MOVL DX,autotmp_0032+-24(SP) 0184 (bufio.go:91) ADDL autotmp_0032+-32(SP),CX 0185 (bufio.go:91) MOVL CX,autotmp_0032+-32(SP) 0186 (bufio.go:91) LEAL autotmp_0032+-32(SP),SI 0187 (bufio.go:91) LEAL 12(SP),DI 0188 (bufio.go:91) CLD , 0189 (bufio.go:91) MOVSL , 0190 (bufio.go:91) MOVSL , 0191 (bufio.go:91) MOVSL , 0192 (bufio.go:91) MOVL AX,SI 0193 (bufio.go:91) LEAL (SP),DI 0194 (bufio.go:91) CLD , 0195 (bufio.go:91) MOVSL , 0196 (bufio.go:91) MOVSL , 0197 (bufio.go:91) MOVSL , 0198 (bufio.go:91) MOVL $1,24(SP) 0199 (bufio.go:91) CALL ,runtime.slicecopy+0(SB) 0200 (bufio.go:91) MOVL b+0(FP),CX 0201 (bufio.go:92) MOVL CX,AX 0202 (bufio.go:92) ADDL $20,CX 0203 (bufio.go:92) MOVL (CX),BX 0204 (bufio.go:92) MOVL AX,autotmp_0043+-76(SP) 0205 (bufio.go:92) ADDL $24,autotmp_0043+-76(SP) 0206 (bufio.go:92) MOVL autotmp_0043+-76(SP),BP 0207 (bufio.go:92) MOVL (BP),SI 0208 (bufio.go:92) MOVL SI,autotmp_0045+-84(SP) 0209 (bufio.go:92) MOVL autotmp_0045+-84(SP),SI 0210 (bufio.go:92) SUBL BX,SI 0211 (bufio.go:92) MOVL SI,autotmp_0044+-80(SP) 0212 (bufio.go:92) MOVL autotmp_0044+-80(SP),SI 0213 (bufio.go:92) MOVL SI,(BP) 0214 (bufio.go:93) MOVL AX,CX 0215 (bufio.go:93) ADDL $20,CX 0216 (bufio.go:93) MOVL $0,(CX) 0217 (bufio.go:90) JMP ,218 0218 (bufio.go:97) MOVL AX,SI 0219 (bufio.go:97) ADDL $12,SI 0220 (bufio.go:97) LEAL autotmp_0047+-96(SP),DI 0221 (bufio.go:97) CLD , 0222 (bufio.go:97) MOVSL , 0223 (bufio.go:97) MOVSL , 0224 (bufio.go:97) MOVL AX,SI 0225 (bufio.go:97) LEAL autotmp_0050+-116(SP),DI 0226 (bufio.go:97) CLD , 0227 (bufio.go:97) MOVSL , 0228 (bufio.go:97) MOVSL , 0229 (bufio.go:97) MOVSL , 0230 (bufio.go:97) ADDL $24,AX 0231 (bufio.go:97) MOVL (AX),CX 0232 (bufio.go:97) CMPL CX,autotmp_0050+-112(SP) 0233 (bufio.go:97) JLS ,235 0234 (bufio.go:97) CALL ,runtime.panicslice+0(SB) 0235 (bufio.go:97) MOVL autotmp_0050+-112(SP),BX 0236 (bufio.go:97) SUBL CX,BX 0237 (bufio.go:97) MOVL BX,autotmp_0050+-112(SP) 0238 (bufio.go:97) MOVL autotmp_0050+-108(SP),BX 0239 (bufio.go:97) SUBL CX,BX 0240 (bufio.go:97) MOVL BX,autotmp_0050+-108(SP) 0241 (bufio.go:97) ADDL autotmp_0050+-116(SP),CX 0242 (bufio.go:97) MOVL CX,autotmp_0050+-116(SP) 0243 (bufio.go:97) LEAL autotmp_0050+-116(SP),SI 0244 (bufio.go:97) LEAL 4(SP),DI 0245 (bufio.go:97) CLD , 0246 (bufio.go:97) MOVSL , 0247 (bufio.go:97) MOVSL , 0248 (bufio.go:97) MOVSL , 0249 (bufio.go:97) LEAL autotmp_0047+-96(SP),BX 0250 (bufio.go:97) MOVL 4(BX),BP 0251 (bufio.go:97) MOVL BP,(SP) 0252 (bufio.go:97) MOVL (BX),BX 0253 (bufio.go:97) MOVL 20(BX),BX 0254 (bufio.go:97) CALL ,BX 0255 (bufio.go:97) MOVL b+0(FP),DX 0256 (bufio.go:97) MOVL 16(SP),BX 0257 (bufio.go:97) LEAL 20(SP),SI 0258 (bufio.go:97) LEAL e+-12(SP),DI 0259 (bufio.go:97) CLD , 0260 (bufio.go:97) MOVSL , 0261 (bufio.go:97) MOVSL , 0262 (bufio.go:98) MOVL DX,AX 0263 (bufio.go:98) ADDL $24,AX 0264 (bufio.go:98) MOVL (AX),BP 0265 (bufio.go:98) MOVL BP,autotmp_0061+-160(SP) 0266 (bufio.go:98) MOVL autotmp_0061+-160(SP),BP 0267 (bufio.go:98) ADDL BX,BP 0268 (bufio.go:98) MOVL BP,autotmp_0060+-156(SP) 0269 (bufio.go:98) MOVL autotmp_0060+-156(SP),BP 0270 (bufio.go:98) MOVL BP,(AX) 0271 (bufio.go:99) JMP ,273 0272 (bufio.go:99) JMP ,284 0273 (bufio.go:99) LEAL e+-12(SP),BX 0274 (bufio.go:99) CMPL (BX),$0 0275 (bufio.go:99) JEQ ,272 0276 (bufio.go:100) MOVL DX,DI 0277 (bufio.go:100) ADDL $28,DI 0278 (bufio.go:100) LEAL e+-12(SP),SI 0279 (bufio.go:100) MOVL DI,AX 0280 (bufio.go:100) CLD , 0281 (bufio.go:100) MOVSL , 0282 (bufio.go:100) MOVSL , 0283 (bufio.go:99) JMP ,284 0284 (bufio.go:102) RET , |
The most likely explanation for what you're seeing is that two goroutines are calling b.Read for the same *bufio.Reader b simultaneously. Are you using multiple goroutines? Are they reading from a single *bufio.Reader? Owner changed to @rsc. Status changed to WaitingForReply. |
Yes, there are 20 goroutines which go through the ~3000 urls. No, they should not be sharing *bufio.Readers. Each goroutine does its own http.Get() and then keeps some slices of the data Read() from response.Body and passes a struct with those slices on a channel. By the time the data is passed on the channel the response.Body should be closed. I did try to copy() the slices to reduce fragmentation and maybe fix the problem, but that did not seem to make a difference. But, note that I am not the direct creator (or user) of the *bufio.Reader. That is something in the http code. See the first backtrace of the first goroutine in the panic. I do wonder if there is anything in http.*persistConn·readLoop that might share a pool or something? The failures did seem to often come up 1500-2500 urls in. At the time I reported it, I also tried to run with GOGC=off and it ran out of memory after about 1600 urls if I remember right. On the other hand, I have reproduced it on amd64 which I believe has a more reliable GC than i386. Also I did see it occur once after about 100 or 200 urls. I can also reproduce issue #1650 with godoc, so if you find a fix for 1650, I can see if it fixes this issue. |
re issue #1650, try hg clpatch 4446062 |
As of right now, 4446062 does not fix this issue nor does it change the backtrace (still
looks almost identical to the one at the top of this issue). After 600 urls, it still
panicked at the same place.
I think I applied 4446062 correctly because the godoc panic is now
2011/04/25 23:59:18 [::1]:55170 /pkg/reflect/
throw: out of memory
runtime.throw+0x40 /Users/tarm/devel/go.hg/src/pkg/runtime/runtime.c:102
runtime.throw(0x2d1335, 0xe)
runtime.mallocgc+0x2af /Users/tarm/devel/go.hg/src/pkg/runtime/malloc.c:60
runtime.mallocgc(0xdc50, 0x1, 0x1, 0x1, 0x308524c0, ...)
makeslice1+0x6e /Users/tarm/devel/go.hg/src/pkg/runtime/slice.c:44
makeslice1(0x12b93c, 0xdc50, 0xdc50, 0x76cfd0, 0x0, ...)
runtime.makeslice+0x92 /Users/tarm/devel/go.hg/src/pkg/runtime/slice.c:24
runtime.makeslice(0x12b93c, 0xdc50, 0x0, 0xdc50, 0x0, ...)
----- stack segment boundary -----
bytes.*Buffer·grow+0x178 /Users/tarm/devel/go.hg/src/pkg/bytes/buffer.go:85
bytes.*Buffer·grow(0x1150c360, 0x28, 0x0, 0x1)
etc
|
If the last time you ran this was months ago, that would probably be before the HTTP/1.1
persist connection support in the http.Client/http.Transport.
Could you try disabling that to see what happens?
tr := &http.Transport{DisableKeepAlives: true}
client := &http.Client{Transport: tr}
client.Get(...)
|
Here is a complete backtrace built from hg de525810c69b (includes the fix for 1650):
("Comment is too long" so had to attach)
Attachments:
|
Actually now that it's running to completion again, I see that it's actually 4100 Get()s. They are all http to a single host. (I'm embarrassed to say, but it's an obnoxious screen scraper, though it does obey robots.txt for the site.) Similar to the lottery number fetching program discussed on golang-nuts, in a loop, I Read() from the response.Body into a []byte buffer and run a regex on that buffer. If the regex matches, I break out of the loop and close the body before EOF. So I probably Close() the body before reading the whole response about 90% of the time to save time and bandwidth. |
Relevant details from looking at the code & server in question: URL is (typically?) a redirect. First response is an HTTP/1.1 301 with a Location & Content-Length + body. Second response is an HTTP/1.1 200 chunked response with a *very short* keep-alive timeout. Something like one second, +/- couple hundred milliseconds. |
New report: this only happens when both the Transport's persistent connections + compression are enabled. Disabling either fixes it. Labels changed: added pkg-http. Owner changed to @bradfitz. |
I can now reliably reproduce this with OP's test program (mailed privately) but I haven't yet been able to reproduce it in a small, self-contained unit test. This fixes it: http://golang.org/cl/4442086 ... but despite all the new tests in there, none actually trigger the bug if I remove the transport.go part of the patch. So I'm reluctant to send this out for review yet. Back to working on a test... |
|
This issue was closed by revision 6e71e1c. Status changed to Fixed. |
Ho hum, I think it's back.
revision 22fa8d29da35 did seem to fix the issue with my original program.
However, a one line changed to my program (change only the URL fetched!) causes this
panic right away.
Updating to the lastest tip b7162c26f2a2 ("http: fix typo in URL.String doc comment.")
does not resolve the issue.
The problem occurs much faster with my modified test program. I hope this will ease
diagnosis and the creation of a test case.
Panic still looks similar to above.
I will email the modified program to Brad privately.
|
Here is a much smaller test program that GET's from a different site than my original program. The panic is a bit of a mess because of all the different goroutines, but having more goroutines seems to make it panic faster. Also changing 'size' to 1<<14 makes the problem go away. Attachments:
|
Patch: http://golang.org/cl/4523058/ With that, main.go works reliably. Thanks for the great test! |
|
This issue was closed by revision fa23a70. Status changed to Fixed. |
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
by tarmigan:
I have a go program that does about 3000 http.Get() calls and parses the results. This program used to work (last time I rebuilt it was few months ago), but after updating to hg 14e85394f65e ("http: close underlying gzip Reader too") it panics. Looking at the backtrace, I don't think my program is at fault. Looks like bufio got into an inconsistent state somehow. I have reproduced it twice now, so I may be able to add debugging prints if needed. GOOS=darwin GOARCH=386 Thanks, Tarmigan panic: runtime error: slice bounds out of range runtime.panic+0x92 /Users/tarm/devel/go/src/pkg/runtime/proc.c:1034 runtime.panic(0xf1a8c, 0x10b19018) runtime.panicstring+0x8f /Users/tarm/devel/go/src/pkg/runtime/runtime.c:116 runtime.panicstring(0x1d5879, 0x0) runtime.panicslice+0x24 /Users/tarm/devel/go/src/pkg/runtime/runtime.c:79 runtime.panicslice() bufio.*Reader·ReadSlice+0x6b /Users/tarm/devel/go/src/pkg/bufio/bufio.go:252 bufio.*Reader·ReadSlice(0x10906f90, 0xa, 0x0, 0x0, 0x0, ...) bufio.*Reader·ReadBytes+0x84 /Users/tarm/devel/go/src/pkg/bufio/bufio.go:327 bufio.*Reader·ReadBytes(0x10906f90, 0xa, 0x0, 0x0, 0x0, ...) net/textproto.*Reader·ReadLineBytes+0x39 /Users/tarm/devel/go/src/pkg/net/textproto/reader.go:43 net/textproto.*Reader·ReadLineBytes(0x10b19010, 0x0, 0x0) net/textproto.*Reader·ReadLine+0x26 /Users/tarm/devel/go/src/pkg/net/textproto/reader.go:36 net/textproto.*Reader·ReadLine(0x10b19010, 0x3, 0x105a98) http.ReadResponse+0x96 /Users/tarm/devel/go/src/pkg/http/response.go:88 http.ReadResponse(0x10906f90, 0x105a98, 0x3, 0x10855300, 0x0, ...) http.readResponseWithEOFSignal+0x3b /Users/tarm/devel/go/src/pkg/http/transport.go:577 http.readResponseWithEOFSignal(0x10906f90, 0x105a98, 0x3, 0x0, 0x0, ...) http.*ClientConn·Read+0x2fd /Users/tarm/devel/go/src/pkg/http/persist.go:368 http.*ClientConn·Read(0x10a7d930, 0x10d04e00, 0x0, 0x0, 0x0, ...) http.*persistConn·readLoop+0x246 /Users/tarm/devel/go/src/pkg/http/transport.go:450 http.*persistConn·readLoop(0x108fbd00, 0xea23) runtime.goexit /Users/tarm/devel/go/src/pkg/runtime/proc.c:178 runtime.goexit() ----- goroutine created by ----- http.*Transport·getConn+0x4db /Users/tarm/devel/go/src/pkg/http/transport.go:289 goroutine 5470 [4]: runtime.gosched+0x61 /Users/tarm/devel/go/src/pkg/runtime/proc.c:600 runtime.gosched() runtime.chanrecv+0x3af /Users/tarm/devel/go/src/pkg/runtime/chan.c:329 runtime.chanrecv(0x108c9b40, 0x67ee38, 0x0, 0x0, 0x67ee40, ...) runtime.chanrecv1+0x3f /Users/tarm/devel/go/src/pkg/runtime/chan.c:424 runtime.chanrecv1(0x108c9b40, 0x108551e0) http.*persistConn·readLoop+0x437 /Users/tarm/devel/go/src/pkg/http/transport.go:481 http.*persistConn·readLoop(0x10987200, 0xea23) runtime.goexit /Users/tarm/devel/go/src/pkg/runtime/proc.c:178 runtime.goexit() ----- goroutine created by ----- http.*Transport·getConn+0x4db /Users/tarm/devel/go/src/pkg/http/transport.go:289 goroutine 5469 [4]: runtime.gosched+0x61 /Users/tarm/devel/go/src/pkg/runtime/proc.c:600 runtime.gosched() runtime.chanrecv+0x3af /Users/tarm/devel/go/src/pkg/runtime/chan.c:329 runtime.chanrecv(0x108c96c0, 0x67de38, 0x0, 0x0, 0x67de40, ...) runtime.chanrecv1+0x3f /Users/tarm/devel/go/src/pkg/runtime/chan.c:424 runtime.chanrecv1(0x108c96c0, 0x10855120) http.*persistConn·readLoop+0x437 /Users/tarm/devel/go/src/pkg/http/transport.go:481 http.*persistConn·readLoop(0x108c9800, 0xea23) runtime.goexit /Users/tarm/devel/go/src/pkg/runtime/proc.c:178 runtime.goexit() ----- goroutine created by ----- http.*Transport·getConn+0x4db /Users/tarm/devel/go/src/pkg/http/transport.go:289 goroutine 5420 [1]: runtime.gosched+0x61 /Users/tarm/devel/go/src/pkg/runtime/proc.c:600 runtime.gosched() runtime.exitsyscall+0x64 /Users/tarm/devel/go/src/pkg/runtime/proc.c:665 runtime.exitsyscall() syscall.Syscall6+0x4b /Users/tarm/devel/go/src/pkg/syscall/asm_darwin_386.s:61 syscall.Syscall6() syscall.kevent+0x58 /Users/tarm/devel/go/src/pkg/syscall/zsyscall_darwin_386.go:158 syscall.kevent(0x11, 0x1082cfd8, 0x1, 0x1082cfd8, 0x1, ...) syscall.Kevent+0x84 /Users/tarm/devel/go/src/pkg/syscall/syscall_bsd.go:441 syscall.Kevent(0x11, 0x1082cfd8, 0x1, 0x1, 0x1082cfd8, ...) net.*pollster·AddFD+0x133 /Users/tarm/devel/go/src/pkg/net/fd_darwin.go:54 net.*pollster·AddFD(0x1082cf00, 0x29, 0x72, 0x0, 0x8, ...) net.*pollServer·AddFD+0x1b0 /Users/tarm/devel/go/src/pkg/net/fd.go:125 net.*pollServer·AddFD(0x108a1f90, 0x10be6e00, 0x72, 0x4) net.*pollServer·WaitRead+0x36 /Users/tarm/devel/go/src/pkg/net/fd.go:254 net.*pollServer·WaitRead(0x108a1f90, 0x10be6e00, 0x1000) net.*netFD·Read+0x1be /Users/tarm/devel/go/src/pkg/net/fd.go:380 net.*netFD·Read(0x10be6e00, 0x10a76000, 0x1000, 0x1000, 0x0, ...) net.*TCPConn·Read+0x6d /Users/tarm/devel/go/src/pkg/net/tcpsock.go:94 net.*TCPConn·Read(0x109e1bf8, 0x10a76000, 0x1000, 0x1000, 0x0, ...) bufio.*Reader·fill+0x14f /Users/tarm/devel/go/src/pkg/bufio/bufio.go:97 bufio.*Reader·fill(0x10ba1900, 0x20) bufio.*Reader·Peek+0xb5 /Users/tarm/devel/go/src/pkg/bufio/bufio.go:116 bufio.*Reader·Peek(0x10ba1900, 0x1, 0x0, 0x0) http.*persistConn·readLoop+0x7f /Users/tarm/devel/go/src/pkg/http/transport.go:433 http.*persistConn·readLoop(0x10ab6600, 0xea23) runtime.goexit /Users/tarm/devel/go/src/pkg/runtime/proc.c:178 runtime.goexit() ----- goroutine created by ----- http.*Transport·getConn+0x4db /Users/tarm/devel/go/src/pkg/http/transport.go:289 goroutine 5286 [4]: runtime.gosched+0x61 /Users/tarm/devel/go/src/pkg/runtime/proc.c:600 runtime.gosched() runtime.chanrecv+0x14d /Users/tarm/devel/go/src/pkg/runtime/chan.c:361 runtime.chanrecv(0x109873c0, 0x673b78, 0x0, 0x0, 0x1000, ...) runtime.chanrecv1+0x3f /Users/tarm/devel/go/src/pkg/runtime/chan.c:424 runtime.chanrecv1(0x109873c0, 0x10f37c00) net.*pollServer·WaitRead+0x47 /Users/tarm/devel/go/src/pkg/net/fd.go:255 net.*pollServer·WaitRead(0x108a1f90, 0x10f37c00, 0x1000) net.*netFD·Read+0x1be /Users/tarm/devel/go/src/pkg/net/fd.go:380 net.*netFD·Read(0x10f37c00, 0x111ba000, 0x1000, 0x1000, 0x0, ...) net.*TCPConn·Read+0x6d /Users/tarm/devel/go/src/pkg/net/tcpsock.go:94 net.*TCPConn·Read(0x109e1858, 0x111ba000, 0x1000, 0x1000, 0x0, ...) bufio.*Reader·fill+0x14f /Users/tarm/devel/go/src/pkg/bufio/bufio.go:97 bufio.*Reader·fill(0x10f2a8d0, 0x20) bufio.*Reader·Peek+0xb5 /Users/tarm/devel/go/src/pkg/bufio/bufio.go:116 bufio.*Reader·Peek(0x10f2a8d0, 0x1, 0x0, 0x0) http.*persistConn·readLoop+0x7f /Users/tarm/devel/go/src/pkg/http/transport.go:433 http.*persistConn·readLoop(0x10c52840, 0xea23) runtime.goexit /Users/tarm/devel/go/src/pkg/runtime/proc.c:178 runtime.goexit() ----- goroutine created by ----- [ more backtrace. Can include if needed]The text was updated successfully, but these errors were encountered: