x/net/http2: stream flow control window is 0, body can't send #59969
Comments
|
we don't support 1.10 anymore |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
NA
What operating system and processor architecture are you using (
go env)?rehl7 x86
go envOutputWhat did you do?
It was found that the K8s APIServer was blocked, which made it unable to respond to some HTTP2 request. After analyzing it through delve, it was found that the serverConn.flow.n = 0，We found that the header of this response has been sent, but the body has not been sent because the window is 0, causing the client to be hung
relevant stack
stream value
dlv) p sc *github.com/openshift/origin/vendor/golang.org/x/net/http2.serverConn { srv: *github.com/openshift/origin/vendor/golang.org/x/net/http2.Server { MaxHandlers: 0, MaxConcurrentStreams: 250, MaxReadFrameSize: 262144, PermitProhibitedCipherSuites: false, IdleTimeout: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector.ResourceResyncTime (0), MaxUploadBufferPerConnection: 65536000, MaxUploadBufferPerStream: 262144, NewWriteScheduler: nil, state: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.serverInternalState")(0xc430c0c000),}, hs: *net/http.Server { Addr: "0.0.0.0:8419", Handler: net/http.Handler(*github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.APIServerHandler) ..., TLSConfig: ("crypto/tls.Config")(0xc430c08000), ReadTimeout: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector.ResourceResyncTime (0), ReadHeaderTimeout: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector.ResourceResyncTime (0), WriteTimeout: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector.ResourceResyncTime (0), IdleTimeout: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector.ResourceResyncTime (0), MaxHeaderBytes: 1048576, TLSNextProto: map[string]func(*net/http.Server, *crypto/tls.Conn, net/http.Handler) [...], ConnState: nil, ErrorLog: *log.Logger nil, disableKeepAlives: 0, inShutdown: 0, nextProtoOnce: (*sync.Once)(0xc42bd7a070), nextProtoErr: error nil, mu: (*sync.Mutex)(0xc42bd7a090), listeners: map[net.Listener]struct {} [...], activeConn: map[*net/http.conn]struct {} [...], doneChan: chan struct {} {}, onShutdown: []func() len: 1, cap: 1, [github.com/openshift/origin/vendor/golang.org/x/net/http2.(*serverInternalState).(github.com/openshift/origin/vendor/golang.org/x/net/http2.startGracefulShutdown)-fm],}, conn: net.Conn(*crypto/tls.Conn) *{ conn: net.Conn(*net.TCPConn) ..., isClient: false, handshakeMutex: (*sync.Mutex)(0xc42dc06714), handshakeCond: *sync.Cond nil, handshakeErr: error nil, vers: 771, haveVers: true, config: ("crypto/tls.Config")(0xc430c08000), handshakeComplete: true, handshakes: 1, didResume: false, cipherSuite: 52392, ocspResponse: []uint8 len: 0, cap: 0, nil, scts: [][]uint8 len: 0, cap: 0, nil, peerCertificates: []crypto/x509.Certificate len: 1, cap: 1, [ ("crypto/x509.Certificate")(0xc434c40000), ], verifiedChains: [][]crypto/x509.Certificate len: 0, cap: 0, nil, serverName: "masterlb.rsxn02.ccnp.cib", secureRenegotiation: false, clientFinishedIsFirst: true, closeNotifyErr: error nil, closeNotifySent: false, clientFinished: [12]uint8 [221,30,142,255,162,136,59,69,220,252,131,26], serverFinished: [12]uint8 [0,0,0,0,0,0,0,0,0,0,0,0], clientProtocol: "h2", clientProtocolFallback: false, in: ("crypto/tls.halfConn")(0xc42dc06820), out: ("crypto/tls.halfConn")(0xc42dc068d0), rawInput: ("crypto/tls.block")(0xc4289442d0), input: *crypto/tls.block nil, hand: (*bytes.Buffer)(0xc42dc06990), buffering: false, sendBuf: []uint8 len: 0, cap: 0, nil, bytesSent: 3923644218, packetsSent: 47, warnCount: 0, activeCall: 0, tmp: [16]uint8 [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],}, bw: *github.com/openshift/origin/vendor/golang.org/x/net/http2.bufferedWriter { w: io.Writer(*crypto/tls.Conn) ..., bw: bufio.Writer nil,}, handler: net/http.Handler(net/http.initNPNRequest) { c: ("crypto/tls.Conn")(0xc42dc06700), h: ("net/http.serverHandler")(0xc435a26b48),}, baseCtx: github.com/openshift/origin/vendor/golang.org/x/net/http2.contextContext(*context.valueCtx) *{ Context: context.Context(*context.valueCtx) ..., key: interface {}(*net/http.contextKey) ..., val: interface {}(*net/http.Server) ...,}, framer: *github.com/openshift/origin/vendor/golang.org/x/net/http2.Framer { r: io.Reader(*crypto/tls.Conn) ..., lastFrame: github.com/openshift/origin/vendor/golang.org/x/net/http2.Frame(*github.com/openshift/origin/vendor/golang.org/x/net/http2.HeadersFrame) ..., errDetail: error nil, lastHeaderStream: 0, maxReadSize: 262144, headerBuf: [9]uint8 [0,0,100,1,5,0,0,74,105], getReadBuf: github.com/openshift/origin/vendor/golang.org/x/net/http2.NewFramer.func1, readBuf: []uint8 len: 831, cap: 831, [65,149,164,104,73,108,162,53,236,71,154,128,37,200,74,171,92,134,142,227,205,5,255,130,69,197,96,117,153,12,84,84,248,61,157,87,61,101,169,19,154,84,174,99,177,220,44,78,116,37,22,199,81,82,143,242,194,160,246,216,66,241,45,136,...+767 more], maxWriteSize: 0, w: io.Writer(*github.com/openshift/origin/vendor/golang.org/x/net/http2.bufferedWriter) ..., wbuf: []uint8 len: 37, cap: 18432, [0,0,28,1,4,0,0,74,105,136,221,251,97,151,221,109,95,74,9,149,33,174,197,4,1,50,160,28,184,211,151,25,109,76,90,55,255], AllowIllegalWrites: false, AllowIllegalReads: false, ReadMetaHeaders: ("github.com/openshift/origin/vendor/golang.org/x/net/http2/hpack.Decoder")(0xc43595d0a0), MaxHeaderListSize: 1048896, logReads: false, logWrites: false, debugFramer: *github.com/openshift/origin/vendor/golang.org/x/net/http2.Framer nil, debugFramerBuf: *bytes.Buffer nil, debugReadLoggerf: log.Printf, debugWriteLoggerf: log.Printf, frameCache: *github.com/openshift/origin/vendor/golang.org/x/net/http2.frameCache nil,}, doneServing: chan struct {} { qcount: 0, dataqsiz: 0, buf: *[0]struct struct {} [], elemsize: 0, closed: 0, elemtype: *runtime._type { size: 0, ptrdata: 0, hash: 670477339, tflag: tflagExtraStar (2), align: 1, fieldalign: 1, kind: 153, alg: *(*runtime.typeAlg)(0xa37ef70), gcdata: *1, str: 212108, ptrToThis: 3762208,}, sendx: 0, recvx: 0, recvq: waitq { first: *(*sudog)(0xc46b526de0), last: *(*sudog)(0xc5137b2420),}, sendq: waitq { first: *sudog nil, last: *sudog nil,}, lock: runtime.mutex {key: 0},}, readFrameCh: chan github.com/openshift/origin/vendor/golang.org/x/net/http2.readFrameResult { qcount: 0, dataqsiz: 0, buf: [0]struct github.com/openshift/origin/vendor/golang.org/x/net/http2.readFrameResult [], elemsize: 40, closed: 0, elemtype: runtime._type { size: 40, ptrdata: 40, hash: 512634742, tflag: tflagUncommon|tflagExtraStar|tflagNamed (7), align: 8, fieldalign: 8, kind: 25, alg: (runtime.typeAlg)(0xa37ef60), gcdata: 31, str: 925566, ptrToThis: 3318560,}, sendx: 0, recvx: 0, recvq: waitq { first: ("sudog")(0xc4740ca720), last: ("sudog")(0xc4740ca720),}, sendq: waitq { first: sudog nil, last: sudog nil,}, lock: runtime.mutex {key: 0},}, wantWriteFrameCh: chan github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest { qcount: 0, dataqsiz: 8, buf: [8]struct github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest [ ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a86100), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a86120), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a86140), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a86160), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a86180), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a861a0), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a861c0), ("github.com/openshift/origin/vendor/golang.org/x/net/http2.FrameWriteRequest")(0xc435a861e0), ], elemsize: 32, closed: 0, elemtype: *runtime._type { size: 32, ptrdata: 32, hash: 2188651780, tflag: tflagUncommon|tflagExtraStar|tflagNamed (7), align: 8, fieldalign: 8, kind: 25, alg: *(*runtime.typeAlg)(0x40553b0), gcdata: *15, str: 1052780, ptrToThis: 11486080,}, sendx: 1, recvx: 1, recvq: waitq { first: ("sudog")(0xc50b2c2c60), last: ("sudog")(0xc50b2c2c60),}, sendq: waitq { first: *sudog nil, last: *sudog nil,}, lock: runtime.mutex {key: 0},}, wroteFrameCh: chan github.com/openshift/origin/vendor/golang.org/x/net/http2.frameWriteResult { qcount: 0, dataqsiz: 1, buf: [1]struct github.com/openshift/origin/vendor/golang.org/x/net/http2.frameWriteResult [ ("github.com/openshift/origin/vendor/golang.org/x/net/http2.frameWriteResult")(0xc427cd0240), ], elemsize: 48, closed: 0, elemtype: *runtime._type { size: 48, ptrdata: 48, hash: 3195946954, tflag: tflagUncommon|tflagExtraStar|tflagNamed (7), align: 8, fieldalign: 8, kind: 25, alg: *(*runtime.typeAlg)(0x40554b0), gcdata: *63, str: 987529, ptrToThis: 3318304,}, sendx: 0, recvx: 0, recvq: waitq { first: ("sudog")(0xc42ec1f680), last: ("sudog")(0xc42ec1f680),}, sendq: waitq { first: *sudog nil, last: *sudog nil,}, lock: runtime.mutex {key: 0},}, bodyReadCh: chan github.com/openshift/origin/vendor/golang.org/x/net/http2.bodyReadMsg { qcount: 0, dataqsiz: 0, buf: *[0]struct github.com/openshift/origin/vendor/golang.org/x/net/http2.bodyReadMsg [], elemsize: 16, closed: 0, elemtype: *runtime._type { size: 16, ptrdata: 8, hash: 555585178, tflag: tflagUncommon|tflagExtraStar|tflagNamed (7), align: 8, fieldalign: 8, kind: 25, alg: *(*runtime.typeAlg)(0xa37efc0), gcdata: *1, str: 686786, ptrToThis: 3317920,}, sendx: 0, recvx: 0, recvq: waitq { first: ("sudog")(0xc43e388c00), last: ("sudog")(0xc43e388c00),}, sendq: waitq { first: *sudog nil, last: *sudog nil,}, lock: runtime.mutex {key: 0},}, serveMsgCh: chan interface {} { qcount: 0, dataqsiz: 8, buf: *[8]interface {} [ nil, nil, nil, nil, nil, nil, nil, nil, ], elemsize: 16, closed: 0, elemtype: *runtime._type { size: 16, ptrdata: 16, hash: 413161447, tflag: tflagExtraStar (2), align: 8, fieldalign: 8, kind: 20, alg: *(*runtime.typeAlg)(0xa37eff0), gcdata: *3, str: 450160, ptrToThis: 3575008,}, sendx: 0, recvx: 0, recvq: waitq { first: *(*sudog)(0xc4725bdec0), last: *(*sudog)(0xc4725bdec0),}, sendq: waitq { first: *sudog nil, last: *sudog nil,}, lock: runtime.mutex {key: 0},}, flow: github.com/openshift/origin/vendor/golang.org/x/net/http2.flow { n: 0, conn: *github.com/openshift/origin/vendor/golang.org/x/net/http2.flow nil,}, inflow: github.com/openshift/origin/vendor/golang.org/x/net/http2.flow { n: 65536000, conn: *github.com/openshift/origin/vendor/golang.org/x/net/http2.flow nil,}, tlsState: *crypto/tls.ConnectionState { Version: 771, HandshakeComplete: true, DidResume: false, CipherSuite: 52392, NegotiatedProtocol: "h2", NegotiatedProtocolIsMutual: true, ServerName: "masterlb.rsxn02.ccnp.cib", PeerCertificates: []*crypto/x509.Certificate len: 1, cap: 1, [ ("crypto/x509.Certificate")(0xc434c40000), ], VerifiedChains: [][]*crypto/x509.Certificate len: 0, cap: 0, nil, SignedCertificateTimestamps: [][]uint8 len: 0, cap: 0, nil, OCSPResponse: []uint8 len: 0, cap: 0, nil, TLSUnique: []uint8 len: 12, cap: 12, [221,30,142,255,162,136,59,69,220,252,131,26],}, remoteAddrStr: "10.33.5.31:46312", writeSched: github.com/openshift/origin/vendor/golang.org/x/net/http2.WriteScheduler(*github.com/openshift/origin/vendor/golang.org/x/net/http2.randomWriteScheduler) { zero: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.writeQueue")(0xc435a2cdc0), sq: map[uint32]*github.com/openshift/origin/vendor/golang.org/x/net/http2.writeQueue [...], queuePool: github.com/openshift/origin/vendor/golang.org/x/net/http2.writeQueuePool len: 0, cap: 4, [],}, serveG: 0, pushEnabled: false, sawFirstSettings: true, needToSendSettingsAck: false, unackedSettings: 0, clientMaxStreams: 4294967295, advMaxStreams: 250, curClientStreams: 9, curPushedStreams: 0, maxClientStreamID: 19049, maxPushPromiseID: 0, streams: map[uint32]*github.com/openshift/origin/vendor/golang.org/x/net/http2.stream [ 15265: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc4f7762a50), 14253: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc49ab3f760), 14255: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc42e1d20b0), 14259: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc449f1b600), 14229: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc4310d8dc0), 14225: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc468bd96b0), 15267: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc479506f20), 19049: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc4ba47c8f0), 15121: ("github.com/openshift/origin/vendor/golang.org/x/net/http2.stream")(0xc45b46efd0), ], initialStreamSendWindowSize: 4194304, maxFrameSize: 16384, headerTableSize: 4096, peerMaxHeaderListSize: 10485760, canonHeader: map[string]string nil, writingFrame: false, writingFrameAsync: false, needsFrameFlush: false, inGoAway: false, inFrameScheduleLoop: false, needToSendGoAway: false, goAwayCode: ErrCodeNo (0), shutdownTimer: *time.Timer nil, idleTimer: *time.Timer nil, headerWriteBuf: bytes.Buffer { buf: []uint8 len: 28, cap: 132, [136,221,251,97,151,221,109,95,74,9,149,33,174,197,4,1,50,160,28,184,211,151,25,109,76,90,55,255], off: 0, bootstrap: [64]uint8 [78,3,52,48,51,88,134,168,235,33,39,176,191,95,153,29,117,208,98,13,38,61,76,119,170,69,250,219,25,108,168,169,42,23,174,195,164,241,219,47,64,144,242,177,15,82,75,82,86,79,170,202,177,235,73,143,82,63,133,168,232,168,210,203], lastRead: opInvalid (0),}, hpackEncoder: github.com/openshift/origin/vendor/golang.org/x/net/http2/hpack.Encoder { dynTab: ("github.com/openshift/origin/vendor/golang.org/x/net/http2/hpack.dynamicTable")(0xc4359f7800), minSize: 4294967295, maxSizeLimit: 4096, tableSizeUpdate: false, w: io.Writer(*bytes.Buffer) ..., buf: []uint8 len: 25, cap: 64, [97,151,221,109,95,74,9,149,33,174,197,4,1,50,160,28,184,211,151,25,109,76,90,55,255],}, shutdownOnce: sync.Once { m: (*sync.Mutex)(0xc4224aa528), done: 0,},} (dlv) p sc.flow github.com/openshift/origin/vendor/golang.org/x/net/http2.flow { n: 0, conn: *github.com/openshift/origin/vendor/golang.org/x/net/http2.flow nil,} (dlv) p sc.inflow github.com/openshift/origin/vendor/golang.org/x/net/http2.flow { n: 65536000, conn: *github.com/openshift/origin/vendor/golang.org/x/net/http2.flow nil,}What did you expect to see?
What did you see instead?
The text was updated successfully, but these errors were encountered: