Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

Open
bcmills opened this issue Feb 22, 2022 · 34 comments
Open

net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 #51323

bcmills opened this issue Feb 22, 2022 · 34 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 22, 2022 •
edited 

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

The builders are seeing a lot of INTERNAL_ERROR results since 2021-10-06, typically for google.golang.org/protobuf but sometimes for other golang.org paths as well.

This may be related to #50541, which saw an uptick in errors from go.googlesource.com in about the same timeframe.

CC @golang/release

../../../../pkg/mod/cloud.google.com/go/datastore@v1.2.0/client.go:25:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/internal/trace/trace.go:23:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.39.0/status/status.go:34:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/datastore@v1.2.0/save.go:26:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/cloudbuild/apiv1/v2/cloud_build_client.go:33:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/longrunning/autogen/operations_client.go:31:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/iam/iam.go:30:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/storage@v1.10.0/iam.go:24:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer

greplogs --dashboard -md -l -e 'golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*: stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2021-01-01

2022-02-19T16:23:54-3962a08-0261fa6/dragonfly-amd64
2022-02-10T22:36:44-59536be-0a6cf87/linux-amd64-wsl
2022-02-09T18:09:24-9862752-6a70ee2/darwin-arm64-11_0-toothrot
2022-02-09T16:29:28-dad3315-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-09T16:29:28-095f870-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-08T17:01:10-9b156ee-ef06a5f/plan9-amd64-0intro
2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot
2022-02-03T19:55:40-cd36cc0-896df42/plan9-amd64-0intro
2022-01-25T22:56:45-c20fd7c-6eb58cd/plan9-amd64-0intro
2022-01-24T21:27:33-6944b10-671e115/plan9-amd64-0intro
2022-01-24T21:27:33-5e0467b-671e115/plan9-amd64-0intro
2022-01-14T16:41:18-4bd3f69-b41185c/plan9-386-0intro
2022-01-07T02:32:39-e83268e-2bb7f6b/linux-amd64-wsl
2022-01-05T22:15:55-43762dc-a845a56/darwin-arm64-12_0-toothrot
2022-01-03T23:45:12-be6af36-95b240b/plan9-amd64-0intro
2022-01-02T14:27:43-6944b10-c886143/plan9-amd64-0intro
2021-12-15T20:26:19-be6af36-07ed86c/darwin-arm64-12_0-toothrot
2021-12-14T18:18:45-ecdc095-becaeea/plan9-arm
2021-12-14T01:48:22-6944b10-1afa432/plan9-amd64-0intro
2021-11-10T19:35:55-03971e3-b954f58/darwin-arm64-11_0-toothrot
2021-10-28T14:25:03-103d89b-a3bb28e/darwin-amd64-10_15
2021-10-26T11:58:05-d418f37-1e2820a/openbsd-arm64-jsing
2021-10-14T15:16:55-a66eb64-ad99d88/linux-amd64-wsl
2021-10-14T01:51:22-e13a265-276fb27/windows-arm64-10
2021-10-13T00:11:47-281050f-4fb2e1c/darwin-amd64-11_0
2021-10-11T20:46:14-089bfa5-7023535/darwin-amd64-11_0
2021-10-07T20:37:43-86761ae-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-07T20:37:43-59d4e92-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-06T18:44:56-40a54f1-195945a/darwin-amd64-nocgo
@gopherbot gopherbot added this to the Unreleased milestone Feb 22, 2022
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 22, 2022
@dmitshur
Copy link
Contributor

From logs, it's also happened for github.com dependencies such as github.com/yuin/goldmark in 2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot.

Downloading happens with GOPROXY=https://proxy.golang.org set, so perhaps the go.googlesource.com and golang.org servers aren't relevant (they just happen to be where most of dependencies are coming from).

I've learned that INTERNAL_ERROR refers to an HTTP/2 error code that is described as "Implementation fault", but it's not clear to me if the server or client is at fault. CC @neild.

@dmitshur dmitshur changed the title x/website: frequent HTTP2 INTERNAL_ERROR errors from *.golang.org since 2021-10-06 net/http, x/net/http2, proxy.golang.org: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 Feb 22, 2022
@olix0r
Copy link

olix0r commented Mar 21, 2022

Recently, we've encountered this fairly frequently (a few times per day) in GitHub Actions, usually for github.com URLs. For example:

#11 39.00 go mod download: github.com/coreos/etcd@v3.3.13+incompatible: stream error: stream ID 4057; INTERNAL_ERROR; received from peer

@heschi
Copy link
Contributor

heschi commented Mar 21, 2022

The best explanation I have for this is some kind of infrequent incompatibility between the Go HTTP libraries and the Google front end servers.

@dmitshur dmitshur mentioned this issue Mar 30, 2022
Closed
@bcmills
Copy link
Contributor Author

bcmills commented Mar 30, 2022

It occurs to me that this could be related to #49645 (CC @neild), although the timeline doesn't quite match up because the fix for that landed Nov. 17.

greplogs --dashboard -md -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-02-23 --omit=plan9

2022-03-29T15:43:06-e96d8cf-ae9ce82/windows-amd64-longtest
2022-03-23T17:12:00-87a8611-dd211cf/freebsd-arm64-dmgk

@mintunitish
Copy link

Is there any update on this issue! Experiencing too frequently

@bcmills bcmills mentioned this issue May 3, 2022
Closed
@bcmills
Copy link
Contributor Author

bcmills commented May 6, 2022

greplogs -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-03-30 --omit=plan9
2022-05-05T16:31:21-15f5199-fdb640b/darwin-arm64-12_0-toothrot
2022-04-26T16:21:18-7b82a4e-e357634/solaris-amd64-oraclerel
2022-04-23T05:38:56-84e58bf-9717e8f/linux-mips64le-mengzhuo
2022-04-12T22:05:26-2897e13-d85694a/windows-amd64-longtest
2022-04-04T04:01:17-a8550c1-a92ca51/linux-mips64le-mengzhuo
2022-03-31T14:57:35-27fe37a-378221b/linux-amd64-longtest
2022-03-31T14:51:17-27fe37a-2ea9376/linux-amd64-longtest

@bcmills bcmills mentioned this issue May 6, 2022
Closed
@bcmills
Copy link
Contributor Author

bcmills commented May 18, 2022

greplogs -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --omit=plan9 --since=2022-05-05
2022-05-18T01:02:54-96df98a-5f2fdbe/windows-amd64-longtest
2022-05-17T21:28:43-405b155-1170771/linux-ppc64le-buildlet
2022-05-17T21:15:15-44db4eb-cf5fa2b/linux-ppc64le-power9osu
2022-05-17T20:59:58-b2400d8-8953302/linux-ppc64le-power9osu
2022-05-17T20:59:58-92f7ca4-8953302/linux-amd64-longtest
2022-05-17T20:51:53-85d78b3-b2ae2f5/linux-ppc64le-power9osu
2022-05-17T20:18:25-e5f449a-7607888/linux-ppc64le-power9osu
2022-05-17T19:40:11-e5f449a-d23c80b/linux-ppc64-buildlet
2022-05-17T18:19:34-85d78b3-076039a/linux-ppc64-buildlet
2022-05-17T18:19:34-84e58bf-076039a/linux-ppc64le-buildlet
2022-05-17T03:26:28-b2400d8-41b9d8c/linux-ppc64-buildlet
2022-05-17T01:50:34-1e55371-aedf298/linux-ppc64-buildlet
2022-05-16T20:19:45-403b017-2540f4e/aix-ppc64
2022-05-16T16:15:58-7413a13-db875f4/darwin-arm64-11_0-toothrot
2022-05-12T21:45:28-4aa4d2e-6365efb/linux-amd64-longtest
2022-05-05T16:31:21-15f5199-fdb640b/darwin-arm64-12_0-toothrot

@bcmills bcmills mentioned this issue May 19, 2022
Open
@bcmills
Copy link
Contributor Author

bcmills commented May 19, 2022

This could possibly be a duplicate of #36759 (reported as a regression Jan. 2020).

@bcmills
Copy link
Contributor Author

bcmills commented May 23, 2022

Assigning to @neild (net/http owner) on the theory that this may be reproducing a previously-reported net/http client bug (#36759).

@bcmills bcmills changed the title net/http, x/net/http2, proxy.golang.org: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06 May 23, 2022
@bcmills bcmills modified the milestones: Unreleased, Go1.19 May 23, 2022
@neild
Copy link
Contributor

neild commented Jun 9, 2022

stream error: stream ID N; INTERNAL_ERROR; received from peer means an HTTP/2 stream (a request) was closed by the remote endpoint with the error code INTERNAL_ERROR.

There are various other error codes to tell the client that it is misbehaving, such as PROTOCOL_ERROR. INTERNAL_ERROR is supposed to indicate an internal error on the remote side.

So whatever is doing the module zip download is claiming that it's unhealthy. It's not inconceivable that this is somehow the result of a net/http client bug, but if it is the remote side is doing a very bad job of telling us about it.

Maybe if we can reproduce the failure with GODEBUG=http2debug=2 enabled we might see something that helps indicate the source of the problem, but at the moment my suspicion is that the problem lies on the remote end of the connection.

@neild
Copy link
Contributor

neild commented Jun 9, 2022

googleapis/google-cloud-go#784 seems like the same issue. From 2017, eventual "fix" was to notice INTERNAL_ERROR and retry the read.

@heschi
Copy link
Contributor

heschi commented Jun 9, 2022

The other end of the connection here is the GFEs; should we try to escalate to them internally?

@neild
Copy link
Contributor

neild commented Jun 9, 2022

Yeah, I think so.

The possibilities I see are:

  • The net/http client is conjuring a peer-originated error out of thin air. This seems incredibly unlikely. I can find only one code path that adds "received from peer" and I don't see any way for it to lie.
  • The net/http client is misbehaving (very easy to believe) and the GFE is somehow responding with INTERNAL_ERROR. That's still a GFE bug, even if it's our fault for tickling it.
  • The net/http client is doing everything right, and the problem is entirely at or beyond the GFE.

At the same time, perhaps we should have a retry loop on module downloads. Connections break for various other reasons; doesn't seem like it would hurt to retry when they do.

@Tomperez98
Copy link

Any idea how to solve this? Getting the following error when running go install golang.org/x/tools/gopls@latest.

/hpath/to/go/pkg/mod/golang.org/x/tools@v0.1.11-0.20220513164230-dfee1649af67/internal/lsp/source/hover.go:23:2: golang.org/x/text@v0.3.7: stream error: stream ID 9; INTERNAL_ERROR; received from peer

@ianlancetaylor
Copy link
Contributor

@neild What is the current status here? This issue is currently in the 1.19 milestone. Should it move to 1.20? To Backlog? Thanks.

@neild
Copy link
Contributor

neild commented Jun 24, 2022

Our current belief is that the root cause is the GFE sending an INTERNAL_ERROR, probably due to the backend behind the GFE breaking the response stream.

This is not an net/http bug; net/http seems to be accurately reporting the error code sent by the GFE. (The error message could perhaps be clearer, but that's a secondary issue.)

Whatever tools are using net/http to fetch large files should be more robust in the handling of stream errors. There are many reasons a large fetch might break mid-stream, and it's unlikely we can entirely address this situation entirely by reducing the number of broken streams. This means we should add some form of retry to module zip downloads.

I think it's okay to move this to the 1.20 milestone.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.19, Go1.20 Jun 24, 2022
@neild neild removed their assignment Jul 18, 2022
@gopherbot gopherbot added this to the Go1.21 milestone Feb 1, 2023
@devopstagon devopstagon mentioned this issue Jun 29, 2023
Open
1 task
@gopherbot gopherbot modified the milestones: Go1.21, Go1.22 Aug 8, 2023
@tomwardill-payoneer tomwardill-payoneer mentioned this issue Aug 11, 2023
Merged
leg100 added a commit to leg100/otf that referenced this issue Aug 13, 2023
@tomwardill-payoneer @leg100
Return error on stream error for retry (#565)
4c2c130 
`handleEvent` was handling the an `error` type by printing something,
but then not continuing to raise the error.

This means that in a stream error, particuarly of type INTERNAL_ERROR,
the backoff/retry mechanism never took place as the return was `nil` and
the agent just stopped.

The problem is particularly relevant with Google Load Balancers and
their handling of http2/websocket long lived connections:
golang/go#51323

---------

Co-authored-by: Louis Garman <louisgarman@gmail.com>
@bcmills bcmills mentioned this issue Sep 19, 2023
Closed
@bcmills bcmills modified the milestones: Go1.22, Backlog Sep 19, 2023
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-08-06 13:51 linux-riscv64-unmatched telemetry@6b1207f8 go@460dc37c x/telemetry/godev/internal/storage [build] (log) 
internal/storage/storage_test.go:14:2: github.com/fullstorydev/emulators/storage@v0.0.0-20230523204811-eccb7d2267b0: read "https://goproxy.cn/@v/v0.0.0-20230523204811-eccb7d2267b0.mod": stream error: stream ID 29; INTERNAL_ERROR; received from peer
2023-08-28 22:24 linux-riscv64-unmatched pkgsite@ad6c6394 go@626a3cd9 (log) 
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/apierror/apierror.go:41:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/call_option.go:36:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.1.1/call_option.go:37:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/go.opencensus.io@v0.23.0/plugin/ocgrpc/client_stats_handler.go:23:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/go.opencensus.io@v0.23.0/plugin/ocgrpc/trace_common.go:22:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/go.opencensus.io@v0.23.0/plugin/ocgrpc/client.go:21:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/api@v0.63.0/transport/grpc/dial.go:31:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/api@v0.63.0/transport/grpc/dial.go:26:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/api@v0.63.0/transport/grpc/dial.go:27:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/api@v0.63.0/transport/grpc/dial.go:28:2: google.golang.org/grpc@v1.43.0: read "https:/goproxy.cn/@v/v1.43.0.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
2023-08-30 14:09 darwin-amd64-11_0 build@762ea330 go@a8d45a72 (log)
2023-08-30 16:14 netbsd-arm-bsiegert telemetry@03a44786 go@06df3292 (log) 
internal/content/esbuild.go:14:2: github.com/evanw/esbuild@v0.17.19: read "https:/proxy.golang.org/@v/v0.17.19.zip": stream error: stream ID 5; INTERNAL_ERROR; received from peer
2023-09-10 18:25 netbsd-arm64-bsiegert crypto@3f0842a4 go@d110d7c4 (log)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-09-19 17:00 darwin-amd64-13 build@c8fdba98 go@2b462646 [build] (log) 
../../../../pkg/mod/cloud.google.com/go/iam@v1.0.1/apiv1/iampb/iam_policy.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/iam@v1.0.1/apiv1/iampb/policy.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.57.0/status/status.go:35:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.11.0/apierror/internal/proto/error.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.11.0/apierror/apierror.go:45:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/pubsub@v1.30.0/snapshot.go:24:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/datastore@v1.11.0/client.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/datastore@v1.11.0/save.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/errorreporting@v0.3.0/apiv1beta1/error_group_client.go:35:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/secretmanager@v1.10.0/apiv1/secretmanagerpb/service.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
internal/secret/flag.go:16:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/notification.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/common.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/metric.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/metric.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/group_service.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/service.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.5/monitoredresource/gcp/gcp_metadata_config.go:26:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.5/trace.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.5/metrics.go:36:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/security@v1.15.0/privateca/apiv1/certificate_authority_client.go:41:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/cloudbuild@v1.9.0/apiv1/v2/cloudbuildpb/cloudbuild.pb.go:30:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/proxy.golang.org/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 141; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-09-08 20:30 linux-riscv64-unmatched pkgsite@8025830b go@bda5e6c3 (log) 
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.11.0/apierror/internal/proto/error.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.11.0/apierror/apierror.go:45:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.57.0/status/status.go:35:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/profiler@v0.1.1/profiler.go:59:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/errorreporting@v0.3.0/apiv1beta1/error_group_client.go:35:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/log_entry.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/logging_metrics.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/logging_metrics.pb.go:30:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/log_entry.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/log_entry.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/logging@v1.7.0/apiv2/loggingpb/logging_config.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/iam@v0.13.0/apiv1/iampb/policy.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/storage@v1.29.0/internal/apiv2/stubs/storage.pb.go:29:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/storage@v1.29.0/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/notification.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/metric.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/monitoring@v1.13.0/apiv3/v2/monitoringpb/service.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.4/monitoredresource/gcp/gcp_metadata_config.go:26:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.4/trace.go:28:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
../../../../pkg/mod/contrib.go.opencensus.io/exporter/stackdriver@v0.13.4/metrics.go:36:2: google.golang.org/genproto@v0.0.0-20230530153820-e85fd2cbaebc: read "https:/goproxy.cn/@v/v0.0.0-20230530153820-e85fd2cbaebc.zip": stream error: stream ID 57; INTERNAL_ERROR; received from peer
2023-09-19 20:58 darwin-amd64-nocgo pkgsite@14233af4 go@bf5d8c02 (log) 
../../../../pkg/mod/google.golang.org/grpc@v1.57.0/server.go:36:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/controlbuf.go:31:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/controlbuf.go:32:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/appengine@v1.6.7/appengine.go:15:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
internal/source/source.go:34:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/microcosm-cc/bluemonday@v1.0.16/sanitize.go:41:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer
internal/frontend/overview.go:16:2: golang.org/x/net@v0.15.0: read "https:/proxy.golang.org/@v/v0.15.0.zip": stream error: stream ID 31; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

gopherbot commented Oct 1, 2023 •
edited by dmitshur

Change https://go.dev/cl/532035 mentions this issue: cmd/go/internal/modfetch: show real URL in response body read errors

@vikasmulaje vikasmulaje mentioned this issue Oct 3, 2023
Closed
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-09-25 14:46 linux-s390x-ibm benchmarks@1082dde5 go@e47cab13 (log) 
../../../../pkg/mod/golang.org/x/net@v0.15.0/idna/idna10.0.0.go:26:2: golang.org/x/text@v0.13.0: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/net@v0.15.0/idna/idna10.0.0.go:27:2: golang.org/x/text@v0.13.0: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/net@v0.15.0/idna/idna10.0.0.go:28:2: golang.org/x/text@v0.13.0: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer
../../../../pkg/mod/gitlab.com/golang-commonmark/linkify@v0.0.0-20191026162114-a0c2df6c8f82/charset.go:10:2: golang.org/x/text@v0.13.0: read "https:/proxy.golang.org/@v/v0.13.0.zip": stream error: stream ID 185; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-10-25 13:59 netbsd-arm-bsiegert telemetry@5ab6b58d go@0046c141 (log) 
../../../../../pkg/mod/golang.org/x/net@v0.10.0/idna/idna10.0.0.go:26:2: golang.org/x/text@v0.9.0: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/golang.org/x/net@v0.10.0/idna/idna10.0.0.go:27:2: golang.org/x/text@v0.9.0: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/golang.org/x/net@v0.10.0/idna/idna10.0.0.go:28:2: golang.org/x/text@v0.9.0: read "https:/proxy.golang.org/@v/v0.9.0.zip": stream error: stream ID 99; INTERNAL_ERROR; received from peer

— watchflakes

gopherbot pushed a commit that referenced this issue Nov 3, 2023
@dmitshur @gopherbot
cmd/go/internal/modfetch: show real URL in response body read errors
8b14e99 
CL 233437 added a redactedURL field to proxyRepo, a struct that already
had a field named 'url'. Neither fields were documented, so the similar
names suggest the most natural interpretation that proxyRepo.redactedURL
is equivalent to proxyRepo.url.Redacted() rather than something else.
That's possibly why it was joined with the module version in CL 406675.

It turns out the two URLs differ in more than just redaction: one is the
base proxy URL with (escaped) module path joined, the other is just the
base proxy URL, in redacted form.

Document and rename the fields to make the distinction more clear, and
include all 3 of base module proxy URL + module path + module version
in the reported URL, rather than just the first and third bits as seen
in the errors at https://go.dev/issue/51323#issuecomment-1735812250.

For #51323.
Updates #38680.
Updates #52727.

Change-Id: Ib4b134b548adeec826ee88fe51a2cf580fde0516
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/532035
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@DmitriyLewen DmitriyLewen mentioned this issue Nov 7, 2023
Open
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-10 21:13 solaris-amd64-oraclerel pkgsite@545ce2ad go@31f0af1e (log) 
../../../../pkg/mod/golang.org/x/net@v0.18.0/idna/idna10.0.0.go:25:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/net@v0.18.0/idna/idna10.0.0.go:26:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/golang.org/x/net@v0.18.0/idna/idna10.0.0.go:27:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/google/safehtml@v0.0.3-0.20211026203422-d6f0e11a5516/html.go:14:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
../../../../pkg/mod/github.com/jackc/pgconn@v1.10.1/auth_scram.go:27:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/godoc/dochtml/dochtml.go:32:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/godoc/dochtml/dochtml.go:33:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer
internal/frontend/imports.go:16:2: golang.org/x/text@v0.14.0: read "https://proxy.golang.org/golang.org/x/text/@v/v0.14.0.zip": stream error: stream ID 109; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-14 16:31 linux-riscv64-unmatched telemetry@69313e64 go@1c15291f (log) 
../../../../../pkg/mod/cloud.google.com/go/iam@v1.0.1/apiv1/iampb/iam_policy.pb.go:28:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/iam@v1.0.1/apiv1/iampb/policy.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/google.golang.org/grpc@v1.55.0/status/status.go:35:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.8.0/apierror/internal/proto/error.pb.go:27:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.8.0/apierror/apierror.go:41:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/stubs/storage.pb.go:30:2: google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1: read "https://goproxy.cn/google.golang.org/genproto/@v/v0.0.0-20230410155749-daa745c078e1.zip": stream error: stream ID 35; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-21 19:04 windows-amd64-longtest go@ebca52ee cmd/go.TestScript (log) 
vcs-test.golang.org rerouted to http://127.0.0.1:54795
https://vcs-test.golang.org rerouted to https://127.0.0.1:54796
go test proxy running at GOPROXY=http://127.0.0.1:54797/mod
--- FAIL: TestScript (0.14s)
    --- FAIL: TestScript/mod_sumdb_file_path (91.40s)
        script_test.go:132: 2023-11-21T19:43:03Z
        script_test.go:134: $WORK=C:\Users\gopher\AppData\Local\Temp\1\cmd-go-test-1865174972\tmpdir2564125123\mod_sumdb_file_path3035018066
        script_test.go:156: 
            > [!net:proxy.golang.org] skip
            [condition not met]
...
            go: golang.org/x/text@v0.3.2: reading file:///C:%5CUsers%5Cgopher%5CAppData%5CLocal%5CTemp%5C1%5Ccmd-go-test-1865174972%5Ctmpdir2564125123%5Cmod_sumdb_file_path3035018066/sumproxy/golang.org/x/text/@v/v0.3.2.mod: The system cannot find the path specified.
            [exit status 1]
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > go get golang.org/x/text@v0.3.2
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/text@v0.3.2: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 11; INTERNAL_ERROR; received from peer
        script_test.go:156: FAIL: testdata\script\mod_sumdb_file_path.txt:46: go get golang.org/x/text@v0.3.2: exit status 1
2023-11-21 19:26 linux-386-longtest go@b3b44244 cmd/go.TestScript (log) 
vcs-test.golang.org rerouted to http://127.0.0.1:41933
https://vcs-test.golang.org rerouted to https://127.0.0.1:38623
go test proxy running at GOPROXY=http://127.0.0.1:41757/mod
2023/11/21 19:54:24 http: TLS handshake error from 127.0.0.1:56838: EOF
2023/11/21 19:54:24 http: TLS handshake error from 127.0.0.1:56852: EOF
2023/11/21 19:55:20 http: TLS handshake error from 127.0.0.1:36984: read tcp 127.0.0.1:38623->127.0.0.1:36984: read: connection reset by peer
--- FAIL: TestScript (0.05s)
    --- FAIL: TestScript/mod_sumdb_file_path (2.36s)
        script_test.go:132: 2023-11-21T19:54:20Z
        script_test.go:134: $WORK=/workdir/tmp/cmd-go-test-791329614/tmpdir2447884709/mod_sumdb_file_path3863367325
...
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            > ! go get golang.org/x/text@v0.3.2
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/text@v0.3.2: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 7; INTERNAL_ERROR; received from peer
            [exit status 1]
            > stderr '^go: golang.org/x/text@v0.3.2: verifying module: golang.org/x/text@v0.3.2: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/text@v0.3.2: (no such file or directory|.*cannot find the path specified.*)'
        script_test.go:156: FAIL: testdata/script/mod_sumdb_file_path.txt:18: stderr '^go: golang.org/x/text@v0.3.2: verifying module: golang.org/x/text@v0.3.2: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/text@v0.3.2: (no such file or directory|.*cannot find the path specified.*)': no match for `(?m)^go: golang.org/x/text@v0.3.2: verifying module: golang.org/x/text@v0.3.2: reading file://.*/sumdb/sum.golang.org/lookup/golang.org/x/text@v0.3.2: (no such file or directory|.*cannot find the path specified.*)` in stderr

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-27 16:49 darwin-amd64-longtest go@6317fb41 cmd/go.TestScript (log) 
vcs-test.golang.org rerouted to http://127.0.0.1:49949
https://vcs-test.golang.org rerouted to https://127.0.0.1:49950
go test proxy running at GOPROXY=http://127.0.0.1:49951/mod
--- FAIL: TestScript (0.45s)
    --- FAIL: TestScript/mod_sumdb_file_path (29.25s)
        script_test.go:132: 2023-11-27T17:59:36Z
        script_test.go:134: $WORK=/tmp/buildlet/tmp/cmd-go-test-4127345601/tmpdir881471550/mod_sumdb_file_path2607082938
        script_test.go:156: 
            > [!net:proxy.golang.org] skip
            [condition not met]
...
            go: golang.org/x/text@v0.3.2: reading file:///tmp/buildlet/tmp/cmd-go-test-4127345601/tmpdir881471550/mod_sumdb_file_path2607082938/sumproxy/golang.org/x/text/@v/v0.3.2.mod: no such file or directory
            [exit status 1]
            > [GOOS:windows] env GOPROXY=file:///$WORK/sumproxy,https://proxy.golang.org
            [condition not met]
            > [!GOOS:windows] env GOPROXY=file://$WORK/sumproxy,https://proxy.golang.org
            > go get golang.org/x/text@v0.3.2
            [stderr]
            go: downloading golang.org/x/text v0.3.2
            go: golang.org/x/text@v0.3.2: read "https://proxy.golang.org/golang.org/x/text/@v/v0.3.2.zip": stream error: stream ID 11; INTERNAL_ERROR; received from peer
        script_test.go:156: FAIL: testdata/script/mod_sumdb_file_path.txt:46: go get golang.org/x/text@v0.3.2: exit status 1

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-30 03:00 linux-riscv64-unmatched pkgsite@9f20944f go@b4fa5b16 (log) 
internal/postgres/delete.go:11:2: github.com/Masterminds/squirrel@v1.5.2: read "https://goproxy.cn/github.com/%21masterminds/squirrel/@v/v1.5.2.zip": stream error: stream ID 169; INTERNAL_ERROR; received from peer

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2023-11-27 18:10 netbsd-arm64-bsiegert perf@b5375226 go@bda1ef13 (log) 
benchseries/chart.go:22:2: gonum.org/v1/plot@v0.10.1: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:23:2: gonum.org/v1/plot@v0.10.1: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:24:2: gonum.org/v1/plot@v0.10.1: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:16:2: gonum.org/v1/plot@v0.10.1: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer
benchseries/chart.go:17:2: gonum.org/v1/plot@v0.10.1: read "https://proxy.golang.org/gonum.org/v1/plot/@v/v0.10.1.zip": stream error: stream ID 23; INTERNAL_ERROR; received from peer

— watchflakes

@zyxkad zyxkad mentioned this issue Feb 23, 2024
Closed
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`
2024-03-06 21:06 linux-riscv64-unmatched telemetry@d5a85b27 go@90d24b4f (log) 
../../../../../pkg/mod/github.com/googleapis/gax-go/v2@v2.12.0/apierror/apierror.go:44:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/storage_client.go:30:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/doc.go:91:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/storage_client.go:32:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/internal/apiv2/storage_client.go:33:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/bucket.go:32:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/acl.go:24:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/http_client.go:41:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer
../../../../../pkg/mod/cloud.google.com/go/storage@v1.30.1/http_client.go:42:2: google.golang.org/api@v0.149.0: read "https://goproxy.cn/google.golang.org/api/@v/v0.149.0.zip": stream error: stream ID 25; INTERNAL_ERROR; received from peer

— watchflakes

@Sebastin-cr
Copy link

I am facing the below errors
http2: Framer 0xc00029b5e0: read RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
http2: Transport received RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
RoundTrip failure: stream error: stream ID 3; INTERNAL_ERROR; received from peer

Any root cause for this?

@neild
Copy link
Contributor

neild commented Mar 11, 2024

In stream error: stream ID 3; INTERNAL_ERROR; received from peer, the "received from peer" indicates that the INTERNAL_ERROR code was received from the remote endpoint.

That is, the server you are talking to sent back an INTERNAL_ERROR code, presumably indicating that it has encountered an internal error of some form. net/http is just relaying that fact to you. The root cause is presumably on the server side.

@Sebastin-cr
Copy link

Sebastin-cr commented Mar 12, 2024 •
edited

is server sending RST_STREAM incase of any internal error introduced in latest go version or was it existing in earlier versions because when we were using x/net/v0.0.0-20211029224645-99673261e6eb this behaviour was not coming and when we updated it to x/net/v0.19 for fortify fixes,we are encountering this error @neild

@neild
Copy link
Contributor

neild commented Mar 18, 2024

I'm sorry, but I don't have enough details to attempt to answer that question.

The error is coming from the remote server. Did you update your server to a newer version of x/net, your client, or both? If it's just the client, then you want to ask why your server is producing INTERNAL_ERROR errors, not why the client is surfacing them. If you did update the server, then I'd really need a reproduction case to attempt to debug the issue (and we should do that on a new issue, not this one, which is not about errors being produced by Go servers).

v0.0.0-20211029224645-99673261e6eb to v0.19.0 is two years of changes. I'm afraid I don't remember everything that happened in that time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Release Status & Release Blockers
Status: No status
Test Flakes
Status: No status
Milestone
Backlog
Development

No branches or pull requests
10 participants
@neild @olix0r @dmitshur @ianlancetaylor @bcmills @gopherbot @mintunitish @Sebastin-cr @heschi @Tomperez98