Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

Closed
bcmills opened this issue Feb 15, 2022 · 7 comments
Closed

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

bcmills opened this issue Feb 15, 2022 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release Team
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 15, 2022 

2022/02/14 19:12:27 Downloading 16776793 bytes of https://storage.googleapis.com/golang-maintner-log/0126.ddc03bfdacbc6aa8a32f5987fd7fcad8a1ec1ee5da521071200b2ae6.mutlog ...
2022/02/14 19:12:27 wrote C:\Users\gopher\.cache\golang-maintner\0126.ddc03bfdacbc6aa8a32f5987fd7fcad8a1ec1ee5da521071200b2ae6.mutlog
2022/02/14 19:12:27 Downloading 16776890 bytes of https://storage.googleapis.com/golang-maintner-log/0127.cf0c3c7009d2d5988293283b748239bcd8b18acfb086db5c390df4eb.mutlog ...
2022/02/14 19:12:27 wrote C:\Users\gopher\.cache\golang-maintner\0127.cf0c3c7009d2d5988293283b748239bcd8b18acfb086db5c390df4eb.mutlog
2022/02/14 19:12:27 Downloading -5369 bytes of https://maintner.golang.org/logs/128 ...
2022/02/14 19:12:27 Corpus GetMutations: syncing segment 128: https://maintner.golang.org/logs/128: 416 Requested Range Not Satisfiable
--- FAIL: TestCorpusCheck (33.65s)
    godata_test.go:52: getting corpus: syncing segment 128: https://maintner.golang.org/logs/128: 416 Requested Range Not Satisfiable
2022/02/14 19:12:27 Loading data from log *maintner.netMutSource ...
2022/02/14 19:13:01 Reloaded data from log *maintner.netMutSource.
FAIL
FAIL	golang.org/x/build/maintner/godata	68.324s

This failure mode seems to occur fairly frequently on windows/amd64, and somewhat less frequently on linux/amd64:

greplogs --dashboard -md -l -e 'syncing segment \d+: .*: 416 Requested Range Not Satisfiable'

2022-02-14T19:10:36-75546d9-ecf3b39/windows-amd64-longtest
2021-11-02T05:35:11-fd71dbe-6f327f7/windows-amd64-longtest
2021-10-26T19:04:53-a72dd9f-1dc77a3/windows-amd64-longtest
2021-10-25T14:50:31-af7dff6-1ed060c/windows-amd64-longtest
2021-10-20T17:23:34-0c6ea36-27a1454/windows-amd64-longtest
2021-10-18T16:06:08-3025285-417100e/windows-amd64-longtest
2021-10-06T20:21:49-8630382-f580b75/windows-amd64-longtest
2021-10-06T19:59:27-8630382-17c513e/linux-amd64-longtest
2021-10-05T19:17:53-1409edd-c1a0aa3/windows-amd64-longtest
2021-10-05T15:58:05-1409edd-f1f626d/windows-amd64-longtest
2021-09-30T20:30:12-71603fa-c035d82/windows-amd64-longtest
2021-09-20T16:55:47-6f7df8f-9e60c37/linux-amd64-longtest
2021-09-20T16:09:39-6f7df8f-a81b0dc/windows-amd64-longtest
2021-09-14T23:35:36-fc41926-bf26e43/windows-amd64-longtest
2021-09-14T02:53:17-14e3e09-ee91bb8/linux-amd64-longtest
2021-09-13T18:38:15-cc073c5-ac40c98/windows-amd64-longtest
2021-09-10T17:17:53-71eba72-cf2fe5d/linux-amd64-longtest
2021-09-10T16:08:21-4026ac5-da1aa65/windows-amd64-longtest
2021-09-08T18:48:02-3ae692a-65f0d24/windows-amd64-longtest
2021-09-02T21:07:55-dc54d45-065f380/windows-amd64-longtest
2021-09-01T14:30:33-3b206e5-717f337/windows-amd64-longtest
2021-08-31T16:43:46-c82f902-6815235/windows-amd64-longtest
2021-08-20T19:58:13-8b42ff7-f67e31d/windows-amd64-longtest
2021-08-18T22:34:31-808d0cd-687f2ac/windows-amd64-longtest
2021-08-13T20:47:06-4f8007b-641e8bc/windows-amd64-longtest
2021-07-31T23:59:40-12eae6d-b8ca6e5/windows-amd64-longtest
2021-07-30T14:01:30-b39e5bc-b7a85e0/windows-amd64-longtest
2021-07-28T14:06:22-4dfbbed-b39e0f4/windows-amd64-longtest
2021-07-02T19:26:52-76eb641-743f03e/windows-amd64-longtest
2021-06-21T15:39:45-df58bba-ced0fdb/windows-amd64-longtest
2021-06-16T18:06:28-daa065c-a294e4e/windows-amd64-longtest
2021-06-15T18:05:29-1bda1af-4d2d89f/windows-amd64-longtest
2021-06-15T17:05:31-1bda1af-033d885/windows-amd64-longtest
2021-06-14T20:37:40-1bda1af-6bbb0a9/windows-amd64-longtest
2021-06-12T00:12:55-4c0be6e-9d46ee5/windows-amd64-longtest
2021-06-04T20:24:30-9477c7e-831f937/windows-amd64-longtest
2021-06-02T14:28:15-10d5d32-d743e67/windows-amd64-longtest
2021-05-26T16:11:00-f2096ad-bfd7798/windows-amd64-longtest
2021-05-21T17:49:01-d0819ed-4fb10b2/windows-amd64-longtest
2021-05-18T15:00:55-48b0b7a-690a8c3/windows-amd64-longtest
2021-05-17T15:57:52-0536f73-bade680/windows-amd64-longtest
2021-05-04T03:22:47-ce4d62a-138d2c9/windows-amd64-longtest
2021-04-29T17:47:48-7eefc07-6d95e5a/windows-amd64-longtest
2021-04-28T17:39:34-f9ec3ca-a547625/windows-amd64-longtest
2021-04-28T15:47:34-f9ec3ca-12af403/windows-amd64-longtest
2021-04-28T00:49:58-f9ec3ca-c9f4350/windows-amd64-longtest
2021-04-27T19:40:24-2b6edb4-b9dfaf7/windows-amd64-longtest
2021-04-27T19:12:41-2b6edb4-0c3557e/windows-amd64-longtest
2021-04-21T23:21:55-4600b17-7bedd47/windows-amd64-longtest
2021-04-16T16:40:34-c328b27-acb189e/windows-amd64-longtest
2021-04-15T12:38:13-43d5804-7ad496b/windows-amd64-longtest
2021-04-13T20:07:47-43d5804-69262d4/windows-amd64-longtest
2021-03-31T16:39:16-305e1e3-46fa8af/windows-amd64-longtest
2021-03-25T02:50:11-d7600ec-a95e2ae/windows-amd64-longtest
2021-03-18T21:10:31-884a8de-79d03ad/windows-amd64-longtest
2021-03-16T00:27:31-0364c52-f02a26b/windows-amd64-longtest
2021-03-10T21:49:24-c79c810-aa26687/windows-amd64-longtest
2021-03-09T18:26:24-baa7b38-e8e425c/windows-amd64-longtest
2021-03-08T20:03:01-baa7b38-bd37284/windows-amd64-longtest
2021-02-24T18:49:08-82b1f4c-b7f62da/windows-amd64-longtest
2021-02-24T16:38:56-82b1f4c-27684ea/windows-amd64-longtest
2021-02-23T18:39:53-f539cd3-c584f42/windows-amd64-longtest
2021-02-23T00:20:38-f539cd3-f113e9a/windows-amd64-longtest
2021-02-22T20:27:09-f539cd3-0490347/windows-amd64-longtest
2021-01-22T21:16:10-024bd71-a2cef9b/windows-amd64-longtest
2021-01-20T14:59:52-789b929-be28e5a/windows-amd64-longtest
2021-01-12T21:39:59-789b929-ba76567/windows-amd64-longtest
2021-01-08T17:14:20-a4cf489-6598c65/windows-amd64-longtest
2021-01-08T01:33:44-0a4bf69-54bd1cc/windows-amd64-longtest
2020-12-04T16:04:44-b6220de-21cfadf/windows-amd64-longtest
2020-12-01T20:11:50-b6220de-933ce97/windows-amd64-longtest
2020-11-25T15:59:35-2e05377-df68e01/windows-amd64-longtest
2020-11-24T00:50:11-302a384-762eda3/windows-amd64-longtest
2020-11-20T17:31:50-07fdb09-c306fd6/windows-amd64-longtest
2020-11-20T15:19:17-07fdb09-cb3f84a/windows-amd64-longtest
2020-11-20T02:27:53-07fdb09-0dcc7d6/windows-amd64-longtest
2020-11-19T02:17:10-07fdb09-0bb6115/windows-amd64-longtest
2020-11-17T18:28:55-07fdb09-01df2fe/windows-amd64-longtest
2020-11-16T17:27:26-5f1e306-d834ece/windows-amd64-longtest
2020-11-07T03:55:32-788b5c2-d51ae66/windows-amd64-longtest
2020-11-03T00:50:57-7830753-ebc1b8e/windows-amd64-longtest
2020-10-30T20:21:14-3b7f115-9393b5b/windows-amd64-longtest
2020-10-29T13:49:26-d43b1a5-96bd0b1/windows-amd64-longtest
2020-10-27T23:36:05-d43b1a5-70e022e/windows-amd64-longtest
2020-10-25T20:56:22-d43b1a5-c800222/windows-amd64-longtest
2020-10-23T20:38:27-1a4bcd6-75032ad/windows-amd64-longtest
2020-10-23T14:59:03-1a4bcd6-b5ddc42/windows-amd64-longtest
2020-10-22T15:30:02-2476803-91b7619/windows-amd64-longtest
2020-10-17T19:22:49-8ea6a2c-30119bc/windows-amd64-longtest
2020-10-16T17:52:59-8ea6a2c-03f181a/windows-amd64-longtest
2020-10-15T15:40:06-cd657d3-aa161e7/windows-amd64-longtest
2020-10-14T08:00:39-ab40b66-e293161/windows-amd64-longtest
2020-10-12T21:13:04-ab40b66-15a11ce/windows-amd64-longtest
2020-10-10T00:55:54-d7e2e4d-d317ba5/windows-amd64-longtest
2020-10-06T01:14:39-df6c66f-694025e/windows-amd64-longtest
2020-09-30T18:07:33-da0cd80-f811663/windows-amd64-longtest
2020-09-30T17:24:53-da0cd80-ae329ab/windows-amd64-longtest
2020-09-29T16:30:26-da0cd80-39dde09/windows-amd64-longtest
2020-09-28T18:46:39-034e344-f33263d/windows-amd64-longtest
2020-09-28T08:59:02-034e344-874b313/windows-amd64-longtest
2020-09-18T14:36:22-43d625a-789d77a/windows-amd64-longtest
2020-09-18T00:39:35-43d625a-9e9c030/windows-amd64-longtest
2020-09-17T19:25:09-43d625a-6796a7f/windows-amd64-longtest
2020-09-14T19:45:28-301a72c-5764653/windows-amd64-longtest
2020-09-09T17:48:56-301a72c-34835df/linux-amd64-longtest
2020-09-08T15:09:48-301a72c-e8f9185/windows-amd64-longtest
2020-08-27T19:42:42-d6a7ee8-2013f70/windows-amd64-longtest
2020-08-25T17:37:38-cec1aa1-5e1e8c4/windows-amd64-longtest
2020-07-29T21:18:03-a6019d6-85afa2e/windows-amd64-longtest
2020-07-23T18:07:28-70303fc-78c20c8/windows-amd64-longtest
2020-06-03T02:59:25-e566a70-9bbe899/windows-amd64-longtest
2020-06-02T22:20:47-e566a70-a9cc105/windows-amd64-longtest
2020-05-21T21:17:48-92427f6-9f4aeb3/windows-amd64-longtest
2020-05-18T23:05:20-02551af-0d39dba/windows-amd64-longtest
2020-05-08T21:21:40-bf27e27-7cfa7d6/windows-amd64-longtest
2020-04-14T20:50:19-9a1cfea-a55645f/windows-amd64-longtest
2020-04-10T15:31:53-f87aadf-782fcb4/windows-amd64-longtest
2020-04-09T01:14:53-5bbd558-7a4247c/windows-amd64-longtest
2020-04-08T21:34:49-fdec4c9-7694bf3/windows-amd64-longtest
2020-04-08T21:07:59-fdec4c9-e3323f5/windows-amd64-longtest
2020-04-08T21:07:59-fdec4c9-b2790a2/windows-amd64-longtest
2020-01-06T17:40:38-17a7d87-562fabc/linux-amd64-longtest
2019-12-24T03:42:15-17a7d87-7cab55d/windows-amd64-longtest

(CC @golang/release)
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 15, 2022
@gopherbot gopherbot added this to the Unreleased milestone Feb 15, 2022
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 15, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Mar 30, 2022

Curiously, a similar failure mode has also occurred with 502 Bad Gateway:

greplogs --dashboard -md -l -e 'getting corpus: .*: 502 Bad Gateway'

2022-03-17T19:24:19-8c691c3-c6244b5/windows-amd64-longtest

@dmitshur dmitshur added this to Planned in Go Release Team Apr 5, 2022
@dmitshur
Copy link
Contributor

dmitshur commented Apr 5, 2022

We can mark the test as known flaky sooner.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 5, 2022
@bcmills
Copy link
Contributor Author

bcmills commented May 9, 2022

greplogs -l -e 'syncing segment \d+: .*: 416 Requested Range Not Satisfiable' --since=2022-02-16
2022-05-06T16:18:34-b8613e0-5fcd1ba/windows-amd64-longtest
2022-03-17T18:03:05-e0ed2a2-0fca8a8/linux-amd64-longtest

@dmitshur
Copy link
Contributor

The 502 Bad Gateway from /logs is another (non-serious) issue, filed #53569 for it.

I was going to send a small change to add a log line, to learn what range value causes the server to respond with 416, but we already have this log line and can see from the log that this is caused by issue #26922. Sent CL 414434 for it.

@gopherbot
Copy link

Change https://go.dev/cl/414434 mentions this issue: maintner: don't try to download negative bytes

gopherbot pushed a commit to golang/build that referenced this issue Jun 29, 2022
@dmitshur @gopherbot
maintner: don't try to download negative bytes
4443b10 
If the maintnerd server restarts with some uncommitted transactions,
clients may end up having more downloaded bytes than the server, after
the server restarts. We're invariably heading towards an ErrSplit if
this happens (which is fine, otherwise we'd need to add more code to
avoid duplicate event processing and it's unlikely to be worthwhile).

In such a situation, don't try to download new data from the server,
since that's guaranteed to fail with a 416 Range Not Satisfiable error.
Do try to reuse the prefix of the existing data as long as its checksum
matches, otherwise download the new version of the growing file.

Fixes golang/go#26922.
Updates golang/go#51211.

Change-Id: Ide099ee0740e854cfe764db8c3b4341836a237f4
Reviewed-on: https://go-review.googlesource.com/c/build/+/414434
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Alex Rakoczy <jenny@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
@heschi
Copy link
Contributor

heschi commented Jul 6, 2022

2022-06-28T19:45:04-950e098-a30f434/linux-amd64-longtest

@dmitshur
Copy link
Contributor

dmitshur commented Jul 6, 2022

The aforementioned failure is from golang/build@950e098, which is a few commits before CL 414434 (golang/build@4443b10).

Tentatively closing since I expect this should be completely fixed and there's nothing more to do here, but please reopen if there are further occurrences after the aforementioned commit.

@dmitshur dmitshur closed this as completed Jul 6, 2022
Go Release Team automation moved this from Planned to Done Jul 6, 2022
@golang golang locked and limited conversation to collaborators Jul 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Go Release Team
Done
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@dmitshur @bcmills @gopherbot @heschi