Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/pkgsite: dealing with excessive load on the worker #41452

Closed
jba opened this issue Sep 17, 2020 · 6 comments
Closed

x/pkgsite: dealing with excessive load on the worker #41452

jba opened this issue Sep 17, 2020 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Milestone
pkgsite/unplanned

Comments

@jba
Copy link
Contributor

jba commented Sep 17, 2020

When dealing with many large modules at once, the worker becomes overloaded and OOMs.

Reducing the max concurrency of the queue and/or increasing the number of worker instances can help, but that requires tuning, and often the characteristics change. For example, workers can handle much higher rates when processing many small modules.

Investigate load-shedding and other techniques for making the worker more resilient.
@jba jba added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite labels Sep 17, 2020
@jba jba added this to the pkgsite/unplanned milestone Sep 17, 2020
@jba jba self-assigned this Sep 17, 2020
@jba
Copy link
Contributor Author

jba commented Sep 17, 2020

We're going to start by tracking the total size of all zip files that are in flight, as a signal for the amount of memory the worker will eventually need to process them. It makes sense to track this rather than actual memory usage because:

  • we can obtain it early in the request path;
  • it leads memory allocation, rather than lagging as would a direct measurement of heap size;
  • it isn't confounded by GC effects.

/cc @findleyr

@gopherbot
Copy link

Change https://golang.org/cl/255738 mentions this issue: internal/fetch: refactor to get info separately

@gopherbot
Copy link

Change https://golang.org/cl/255737 mentions this issue: internal/proxy: implement GetZipSize

@gopherbot
Copy link

Change https://golang.org/cl/255757 mentions this issue: internal/stdlib: Zip takes resolved versionn

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 17, 2020
@jba
internal/proxy: implement GetZipSize
d5afe0e 
Provide an efficient way to get the size of a module zip file
without downloading it.

For golang/go#41452

Change-Id: I9a996efb83146109a31e73b302af1b49f197b43f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255737
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/255759 mentions this issue: internal/worker: implement load-shedding

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
@jba
internal/fetch: refactor to get info separately
21c3920 
Reorganize FetchModule into two functions.

First, one calls GetModuleInfo to get some prelimary,
cheap information, including the size of the zip.

Then one calls FetchModule with the result of GetModuleInfo.

This allows us to make decisions based on the zip size
before downloading the entire zip.

For golang/go#41452

Change-Id: Ifbffa5882488e31a72e78670caf69c3d97a80546
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255738
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
@jba
internal/stdlib: Zip takes resolved version
89b8b39 
In the only call to this function, we already know the resolved
version.

For golang/go#41452

Change-Id: I96e5ebdf5acb0aeec45288b25b555e5c6cbb52ee
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255757
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 18, 2020
@jba
internal/worker: implement load-shedding
1a9ed2e 
Don't fetch a module if the total zip size currently being fetched
exceeds a threshold.

For golang/go#41452

Change-Id: I9f3f12c1daf036754877b78b1eb3607404dc676b
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/255759
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
@julieqiu julieqiu removed the website label Sep 19, 2020
@julieqiu julieqiu removed the website label Sep 19, 2020
@jba
Copy link
Contributor Author

jba commented Sep 23, 2020

Load-shedding is implemented and seems successful.

@jba jba closed this as completed Sep 23, 2020
@golang golang locked and limited conversation to collaborators Sep 23, 2021
@rsc rsc unassigned jba Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Projects
None yet
Milestone
pkgsite/unplanned
Development

No branches or pull requests
3 participants
@julieqiu @gopherbot @jba