Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/pkgsite-metrics: failed to create new OS thread with new gvisor version #65215

Open
zpavlinovic opened this issue Jan 22, 2024 · 2 comments
Open

x/pkgsite-metrics: failed to create new OS thread with new gvisor version #65215

zpavlinovic opened this issue Jan 22, 2024 · 2 comments
Assignees
@zpavlinovic
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@zpavlinovic
Copy link
Contributor

Go version

1.21.2

Output of go env in your module/workspace:

-

What did you do?

Run pkgsite-metrics in production.

What did you see happen?

'go clean' failed with output runtime: failed to create new OS thread

The number of long-lived processes grows with each served request. After about 500 requests, the number of alive processes is > 32k and that kills the server. This happens with a new version of runsc 20240115.0

What did you expect to see?

The number of processes should remain near constant.
@zpavlinovic zpavlinovic added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 22, 2024
@zpavlinovic zpavlinovic added this to the Unplanned milestone Jan 22, 2024
@zpavlinovic zpavlinovic self-assigned this Jan 22, 2024
@zpavlinovic
Copy link
Contributor Author

Probably related to google/gvisor#9834

@gopherbot
Copy link

Change https://go.dev/cl/557555 mentions this issue: internal/worker: reset server after certain number of requests

gopherbot pushed a commit to golang/pkgsite-metrics that referenced this issue Jan 23, 2024
@zpavlinovic
internal/worker: reset server after certain number of requests
b1f3e84 
This CL updates gvisor to the newest available version and adds a
workaround for an issue where processes keep getting created by runsc
and stay alive until the limit is reached and server is killed.

Updates golang/go#65215

Change-Id: I9ac46bdadde3adf78285cd5ed8b42837394ec886
Reviewed-on: https://go-review.googlesource.com/c/pkgsite-metrics/+/557555
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Zvonimir Pavlinovic <zpavlinovic@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zpavlinovic zpavlinovic

Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
2 participants
@zpavlinovic @gopherbot