runtime: automatically bump RLIMIT_NOFILE on Unix #46279

bradfitz · 2021-05-20T02:44:32Z

I just read http://0pointer.net/blog/file-descriptor-limits.html which in a nutshell says:

don't use select
systemd sets the RLIMIT_NOFILE soft limit to 1024 for compatibility reasons, to not break select users
systemd sets the RLIMIT_NOFILE hard limit 512K, for programs that want more (without escalation), but by raising their soft limit past 1024, they're implicitly acknowledging that select won't work.

I realize that since Go doesn't use select, the Go runtime could automatically do this fd soft limit bumping on Linux.

We do have a Select wrapper at https://pkg.go.dev/golang.org/x/sys/unix#Select, though, so perhaps we could do the same thing we did for #42347 in 18510ae (https://go-review.googlesource.com/c/go/+/299671) and do the bumping conditionally based on whether the unix.Select func is in the binary. Or cgo too, I suppose.

I suspect many users are unaware of this 512K hard limit that's free to bump up to. I certainly was unaware. (I normally have to go in and manual tweak my systemd limits instead, usually in response to problems once I hit the limit...) I think fixing it automatically would help more users than it'd hurt. (I actually can't think how it'd hurt anybody?)

I don't think we need it as a backpressure mechanism. As the blog post mentions, memory limits are already that mechanism.

/cc @ianlancetaylor @aclements @rsc @randall77

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2021-05-20T04:01:36Z

The limitation on select is not a kernel limitation. It's a limitation on the implementation of fd_set in glibc. And we've inherited that limitation in x/sys/unix.FdSet, but perhaps we could fix that. If we did, then we could raise the soft limit to the hard limit unconditionally.

I note that on my Debian system the soft and hard limits are both 131072. On my CentOS 6 system the soft limit is 1024 and the hard limit is 4096. On my recent Fedora system the soft limit is 1024 and the hard limit is 524288.

bradfitz · 2021-05-20T04:32:21Z

Yeah, I saw FdSet was struct { Bits [16]int64 }. Make it opaque with a [16]int64 used by default and a spill-over lazily-allocated bitmap when (*FDSet).Set(fd int) is called with a "big" fd? Doable, but I wonder if it's worth the effort. Does anybody actually use unix.Select?

We'd still need a conditional mechanism regardless for cgo I assume, as we wouldn't know whether C code was using select as easily?

FWIW, on my various Debian (buster) & Ubuntu (focal LTS, hirsute) machines here, I see 1024 & 1048576.

bradfitz · 2021-05-20T04:36:24Z

Does anybody actually use unix.Select?

GitHub code search says https://github.com/search?l=&p=2&q=unix.Select+language%3AGo&type=Code .... it's mostly wireguard-go's rwcancel package.

(cc @zx2c4 as FYI)

zx2c4 · 2021-05-20T13:36:21Z

~~I'm happy to get rid of that and replace it with poll. (Want to send a patch?)~~ Done: https://git.zx2c4.com/wireguard-go/commit/?id=a9b377e9e10eb5194c0bdff32136c11b17253bfd

This proposal sounds like a good idea, with the caveat that we probably shouldn't do it in initialization for -buildmode=shared.

rsc · 2021-11-10T22:40:51Z

What happens today, even in programs that do nothing but file I/O (no select etc), is that if you open too many files you get errors. Auto-bumping would let those programs run longer.

If Go did it at startup, it would be inherited by non-Go programs that we fork+exec. That is a potential incompatibility, but probably not a large one. Technically, I suppose we could undo it in the subprocess between fork and exec.

rsc · 2021-11-10T23:00:23Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

nightlyone · 2021-11-11T06:04:29Z

To summarize the limitating use cases where we should not be raising the soft limit.

select implementation from glibc (or other libc like musl too?) is used
cgo is used because dynamic linking can load anything with dlopen and also can cause exec calls in places we don't know.
select implementation from our own syscalls or unix package is used, unless that one is changed as suggested above.
NSS (user/group lookup and DNS lookup) is used from glibc.
and we need to reset, if we call the exec family of syscalls

rsc · 2021-12-01T18:27:28Z

One problem with restoring the limit in exec is we won't know if the limit was intentionally changed by the program in the interim. What about programs that explicitly raise the limit and then exec today? We would be dropping it back down.

It seems like if we are going to raise the limit, we should just do that, not try to put it back.

I just ran into this problem with gofmt on my Mac, where the limit defaults to 256 (and gofmt was editing many files in parallel). I'd love for Go to raise the limit there too.

How much does it really matter if we raise the limit for a subprocess?

People can always set the hard limit if they want Go not to try to bump the soft limit up.

rsc · 2021-12-08T18:37:59Z

It's pretty awful that the limit is breaking completely reasonable Go programs like gofmt -w.
It seems very wrong for gofmt to have to put a bump in.
It seems like we should bump the limit at startup - Go doesn't use select.

It's very hard to see any programs benefiting from this limit in practice anymore.
I understand that systemd can't make such a global decision, but I think Go can.

zx2c4 · 2021-12-08T20:31:33Z

I think that seems quite reasonable.

We can even document this in unix.Select/syscall.Select and mark them as deprecated so that editors bring attention to them and maybe add something to go vet too. It seems always possible to move to poll or similar.

rsc · 2021-12-15T18:26:52Z

Not sure anyone is using syscall.Select for fd's anyway.
Every time I've used it in the past decade it has been to get a sub-second-resolution sleeping API (selecting on no fds).

rsc · 2021-12-15T18:48:07Z

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

AlekSi · 2021-12-15T19:47:56Z

Should the title be updated to mention Unix or something instead of Linux?
Personally, I constantly run into that limitation on macOS; would like to see that resolved.

ianlancetaylor · 2021-12-15T19:51:57Z

The considerations may be different on different Unix systems. On Linux the details are somewhat specific to systemd.

It may well be appropriate to do this on macOS also, but I don't know what the tradeoffs are there. Why does macOS have a default low limit?

AlekSi · 2021-12-15T20:08:02Z

From what I was able to find, that default goes back to the very first OS X release and probably even back to BSD. The constant is there.

Of course, not doing that on macOS is not a deal-breaker but an annoyance.

kolyshkin · 2021-12-16T02:20:48Z

The only issue I am aware of that can arise if RLIMIT_NOFILE is set to a very high value is, some binaries (that may be executed from a Go program and thus inherit the limit) want to do something like this (pseudocode):

for fd := 3; fd < getrlimit(RLIMIT_NOFILE); fd++ {
      close(fd) // or set CLOEXEC flag
}

For a specific example, rpm package manager used to do that (fixed by rpm-software-management/rpm@5e6f05c), and also some older version of Python (but I'm not sure).

Most probably this should not be an issue, since Docker also does a similar thing (moby/moby#38814) and since everyone seems to be using containers now, let's hope that issues like this are fixed (yet better, maybe some programs have even started using close_range()).

Also, this is surely not a showstopper to accept the proposal -- just something to keep in mind.

rsc · 2022-01-05T19:17:57Z

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

gopherbot · 2023-03-15T20:44:59Z

Backport issue(s) opened: #59063 (for 1.19), #59064 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

As of Go 1.21 syscall.Setrlimit and syscall.prlimit can affect starting a new process, by restoring the original NOFILE rlimit. That is recorded locally in the syscall package, so just always call the syscall functions. For golang/go#46279 Change-Id: I2f3dafe5562a7dde1297bad6f5d34a80af5d620b Reviewed-on: https://go-review.googlesource.com/c/sys/+/476695 Run-TryBot: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com>

gopherbot · 2023-03-22T20:21:06Z

Change https://go.dev/cl/478659 mentions this issue: [release-branch.go1.20] syscall: restore original NOFILE rlimit in child process

gopherbot · 2023-03-22T20:43:30Z

Change https://go.dev/cl/478660 mentions this issue: [release-branch.go1.19] syscall: restore original NOFILE rlimit in child process

…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For #46279 Fixes #59064 Change-Id: Ib813de896de0a5d28fa2b29afdf414a89fbe7b2a Reviewed-on: https://go-review.googlesource.com/c/go/+/478659 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>

…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For #46279 Fixes #59063 Change-Id: I9be6ecc52d4f82eb226907611aec9df808e1da84 Reviewed-on: https://go-review.googlesource.com/c/go/+/478660 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>

gopherbot · 2023-04-19T22:07:55Z

Change https://go.dev/cl/486576 mentions this issue: syscall: add prlimit

Now Go only sets the rlimit for the parent and any fork/exec'ed process gets the rlimit that was the default before fork/exec. Ref: golang/go#46279 This fix got backported to [Go 1.20.4](golang/go@ecf7e00) breaking Talos. Talos used to set rlimit in the [`SetRLimit`](https://github.com/siderolabs/talos/blob/v1.4.2/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L302) sequencer task. This means any process started by `wrapperd` gets the default Rlimit (1024). Fix this by explicitly setting `rlimit` in `wrapperd` before we drop any capabilities. Fixes: siderolabs#7198 Signed-off-by: Noel Georgi <git@frezbo.dev>

Now Go only sets the rlimit for the parent and any fork/exec'ed process gets the rlimit that was the default before fork/exec. Ref: golang/go#46279 This fix got backported to [Go 1.20.4](golang/go@ecf7e00) breaking Talos. Talos used to set rlimit in the [`SetRLimit`](https://github.com/siderolabs/talos/blob/v1.4.2/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L302) sequencer task. This means any process started by `wrapperd` gets the default Rlimit (1024). Fix this by explicitly setting `rlimit` in `wrapperd` before we drop any capabilities. Fixes: siderolabs#7198 Signed-off-by: Noel Georgi <git@frezbo.dev> (cherry picked from commit a2565f6)

As of https://go.dev/cl/476695 golang.org/x/sys/unix can call syscall.prlimit, so we need such a function in libgo. For golang/go#46279 Fixes golang/go#59712 Change-Id: I87ad6daaba68c188fb0abecb30f7d574db1f2600 Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/486576 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Than McIntosh <thanm@google.com>

As of https://go.dev/cl/476695 golang.org/x/sys/unix can call syscall.prlimit, so we need such a function in libgo. For golang/go#46279 Fixes golang/go#59712 Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/486576

…ild process If we increased the NOFILE rlimit when starting the program, restore the original rlimit when forking a child process. In CL 393354 the os package was changed to raise the open file rlimit at program start. That code is not inherently tied to the os package. This CL moves it into the syscall package. This is a backport of CLs 476096 and 476097 from trunk. For golang#46279 Fixes golang#59064 Change-Id: Ib813de896de0a5d28fa2b29afdf414a89fbe7b2a Reviewed-on: https://go-review.googlesource.com/c/go/+/478659 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>

Stop managing limitnofile, Bookworm systemd default is soft 1024 and hard 524288. Further, golang >= 1.19 'os' package bumps soft to hard limit automatically (cfr golang/go#46279) For example thanos-query-frontend where we don't explicitly set limitnofile: root@titan2001:~# cat /proc/$(pgrep -f "thanos query-frontend")/limits | grep files Max open files 524288 524288 files root@titan2001:~# systemctl show thanos-query-frontend | grep -i limitnofile LimitNOFILE=524288 LimitNOFILESoft=1024 Bug: T346950 Change-Id: I3df68d63c66293e4425cb7e67670e44459c4d474

gopherbot · 2023-11-02T18:54:56Z

Change https://go.dev/cl/539435 mentions this issue: syscall: copy rlimit.go's build constraint to rlimit_test.go

Tests in rlimit_test.go exist to test the behavior of automatically bumping RLIMIT_NOFILE on Unix implemented in rlimit.go (issue #46279), with darwin-specific behavior split out into rlimit_darwin.go and the rest left empty in rlimit_stub.go. Since the behavior happens only on Unix, it doesn't make sense to test it on other platforms. Copy rlimit.go's 'unix' build constraint to rlimit_test.go to accomplish that. Also simplify the build constraint in rlimit_stub.go while here, so that its maintenance is easier and it starts to match all non-darwin Unix GOOS values (previously, 'hurd' happened to be missed). In particular, this fixes a problem where TestOpenFileLimit was failing in some environments when testing the wasip1/wasm port. The RLIMIT_NOFILE bumping behavior isn't implemented there, so the test was testing the environment and not the Go project. Updates #46279. For #61116. Change-Id: Ic993f9cfc021d4cda4fe3d7fed8e2e180f78a2ca Reviewed-on: https://go-review.googlesource.com/c/go/+/539435 Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

gopherbot · 2023-11-07T19:32:09Z

Change https://go.dev/cl/540615 mentions this issue: [release-branch.go1.21] syscall: copy rlimit.go's build constraint to rlimit_test.go

… rlimit_test.go Tests in rlimit_test.go exist to test the behavior of automatically bumping RLIMIT_NOFILE on Unix implemented in rlimit.go (issue #46279), with darwin-specific behavior split out into rlimit_darwin.go and the rest left empty in rlimit_stub.go. Since the behavior happens only on Unix, it doesn't make sense to test it on other platforms. Copy rlimit.go's 'unix' build constraint to rlimit_test.go to accomplish that. Leave out the simplification of the build constraint in rlimit_stub.go so that this CL remains a test-only fix. In particular, this fixes a problem where TestOpenFileLimit was failing in some environments when testing the wasip1/wasm port. The RLIMIT_NOFILE bumping behavior isn't implemented there, so the test was testing the environment and not the Go project. Updates #46279. For #61116. Fixes #63994. Change-Id: Ic993f9cfc021d4cda4fe3d7fed8e2e180f78a2ca Cq-Include-Trybots: luci.golang.try:go1.21-wasip1-wasm_wasmtime Reviewed-on: https://go-review.googlesource.com/c/go/+/539435 Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> (cherry picked from commit b7cbcf0) Reviewed-on: https://go-review.googlesource.com/c/go/+/540615 Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Heschi Kreinick <heschi@google.com>

bradfitz added OS-Linux Proposal labels May 20, 2021

gopherbot added this to the Proposal milestone May 20, 2021

ianlancetaylor added this to Incoming in Proposals (old) May 20, 2021

komuw mentioned this issue May 25, 2021

bump the RLIMIT_NOFILE soft limit to the hard limit komuw/ong#2

Closed

rsc moved this from Incoming to Active in Proposals (old) Nov 10, 2021

rsc moved this from Active to Likely Accept in Proposals (old) Dec 15, 2021

rsc added the Proposal-FinalCommentPeriod label Dec 15, 2021

rsc changed the title ~~proposal: runtime: automatically bump RLIMIT_NOFILE on Linux?~~ proposal: runtime: automatically bump RLIMIT_NOFILE on Unix Jan 5, 2022

rsc moved this from Likely Accept to Accepted in Proposals (old) Jan 5, 2022

rsc changed the title ~~proposal: runtime: automatically bump RLIMIT_NOFILE on Unix~~ runtime: automatically bump RLIMIT_NOFILE on Unix Jan 5, 2022

rsc added the Proposal-Accepted label Jan 5, 2022

rsc removed this from the Proposal milestone Jan 5, 2022

This was referenced Mar 15, 2023

runtime: automatically bump RLIMIT_NOFILE on Unix [1.19 backport] #59063

Closed

runtime: automatically bump RLIMIT_NOFILE on Unix [1.20 backport] #59064

Closed

frezbo mentioned this issue May 9, 2023

NO_FILE ulimit decreases from 1048576 to 1024 when upgrading to v1.4.2 siderolabs/talos#7198

Closed

frezbo mentioned this issue May 9, 2023

fix: set rlimit explicitly in wrapperd siderolabs/talos#7199

Merged

zhsj mentioned this issue Jul 28, 2023

TestGetconf fails with OPEN_MAX tklauser/go-sysconf#77

Closed

MichaelEischer mentioned this issue Aug 5, 2023

Restrict rest-server.service SystemCallFilter (disallow @resources and @privileged) restic/rest-server#249

Closed

7 tasks

dmitshur mentioned this issue Nov 7, 2023

syscall: TestOpenFileLimit unintentionally runs on non-Unix platforms #63993

Closed

cartermckinnon mentioned this issue Dec 22, 2023

File descriptor limit change in AMI release v20231220 awslabs/amazon-eks-ami#1551

Closed

thaJeztah mentioned this issue Dec 22, 2023

Remove LimitNOFILE from containerd.service containerd/containerd#8924

Merged

howardjohn mentioned this issue Mar 6, 2024

fix warnings getting fd limit in distroless istio/istio#49761

Merged

ianlancetaylor mentioned this issue Apr 12, 2024

syscall: caching RLIMIT_NOFILE is wrong implementation #66797

Open

fancycode mentioned this issue Apr 18, 2024

'Failed to start: core-dump' after upgrade strukturag/nextcloud-spreed-signaling#615

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: automatically bump RLIMIT_NOFILE on Unix #46279

runtime: automatically bump RLIMIT_NOFILE on Unix #46279

bradfitz commented May 20, 2021 •

edited

ianlancetaylor commented May 20, 2021

bradfitz commented May 20, 2021 •

edited

bradfitz commented May 20, 2021 •

edited

zx2c4 commented May 20, 2021 •

edited

rsc commented Nov 10, 2021

rsc commented Nov 10, 2021

nightlyone commented Nov 11, 2021

rsc commented Dec 1, 2021

rsc commented Dec 8, 2021

zx2c4 commented Dec 8, 2021

rsc commented Dec 15, 2021

rsc commented Dec 15, 2021

AlekSi commented Dec 15, 2021

ianlancetaylor commented Dec 15, 2021

AlekSi commented Dec 15, 2021

kolyshkin commented Dec 16, 2021

rsc commented Jan 5, 2022

gopherbot commented Mar 15, 2023

gopherbot commented Mar 22, 2023

gopherbot commented Mar 22, 2023

gopherbot commented Apr 19, 2023

gopherbot commented Nov 2, 2023

gopherbot commented Nov 7, 2023

runtime: automatically bump RLIMIT_NOFILE on Unix #46279

runtime: automatically bump RLIMIT_NOFILE on Unix #46279

Comments

bradfitz commented May 20, 2021 • edited

ianlancetaylor commented May 20, 2021

bradfitz commented May 20, 2021 • edited

bradfitz commented May 20, 2021 • edited

zx2c4 commented May 20, 2021 • edited

rsc commented Nov 10, 2021

rsc commented Nov 10, 2021

nightlyone commented Nov 11, 2021

rsc commented Dec 1, 2021

rsc commented Dec 8, 2021

zx2c4 commented Dec 8, 2021

rsc commented Dec 15, 2021

rsc commented Dec 15, 2021

AlekSi commented Dec 15, 2021

ianlancetaylor commented Dec 15, 2021

AlekSi commented Dec 15, 2021

kolyshkin commented Dec 16, 2021

rsc commented Jan 5, 2022

gopherbot commented Mar 15, 2023

gopherbot commented Mar 22, 2023

gopherbot commented Mar 22, 2023

gopherbot commented Apr 19, 2023

gopherbot commented Nov 2, 2023

gopherbot commented Nov 7, 2023

bradfitz commented May 20, 2021 •

edited

bradfitz commented May 20, 2021 •

edited

bradfitz commented May 20, 2021 •

edited

zx2c4 commented May 20, 2021 •

edited