Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/gorename: too many open files #10306

Closed
anacrolix opened this issue Apr 1, 2015 · 11 comments
Closed

cmd/gorename: too many open files #10306

anacrolix opened this issue Apr 1, 2015 · 11 comments

Comments

@anacrolix
Copy link
Contributor

gorename frequently aborts with the error "too many open files". It doesn't always happen. That then triggers a long cascade of failures of "undeclared name".

open /Users/anacrolix/src/go/src/net/ipsock.go: too many open files
open /Users/anacrolix/src/go/src/net/ipsock_posix.go: too many open files
/Users/anacrolix/src/go/src/net/sock_posix.go:20:2: undeclared name: netaddr
@anacrolix
Copy link
Contributor Author

This seems relevant, it's the default on my system:

$ ulimit -a
open files                      (-n) 256

@adonovan
Copy link
Member

adonovan commented Apr 1, 2015

Thanks for the report. I thought I had fixed this already. Will take a look tomorrow.

@anacrolix
Copy link
Contributor Author

I was on revision 473fd85 of go.tools

@mikioh mikioh changed the title gorename: too many open files cmd/gorename: too many open files Apr 2, 2015
@alandonovan
Copy link
Contributor

I was unable to reproduce the problem by setting ulimit -n; I only noticed problems when the limit was so low that it couldn't even get started. But this CL should reduce the concurrency of file I/O:

https://go-review.googlesource.com/#/c/8421/

(The earlier fix I mentioned was for a different parallel I/O operation in the same tool.)

Could you try patching this change to see if it helps? Thanks.

@bakineggs
Copy link

I ran into this issue a lot last week when using errcheck, so I submitted CL #8475 as a fix. I didn't realize there was an open issue and a different fix until I saw the conflicting change listed in Gerrit.

On a project I work on, errcheck fails with this error the majority of the time using the master revision of go/loader. It does not fail with this error using the branch from CL 8421 or CL 8475 (although with CL 8421, it should still be possible to have this error occur in conf.FindPackage).

alandonovan pushed a commit to golang/tools that referenced this issue Apr 6, 2015
This is a putative fix for the file descriptor exhaustion problem
described in golang/go#10306.

Change-Id: If603fb9bbaec1b53f6b44d15b2c202e4670035ce
Reviewed-on: https://go-review.googlesource.com/8421
Reviewed-by: Matt Joiner <anacrolix@gmail.com>
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@alandonovan
Copy link
Contributor

I agree that it's still possible in principle for the error to occur. (It's an unfortunate consequence of ignoring the possibility of I/O failures in functions like (*go/build.Context) isDir.) But I think the CL 8421 fix should eliminate it in practice; I suggest you revert CL #8475.

If the problem recurs, we'll revisit that change.

@pmezard
Copy link
Contributor

pmezard commented Oct 18, 2015

FWIW, I reproduced it running go/master db2e73f, with ulimit -n == 256. The renaming happened but a lot of "too many open files" errors were displayed.

@anacrolix
Copy link
Contributor Author

I still get it on occasion with go1.5.

@anacrolix
Copy link
Contributor Author

anacrolix@Matts-MacBook-Pro:~/tmp$ gorename -from '"github.com/anacrolix/torrent".torrent.DisplayName' -to displayName
While scanning Go workspace:
Package "github.com/mattn/go-gtk/example/expander": open /Users/anacrolix/gopath/src/github.com/mattn/go-gtk/example/expander/expander.go: too many open files.
Renamed 3 occurrences in 2 files in 1 package.

@mbertschler
Copy link

I also have this problem often, but not every time I run it.
go version go1.5.1 darwin/amd64

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18050 mentions this issue.

alandonovan pushed a commit to golang/tools that referenced this issue Dec 18, 2015
Each string in build.Package.{,Test,XTest}Imports must now be
interpreted relative to the Package.Dir.  This adds a ton of
redundant I/O.

Also:
- use a counting semaphore to limit concurrent I/O calls
  (Fixes golang/go#10306)
- remove obsolete call to runtime.GOMAXPROCS from the test.

Change-Id: Ic556c4cf41cce7a88c0158800c992e66f354c484
Reviewed-on: https://go-review.googlesource.com/18050
Reviewed-by: Russ Cox <rsc@golang.org>
@golang golang locked and limited conversation to collaborators Dec 29, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants