Skip to content

x/tools/cmd/godoc: ignored "too many open files" errors can cause incomplete /pkg/ subdirectories listings #14252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmitshur opened this issue Feb 8, 2016 · 6 comments
Closed

x/tools/cmd/godoc: ignored "too many open files" errors can cause incomplete /pkg/ subdirectories listings #14252

dmitshur opened this issue Feb 8, 2016 · 6 comments
Labels
FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.10

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Feb 8, 2016

What version of Go are you using (go version)?

go version go1.6rc2 darwin/amd64

What operating system and processor architecture are you using?

OS X 10.11.3 (64-bit).

What did you do?

  1. Have a GOPATH workspace with many, many packages.
  2. Run godoc -http=localhost:6060.
  3. Visit http://localhost:6060/pkg/crypto/ (it's used as an example, it affects all packages equally).

What did you expect to see?

A complete list of subdirectories, with their package synopses.

image

What did you see instead?

Randomly incomplete listing of subdirectories, and some missing package synopses.

image

image
@dmitshur
Copy link
Contributor Author

dmitshur commented Feb 8, 2016

I've looked into why it happens. At first I thought it might be some race condition, but I've learned it's not.

Instead, the problem is here:

https://github.com/golang/tools/blob/fe74a4186116b8d7dd38a723993e0d84f8834b34/godoc/dirtrees.go#L107-L127

Because all GOROOT and GOPATH workspaces are walked in parallel (likely with some concurrency limits), it may run into "too many files open" limits and erroring out. The linked code segment ignores all situations where parsing the .go files results in errors (there's no if err != nil { ... } section).

I've added a simple else if err != nil { log.Println(err) } section there and I saw many "too many files open" errors occur both on .go files in GOROOT as well as in GOPATH during my test.

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Feb 8, 2016
@rsc rsc modified the milestones: Go1.8, Go1.7 May 18, 2016
@quentinmit quentinmit added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 10, 2016
@rsc rsc modified the milestones: Go1.9, Go1.8 Nov 11, 2016
@bradfitz
Copy link
Contributor

bradfitz commented Jul 6, 2017

@adg improved this in https://go-review.googlesource.com/24060

I'll send a bit more.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/47651 mentions this issue.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jul 6, 2017
gopherbot pushed a commit to golang/tools that referenced this issue Jul 6, 2017
@bradfitz
godoc: include dir listing in rate limit, log dir reading errors
72ed06f 
Updates golang/go#14252

Change-Id: I3ce2a410708900f5c80d15e722b410855fd9c571
Reviewed-on: https://go-review.googlesource.com/47651
Reviewed-by: Dmitri Shuralyov <shurcool@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@kevinburke
Copy link
Contributor

kevinburke commented Oct 2, 2017 •
edited
Loading

I think this has been addressed. At the very least, I have a pretty large source tree (every Github checkout I do in any language is in GOPATH) and I've never seen this issue, nor a godoc server with randomly missing files.

@kevinburke
Copy link
Contributor

Closing, feel free to reopen if this is still a problem, or there's a way to reproduce on tip.

@dmitshur
Copy link
Contributor Author

dmitshur commented Oct 2, 2017 •
edited
Loading

I suspect that the 2 commits to help this issue (golang/tools@a2a5522 and golang/tools@72ed06f) were sufficient to resolve it. If I ever run into this again, I'll post an update.

@golang golang locked and limited conversation to collaborators Oct 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.10
Development

No branches or pull requests
7 participants
@bradfitz @rsc @quentinmit @kevinburke @dmitshur @ianlancetaylor @gopherbot