New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: filepath.Walk panics on huge tree traversal #15615
Comments
Let's keep this bug focused on the crash (which should not happen) and not alternate implementations of filepath.Walk. That has been discussed elsewhere, IIRC. Feel free to open a separate bug for that and we can de-dup if necessary. |
Which version of Windows are you using? Please run with /cc @alexbrainman (for Windows) |
Windows 10, 16 GB of RAM. Here's the full traceback:
|
Thanks. 17-18 frames isn't very deep. |
I also updated the sample code — it's actually the |
Thanks for the update. For future reference, try not to update previous comments, as it makes following threads difficult. In this case it's probably, since it's already been done, and the thread is new. |
OK, that's expected then. callback is called with a nil info and non-nil error. Closing, please comment if you disagree. |
Indeed, you can see the nil os.FileInfo here and the non-nil error:
(The first two are the path, the next two are the FileInfo, and the 5th/6th are the non-nil error) |
Indeed, the culprit was that for certain paths which are more than 255 characters long (and this is a path size limit in non-Unicode Windows API) there was an error reported: A solution (for others who stumble into this) would be to prepend the passed path with filepath.Walk("\\\\?\\C:\\path\\to\\some\\dir") This will make sure that |
#3358 strikes again. |
I bumped into this bug also, on windows 7, with go 1.6, as well as go 1.7.3. |
The fix for this (231aa9d) will be in Go 1.8. |
My environment:
Test script:
When I run this script against a directory that contains ~150K folders and ~1M files (the actual folder I tried this on contained about 50 big project checkouts with pretty deep tree structure), the program works for several seconds, then crashes:
panic: runtime error: invalid memory address or nil pointer dereference
Experiment 1: if I comment out the
if info.IsDir() { ... }
clause (replace it with simplyfolderCount++
), there will be no crash.Experiment 2: if I replace the
filepath.Walk
with my own implementation of this function that doesn't use recursion, the program won't crash either.So my current understanding is that this happens due to a combination of a recursive algorithm (which might take much stack/memory) and some specific code inside a callback.
A related question: did you consider getting rid of the recursive algorithm in
filepath.Walk
? I experimented with an alternative implementation of this function that uses goroutines/workers, and it seems not only to work correctly on huge directory trees, but also gives performance boost due to parallelism.The text was updated successfully, but these errors were encountered: