Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools: unify directory scanning code between gorename, goimports, etc? #16427

Closed
bradfitz opened this issue Jul 19, 2016 · 3 comments
Closed

x/tools: unify directory scanning code between gorename, goimports, etc? #16427

bradfitz opened this issue Jul 19, 2016 · 3 comments
Labels
FrozenDueToAge Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Jul 19, 2016 •
edited

goimports has its own optimized directory scanning code.

Reportedly, gorename is quite a bit slower.

They should probably share code.

/cc @alandonovan
@bradfitz bradfitz added this to the Unreleased milestone Jul 19, 2016
@bradfitz bradfitz mentioned this issue Jul 19, 2016
Closed
@thomasf
Copy link

thomasf commented Jul 19, 2016 •
edited

It does take up to a minute for me to run gorename and the default output does
complain about some things as well. This is maybe not the same issue as the
general slowness one but it's maybe related.

I have a bunch of gcc sources and builds here and there under my src/GOPATH
which seems to confuse gorename a bit, maybe it has something todo with the
fact that the gcc sources includes the go compiler and tests (?)

A normal run of gorename spits out somewhere around a thousand lines like the
ones below this while running, there are also some errors about the tests of my
Go repository checkout which I have under src for reference. The gorename -v
argument doesn't seem to add any interesting additional information to let me
know what's going on.

Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/complex.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/bin_search_tree_/bin_search_tree_.hpp: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/parallel/algo.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/backward/auto_ptr.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/thumb2/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/armv6-m/libstdc++-v3/include/arm-none-eabi/bits/atomic_word.h: no such file or directory.
Package "github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_": open /home/thomasf/src/github.com/jsnyder/arm-eabi-toolchain/build/gcc-final/arm-none-eabi/libstdc++-v3/include/ext/pb_ds/detail/binary_heap_/binary_heap_.hpp: no such file or directory.
...

EDIT: I temporarily moved the three gcc's which gorename was complaining about
out of my src directory and gorename finished a good portion of seconds earlier.

My ~/src has 1800000 files in total with each GCC installation having
up to ~19000 files each so they do represent a sizable portion of all my files in ~src (~30%).

(I'll revisit these numbers after I've had some sleep, a bit too tired for this atm)

@haya14busa
Copy link
Contributor

haya14busa commented Sep 6, 2016 •
edited

Hi! Thank you for maintaining go and making goimports fast. I really like it 💓

For what it's worth, I created https://github.com/haya14busa/gopkgs
which just list all packages by using the same implementation as goimports.

In my environment ($GOPATH/src has 725045 files), go list ... takes about 10 seconds to list packages, but
gopkgs command with .goimportsignore takes under 1 sec.

go list ... is slow but gopkgs is fast enough for daily use and can use it with filtering tool or something (if you are interested in some usages, please see README).
I'm very happy with it.

I implemented gopkgs by coping and modifing https://godoc.org/golang.org/x/tools/imports to export some interface,
but it's not clever way.

So I'm really +1 for this issue.

I think it might be great to use the scanning code in go list as well if possible and
it's also great to provide the code as a public library.

Thanks

@thomasf thomasf mentioned this issue Aug 18, 2017
Open
@agnivade agnivade mentioned this issue Feb 28, 2019
Closed
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019
@heschi
Copy link
Contributor

heschi commented Nov 7, 2019

I extracted gopathwalk out from goimports, so it's available for use if we want it. But with gopls taking over, and modules making directory scanning very very complicated, I doubt this is going to go much further.

@heschi heschi closed this as completed Nov 7, 2019
@golang golang locked and limited conversation to collaborators Nov 6, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @thomasf @haya14busa @gopherbot @heschi