go clean currently removes binaries and certain legacy files from the current directory or specified packages, plus any additional caches as indicated by the -cache, -testcache, and -modcache flags.

However, the user may want to clean the caches from outside of any package directory. The resulting package-loading behavior leads to confusion and delay (#26432, #28680, and perhaps others).

To avoid that confusion, we should make go clean only clean the current directory if either some package is specified as an explicit argument to go clean or the -*cache flags are not set.

(This won't fix #28680 completely, since it will still trigger a fetch if an explicit package argument is given, but it will at least address the common case.)