Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/covdata: traverse subdirectories of those passed via the -i argument #63275

Closed
thesilentg opened this issue Sep 28, 2023 · 6 comments
Labels
Milestone

Comments

@thesilentg
Copy link

Currently a number of the subcommands of go tool covdata (textfmt, percent, ...) require users to pass the-i argument which specifies the input dirs to examine (comma separated) for coverage data. Only the files in the named directory/directories are examined, and no subdirectories are visited. This lack of subdirectory traversal leads to some (admittedly minor) inconveniences for users of this command.

Example 1: Multiple static locations, unchanging

/ coveragedata 
    / unittests
        <coverage files from the unit test suite go here>
    / integrationtests
        <coverage files from the integration test suite go here>

Users may want to examine the unit test coverage (go tool covdata func -i=coveragedata/unittests), the integration test coverage (go tool covdata func -i=coveragedata/integrationtests), or the combined coverage. Today, getting the combined coverage requires either manually specifying both directories as part of the func subcommand (go tool covdata func -i=coveragedata/unittests,coveragedata/integrationtests), or by manually specifying both directories as part of the merge subcommand (go tool covdata merge -i=coveragedata/unittests,coveragedata/integrationtests -o merged) and then later running the func command on the merged result (go tool covdata func -i=merged).

It would be easier to allow for -i to traverse the passed directories (including any subdirectories), such that users wishing to view coverage data for both could simply run go tool covdata func -i=coveragedata or go tool covdata merge -i=coveragedata -o merged && go tool covdata func -i=merged. However, for two static directory locations which don't change, this represents a very minor papercut and can easily be worked around with existing options.

Example 2: Multiple static locations, changing

/ coveragedata 
    / windows
    / macos
    / linux 
    ... 

The testing coverage docs specifically reference an example of:

For example, consider a program that runs on both macOS and on Windows. The author of this program might want to combine coverage profiles from separate runs on each operating system into a single profile corpus, so as to produce a cross-platform coverage summary.

In practice, this could become a bit unwieldily, as the every time a new platform is added for testing the call to the covdata command would have to be updated to manually pass in this new directory location. Updating the coverage command may be forgotten, silently missing being included in top level coverage. If directory traversal was included, this command could again be simplified to go tool covdata func -i=coveragedata eliminating the risk of new platforms being missed. Again, the current tools provide the necessary workaround (-i=coveragedata/windows,coveragedata/macos, ...), and the behavior still remains a minor papercut.

Example 3: Dynamic Locations

/ coveragedata 
    / us-windows-paypal
    / us-macos-stripe
    / ca-windows-paypal
    / ca-macos-stripe
    / es-windows-paypal
    / es-macos-stripe
    ... 

Consider a hypothetical company which handles user purchases which wants to run its integration test suite for a number of countries (EN / CA / ES / ...), platforms (windows, macos, ...), and payment vendors (paypal, stripe, ...) to measure overall coverage for all supported combinations of purchase flows. This generates a dynamic number of directory locations, which all need to be fed into the -i argument. This can certainly be accomplished with the current tooling, but would likely be much simpler (and less error-prone) if they could just specify go tool covdata func -i=coveragedata and avoid having to dynamically build a list of every possible directory location.

The proposed change would be to update this to instead call filepath.WalkDir , with no other changes to the parsing logic.

@gopherbot gopherbot added this to the Proposal milestone Sep 28, 2023
@seankhliao seankhliao changed the title proposal: coverage: traverse subdirectories of those passed via the -i argument proposal: cmd/covdata: traverse subdirectories of those passed via the -i argument Sep 28, 2023
@ianlancetaylor
Copy link
Contributor

CC @thanm

@thesilentg
Copy link
Author

thesilentg commented Sep 28, 2023

Thanks for the prompt intake on this proposal! I also wanted to clarify that if accepted, I'd be willing to author the CR for this

@thanm
Copy link
Contributor

thanm commented Sep 28, 2023

It seems to me that this could be avoided easily with a very minimal amount of scripting, e.g.

$ find . -name "covmeta.*" -print
...
./main/covdata.amd64/covmeta.b245ad845b5068d116a4e25033b429fb
./main/covdata.386/covmeta.f3833f80c91d8229544b25a855285890
./main/covdata.arm64/covmeta.d7050703cbff64bbf8712eae417d5853
...
$ find . -name "covmeta.*" -print | xargs dirname | sort -u > dirs.txt
$ cat dirs.txt
./main/covdata.amd64
./main/covdata.386
./main/covdata.arm64
$ go tool covdata merge  -o merged -i=`cat dirs.txt | paste -sd ","`
$

something along those lines.

@thesilentg
Copy link
Author

thesilentg commented Sep 28, 2023

It seems to me that this could be avoided easily with a very minimal amount of scripting, e.g.

Absolutely, which is why I mentioned this is more a minor papercut. It is very much a solvable problem with the existing tools, and more of a nit with regards to developer ease-of-use.

Are you mostly working backwards from keeping the surface area / scope of behavior for go tool covdata as simple as possible, or are there technical concerns you have with regards to directory traversal? You know much better than I the internals of how these coverage profiles are managed / merged / read, so looking to understand more about what could go wrong here if traversal was permitted. Thanks!

@thesilentg
Copy link
Author

I'll leave another potential developer mistake for consideration below:

/ coveragedata 
    covmeta.b245ad845b5068d116a4e25033b429fb
    covmeta.f3833f80c91d8229544b25a855285890
   / amd64
      <more profiles>
   / arm64
      <more profiles>

In this case, some general profiles have been written to the top level directory, and other platform-specific profiles to the platform-specific subdirectories below. If the developer runs go tool covdata func -i=coveragedata it will report success (since it reads the profiles in /coveragedata), but gives no indication to them that the profiles inside of / amd64 or /arm64 were (mistakenly) excluded. In contrast, if the top level profiles are removed and the go tool covdata func command is rerun, it will instead report warning: no applicable files found in input directories, which gives the user an indication that something is incorrect.

@thesilentg
Copy link
Author

Closing this as I'm fine with the current behavior upon deeper reflection.

@thesilentg thesilentg closed this as not planned Won't fix, can't repro, duplicate, stale Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants