Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/telemetry: add garbage collection #63884

Open
findleyr opened this issue Nov 1, 2023 · 1 comment
Open

x/telemetry: add garbage collection #63884

findleyr opened this issue Nov 1, 2023 · 1 comment
Labels
NeedsFix The path to resolution is known, but the work has not been done. telemetry x/telemetry issues
Milestone
Unreleased

Comments

@findleyr
Copy link
Contributor

findleyr commented Nov 1, 2023

An interesting point that came up in the context of discussing #63832 is that in some organizations data that is persisted "forever" is treated differently from data that is transient. Right now, local telemetry data is persisted until it is manually deleted (see also #63142).

We don't need local data forever: it is only needed for (1) weekly uploading, if telemetry uploading is enabled, (2) local viewing, to see what would be uploaded while making the decision to opt in, and (3) local debugging. For all of those use cases, keeping a few months of recent data is sufficient.

We should therefore garbage collect old data by default.
@gopherbot gopherbot added the telemetry x/telemetry issues label Nov 1, 2023
@gopherbot gopherbot added this to the Unreleased milestone Nov 1, 2023
@cherrymui cherrymui added the NeedsFix The path to resolution is known, but the work has not been done. label Nov 2, 2023
@hyangah
Copy link
Contributor

hyangah commented Nov 3, 2023

Another interesting point from #63832 (comment)

This data is necessary for the operation of the tool. It is also stored in a directory that is designed to hold ephemeral data, so it can be removed without issue and excluded by name from backups.

  • go or gopls's build caches are usually os.UserCacheDir(). They are ephemeral so 'cache' makes sense.
  • the default locations for GOMODCACHE, checksum data, cached vulndb info, GOPATH are under os.UserHomeDir. go offers env vars to override some of them.
  • On the other hand, the collected counters and processed json files are stored under os.UserConfigDir().

I wonder if we also need to consider a different location or configuration knob for collected data, or if just garbage collection is sufficient.

I think telemetry metadata files like mode and weekends fit well to the purpose of os.UserConfigDir(), but I am not sure about the collected data. Should we eventually consider to move them to other place outside the config dir?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsFix The path to resolution is known, but the work has not been done. telemetry x/telemetry issues
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@hyangah @gopherbot @cherrymui @findleyr