Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/pkgsite: experiment rollout imprecise and unstable #40314

Closed
abursavich opened this issue Jul 20, 2020 · 1 comment
Closed

x/pkgsite: experiment rollout imprecise and unstable #40314

abursavich opened this issue Jul 20, 2020 · 1 comment
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Milestone
Unreleased

Comments

@abursavich
Copy link

What is the URL of the page with the issue?

N/A

What is your user agent?

N/A

Screenshot

N/A

What did you do?

Read the experiments code.

What did you expect to see?

Experiments that rollout with precision and per user stability.

What did you see instead?

The chosen arithmetic uint(h.Sum32())%(100/e.Rollout) == 0 has two problems.

  • The integer division of 100/e.Rollout greatly reduces the precision available (consider that e.Rollout = 51 actually results in 100% rollout).
  • Varying the modulus divisor with the rollout threshold shuffles users in and out of an experiment rather than making a user enter and remain in the experiment once at a given threshold.
@gopherbot gopherbot added this to the Unreleased milestone Jul 20, 2020
@gopherbot
Copy link

Change https://golang.org/cl/243659 mentions this issue: internal/middleware: fix experiment assignment precision and stability

@julieqiu julieqiu added pkgsite NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 20, 2020
@golang golang locked and limited conversation to collaborators Jul 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@abursavich @julieqiu @gopherbot