Value Metric Updates - #181
Conversation
- Updated docs/experiments.mdx Mintlify-Source: dashboard-editor
Confidence Score: 4/5The PR is not ready to merge because the Value Metric denominator guidance remains internally contradictory. The page simultaneously says lift and group-identifier calculations always normalize by exposures while the revised Value Metric section says group rates and variance use only users with observed values. Files Needing Attention: docs/experiments.mdx
|
| Filename | Overview |
|---|---|
| docs/experiments.mdx | Updates Value Metric denominator guidance, but the general lift and group-identifier notes still contradict the new exception. |
Reviews (2): Last reviewed commit: "Merge branch 'main' into mb-experiments-..." | Re-trigger Greptile
![greptile-apps[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| | ------------ | ---------------- | | ||
| | Small (< 100k) | 50-100% | | ||
|
|
||
| | **MTU bucket** | \*\*Estimated MEU (% MTU) \*\* | |
There was a problem hiding this comment.
The escaped asterisks render as literal ** characters rather than bolding the second heading, leaving the MEU table header visibly inconsistent with the first column.
| | **MTU bucket** | \*\*Estimated MEU (% MTU) \*\* | | |
| | **MTU bucket** | **Estimated MEU (% MTU)** | |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| - **Variance:** Calculated using Bernoulli distribution: p × (1-p) | ||
| - **Example:** If 25 out of 100 exposed users convert, group rate = 0.25 (25%). Note that this denominator is exposures — not funnel entrants. This is different from the funnel conversion rate you see in the Totals or Chart tabs, which uses funnel entrants as the denominator. | ||
|
|
||
| **Value Metrics (Averages, Sums):** |
There was a problem hiding this comment.
The Value Metrics section is clearer, but could we add one short sentence defining what counts as an “observed value”? Readers may not know whether that means only users who triggered the underlying event, and it would help to explicitly say how missing or zero values are handled.
| - **Variance:** Calculated from the distribution of individual property values, over that same observed-value population | ||
| - **Example:** If the treatment group spent \$5,000 total from 20 users who had a value (out of 100 exposed), the group rate = \$250 average per user with an observed value | ||
|
|
||
| **Why This Matters:** Value metrics report the average based only on users who had an observed value for the metric, not all exposed users, since there's no natural zero for someone who never triggered the underlying event. The exposed count still matters; it's used downstream in the standard error and confidence interval calculation, so overall reach still factors into how confident you can be in the result, just not into the group rate or variance themselves. |
There was a problem hiding this comment.
This is a helpful clarification, but it may still read as if exposures are no longer part of the Value Metrics calculation at all. Could we make the distinction explicit that exposures are excluded from the group rate/variance, but still used in downstream standard error and confidence interval calculations?
|
|
||
| Lift is the percentage difference between the control and variant(s) metrics. | ||
| $Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}$ | ||
| Lift is the percentage difference between the control and variant(s) metrics. $$Lift= { (variant \,group\,rate - control \,group\,rate) \over (control \,group\,rate)}$$ |
There was a problem hiding this comment.
The lift explanation is more explicit now, but the formula and the metric-type-specific denominator rules are spread across several paragraphs. Could we consolidate this a bit so readers don’t miss that lift behaves differently depending on the metric type?
| @@ -384,9 +385,10 @@ Benjamini-Hochberg controls the **false discovery rate**, the share of your flag | |||
| This correction operates on the values in your breakdown, which is a different axis from the [Multiple Testing Correction](#bonferroni-correction) you apply across metrics and variants. That means adding more breakdown values doesn't inflate your false positive rate. | |||
|
|
|||
| ### When do we say the Experiment is ready to review? | |||
There was a problem hiding this comment.
“Experiment is ready to review” could be interpreted as statistically ready, not just setup-complete. Could we clarify that this banner reflects configuration completion rather than a recommendation to end the experiment?
myronkaifung
left a comment
myronkaifung
left a comment
There was a problem hiding this comment.
Looks good formatting wise. @mbocianski feel free to merge after reviewing Elliot's comment.
3 participants
See Slack thread for details.
https://mixpanel.slack.com/archives/C0547DQTVCN/p1786744670689369
Mintlify
0 threads from 0 users in Mintlify