Skip to content

Document Bayesian model as closed beta, fix broken #bayesian anchor #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
russell-loube-mixpanel wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Document Bayesian model as closed beta, fix broken #bayesian anchor #198

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
907ba79
Document Bayesian model as closed beta, fix #bayesian anchor
russell-loube-mixpanel Aug 27, 2026
4669731
Merge branch 'main' into docs/bayesian-closed-beta
russell-loube-mixpanel Aug 31, 2026
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/experiments.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Enter either the sample size (the number of users to be exposed to the experimen

Mixpanel has set default automatic configurations, seen below. If required, please modify them as needed for the experiment

1. **Experiment Model type**: Sequential. You can also choose Frequentist or Bayesian; see [Experiment Model Types](#experiment-model-types) for how they differ.
1. **Experiment Model type**: Sequential. You can also choose Frequentist, or Bayesian if your project is in the [closed beta](#bayesian); see [Experiment Model Types](#experiment-model-types) for how they differ.
2. **Confidence Threshold**: 95%. On a Bayesian experiment this setting is labeled **Decision Threshold** and sets the minimum Chance to Beat Control required to highlight a variant as a winner.
3. **Experiment Start Date**: Date of the first user exposed to the experiment

Expand Down Expand Up @@ -275,8 +275,14 @@ To illustrate how these calculations work in practice, let's walk through a conc

This example shows why larger sample sizes are crucial—with only 10 users per group, even a 10-point difference in conversion rate isn't statistically significant.

<span id="bayesian" />

@myronkaifung myronkaifung Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps using #### for Bayesian is a cleaner fix than adding a custom span

##### Bayesian

<Note>
**Closed beta.** The Bayesian model is in closed beta and is not enabled for all projects. If you don't see it in the Experiment Model type picker, reach out to your account team to request access. Behavior described below may change before general availability.
</Note>

The Bayesian model answers a different question than the Frequentist and Sequential models. Instead of asking "how surprising would this data be if the treatment had no effect?" (a p-value), it estimates a distribution of plausible values for each variant's metric and reports **how likely it is that the variant is actually better than control**. That probability is surfaced as the **Chance to Beat Control** column.

**Priors:**
Expand Down Expand Up @@ -766,7 +772,7 @@ You can see your experiment MEU usage by going to Organization Settings \> Plan

- Frequentist: Capable of detecting smaller lifts, but requires you to keep experiments for the full duration. You’re discouraged from preemptively making decisions before the test duration is complete. **When to use?** For very small changes (~1% lift) when precision matters.

- Bayesian: Reports the probability that each variant beats control instead of a p-value, and can be read at any point during the experiment. **When to use?** When you're reporting results to an audience that doesn't work in p-values. Note that a few metric configurations aren't supported, listed under [Bayesian](#bayesian).
- Bayesian (closed beta): Reports the probability that each variant beats control instead of a p-value, and can be read at any point during the experiment. **When to use?** When you're reporting results to an audience that doesn't work in p-values. Note that this model is in [closed beta](#bayesian) and a few metric configurations aren't supported.

### Experiment metric types
- **Primary Metrics:** Main goals you're trying to improve. These are metrics used to determine if the experiment succeeded. Examples: revenue, conversion rates, ARPU.
Expand Down