Use this method when you need to compare two groups based on conversion rates or counts. It determines which variation is better by calculating the direct probability that one group outperforms the other, helping you make confident product or marketing decisions without complex setup or long wait times.
Understanding conversion probabilities
This tool models your success counts as a Beta distribution. By using the Beta-Bernoulli conjugate model, we can analytically calculate the probability distribution for each group’s conversion rate. This allows you to see the range of likely values for each variation, represented by their mean and high-density intervals.
Unlike methods that rely on p-values, this approach provides a direct statement about the parameters of interest. You receive the average conversion rate and a 95% interval where the true conversion rate most likely resides.
Evaluating performance with lift
Beyond comparing raw rates, the tool calculates the 'lift,' which measures the relative improvement of one variation over the other. By sampling from the posterior distributions, it provides an expected percentage increase and a 95% interval for that lift, giving you a clear view of the potential impact of your changes.
Risk management via expected loss
To help you make safer decisions, the method computes the expected loss. This value represents the average 'cost' or penalty you would incur if you chose one variation but it turned out to be inferior. Low expected loss provides higher confidence when you are ready to declare a winner.
Reproducibility and speed
Because this method uses analytical solutions rather than complex simulation techniques like MCMC, it is extremely fast and provides identical results every time the same data is provided. This deterministic nature ensures that your analysis is perfectly reproducible and audit-friendly.
1 · Intent → method
An LLM picks bayesian_ab_test from a fixed catalog.
2 · Method → numbers
Deterministic Python engine runs the math. Same input → same output.
3 · Numbers → plain language
A second LLM translates the result into your domain’s vocabulary.
Why use this method instead of a standard t-test?
This Bayesian A/B test calculates the direct probability that one group is better than the other, providing an intuitive percentage. It also calculates the expected loss, which helps you understand the risk of choosing the wrong variation.
How does the platform decide which variation is the winner?
The method uses a decision threshold. If the probability that one group is better than the other exceeds your threshold (default 95%), it reports a clear winner. If the probability falls in between, it labels the result as inconclusive.
Tool input schema
Schema for bayesian_ab_test not exported yet (run pnpm export:registry).