Sample Size Calculator
Plan experiments with the right amount of data
Test Parameters
Relative improvement you want to detect (e.g., 10% means 5% → 5.5%)
Required Sample Size
Test Duration Estimate
Based on 1,000 daily visitors split across 2 variations
Test Summary
Quick Reference: Sample Sizes by MDE
Sample size per variation at 80% power, 95% confidence (two-tailed)
| Baseline Rate | 5% MDE | 10% MDE | 20% MDE | 50% MDE |
|---|---|---|---|---|
| 1% | 3,139,200 | 784,800 | 196,200 | 31,392 |
| 3% | 1,025,067 | 256,267 | 64,067 | 10,251 |
| 5% | 601,920 | 150,480 | 37,620 | 6,019 |
| 10% | 285,120 | 71,280 | 17,820 | 2,851 |
| 20% | 126,720 | 31,680 | 7,920 | 1,267 |
When to Use Sample Size Calculator
Pre-Test Planning
Before launching any A/B test, calculate the required sample size to ensure your experiment can detect meaningful differences. This prevents wasting time on underpowered tests that won't yield actionable results.
Timeline Estimation
Combine sample size requirements with your traffic data to estimate how long tests will run. This helps with resource planning and setting realistic expectations for stakeholders about when results will be available.
Traffic Allocation
Determine if you have enough traffic to run multiple tests simultaneously or if you need to prioritize. Understanding sample requirements helps optimize your testing roadmap and maximize learning velocity.
Survey Research
Calculate how many survey responses you need for statistically valid conclusions. Whether measuring customer satisfaction or market research, proper sample sizing ensures your findings represent the broader population.
Budget Planning
For paid traffic experiments, knowing the required sample size helps estimate advertising costs. Calculate whether your budget can support the traffic needed to achieve statistical significance within your timeframe.
Risk Assessment
Understand the trade-offs between test duration, detectable effect size, and confidence levels. Make informed decisions about acceptable risk levels when time or traffic constraints limit your testing capacity.
Frequently Asked Questions
What is a sample size calculator?
It determines the minimum number of participants needed for statistically valid experiments. For A/B testing, it calculates how many visitors each variation needs to detect meaningful differences with your desired confidence and power levels.
What is Minimum Detectable Effect?
MDE is the smallest relative improvement you want to detect. A 10% MDE on a 5% baseline means detecting if conversion reaches 5.5%. Smaller MDEs require larger samples because subtle differences are harder to distinguish from random noise.
What is statistical power?
Power is the probability of detecting a real effect when it exists. 80% power means an 80% chance of finding significance if the variation truly performs better. Higher power reduces false negatives but requires more samples.
Why does lower baseline need more samples?
Lower conversion rates have higher relative variance. A 1% rate fluctuates more proportionally than a 20% rate. More data is needed to separate true effects from this natural variation in low-conversion scenarios.
One-tailed vs two-tailed tests?
Two-tailed tests detect differences in either direction and are recommended for most cases. One-tailed tests require fewer samples but assume the variation can only perform better, which is rarely guaranteed.
Can I reduce the required sample size?
Yes, by accepting larger MDE (detecting only big improvements), lower power (more false negatives), or lower confidence (more false positives). Each trade-off has implications for decision quality.
No comments yet. Be the first to share your thoughts!