Conflicting sample-size and duration guidance can cause tests to end too early
Source references: 6The main document and detailed reference give materially different per-variant samples for the same baseline and lift—for example, 150,000 versus 380,000 for a 1% baseline and 10% lift. The reference then gives a duration formula that includes the number of variants, while its quick framework divides a per-variant sample directly by total page traffic, commonly omitting the two-way traffic split.
Using the smaller figures or simplified duration calculation can leave a test seriously underpowered or end it prematurely. Random variation may then be treated as a winner and drive rollout decisions affecting conversion or revenue.
The source supports this risk. For a 1% baseline and 10% relative lift, the main file says 150,000 observations per variant while the detailed guide says 380,000, which could lead users to decide from an underpowered test. The guide’s full duration formula multiplies by the number of variants, but its quick framework divides per-variant sample by page traffic directly. If that traffic is the total split among variants, a standard two-variant test would be estimated at roughly half the needed duration. Users should ask the author to reconcile assumptions, methods, and units, and verify calculations independently.
| Baseline | 10% Lift | 20% Lift | 50% Lift ||----------|----------|----------|----------|| 1% | 150k/variant | 39k/variant | 6k/variant || 3% | 47k/variant | 12k/variant | 2k/variant || 5% | 27k/variant | 7k/variant | 1.2k/variant || 10% | 12k/variant | 3k/variant | 550/variant |Show 5 other places
| Lift to Detect | Sample per Variant | Total Sample ||----------------|-------------------|--------------|| 5% (1% → 1.05%) | 1,500,000 | 3,000,000 || 10% (1% → 1.1%) | 380,000 | 760,000 || 20% (1% → 1.2%) | 97,000 | 194,000 || 50% (1% → 1.5%) | 16,000 | 32,000 || 100% (1% → 2%) | 4,200 | 8,400 |```Duration (days) = (Sample per variant × Number of variants) / (Daily traffic × % exposed)``````Daily traffic to page: _____Baseline conversion rate: _____MDE I care about: _____Sample needed per variant: _____ (from tables above)Days to run: Sample / Daily traffic = _____If days > 60: Consider alternativesIf days > 30: Acceptable for high-impact testsIf days < 14: Likely feasible```Duration (days) = (Sample per variant × Number of variants) / (Daily traffic × % exposed)```Sample needed per variant: _____ (from tables above)Days to run: Sample / Daily traffic = _____