Is your cold email A/B test significant?
Enter sends and replies (or opens) for variant A and variant B. Get both rates, the absolute and relative lift, a two-proportion z-test p-value and how many sends per variant the difference needs, with the formulas. Free, no account.
Not significant yet
Example numbers. Enter your own above.
A: 3% reply rate, B: 5.4%. B minus A is +2.40 percentage points (+80.0% relative). Not significant yet (p = 0.059, needs under 0.05). If the true rates are the ones seen so far, you need about 1,096 sends per variant to detect the difference 80% of the time at 95% confidence: about 596 more sends per variant.
| Variant | Sends | Replies | Reply rate |
|---|---|---|---|
| A | 500 | 15 | 3% |
| B | 500 | 27 | 5.4% |
- Absolute lift
- +2.40 percentage points
- Relative lift
- +80.0%
- z
- 1.892
- p-value
- 0.0585 (two-sided)
- 95% interval for B minus A
- -0.08 to +4.88 percentage points
- Sends per variant needed
- 1,096
Assumptions: each send is independent, recipients were split between A and B at random and sent over the same days, and the sample size treats the rates seen so far as the true rates. Check the result once, at the sample size you planned; checking after every send and stopping at the first significant result overstates it.
How the test works
The calculator runs the standard two-proportion z-test. It pools both variants' results to estimate the rate if there were no real difference, works out how far apart the two rates are in standard errors (z), and turns z into a two-sided p-value: the chance of a gap at least this big if A and B were really the same. Below 0.05 is significant at 95% confidence.
The sample size is the usual formula for comparing two proportions: how many sends each variant needs so that a real difference the size of the one you see would show up as significant 80% (or 90%) of the time. Small differences need far more sends than large ones; halving the gap takes roughly three to four times the sends.
Testing cold email well
- Change one thing at a time (subject, first line or the ask), so you know what made the difference.
- Split one list at random and send both variants over the same days; a list or a weekday difference can look like a copy difference.
- Decide the sample size first and read the result once. Stopping when it first looks significant makes false wins more likely.
- Prefer replies to opens. Apple Mail Privacy Protection loads tracking pixels on its own, so open counts include opens no person made.
Get 67 leads free No card. Search free with contacts masked.
Subject line graderCold email graderCold email campaign calculatorSpintax generatorEmail metrics calculatorWeb design proposal generatorEmail header analyzer67Leads pricing
Questions
Enter the sends and replies for each variant above. A two-sided p-value under 0.05 means the gap is significant at 95% confidence; above it, the difference could be chance.
It depends on the reply rates and the gap between them. The calculator shows the sends per variant for the difference you see, at 80% or 90% power; small gaps need thousands.
Replies, where you can. Opens are counted from a tracking pixel that some mail apps load automatically, so open rates are less reliable than reply rates.
B's rate minus A's, divided by A's. From 3% to 5.4% is +2.4 percentage points absolute and +80% relative.
More free tools
Local business counterNo-website business finderLead list previewCold email template generatorSpintax generatorEmail metrics calculatorWeb design proposal generatorEmail header analyzerAll Calculators & planning toolsAll tools