A/B Test Calculator

What is an A/B Test?

An A/B test (or split test) is a randomized experiment where two versions of a webpage, email, or digital asset are shown to different segments of users simultaneously. Version A serves as the Control, while Version B acts as the Variation with a specific modification. Marketers commonly test metrics like click-through rates using our CTR Calculator and overall conversion performance with our Conversion Rate Calculator.

How A/B Testing Significance is Calculated

To determine whether the observed difference between Control and Variation is real or caused by random chance, statistical hypothesis testing is applied using a two-proportion Z-test.

1. Conversion Rates

First, compute the conversion rate for each group:

Control Conversion Rate ($CR_A$): $CR_A = \frac{C_A}{N_A}$
Variation Conversion Rate ($CR_B$): $CR_B = \frac{C_B}{N_B}$
Relative Uplift: $\text{Uplift} = \left(\frac{CR_B - CR_A}{CR_A}\right) \times 100\%$

2. Pooled Proportion and Standard Error

Under the null hypothesis ($H_0: CR_A = CR_B$), the pooled conversion rate is:

$$p = \frac{C_A + C_B}{N_A + N_B}$$

The standard error ($SE$) of the difference between the two proportions is:

$$SE = \sqrt{p (1 - p) \left( \frac{1}{N_A} + \frac{1}{N_B} \right)}$$

3. Z-Score and p-Value

The Z-score measures how many standard errors the variation rate is away from the control rate (you can also calculate standardized statistical metrics directly with our Z-Score Calculator):

$$Z = \frac{CR_B - CR_A}{SE}$$

The $p$-value represents the probability of observing a result as extreme as (or more extreme than) the test data assuming no real difference exists. If $p \le \alpha$ (where $\alpha = 1 - \text{Confidence Level}$), the result is declared statistically significant.

Frequently Asked Questions

What confidence level should I select for A/B testing?

A 95% confidence level ($\alpha = 0.05$) is the industry standard for most marketing and product tests. A 90% confidence level is acceptable for fast exploratory tests, while a 99% confidence level is used when changes carry high commercial or technical risks.

What is the difference between one-tailed and two-tailed tests?

A two-tailed test checks whether Variation B is significantly different (either better or worse) than Control A. A one-tailed test checks only if Variation B is significantly better than Control A. Two-tailed tests are generally recommended to prevent false positives from negative variations.

How long should an A/B test run before declaring a winner?

An A/B test should run for at least 1 to 2 full business cycles (typically 7 to 14 days) to capture weekday and weekend traffic patterns, even if statistical significance is reached earlier.

What should I do if my A/B test is inconclusive?

An inconclusive result means the variation did not produce a statistically detectable difference. You can choose to run the test longer to gather more sample size, or iterate on a bolder design variation.

Report