Test two versions of a landing page and find out whether the difference in conversion rate is real or just luck. Enter visitors and conversions for version A and version B to get the lift, the p-value and a plain verdict, or switch to planning mode to see how many visitors each version needs before you start.
Every visitor either converts or does not, so each version of the page has a conversion rate that you estimate from a sample. Samples wobble: show the same page to two different groups of 1,000 visitors and the rates will differ a little by chance. The test asks whether the gap between A and B is bigger than that wobble would normally produce.
The z-test answers with a p-value. A small p-value says the gap is unlikely to be luck. A large one says you cannot tell, which is not the same as the pages being equal. With too few visitors, a real improvement will often show a large p-value, so a flat result on a small test mostly means you need more data.
Power is the chance your test notices a real lift of the size you planned for. The usual target is 80 percent, which is what the planning mode uses. Raise the confidence level or shrink the lift you want to catch and the required sample rises quickly, so decide what lift would actually change your decision before you start.
Stopping a test the first time it looks significant is the most common mistake. The p-value moves around as visitors arrive, and if you check often enough it will dip below 0.05 at some point even when both versions are identical. Fix the sample size in advance, wait for it, then read the result once.
The tool also shows a range for the difference between the two rates. If the range runs from 0.1 to 1.0 percentage points, the true gain could be tiny or ten times larger. A wide range means the test is imprecise, even if the p-value is under 0.05.
Remember that a statistical win is not always a business win. A lift of 0.2 points that costs a week of design work may not be worth rolling out, while a lift that is not quite significant on a high-value page may deserve a longer test.
It runs the standard two-proportion z-test. It pools both versions into one conversion rate p, computes the standard error sqrt(p x (1 - p) x (1/nA + 1/nB)), divides the difference in rates by it to get z, and turns z into a two-sided p-value. If the p-value is below 1 minus your confidence level (0.05 at 95 percent), the difference counts as statistically significant.
Version A has 10,000 visitors and 500 conversions (5.0 percent). Version B has 10,000 visitors and 550 conversions (5.5 percent). The pooled rate is 1,050 / 20,000 = 5.25 percent, the standard error is 0.00315, so z = 0.005 / 0.00315 = 1.585 and the two-sided p-value is about 0.113. That is above 0.05, so the 10 percent lift is not significant at 95 percent confidence.
It is the chance of seeing a difference at least this large if the two versions actually convert equally well. A p-value of 0.03 means about a 3 percent chance of a gap this big from luck alone. It is not the chance that B is better, and it says nothing about how large the true lift is.
Planning mode uses the standard two-proportion sample size formula with 80 percent power: n = (z(alpha/2) x sqrt(2 x pbar x (1 - pbar)) + z(beta) x sqrt(p1(1 - p1) + p2(1 - p2)))^2 / (p2 - p1)^2, where p1 is your current rate, p2 is the rate after the lift and pbar is their average. A 10 percent baseline and a 20 percent relative lift (to 12 percent) needs about 3,841 visitors in each version at 95 percent confidence.
The needed sample grows with the inverse square of the difference you want to detect. Halving the lift roughly quadruples the visitors. That is why low-traffic pages can only reliably detect big changes.
It is a quick estimate using a standard textbook method, and it assumes visitors were split randomly and each visitor counted once. It is not professional advice. Results from tests with very few conversions, several variants or repeated peeking need extra care.
Related: Conversion rate calculator, ROI calculator, Landing page copy generator.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.