Your landing-page test ends on Friday. The new version produces more demo requests and the dashboard says “90% confidence”. One person wants to ship it; another says every test must reach 95%. They are both trying to avoid a bad call, but the number alone cannot make the decision.
For a small, reversible website change, a pre-set 90% confidence rule can be reasonable. It can help a team learn without waiting indefinitely for evidence that is costly to collect. The same rule may be too weak for pricing, substantial spend, accessibility, privacy or a promise that affects trust. Before the test starts, agree what success means, how long you will run it, what downside you will watch and what you will do with a close result.
There is a further catch for Barrava readers: a valid comparison needs people to receive genuinely different experiences. Barrava's current basic experiment path assigns “loaded” and “suppressed” labels, but the current widget code constructs the same bar in both variants. Those labels by themselves cannot establish that hiding the CTA caused any difference. Use a testing setup that actually applies the planned change and confirms it on the page before discussing confidence.
Start with the cost of a wrong decision
Imagine two choices. The first changes a heading above a demo form for two weeks. You can restore the old copy in minutes; the page has modest traffic and the change does not alter a material claim. The second moves a large share of ad budget to a new audience and changes the price shown to visitors. The cost of a mistake is very different.
Your decision threshold is how much statistical uncertainty you will accept before acting. Use a two-sided test when you want the method to check for improvement or harm; the new heading could move results either way. In that conventional setup, a 90% confidence rule corresponds to a 10% significance level, often written alpha = 0.10; 95% corresponds to alpha = 0.05. Roughly speaking, if there were no real difference and the test conditions held, the looser rule would flag more apparent differences across repeated tests. It is a trade-off, not a universal grade of evidence.
Choose the rule from the downside:
| Decision | A proportionate approach |
|---|---|
| Low-cost, quickly reversible copy or layout change | A pre-set 90% rule may be enough to support a cautious rollout if the effect is worthwhile and guardrails remain healthy. |
| Material budget, pricing or hard-to-reverse commercial change | Seek stronger evidence, a stricter pre-set rule, replication or a staged commitment. |
| A change that could harm access, privacy, legal accuracy or customer trust | Fix the risk itself; a high statistical score does not excuse a harmful experience. |
This is why “90% for marketing” is a possible operating choice, not an industry law. Google Ads, for example, uses its own certainty language for some lift studies while its experiments guidance uses a 95% two-sided interval. A product's dashboard label must be read in that product's method, not copied into every A/B test. Google Ads lift certainty; Google Ads experiments.
Say what 90% means without overselling it
Suppose a test reports a confidence interval for the difference in booking rates. Think of the interval as the range the method produces around the observed difference. If the same well-run procedure were repeated many times, a 90% interval procedure would contain the true difference in about 90% of those repetitions. It does not mean there is a 90% chance that this particular new page is better. NIST's confidence-interval explanation explains the long-run idea.
Some tools show a p-value instead. It asks how unusual the observed difference would be under a specified “no difference” model. It does not tell you the chance that your idea is true, the size of a commercial gain or whether visitors were treated fairly. The American Statistical Association's statement warns against making a decision from a p-value threshold alone.
You can make a better decision by reading three things together: the estimated effect, the interval around it and the consequence of a wrong rollout. A narrow gain too small to pay for the work may not matter. A wide interval that still allows harm should make you cautious even when the headline says “winner”.
A worked example: a result just over the line
Illustrative numbers, not Barrava or customer data. A team tests a clearer heading above a demo form. Random assignment puts 10,000 eligible visits into each version. The original records 300 confirmed requests; the new heading records 342. The team agreed before launch to read the result after those visits and to use a two-sided 90% interval because the copy is easy to revert. It also agreed that form errors and qualified-lead rate must not worsen.
- Original request rate: 3.00% (300 of 10,000).
- New request rate: 3.42% (342 of 10,000).
- Observed lift: 0.42 percentage points, or 42 extra requests per 10,000 visits in this sample. The relative lift is 14%, but “14%” sounds bigger than the actual difference if you do not show the baseline.
- A simple approximate two-sided 90% interval for the rate difference runs from about +0.01 to +0.83 percentage points. At 95%, the approximate interval runs from −0.07 to +0.91 points and includes zero.
That is a close call. The 90% rule, chosen in advance, puts the result just on the positive side. It does not establish that the new heading will add exactly 42 requests for every future 10,000 visits. The lower end is almost no gain; the higher end is much larger. Check that traffic allocation, form tracking and qualification rules were stable. Then decide whether a cautious rollout is worth the small downside of being wrong. If sales rejects most extra requests, the request count was the wrong success measure.
For a larger budget or a claim that visitors might misunderstand, the same evidence would not justify a confident full rollout. You might continue the pre-planned test, repeat it, or keep the current version while gathering other evidence. You cannot turn an inconclusive result into a clear win by switching from 95% to 90% after seeing the dashboard.
The illustrative interval uses a simple large-sample comparison of two independently assigned rates. Your own design may need a different method, especially with repeated users, uneven traffic, multiple pages or delayed outcomes. Ask the analyst to state the method rather than borrowing these numbers as a template.
Write the rule before the first visitor enters
A credible test plan can fit on one page:
- Name the choice. “If the new heading increases confirmed, suitable demo requests enough to justify the change, we will roll it out in stages.”
- Choose one primary outcome. Record a confirmed request or another business result, not merely a button click. Specify its source system.
- Set a minimum worthwhile effect. How much improvement would pay for the effort or matter to customers? A statistically detectable but trivial gain can still be a poor use of time.
- Set guardrails. These are measures you will refuse to sacrifice for the primary number, such as form errors, mobile usability, complaint rate or lead quality.
- Set the confidence rule and stopping point. Pick 90%, 95% or another justified threshold, along with the planned duration or sample size, before looking at interim outcomes.
- Check the actual experience. Random assignment is only useful if one group really sees the proposed change and the other sees the original. Confirm this on desktop and mobile.
- Decide how to roll back. Record who can reverse the change and when you will review the result after release.
Repeated peeking means checking a conventional fixed-sample result every day and stopping the first time it looks favourable. That changes the odds of a false signal. Follow the stopping rule you chose, or use an analysis method designed for continuous looks. Research on repeated monitoring of A/B tests explains why the ordinary fixed-sample p-value does not survive unlimited peeks.
If you test many CTAs, devices or audience slices and quote only the best result, you create another selection problem. Pre-name the main outcome and the few segments that could change the decision. Use later slices to generate questions, not retroactive proof.
The 95% and 99% myths do not settle this
You may hear that psychology “requires 95%” while medicine “requires 99%”. That is too tidy. Thresholds depend on the design and consequences. UK MHRA clinical-investigation guidance describes a conventional 5% two-sided level in that context; 99% is not a universal medical rule. None of those fields grants marketing an automatic 90% pass. The fair question remains: what happens if we are wrong, and what evidence does this particular decision need?
For a website experiment, start with the visitor experience and the decision. Make the variants real, keep the data clean, choose the rule in advance, then read the estimated effect and its limits together. If the change is low-risk and reversible, 90% may be enough to try and monitor. If it touches money, access or trust in a serious way, demand more.
Barrava can help teams manage website actions in one configurable bar, but its current experiment labels do not by themselves prove a CTA was suppressed for a valid holdout. Explore the current Barrava product and verify the on-page behaviour before using any variant report to make a lift claim.
