How to Run a Holdout Test (And Finally Know If Your Discounts Work)

How to Run a Holdout Test (And Finally Know If Your Discounts Work)

Every store owner has had this conversation:

"Did the discount work?" "Yeah — we got 200 orders." "Would you have gotten them without the discount?" "…"

That last question is the only one that matters, and most stores can't answer it. Not because they're careless — because answering it requires something almost nobody sets up: a control group.

Here's how to run one. It's called a holdout test, and it's the difference between guessing and knowing.

The problem with everything you're measuring now

Total revenue, conversion rate, AOV — none of these tell you whether a discount worked. They tell you what happened, not what would have happened otherwise.

If you ran 20% off and revenue rose, there are two possible explanations:

1. The discount created orders that wouldn't have happened.
2. The discount discounted orders that were coming anyway.

Both look identical in your dashboard. But they mean opposite things: one is growth, the other is margin walking out the door. The only way to tell them apart is to watch a group of shoppers who didn't get the discount.

That group is the holdout.

What a holdout test actually is

You randomize your audience into two groups:

• Treatment: sees the discount.
• Control (holdout): sees no discount.

Then compare. The difference between the two is your incremental result — the part the discount actually caused.

That's it. Same logic as a clinical trial, pointed at your storefront.

How to run one (without a data team)

1. Pick one decision. Don't test everything. Start with a single offer at a single moment — exit-intent, or a welcome code. One variable.

2. Define the metric before you start. Conversion rate? Revenue per visitor? Incremental margin? Pick one and write it down before the test, so you can't move the goalposts later.

3. Split randomly. Randomly assign shoppers at that moment to treatment or control. A 90/10 or 95/5 split is common — you don't need half your traffic in the holdout, just enough to compare. A 5% holdout is usually invisible in your revenue.

4. Run long enough — and don't peek. This is where most tests die. At modest traffic, you may need weeks for a trustworthy result. Checking daily and stopping when it looks good is how you fool yourself. Set the duration up front and hold the line.

5. Compare incremental, not total. Compute the difference between groups. If treatment converted at 2.4% and control at 2.0%, the discount added 0.4 points — and that's what you actually paid for.

How to read the result

• Treatment wins big: the discount is doing its job. Keep it — but only for the shoppers who need it.
• Groups are about equal: you're mostly discounting sales that would've happened. You're paying for nothing.
• Control wins: the offer hurt (friction, annoyance, wrong audience). Drop it.

The most valuable outcome isn't "the discount worked." It's learning which shoppers it worked on — so you can stop paying for the rest.

Three mistakes to avoid

1. Comparing to last week instead of a control. Seasonality and traffic shifts will fool you every time. Only a holdout is a clean comparison.
2. Samples too small, windows too short. A handful of orders isn't a result. Under-powered tests produce confident nonsense.
3. Changing one thing mid-test. New creative, new depth, new audience — any change invalidates the comparison.

The payoff

Once you can measure incrementality, the argument ends. You stop debating discount strategy and start knowing: keep the offers that create sales, cut the ones that don't, and aim the rest at the shoppers who actually need them.

That's the difference between running promotions and running a business.

---

Start small. One offer, one moment, a 5% holdout, and a metric written down before you begin. You'll learn more from that single test than from a year of dashboard-watching.