Ad Tests: How to Know a Winner Is Real, Not Luck

Most small businesses test ads the same way: run two versions for a few days, see which one has more enquiries, and switch everything to the winner. A month later the winner is performing no better than the old ad, and nobody is sure why. Often the honest answer is that it never won. The early lead was luck.

A little discipline turns ad tests from guesswork into decisions you can trust, without any statistics software.

Why early results mislead

With small numbers, chance swings results a lot. If one ad gets 6 enquiries and the other gets 3 in the first week, it looks like a clear win: twice as good. With counts that small, a gap like that turns up by chance all the time. Run the same two ads again the following week and the order can easily flip.

Day-of-week patterns add to the confusion. Many service businesses get more enquiries on Mondays or just after salary day. A test that starts on a busy Monday and ends on a quiet Thursday measures the calendar as much as the ads.

Decide the rules before you start

• Test one change at a time: the headline, or the offer, or the image. If you change three things, a win tells you nothing about which one mattered.
• Pick the measure in advance. Enquiries or bookings are better than clicks, because an ad can attract clicks from people who were never going to buy.
• Set a minimum. Run at least two full weeks so every weekday appears twice, and wait until each version has a few dozen of the result you are measuring, not a handful.
• Split the budget evenly and change nothing else while the test runs.

Write these rules down with the start date. It is much easier to resist calling a winner on day three when you have already agreed not to.

What to do when the numbers stay small

Many local businesses will never get a few dozen bookings per ad version in a fortnight. That is fine; the approach changes rather than the principle.

Measure something that happens more often but still signals intent, such as calls longer than a minute, form submissions or WhatsApp chats that ask about price. Run the test longer, a month if needed. And test bigger differences: two genuinely different offers will separate faster than two slightly different headlines.

Reading the result honestly

When the test ends, look at the gap and the volume together. A version that wins by a wide margin on a solid number of results is a real finding. A small edge on a small number is a tie, and a tie is a useful answer too: keep whichever version is cheaper or easier to run, and test something with more potential next.

Check the quality behind the numbers. If the winning ad brought more enquiries but your team says they were the wrong people, the cost per lead covered earlier will look good while the cost per customer gets worse.

Keep a simple test log

Record each test in one line: dates, what changed, the measure, the result and the decision. After a few months the log shows patterns no single test can, such as offers mattering more than images for your audience. It also stops the team from re-testing the same idea every season.

FAQ

Can I trust the platform's own A/B test tools?

They help, because they split the audience fairly and report confidence. The same rules still apply: one change, a meaningful measure, and enough time and results before deciding.

How many tests should run at once?

On a small budget, one. Several tests at once split the money so thin that none of them reaches a clear answer.

What should we test first?

The offer, then the first line of the ad or the main image. Small wording changes usually matter least, so leave them until the bigger questions are settled.