people-based testing

Everyone wants to optimize their advertising and that means figuring out what works and what doesn’t. To that, you need to figure out not only a winner and a loser but you need confidence that you are right.

Unfortunately, it’s not as simple as just comparing the current performance of one ad with another ad — that’s like saying just because one basketball team is currently ahead in the 1st half of the game that it is the best team. You need enough time and data (enough matchups in basketball) to determine a true champion.

In statistics, confidence comes from having enough “statistical significance” which essentially means knowing the results are likely not just chance but would be likely repeated if the match up happened again. To achieve statistical significance, you need a minimum sample size of data based on a target desired lift in performance that would make one ad a meaningful winner over another.

Thunder has built an easy to use sample size calculator which will allow you to now input basic variables in your creative experiment to determine the minimum sample size necessary whether you’re thinking about how many impressions or how many people need to be in the test to get a meaningful result. Thunder Experience Cloud uses minimum sample sizes to ensure it has enough data for each creative version it is testing before its Dynamic Creative Optimization solution serves the winning creative to all consumers, thereby achieving maximum media impact and efficiency with highest confidence.

If a vendor tells you there isn’t a minimum or that they can test thousands of ad versions, you need to ask them if they have a sample size calculator or how they will achieve statistical signfiance with their results. Otherwise, it is highly suspect they are really testing and optimizing your advertising.

Try out our free testing sample size calculator for A/B and multivariate testing.