A/B Split Testing Sample Size Calculator

Everyone wants to optimize their advertising and that means figuring out what works and what doesn’t. To that, you need to figure out not only a winner and a loser but you need confidence that you are right.

Unfortunately, it’s not as simple as just comparing the current performance of one ad with another ad — that’s like saying just because one basketball team is currently ahead in the 1st half of the game that it is the best team. You need enough time and data (enough matchups in basketball) to determine a true champion.

In statistics, confidence comes from having enough “statistical significance” which essentially means knowing the results are likely not just chance but would be likely repeated if the match up happened again. To achieve statistical significance, you need a minimum sample size of data based on a target desired lift in performance that would make one ad a meaningful winner over another.

Thunder has built an easy to use sample size calculator which will allow you to now input basic variables in your creative experiment to determine the minimum sample size necessary whether you’re thinking about how many impressions or how many people need to be in the test to get a meaningful result. Thunder Experience Cloud uses minimum sample sizes to ensure it has enough data for each creative version it is testing before its Dynamic Creative Optimization solution serves the winning creative to all consumers, thereby achieving maximum media impact and efficiency with highest confidence.

If a vendor tells you there isn’t a minimum or that they can test thousands of ad versions, you need to ask them if they have a sample size calculator or how they will achieve statistical signfiance with their results. Otherwise, it is highly suspect they are really testing and optimizing your advertising.

Try out our free testing sample size calculator for A/B and multivariate testing.

Inputs for Thunder sample size calculator

 

Continue Reading

What is a CMP?

CMP is the hot adtech acronym of 2018. There are actually two meanings to this term: (1) Creative Management Platform and (2) Consent Management Platform. Here’s an overview of both these products and why you may need one.

Creative Management Platform

Introduced in 2016 by Thunder, the CMP acronym original stood for “creative management platform,” a tool for producing and trafficking ad creatives. Rather than just a general purpose creative editor like Adobe Photoshop or Animate, which are applications built for a single designer to use by him or herself, CMPs are meant for an enterprise that has a scale issue with creative.

Many brands, agencies and publishers are increasingly needing to build ads in different sizes and versions for different audiences and media formats. Consequently, creative production demands have grown exponentially while most creative organizations can only scale linearly in their capability by adding more designers and programmers. Because traditional creative editors were built for highly advanced users, a creative bottleneck formed as demand went up and not enough talent or payroll existed to fill the void.

Creative Management Platforms radically simplified ad production by providing easier interfaces and automated production tasks like re-sizing. Forrester began recognizing CMPs in 2017 as part of their broader creative ad tech research which has been timed with the rise in enterprise demand for new marketing creative technologies.

Consent Management Platform

Introduced in 2018, the new CMP acronym stands for “consent management platform.” The European privacy laws known as GDPR required publishers and marketers to obtain explicit consent for certain tracking and targeting data. As a result, a new category of tools emerged to specifically help these enterprises collect and keep track of user consent.

The CMP then feeds that consent information tied to an ID to other selected partners in the digital advertising supply chain. As a result, every party in a publisher’s supply chain understands what data they may use and for what.

Which CMP do I need?

It depends if you’re looking to solve a creative problem or a data privacy problem. Talk to Thunder if you need help with your data-driven creative problems or digital creative production problems. Check out these consent management vendors if you’re looking to solve a privacy preference problem.

Continue Reading

How to Test Ad Creatives: Beginner’s Guide to Optimize Your Display Ad Tests

There are so many creative elements that digital marketers can test in their banner ads – from value propositions to taglines to images and styling – that it can be hard to know where to start.  

A/B testing your creatives take a couple weeks to conduct to get proper statistical significant, so it’s often difficult to test every possible creative variation.  So, how should a digital marketer get started with A/B testing their banner ads?

Thunder has conducted hundreds of A/B tests, and distilled our learnings into the best practices for designing creative tests.  When followed, these tips can reduce the amount of time required to optimize your creative!

What is Test Significance?

Before we begin, we should address a commonly misunderstood concept: test significance. Marketers with no background in statistics often miss a critical fact: your tests may tell you less than you think.  

The reason is simple: our testing approach basically surveys the opinions of a smaller group of people within our target population, and sometimes, these small groups don’t completely represent the true opinion of our target population. This can expose marketers to faulty decisions that are based on false positives, that is, tests in which the apparent winner is not the actual over-performer in the target population.  

Statisticians have overcome these sampling errors with “statistical significance” to correct for this type of error, and you should always ask your A/B test vendor how they control for sampling errors including false positives.  If our goal is to learn from our creative testing, then we must ensure that our outcomes are statistically significant!

#1 Test Hypotheses, Not Ads

The first question to ask when designing a creative A/B test is this: What hypothesis do we want to test?  Common hypotheses to test include:

  • Value Proposition (ex: 10% off vs. $25 off)
  • Image (ex. red car vs. blue car)
  • Tagline (ex. “Just do it” vs. “Do it”)
  • Call to Action Text (ex. “Subscribe now!” vs. “Learn more”)
  • Single Frame vs Multi-Frame

Each test should allow you to answer a question, for example: “do my customers like 10% off, or do they like $25 off?”

Many creative tests make the mistake of testing creatives that were created independently of each other, and thus vary in more than one way.  The reason why these tests are ineffective is that the marketer can’t distill the test into a lesson to be applied to future creative design. The only learning from such a test is that the brand should shift traffic to the winning ad.  But no lessons for the next new ad result from such a test.

For example, the A/B test below is comparing different layouts, images, value propositions and CTA text all at the same time.  Let’s say Creative B wins. What have we learned? Not much, other than in this particular set of ads, Creative B outperforms Creative A.  But we don’t know why, and thus have learned nothing that we can apply to future ads.

A/B Test with No Hypothesis

 

By comparison, the following two A/B tests have specific hypotheses – “do red cars work better than blue cars?”  At the end of this test, we will learn that either red SUV’s or blue sports cars outperform the other, and can apply this learning to future creatives.

Hypothesis-Driven A/B Test: Car Type Drives Performance

 

In this next A/B test, the hypothesis is that the value proposition in the tagline drives performance.  A common first A/B test for a brand is to compare feature-based vs value-based taglines.

Hypothesis-Driven A/B Test: Value Proposition Drives Performance

 

#2 Test Large Changes before Small Changes

Large changes should be tested first because they generate larger differences in performance, so you want these learnings to be uncovered and applied first.  

Larger changes – such as value proposition and image – are also more likely to perform differently for different audience segments that small changes – like the background of the CTA button.  As such, by breaking out your A/B test results by audience segment, you can learn what tagline or image pop with particular segments, which can guide the design of a creative decision tree.

Large changes: Value Proposition, Brand Tagline, Image, Product Category, Price/Value vs Feature, Competitive Claims

Smaller changes: CTA text, CTA background, Styling and formatting, Multiframe vs Single Frame

Small changes are likely to drive small lift.  Only test this after testing bigger changes.

 

#3 Test multiple creative changes with Multivariate Test Design

Multivariate test designs (MVT) sound more complex than they are.  Multivariate tests simply allow you to run 2 or 3 A/B tests at the same time, using the same target population.  They are a statistically rigorous way to break Rule #1 above that says you should test a single change at a time.  In the case of MVT test design, you can more than one change by creating a separate creative for every combination of changes, and then learning from these tests.  

For example, if, as below, you are testing 2 changes – message and image – each of which have 2 variations, you have a 2×2 MVT test and need to create 4 ads.

Multivariate test that tests Image and Message at the same time

 

When the test is done, aggregate test results along each dimension to evaluate the results of each A/B test independently. If you have enough sample, you can even evaluate all the individual creatives against each other to look for particular interactions of message and image that drive performance.

To Summarize:

To drive more optimizations more quickly and generate demand and budget for more testing, following these simple tips:

  1. Test hypotheses that generate learnings for subsequent creative design
  2. Test large changes first and setting up multiple variate tests
  3. Test one change at a time, or set up a multivariate test framework

Happy testing!

Continue Reading

Panel discussion at OMMA Programmatic –  Can Robots Fix Programmatic Creative?

Thunder CEO Victor Wong had the pleasure of sitting in on a panel about programmatic creative at this year’s OMMA Programmatic event.

“Less than 1/3rd of online users today feel that internet advertising is relevant to them.

“Through programmatic creative and using data, if we can bring that up to 50%, that’s going to lead to hugely impactful outcomes for our clients and hopefully cut down on ad blocking and improve the way people perceive advertising in the future,” said Andrew Sandoval, Director, Biddable Media at The Media Kitchen in an opening statement.

Watch the video recording:

Below is a summary version of the main takeaways from the talk.

Continue Reading

What is the difference between dynamic creative and data-driven creative?

dynamic creative vs data-driven creative

Dynamic creatives are ads that can change content on the fly at any time.

Data-driven creatives use information about a customer to inform creative messaging.

Thus, a creative can be dynamic and data-driven if the same creative puts content in the ad that can be changed at any time, AND the content was chosen is based on data.

A creative may be dynamic but not data-driven if it simply changes content without regard to who the targeted user is.

Continue Reading

Where Next? Finding Flite Alternatives and Competitors After the CMP Shutdown

Flite alternatives and competitors

Snap’s acquisition of Flite last December led to all Flite ads going dark, leaving some advertisers scrambling to evaluate Flite alternatives and competitors. Although Flite’s website remains online, the creative management platform (CMP) has closed its doors for good.

Ad tech is often considered an area that is oversaturated with similar technologies, and many view Flite’s exit as a positive step for the industry overall. 

Continue Reading

What is Programmatic Creative? – Definition and Tactics

Thunder Creative Management Platform (CMP) designer
Above: Design work being done in Thunder Creative Management Platform

This article is continuously updated to reflect how programmatic creative is evolving.

Programmatic creative enables data-informed, software-assisted creative executions that deliver on the promise of modern digital marketing.

The purpose of programmatic creative is to harness the data that we’re spending billions of dollars to leverage in advertising and activate it to create a more successful execution—one that captures attention and increases campaign results. Studies suggest optimized creative can routinely boost performance by 30-50%, often more.

So to attract eyeballs, programmatic creative is solving the production and creative management challenges in producing a large volume of ads, thereby enabling messages to be hyper-relevant. Rather than showing a generalized creative, these new technologies allow the experience to be specifically tailored and customized to the viewer.

Continue Reading

Programmatic Creative vs. Dynamic Creative Optimization (DCO)

programmatic creative vs. dynamic creative optimization

This post is continuously updated to reflect how programmatic creative and dynamic creative optimization (DCO) are evolving.

When people think of the relationship of programmatic creative vs. DCO, a common misunderstanding is that dynamic creative optimization and programmatic creative are different technologies.

One term is actually a subset of the other. DCO is a form of programmatic creative.

While DCO falls under the programmatic creative umbrella, but it’s not the only way to build creatives that activate the 6+ billion dollars of data in contained in programmatic media.

To illustrate this idea, let’s start with a definition of programmatic creative.

Programmatic Creative Definition

Programmatic creative refers to the set of advertising technologies that add speed, scale, and automation to the creative process. This covers ad production, dynamic ads, and creative optimization.

Programmatic media has unlocked tremendous potential in how we tell stories in paid media online. The purpose of programmatic creative is to enable data and creativity to come together to tell brand stories in a more resonant and effective way than ever before.

Continue Reading

Video: What Data-Driven Creative Means For The Future Of Advertising

Though $6 billion is invested in targeting technologies like Demand Side Platforms (DSPs) and Data Management Platforms (DMPs), programmatic advertisers have not successfully delivered the right message for the right audience.

In order to build long-term relationships, drive brand loyalty and long-term advocacy, advertisers must provide personalized messages that truly resonate with customers.

In this video, learn about how your organization can achieve data-driven creative by activating both data and technology investments.

Continue Reading

What Data-Driven Creative Means For The Future Of Advertising

data-driven advertising future

With over $20 billion being spent globally in programmatic advertising, data-driven creative (or programmatic creative) poses a tremendous opportunity for marketers to reach and engage potential customers.

Though 30%, or $6 billion, of this programmatic spend is invested in targeting technologies like Demand-Side Platforms (DSPs) and Data Management Platforms (DMPs), brands and advertisers have not yet successfully delivered on the promise of the right message for the right audience.

In fact, according to research by AppNexus, up to 97% of programmatic campaigns lack a targeted creative for each audience segment.activate dsp dmp data

This means that once the ad has successfully reached their potential customers, a majority of creatives are generic and untailored. Brands and advertisers need to combat this $6 billion waste by activating their data and delivering personalized messages that truly resonate with customers.

Continue Reading

Define Programmatic Advertising – Concept Breakdown and Insider Tactics

Define Programmatic Advertising

Programmatic advertising has been one of the most transformative advancements in marketing. It has also been one of the biggest disruptors to the ad-powered internet.

Now a dominant way to sell and purchase media, programmatic advertising is expected to rise to encompass 58% of all ad spend on digital display this year. The US is the strongest programmatic market, accounting for 62% of all global programmatic ad spend.

But what exactly is programmatic?

Here’s how we define programmatic advertising.

Continue Reading

What is People-First Marketing?

what is people first marketing

People-first marketing is both an approach and a set of marketing strategies that focus on personalizing customer touchpoints across channels to drive engagement and maximize campaign performance.

The proliferation of digital channels, devices, and platforms have led to a vast shift in consumer behavior and expectations. Gone are the days where advertisers can mass send out generic marketing messages aimed at everyone–which in reality, cater to no one.

Consumers have gained more control over their own journey to purchase, which means brands and advertisers need to do a better job of identifying and reaching the right audience at the right moment. Understanding who your specific target audiences are, as well as their needs, wants, and pain points help put each customer at the forefront of your marketing strategy.

Targeting the right audience, however, is just one part of the equation.

Executing a people-first strategy also means understanding how to capture your audience’s attention with the right message. To activate the $6 billion dollars spent on targeting technologies like DSPs and DMPs, brands and advertisers need to focus on delivering a personalized message that captures their audience’s attention at the right time in their customer journey. Delivering a personalized creative is key to truly engaging consumers, instead of just marketing to them.

Continue Reading

7 Ingredients For The Perfect Programmatic Creative Management Platform

7 Key Ingredients For A Creative Management Platform

The main concepts of advertising have been around for over a century, but the emergence of programmatic creative and the creative management platform (CMP) is a phenomenon that is much newer.

A number of companies have seen incredible results from creative management platforms, including Anheuser-Busch, the biggest brewer in the world, and Intercontinental Hotels Group (IHG), the second-biggest hospitality company in the world.

Programmatic creative technologies combine well with programmatic buying, data management platforms (DMPs) and demand-side platforms (DSPs). In this ad tech stack arrangement, the CMP is the creative arm of a data-driven campaign strategy.

We’re going to explore seven ingredients that can make up the perfect creative management platform that aligns with your programmatic creative strategy. If you’re still learning about what a CMP is, check out the video below before you review the list.

Continue Reading

The 2017 the Ad Creative Landscape for Programmatic

Ad Creative Landscape - 2017 (Thunder)

In 2017, US programmatic digital display ad spending will reach $27.47 billion, with programmatic representing 72% of the total spend in the category. Mobile is driving a significant portion of programmatic growth.

The lion’s share of display ad inventory will be a combination of rich audience data, targeting and placement.

How will the ads that align with these programmatic buys come into being? What tools are services are used to source the ad content and personalize the ads? Once built, how will the ad performance optimized?

To answer these questions we created the Ad Creative Landscape in 2014, now updated for 2017. Inspired by the Display Ad Lumascape, this graphic summarizes many key players and services in the display ad creation process.

Here’s a breakdown of the main components.

Continue Reading

Research: Programmatic Creative Industry Report

Programmatic advertising enables digital marketers to reach the right audience at the right time more quickly and efficiently than ever before. Advertisers have heavily focused on who they’re delivering the message to and how they’re delivering it, but what about the actual content they’re delivering?

Is the creative falling behind in this programmatic revolution?

Last year, Thunder teamed up with Digiday Content Studio to research the creative needs of brands, agencies and publishers. Watch this video to learn the top findings from this report.

Continue Reading

How To Overcome 80% Of Your Programmatic Creative Challenges

pasted-image-at-2016_11_15-02_18-pm
Image Source: ‘Everything was easier at the beginning’: Marketers vent programmatic frustration – Digiday

At this year’s Digiday Programmatic Summit, participants were asked to jot down their greatest challenges in programmatic advertising. The response was enlightening.

Here’s the breakdown of top programmatic challenges:

  • Telling the difference between vendors (30%)
  • Reconciling creative and programmatic (30%)
  • Time required to improve campaign performance vs. client’s budget (10%)
  • Cracking data budget (10%)
  • Duplication of audiences (10%)
  • Managing campaign cost (5%)
  • Terminology discrepancies between digital and TV departments (5%)

Normally on this blog we try to not talk too much about Thunder. But today I’d like to address these challenges, and show you how Thunder is working everyday to help you overcome them.

Continue Reading

Rich Media Advertising: Who’s Killing It And Why

Who is killing rich media advertising

The writing’s on the wall for rich media advertising as we know it today.

When it debuted, the rich functionality in rich media advertising was seen as a victory. At first it was animation, but then interactive elements became available. Banners could play videos, pop over your screen and expand over content.

The vision behind rich media advertising was admirable: make ads that people want to interact with. But it turned out to be a bait and switch.

Instead of endearing consumers to ads, these formats did the opposite. They pestered users with unwanted pop ups, aggravating expansions, and startling autoplay video with audio. They “politely” loaded megabyte after megabyte on people’s mobile data plans. They intruded at every opportunity because of marginal gains in CTR.

As the ad industry transforms, rich media is on the chopping block. Here are the top 5 attackers.

Continue Reading
  • 1
  • 2