A/B testing is a lie. Here’s the truth.

A/B testing has long been used to find the perfect ad creative – but its effectivess is greatly exaggerated. Here's the truth of A/B testing, and how to find the best creative without it.

“In reality, A/B testing wastes huge amounts of money, time and resources in pursuit of the perfectly optimised creative”


The marketing industry has a dirty little secret: A/B testing isn’t effective as we’ve been told

In reality, the vast majority of teams implementing it just aren’t getting reliable data or actionable insights from the approach – and are wasting valuable time and resources in the process.

So, what can we do instead to make sure we’re using the very best ad creatives possible? 

A/B testing: The holy grail of marketing?

Finding the right creative for your audience is incredibly important for the success of your ads. According to Stand Up For Creative, a campaign’s success relies 70% on the creative (and just 30% on the media plan), so it’s critical that each campaign is using the best creative possible

A/B testing is one practice for discovering the best creative that many data-driven marketers swear by. It’s all about finding the perfect image, text and CTA (among other things) for different audiences.

A/B testing works by testing two (or more) versions of an ad on the audience to see which version is the most engaging. Using the approach, teams can test which image, headline, CTA, text and more drives the highest performance for that ad.

However, A/B testing isn’t the absolute answer we’ve been led to believe it is. In reality, the approach wastes huge amounts of money, time and resources in pursuit of the perfectly optimised creative. 

“Only 10% of A/B testing experiments result in actionable change ”

While all the creative elements rotate in the A/B test, thousands, even millions, of possible customers end up seeing un-optimised creatives that don’t push the right message – and that don’t lead to a sale. This equates to many, many lost opportunities, and wasted budget


In fact, some of our clients say they waste up to 70% of their budget on A/B testing before they end up finding the creative they settle on. And many never even find the perfect creative at all

Here’s the truth:

  • Only 25% of A/B tests actually produce significant results (Appsumo)
  • Only 10% of A/B testing experiments result in actionable change  (Northwestern, IDS )
  • 50% of teams could not make decisions from A/B testing experiments due to inconclusive or poorly measured data (Northwestern, IDS )

Since finding the perfect creative is so critical to the success of a campaign, what approaches can marketers use to make sure these creatives will hit the mark, without relying on flawed A/B testing?


Do less A/B testing for longer

One of the big issues with A/B testing is that it’s not often used correctly.

Northwestern IDS concluded that most companies are “running A/B tests too frequently for too little time, contributing to a high failure rate that makes A/B test results less valuable and meaningful”. 

“Less frequent A/B tests are more likely to result in successful experiments”

With this in mind, it makes sense to simply give A/B testing more time

Less frequent A/B tests that last longer than a month are more likely to result in successful experiments that let you know what the best creative is. 


Saying that, we probably don’t need to tell you that a month in a marketing team is a long time. If your message or approach pivots at any point, you’ll need to start the process from the beginning. 

And even if you commit to a lengthy process of A/B testing, waiting a longer time to be sure you’re using a properly optimised creative can seriously delay the time it takes to see and action results.

Persona-led creative selection

Building out personas for your audience is one common approach to crafting creatives that will be engaging for them – especially if those personas are built using the data company’s hold on them. 

With a better idea of who it is you’re trying to appeal to, you’ll have more guidance on what they’ll want to see from your creatives, meaning higher-performing ads, in theory.


However, even if you’ve done all the research possible and you’re using the most detailed personas you can build, you’re still left relying on guesswork to craft and select the best creatives.

Insights from panels 

Hearing from your audience themselves about what they find engaging in creatives can be very helpful in crafting genuinely effective creatives. 

This is why testing your creatives using panels can shed a great deal of the guesswork that goes into making ads. A/B testing no longer has to be the middleman when you know exactly what your customers want from creatives.

However, panels aren’t a magical solution. The creatives you are bringing to them have still been created using guesswork, after all. And they rely on people knowing exactly what their content preferences are. 

In reality, what people find engaging in creatives is not as fixed as we might think. A lot of their content preferences are unconscious, and they may struggle to identify in a panel setting what creatives they would actual click on ‘in the moment’.AB-testing-lies

Pre-optimising with creative data – and Content Atomisation

All of these approaches outlined may be effective, but they all have the same problem: The creatives that are being tested have been created using guesswork in the first place

“These approaches have the same problem: The creatives that are being tested have been created using guesswork in the first place”

So, how can you take guesswork out of the loop altogether? 

Content Atomisation is an approach informed by data which has been proven to be enormously effective at driving performance while removing guesswork from the equation. 

Here’s how it works: Content Atomisation uses AI to analyse the performance of individual creative features. Once you understand what features are driving engagement, you understand exactly what your audience wants to see – from whole groups right down to the individual – and your messaging engages them. Every. Single. Time. 

This means that A/B testing is no longer necessary, because with Content Atomisation you know exactly what creatives perform for your audience

Read more about how Content Atomisation works. 

At Datasine, we’ve created a tool that allows you to easily implement Content Atomisation – and early adopters have seen gains of up to 59% in conversions.  

Our tool lets you:

  • Access to AI predictions into how well any number of creatives you upload to the platform will perform for your audience
  • Select highly-engaging assets from a variety of content libraries
  • See actionable insights into what content works for your brand – and why


Want to learn more? Organise a call with our friendly team to talk about the impact Content Atomisation can have for your brand now.

Filter by Category:
Jergan Callebaut
The AI tool marketers are using to increase Facebook campaig...

A new breed of marketing tools is here that leverages machine learning and psychology to pick high-converting creatives for your Facebook ads.

Use AI to identify high-converting marketing images

Cutting-edge AI (artificial intelligence) research into how computers see marketing images is changing the way marketers build more successful campaigns.

Adtech vs martech: what’s the difference?

Adtech and martech are terms that are regularly conflated. Let’s put aside misconceptions to define adtech and martech – and their differences.

In Conversation with… Bernardo Nunes, Head of Science at G...

Growth Tribe’s Bernardo Nunes joins the conversation to talk artificial intelligence in marketing: how companies can leverage AI and cultivate tech talent.

In Conversation with… Harry Hugo, Co-Founder of The Goat A...

Serial Entrepreneur and Co-Founder of The Goat Agency, Harry Hugo, joins the conversation to talk social media, influencer marketing and founding Goat.

Forms of personalisation in today’s digital world

Personalisation means connecting with your audience on a deeper level. Our Head of Psychology, Jergan Callebaut, looks at its many forms.

What makes us click? Increase ad performance with creative d...

Datasine’s CEO Igor Volzhanin outlines his MAD//Fest presentation, explaining why using creative data and AI leads to a steep increase in ad performance.

Facebook Ads A/B test guide

Facebook Ads A/B testing is crucial to understanding what marketing content works best. Datasine’s PPC Manager outlines how Facebook Ads A/B tests work in this simple guide.

In Conversation with… Mohamed El Hannaoui, Growth Marketer

Experienced Growth Marketer Mohamed El Hannaoui joins the conversation to talk growth marketing in startups and how AI is going to change marketing

Five of the best tools for Facebook Ads in 2020

Datasine looks at the five of the best – and most affordable – tools on the market for creating and optimising Facebook Ads.

What’s the most important part of Facebook Ads? The creati...

The data boom has driven focus on areas such as targeting, brand and reach for Facebook Ads, yet the creative remains the most impactful factor in successful Facebook Ads

Your images matter: five reasons why

Got your attention with the picture above? Great! I’m sure you’ll love the rest of this article. It probably doesn’t come as a surprise to you that we live in the age of visual culture. Ten percent of all photos ever taken by humankind took place in the last 12 months.

Growth hacking Facebook ads

Growth hacking Facebook ads sends engagement skyrocketing and there are many ways to optimise paid ads to drive brand awareness and sales.

Speaker spotlight: Emily Miller, Head of Marketing at Ollie ...

Datasine talks marketing, AI and personalisation with Emily Miller, Head of Marketing at Ollie Quinn, ahead of her fireside chat at #SuperchargeMarketing.

You’ve overlooked creative data. But we’re pioneering a ...

Creative data has been overlooked by marketers. But at Datasine we’ve found a way to drive performance with this powerful data: Content Atomisation – supercharged by AI.

The “integrators”: Why fusing creativity and data drives...

A McKinsey report has concluded that marketers who use creativity and data together achieve a much higher performance for their brand

Speaker spotlight: Mack Grenfell, Biddable Lead at Babylon H...

Datasine talks to Mack Grenfell, Biddable Lead at Babylon Health about falling into the industry, and how AI is helping marketers optimise spend.

Speaker spotlight: Mitch Platt, UK Country Manager at Growth...

Datasine discusses growth marketing, personalisation and AI adoption with Mitch Platt, UK Country Manager at Grow Tribe, ahead of his presentation at #SuperchargeMarketing.

Running a data-driven marketing team

Data-driven marketing is a key focus for marketing teams today. I spoke with Alessio Pieroni, Head of Marketing at Mindvalley about how to practice it successfully.

Semantic content analysis will disrupt marketing – for the...

Discover the benefits semantic content analysis brings, and how to utilise data, psychology and AI to predict the highest-converting content for your brand.

Tackling data-driven disruption as a Creative

Data-driven disruption is sweeping through creative industries. We sit down with Wayne Deakin, Executive Creative Director at Huge EMEA to talk about facing it head on.

Nine Mailchimp integrations for personalisation

Check out our top nine Mailchimp integration picks for empowering marketers to increase campaign metrics and improve customer experiences.

In Conversation with… Meagan Healey, Head of Content &...

Head of Content & Product Marketing at Attest, Meagan Healey, joins the conversation to talk data-driven marketing and why aligning sales and marketing is a game changer.

IWD: 20 marketing leaders in 2020 [1/2]

For International Women’s Day 2020, 20 female marketing leaders weigh in on the advice they’d give their younger selves [1/2]

Speaker spotlight: Karla Rivershaw, Head of Marketing of Tur...

Datasine discusses the tools that make marketing work, and finding the right balance when personalising communications with Karla Rivershaw, Head of Marketing at Turtl.

Which Tokyo 2020 art poster does our AI say is the most enga...

Datasine Connect, Datasine’s AI platform, analysed the 12 official art posters for the Tokyo 2020 Olympic games to discover which was the most engaging for the UK population

3 simple tips for building better Facebook Ads

Datasine’s Performance Marketing Manager gives 3 simple steps for building better Facebook Ads, including optimising creatives and using AI

Three Mailchimp newsletter growth hacks

Email is one of the most effective marketing channels and Mailchimp has long been a favourite amongst marketers and growth hackers. Learn how to up your Mailchimp newsletter game with our growth hacks. 

The three must-ask questions before a rebrand

Datasine’s Marketing Director, Cornel Lazar, shares the questions each brand must ask before rebranding, and Datasine’s journey to change its look and feel.

Speaker spotlight: Emma Wynne, Head of Growth Marketing at P...

Datasine talks about personalising marketing communications and marketing tools with Emma Wynne, Head of Growth Marketing at Pasta Evangelists.

Can AI optimise the creative process?

While artificial intelligence (AI) has had little impact on the creative sector so far, AI has huge potential to optimise ad creatives

How to grow your brand using paid social

Brand Manager and Founder of, Magda Urbaniak, shares her best practices for brands paid and organic social efforts

Which images make the most successful Facebook ads?

Images are the cornerstone of successful Facebook ads. We look at how to find the best images to create more effective Facebook ads.

Marketing 20:20: meet the Speakers

Datasine sits down with the two guest speakers of Marketing 20:20, Becky Simms, Founder and CEO of Reflect Digital and Social Media Expert Kieran S Lawler

What is a rebrand? (And what is it not?)

I sat down with our Marketing Director to discuss what a rebrand is, what goes into a rebrand and how Datasine approached the project.

MAD//Fest Day One: the recap

We look back on our first day at MAD//Fest with presentations from Burger King, Virgin Trains and our own CEO Igor Volzhanin.

The life of a Creative Director – in the age of data and A...

Data and AI are becoming more vital to the role of the Creative Director. Datasine sits down with Alex Colley, Creative Director of ikon, to find out more about the impact tech is having.

IWD: 20 marketing leaders in 2020 [2/2]

For International Women’s Day 2020, 20 female marketing leaders weigh in on the advice they’d give their younger selves [2/2]

Mirroring at scale

How mirroring can help you level up the impact of your one-to-many communications, offering personalised content and customer experience.

Delivering truly personalised email marketing experiences

Datasine sits down with Lemlist CEO Guillaume Moubeche to talk about delivering personalised email marketing experiences and building relationships.

A quick guide to remarketing with Facebook dynamic ads

Facebook dynamic ads, or Facebook product ads, let you remarket to your audience. Find out how to use them in this simple guide to DPA ads.

How AI is set to transform email marketing

Artificial intelligence (AI) is increasingly used in email marketing, learn how machine learning enhances personalised email campaigns.

What is segmentation in marketing?

Segmentation in marketing allows marketers to place customers into groups for targeting. We take a look at the types of market segmentation.

Make your marketing pandemic proof

In crisis times, marketing budgets get slashed. Here’s how we can support you to stretch your budget further and drive sales – even in the face of a pandemic.

Five things we learned at Supercharge Marketing

Datasine looks back at the five things we learned at Supercharge Marketing, from data-driven marketing to investing in learning new skills.

What does your audience want to see from your branding?

Datasine sits down with Flavilla Fongang, Creative Brand Strategist and Founder 3 Colours Rule to talk branding, neuromarketing and the future of the industry

From binary baby to “Friends selfie”: A (brief) ...

We see thousands of images online every day. We take a trip back over the history of the digital image to see how we got to this point.

Are you blinded by data?

Learn how developing an Integrator Mindset can help you to creatively personalise and build genuine relationships with your customers

How to send the right message at the right time, in travel m...

We speak Joe Stevens, Corporate Digital Marketing Manager at Flight Centre about using data to send the right message to the right person at the right time

AI-powered marketing needs interpretability – and collabor...

Artificial intelligence (AI) should be taking marketing by storm, but its lack of interpretability for most marketers is holding back innovation and adoption.

How to use AI to optimise your marketing content

AI can drive marketing success by optimising audience targeting and reach but, more importantly, AI optimise your creative content through Content Atomisation.

How to use data to inspire customer loyalty in the travel in...

Lack of customer loyalty is a key problem in the travel industry. We take a look at some of the ways data can tackle the issue.

Creativity back on top: Why creative data will dominate this...

Data-driven marketing has swept through the industry, but the data focus has led to a decline in creativity. But you can combine data and creativity by uncovering your campaign’s creative data.