A/B Testing: The Value of Knowing What Works
Insights, delivered.
You’ve probably heard us say it before: Research kills opinions.
Virtually anything you want to know about your audiences can be proven. As a communicator, you should have a baseline for how much your audiences are engaging with you, and always be thinking of new elements to increase that engagement. That could be something as small as a line of text on your homepage to something as big as your organization’s overall messaging. As long as you’re routinely testing those new elements against your baseline, your marketing efforts will be refined over time and become more effective.
That’s the power of A/B testing. In fact, 60% of organizations find A/B testing to be highly valuable for optimizing their conversion rates.
What Is A/B Testing?
A/B testing, also known as split testing, is a randomized experimentation process where two or more versions of a variable are shown to different segments of users at the same time to determine which version is most effective. The A refers to the control (or baseline), and the B represents the altered or new variable.
In the big umbrella of marketing, A/B testing usually falls under a Conversion Rate Optimization (CRO) strategy. It’s a method designed to illuminate what your audiences will respond to the most.
Always Be Testing
Always. Be. Testing.
The best marketing teams lead with data. With A/B testing, you’re always challenging that data to see if you can get better results—and you can A/B test virtually anything. You have to be intentional about testing, which means allocating part of your budget to it, allocating enough time, being open to testing new things, and being willing to be wrong or to fail. You should strive to build a culture that prioritizes testing because you understand how it can influence your outcomes.
A/B testing is optimal for low-risk changes and modifications. A (non-exhaustive) list of those include:
Headlines/Descriptions of landing pages or ads
Calls to action (CTAs) on an ad or landing page
Email subject lines
Images on ads
Simple design tweaks on ads/landing pages
Landing pages forms (ex: required vs non-required fields, progressive fields, types of fields asked)
Target audiences for ads (ex: countries, regions, age ranges, etc.)
Ad bidding methods
Links to different landing pages
With testing results and goal conversions, you’ll want to consider these metrics:
Website traffic
Conversions & conversion rate
Bounce rates
Abandonment rate
Engagement (likes, comments, shares)
A/B Testing vs. Iteration Testing
While it’s good to regularly practice A/B testing, it may not be able to answer all your questions. Keep in mind that A/B tests can only test one element or variable at once. If you test too many variables, you won’t be able to prove causation for a change in user behavior. Instead, multivariate testing or iterative testing may be a better option for larger projects.
Tips for Building Your Testing Criteria
First things first: Create a hypothesis.
If A is what you’re doing, what influence are you proposing B will have on your audiences? Do you have any existing data that supports this (It’s ok if you don’t!)?
Remember—you’re only testing one variable at a time. For example, you could test:
Two different CTAs on an ad or landing page
Two different images in an ad
Two different headlines on a landing page (not two different headlines and two different sets of body copy)
With any test, the audience sizes exposed to the different variables should be of similar size—ideally equal. More than that, your audiences should be randomly selected to ensure no other errors in sample quality. Always run the two variables simultaneously to avoid potential variances in time. Data can vary from week to week or even from day to day!
Lastly, never ever touch your A/B test while it’s running. If you do, you’ll manipulate the results and it will all be for nothing! Let your test run to completion. You can always test something else after.
A Quick Note on Statistical Significance
Not to get too technical here, but an important part of the methodology of testing is statistical significance. To put it plainly: If the size of your audience (the sample size) is not large enough, your data won’t actually be representative of your audience as a whole. It’s up to you to ensure that your sample size is large enough to be statistically significant.
For example, if you’re running an A/B test on a social ad, make sure your budget is large enough to garner enough impressions so your data is accurate. You can use this resource to calculate the statistical significance of your test.
As a rule of thumb, you want to shoot for at least a 95% confidence level, ideally 98%.
After the Test
After conducting your A/B test, revisit your initial hypothesis and analyze your results. Then, take action based on the data. If your B variable created a drastic, positive change in engagement, your next move is to implement that variable for your entire audience.
It is possible that your A/B test will come back as inconclusive, or as a false positive. In that case, test again! Go back to your hypothesis and start again with a new variable.
Tools for A/B Testing
There are a handful of effective tools you can use to conduct or support your testing. At Mighty Citizen, we’re big fans of Google Optimize. Google Optimize is a free tool you can use to compare variations of your website. With it, you can increase conversions and overall user experience.
Some other tools you can use for testing are:
Our Work with Humanity & Inclusion
In our award-winning work with Humanity & Inclusion (HI), we used paid social ads to help HI gain insights into their target audience. Read how we used A/B testing on social media advertising to research the messaging that best engaged HI’s donors and supporters:
Mighty Citizen Can Help
Does your organization need a comprehensive strategy for A/B testing? Our experts can help you identify what to test, when to test it, and what to do with your results. Drop us a line—we’d love to hear more.
