A Practical Guide to A/B Testing for PPC (+5 Ideas to Test)

Intro

PPC A/B Testing is a powerful way to improve your ad campaigns’ effectiveness.

In this practical guide, you’ll find out what A/B testing for PPC is and learn about the different types of tests and testing statistics necessary for data-driven decisions. You’ll also learn how to set up your first A/B test and get practical high-impact ideas to try yourself.

What is A/B testing for PPC?

A/B testing for PPC is a method of testing 2 or more variants of your ad campaign elements, such as ad copy, landing pages, or targeting, with the goal of providing statistical proof for various hypotheses, which can be leveraged to refine your campaigns and improve the results.

While not entirely different from landing page or email A/B testing, PPC A/B testing requires a dedicated approach due to the ad platforms’ limitations, sample size variance, and risk of affecting the overall performance of your campaigns.

Types of PPC tests

There are four main types of A/B tests in PPC:

A/B tests
An A/B test is an experiment with one hypothesis that leads you to change a single element of your ad campaign and test it against the original control variant. This is the most common test type that helps you narrow down to specific elements and refine your campaigns.
Example of A/B testing: testing 2 text ads with free shipping vs. 15% off as the main offer.
Multivariate tests
Meet Ranktracker
The All-in-One Platform for Effective SEO
Behind every successful business is a strong SEO campaign. But with countless optimization tools and techniques out there to choose from, it can be hard to know where to start. Well, fear no more, cause I've got just the thing to help. Presenting the Ranktracker all-in-one platform for effective SEO
We have finally opened registration to Ranktracker absolutely free!
Create a free account
Or Sign in using your credentials
A multivariate test is an experiment with multiple hypotheses and multiple changes. With this method, you’re testing different combinations of small changes made to your control variant. I rarely use this type as it requires the highest sample size (often impossible for PPC) out of all four test types, and generates the smallest uplift in results, thus decreasing the confidence level (see my definitions of sample size, uplift, and confidence level in the next section)
Example of Multivariate testing: testing 4 creatives with different combinations of headlines and images.
A/B/n tests
An A/B/n test is also an experiment with multiple hypotheses and multiple changes. However, unlike in the case of multivariate testing, the variants can be completely different from each other. It’s one of the test types I frequently use for new accounts or new campaigns where historical data isn’t available, and I want to test altogether different setups or combinations of elements rather than narrowing down my selection with A/B or multivariate testing.
Example of A/B/n testing: testing 2+ sets of creatives with completely different layouts and/or landing pages.
Sequential tests
A sequential test is a type of A/B test that tests campaign element variants in phases or sequences. A sequence can be 2 weeks, 1 month, or longer (I don’t recommend running a test for less than 2 weeks). This is the least preferred test type, as running a test during different time periods introduces outside factors you can’t control, such as seasonality, sample size variance, and targeting deviation. However, it’s also a common type, as not every PPC platform offers full (or any) A/B testing features.
Example: testing Maximize Conversions bidding vs. Maximize Conversion Value in Google Ads

In an ideal scenario, you would employ all tests in the following sequence:

A/B/n testing to find the setup that works best
A/B testing to narrow down and refine your setup
Multivariate testing to narrow down your setup further
Sequential testing to test elements in sequential order when there’s no proper A/B testing functionality

A/B testing statistics

For A/B testing to provide statistically significant data, inform your decisions, and lead to improvements in PPC, there are 4 key statistics you need to consider:

Sample size
Meet Ranktracker
The All-in-One Platform for Effective SEO
Behind every successful business is a strong SEO campaign. But with countless optimization tools and techniques out there to choose from, it can be hard to know where to start. Well, fear no more, cause I've got just the thing to help. Presenting the Ranktracker all-in-one platform for effective SEO
We have finally opened registration to Ranktracker absolutely free!
Create a free account
Or Sign in using your credentials
In PPC, your sample size is the amount of traffic you need to generate for the test results to be representative of your audience. For ad-level metrics (such as CTR or View Rate), impressions will be the source of your samples, but for conversion-specific metrics (such as Conversion Rate, Cost/Conv., or ROAS) you should pick clicks. In general, the larger the sample size, the more accurate your test will be.
Expected uplift
A prediction on how a tested change will affect the final metric, expressed in percentage and ranging from 0 to 100%. For example, based on historical data and conversion research, you might predict that a change in the main offer from 10% Off to Free Shipping will increase the conversion rate by 30%.
P-value
We’re in the advanced statistics territory. To put it simply, the p-value helps determine whether the results deviate significantly from what would be expected, or how statistically significant the results are. It ranges from 0 to 1, and the smaller the value, the more statistically significant the results.
Confidence levels
Confidence levels or confidence intervals are a measure of certainty in test results. For example, a 95% confidence level means that if we repeat the same test multiple times, 95% of the tests would produce similar results.

Why is PPC A/B testing important?

A/B testing affects 3 key areas of your PPC campaigns:

Results
When working on PPC campaigns, you’re constantly facing the question “Will thing A do better than thing B?” (replace ‘thing’ with campaign/ad/copy/audience/angle/etc.). A/B testing equips you with a way to answer such questions, test different hypotheses, and, ultimately, improve your results.
Structure
If, like me, you’ve felt like some of your optimizations were too ad hoc, reactive to the data on hand, or even cosmetic, A/B testing is the approach that will help you add more structure. It can help create performance “footholds” (proven hypotheses) and focus on finding the most impactful optimization opportunities instead of cosmetic changes.
Communication and Engagement
If you’re an agency or an in-house specialist, you’ve most likely experienced communication and engagement issues with clients or executives. A/B testing can help solve some of these issues, as it offers another layer of transparency, awareness, and engagement. If nothing else, it lets you give a quick answer should anyone ask “Have you tested a green button instead?” :)

What can you A/B test?

Deciding what to A/B test in your PPC campaigns is crucial. I recommend starting with the elements that, if improved, could have the highest impact on your results.

Creatives
Examples: layout, color scheme, model vs. no model, short-form video vs. long-form, UGC vs. own assets.
Offer
Examples: free shipping vs. discount, free bonus vs. scarcity, free trial vs. freemium, guarantee vs. no guarantee, webinar vs. ebook.
Ad placement
Examples: Facebook vs. Instagram, mobile vs. desktop, search vs. search partners.
Ad copy
Examples: Long-form vs. short-form copy, bullet list vs. paragraph, including the word ‘free’ vs. not, benefits vs. authority.
Targeting
Examples: new keywords, narrow targeting vs. broad, lookalike vs. cold, older remarketing audiences vs. younger, phrase match keywords vs. broad, narrow location targeting vs. broad.
Campaign/Ad types
Examples: DSA vs. regular search campaigns, dynamic remarketing campaigns vs. regular remarketing, lead ads vs. messenger ads.
Budget allocation
Examples: more budget towards campaign 1 vs campaign 2, more budget towards remarketing vs. acquisition, more budget towards Performance Max vs. Shopping.
Landing pages
Examples: layout, images vs. videos, dynamic keyword insertion, headlines, forms, social proof, ad-to-landing page message match.
Bidding strategies
Examples: Maximize conversions vs. maximize conversion value, Target CPA caps, Target ROAS targets, highest volume vs. highest value.
Campaign structure
Examples: Broad(or Hagakure) structure vs. granular, more dynamic/automated campaigns vs. fewer, best-performers vs. low-performers, SKAGs.

How to A/B test your PPC campaigns

Setting up your A/B test

Once you’ve come up with a list of ideas to A/B test it’s time to form hypotheses and decide on the approaches and tools.

Hypothesis

Your hypothesis is the assumption you are trying to test with the experiment. It expresses the effect you expect to see from making a change, such as revising ad copy, changing your ad creative, or expanding your targeting. To structure my hypotheses, I like to refer to the Hypothesis Kit V4 by Craig Sullivan:

Based on (data/research/observation)
we believe that (change)
for (population)
will cause (impact).
We will know this when we see (metric).
This will be good for customers, partners, or our business (because).

Approach

This is where you decide how to approach your test. Will this be an A/B test? A/B/n? Sequential? It’s important to identify this from the start as it will impact your A/B testing tools, budgets, and outcomes. As mentioned above, I recommend starting with A/B/n tests if you don’t have any historical data and your hypothesis is based on observations. However, with certain tests and ad platforms, you’ll be limited to sequential test approaches (i.e. bidding strategies on Google Ads).

Tools

When it comes to PPC A/B testing a spreadsheet dashboard is your best friend. If you’re not sure where to start with it, you can find my most recent dashboard here. If you’re running just a few tests per quarter, I recommend you fill it out manually. If it’s more than a few, you can automate it using such tools as Supermetrics to pull PPC data.

excel

Launching your A/B test

Your launch instructions will depend on the tested element and the ad platform you selected. One thing will remain the same, though - your experiment needs to produce equal or almost equal sample sizes for both control and test variants, meaning proper A/B tests should never be launched into the same campaign or ad group unless you can control the budget and traffic spread (i.e. Ad set budget optimization campaigns, or ABO, in Facebook Ads).

Here are the test setups I use most often:

Facebook/Instagram/Pinterest/LinkedIn: the native A/B testing feature, new ad sets, new campaigns, sequential launches.
Google/Microsoft: the native campaign experimentation feature, ad copy A/B testing feature, equal ad rotation feature, sequential launches.

Analyzing the data

You crafted a hypothesis, set up the test, and let it run its course. What now?

Fill out your dashboard, and see if your test produced the expected uplift, if your sample size was large enough, if your results are statistically significant, or if your test needs more time to reach higher significance.

You can use a calculator to help with sample size, and confidence/significance calculations.

If you have a clear winner, formulate a conclusion and prepare an action plan for introducing it into your PPC setup.

5 PPC A/B testing ideas to try

1. Offer testing

When it comes to maximizing PPC results, don't underestimate the impact of testing different offers. In my experience, this produces the most significant changes in results.

This can include scarcity (think limited supply), urgency, bonuses, guarantees, or discounts.

When available, remember to use the native ad copy testing feature to ensure more control over sample sizes and traffic splits per variant (such as the ‘Ad variation’ experiment type in Google Ads).

2. Landing page testing

“Wait, I thought this was a practical guide to PPC testing?”. In my experience, landing pages are one of the highest-contributing factors when it comes to success with PPC. If your landing page isn’t well-optimized, it doesn’t matter how good your ads are - your results will still be limited.

For the largest uplifts, I recommend starting with layout and form testing as these could contribute to the most significant uplifts in conversion rate. For example, this credit card company saw a 17% increase in conversion rate after optimizing its form.

Next, consider ad-to-message matching and headline testing to improve your ad-to-conversion flow.

3. Creative testing

According to Nielsen, ad creative quality contributes to 49% of incremental sales and is the most critical driver of advertising effectiveness. This is why I always recommend doing high-frequency creative testing across creative-first channels, such as Facebook and TikTok. It was also a significant contributor to my client’s 54% increase in bookings in just 6 months.

For the largest uplifts, I recommend testing layout changes, messaging, and UGC content.

4. Targeting testing

Targeting testing is yet another idea I recommend trying for the highest potential uplifts. As mentioned in the “What can you A/B test” section, these can include new keywords, narrow targeting vs. broad, and lookalikes vs. saved audiences.

For example, you might want to test a separate long-tail keyword campaign vs. a short-tail one to see if you can improve budget control and reduce your CPA.

For this, I recommend using a tool like RankTracker’s Keyword Finder to help you get more advanced keyword suggestions and filtering than you’d get with Google Keyword Planner.

keyword finder

5. Bidding testing

A/B testing bidding strategies can be a powerful way to optimize your PPC results. This can reveal whether your current bids are too high or low, whether you’re optimizing for the highest-value customers or not, and whether it’s best to go for the highest number of conversions (quality) vs. the highest conversion value (quantity).

For example, you can test increasing your Target CPA limits by 30-50% to see if you’re missing out on clicks that could result in conversions, or decreasing your Target ROAS by 25% to generate a higher volume of conversions during a high-competition period (i.e. Black Friday).

bidding