When you’re optimising campaigns, customer journeys, or product experiences, testing is your best friend.
Recently, I wrote about A/B testing: what it is, why it matters, and how to use it. That post covered how to compare two versions of a page, email, or campaign element to see which performs better.
But A/B testing isn’t the only option. Incremental testing offers an alternative approach to measuring impact, particularly when assessing the incremental lift from a change, rather than a direct side-by-side comparison.
Let’s explore the differences, pros and cons, and where each approach shines.
What is A/B Testing?
A/B testing pits two versions (A and B) against each other. You split your audience, show each group a different version, and measure which performs best.
It’s simple, clear, and perfect for testing things like headlines, call-to-action buttons, or email subject lines.
👉 Full details and practical examples are in my earlier blog: A/B testing explained.
What is Incremental Testing?
Incremental testing measures the added value (or increment) of introducing a new element compared to a control group who sees none of it.For example:
- You run a campaign with a loyalty discount to one group, and another group sees no discount at all.
- Instead of comparing two different creatives, you’re measuring the uplift the discount itself drives.
It’s not about which version wins, but how much extra impact your change brings compared to doing nothing different.
Key Differences
Aspect
A/B Testing
Incremental Testing
Purpose
Compare two (or more) versions
Measure additional lift of a change
Setup
Audience split between variants
Test vs control (with no change)
Focus
Which performs best
How much value is added
Best for
Creative elements, UX tweaks
New features, campaigns, incentives
Pros and Cons
A/B Testing
Pros:
- Simple to set up and explain.
- Great for small, clear changes (e.g. button colours, copy).
- Provides a direct winner.
Cons:
- Can’t always show the true incremental value of an idea.
- Needs high traffic to reach statistical significance.
- Doesn’t account for external factors (seasonality, market changes).
Incremental Testing
Pros:
- Shows the true additional impact of a campaign or feature.
- Useful for understanding ROI on promotions or incentives.
- Works well with lower-volume data.
Cons:
- More complex to design and analyse.
- Results can be harder to communicate to non-data stakeholders.
- Doesn’t always tell you which creative version works better, only whether the tactic adds value.
Where to use each
- Choose A/B testing when you want to refine messaging, design, or user experience. It’s the go-to for quick, tactical improvements.
- Choose incremental testing when you’re assessing the overall value of a campaign or feature. It helps justify investment by proving whether the tactic truly drives extra results.
In practice, most organisations benefit from using both. A/B testing sharpens the details, while incremental testing gives the big-picture ROI.
Final Thoughts
If you only run A/B tests, you might miss whether your new campaign element actually creates a genuine uplift. If you only run incremental tests, you might miss the creative refinements that make experiences smoother and more effective.
As someone recently said on Reddit:
A/B testing helps me spend my budget more efficiently. Incrementality testing helps me justify my budget in the first place.
Both belong in your marketer’s toolkit, and knowing when to use each helps you build strategies grounded in evidence rather than assumption.
#DigitalStrategy #DataDrivenMarketing #MarketingOptimisation
