By Editorial TeamWhen Sarah, the Head of Performance Marketing at “GreenThumb Gardens,” went to her CEO David in early 2025 to talk about integrating AI into their ad strategy, he was skeptical. It was understandable. The Portland-based e-commerce nursery was already burning through nearly $200,000 a month on paid acquisition that felt like a total black box. Organic growth was steady, but trying to pin a specific sale to a specific campaign was a constant headache. David’s question was simple and direct: if we spend on AI, how do we prove it’s bringing in *new* money we wouldn’t have gotten anyway? He wanted hard data on the incremental impact.
Key Takeaways
- Isolate the real lift from your AI campaigns by running strict A/B tests with ghost ads or geo-testing.
- Stop obsessing over last-click. You’ll measure real incrementality by tracking the uplift in conversion rates between your test and control groups.
- You have to spend money to learn. Earmark 10-15% of your marketing budget for these incrementality tests so you get statistically significant results you can act on.
- Get serious with the stats. Use models like causal inference or Bayesian methods to filter out the noise from seasonal trends or competitor sales.
The Attribution Conundrum: More Than Just Last-Click
Sarah knew exactly what David was getting at. The old attribution models, whether last-click or multi-touch, just don’t tell the whole story. A customer might see one of your AI-powered ads, search for GreenThumb Gardens later, and then buy a succulent. The ad gets the credit, but would they have bought it anyway? That’s the core problem of incrementality measurement, and it gets even trickier with the fast-moving, black-box nature of AI marketing.
“We were drowning in data, but starving for insights,” Sarah told me during a consult. “Our dashboards showed a great ROAS on paper, but the overall business growth didn’t always match up. It felt like we were just paying to retarget people who were already on their way to the checkout.” It’s a common story, and it’s why you can’t afford to skip a deep dive into incrementality. If you do, you’re flying blind and can’t tell if your campaigns are effective or just correlated with success.
GreenThumb’s first step with AI was to use an AI bidding strategy on Google Ads and a dynamic creative tool for their Meta campaigns. These tools promised to be more efficient and target better, but proving they added real, incremental value demanded a much more scientific approach than just watching ROAS figures go up.
“AI maturity measures how consistently a go-to-market team turns AI into measurable business outcomes. For marketing, that means AI runs in documented workflows, draws on clean customer data, operates under human oversight, and ties to metrics like MQLs, customer acquisition cost (CAC), and revenue.”
Establishing the Baseline: The Control Group Imperative
My advice to Sarah was blunt: you can’t measure lift without a proper control group. This is the absolute foundation of the whole process. For GreenThumb, we set up a two-part plan. First, we used geo-testing for their Google Ads. We picked three similar cities in the Pacific Northwest, places with comparable demographics but not their main markets. Two cities, Tacoma and Spokane, became the test markets where the AI campaigns ran full blast. The third, Boise, was the control. It got only the bare-minimum baseline ads, or in some cases, no paid ads at all for certain products. We had to isolate the AI’s effect. An IAB report from 2025 noted that companies using geo-testing this way had 12% higher confidence in their incrementality numbers.
For their Meta campaigns, we used what are called “ghost ads” or “holdout groups.” It’s a method where you create an audience segment that looks just like your target audience, based on demographics, past purchases, online behavior, but you intentionally block them from seeing your AI-driven ads. The difference in the conversion rate between the people who saw the ads and this holdout group gives you a direct measurement of the AI’s incremental lift. This requires getting your hands dirty with audience segmentation, and we used Meta Business Manager’s custom audience features to create these precise exclusions for GreenThumb.
Beyond Clicks: Measuring True Business Impact
The results after three months of testing were eye-opening. In the geo-test cities, GreenThumb saw sales of their potted plant collections jump by 15% in Tacoma and 18% in Spokane. Meanwhile, the control city, Boise, just plodded along with 3% organic growth. This was more than just clicks. It was more plants in boxes getting shipped to customers. “The difference was stark,” Sarah said. “Boise just kept chugging along, but Tacoma and Spokane really took off. It made David sit up and pay attention.”
The ghost ads on Meta provided even more specific insights. A campaign promoting new drought-resistant succulents saw a 2.8% conversion rate in the group that saw the ads, compared to just 1.9% in the holdout group. That 0.9 percentage point lift, when you scale it across the entire audience, meant an extra 300 sales that were directly caused by the AI-driven campaign. This is the kind of precision that turns marketing spend from a cost into something you can defend as a real investment.
But proper measurement isn’t just about running an A/B test. You have to do the math. We used a causal inference model to account for all the other stuff that could affect sales, like seasonal gardening trends, a sudden heatwave, or a competitor’s big sale. This is where you need some real data science expertise, because if you just compare two groups without controlling for these outside factors, you can easily fool yourself. If Spokane had a sunny week and Boise had a freak snowstorm, that would skew the plant sales numbers regardless of what ads were running.
The Challenges and Continuous Iteration
Getting these tests running wasn’t exactly a walk in the park. The first major hurdle was getting enough data for the results to mean anything. “We initially ran our geo-test for only a month,” Sarah admitted, “and the results were all over the place. We learned fast that you need longer test periods and bigger groups to get reliable numbers, especially when the lift you’re measuring is small.” A good rule of thumb in e-commerce is to run these tests for at least one full purchase cycle (two is better) to get clean data. A late 2025 eMarketer report pointed out that setting aside 10-15% of your marketing budget for testing isn’t just a good idea, it’s the cost of knowing what’s actually working.
We also got some internal pushback when sales reps started complaining that the control groups were “losing out” on sales. It’s a common fight. My response is always the same: the small, short-term “loss” in a control group is the price you pay for the long-term intelligence that keeps you from wasting huge chunks of your budget. Without that control, you’re just guessing, and guessing is way more expensive. It’s a trade-off that delivers priceless information about your actual campaign ROI.
To make the process smoother, GreenThumb eventually started using a dedicated incrementality testing platform like Measured. These tools helped automate setting up holdout groups and offered more advanced statistical analysis, integrating with their ad channels and CRM to give a complete picture of incremental impact across the board.
The Bottom Line: Actionable Insights for Growth
After six months of testing, GreenThumb Gardens had a crystal-clear picture of what their AI campaigns were actually doing. The AI-powered Google Ads were generating an extra $45,000 in monthly revenue that wouldn’t have happened otherwise, a 22% incremental lift over what they were doing before. On Meta, the dynamic creative was driving a 15% increase in conversion value on key product lines. These were tangible, defensible gains, and Sarah could finally give David a report that showed the true incremental revenue, not just a fuzzy ROAS number.
“The biggest revelation,” Sarah shared, “was realizing that some of our seemingly high-performing campaigns were actually cannibalizing organic sales. The incrementality tests helped us reallocate budget from those campaigns to truly incremental ones, improving our overall efficiency by 18%.” This was a huge win. They didn’t just find what worked. They found what *didn’t* and stopped wasting money on it.
Measuring the incrementality of AI marketing is a discipline, not a project you finish once. The algorithms change, markets shift, and people’s behaviors evolve. You need continuous testing, solid control groups, and real analytical models to build an intelligent marketing program. It takes an upfront investment of time and resources, but the payoff, finally understanding your true return on ad spend and turning marketing into a provable growth engine, is absolutely worth it.
FAQ Section
What is incrementality measurement in marketing?
It’s about determining the true causal impact of a marketing action. You compare a group that saw your campaign to an identical control group that didn’t, which lets you isolate the sales or leads that happened only because of your campaign. It answers the question, “How many of these sales were truly new?”
Why is incrementality particularly important for AI-driven marketing campaigns?
AI campaigns are often a black box of complex algorithms, so traditional attribution can’t easily isolate their impact. Incrementality testing is how you prove the AI is adding real value and not just taking credit for conversions that were already happening or stealing from organic channels.
What are common methods for measuring incrementality?
The most common methods are A/B testing with control groups (like holdout groups or “ghost ads”), geo-testing where you compare different markets, and media mix modeling (MMM), which uses high-level statistical analysis. The right method depends on your campaign type and budget.
How can I set up a control group for incrementality testing?
You need to create a segment of your audience that’s statistically identical to the group you’re targeting but is excluded from the campaign. You can do this using audience tools within your ad platform to create a randomized holdout group, or by picking entire regions to serve as your control in a geo-test.
What metrics are most important when evaluating incrementality?
Move beyond ROAS and CPA. For incrementality, you should be focused on the uplift in conversion rates, the amount of incremental revenue, and the incremental profit. These directly measure the extra value your campaign created. For a longer-term view, you can also track the customer lifetime value (CLTV) of the customers you acquired incrementally.
