Incrementality testing is a scam.

And I say that as someone who used to sell it.

Almost every founder I talk to comes out of an incrementality test feeling one of two ways.

Either they're too dumb to get it, or they were never really meant to. It's the second one.

So let me explain the whole thing from the ground up, because it only works on you if you can't see where you're standing.

And most people can't, because nobody bothers to explain it.

Start with what incrementality testing is even supposed to do.

You spend money on ads. Sales come in. The obvious question is: how many of those sales did the ads actually cause? Not how many showed up after the ad — how many happened only because of it. Because some of those people were going to buy from you regardless.

They already knew you.

They were already looking.

The ad was just the last thing they saw before doing the thing they were going to do anyway.

Incrementality is that gap. The sales that wouldn't exist if you hadn't paid. Everything else is credit the ad is taking for work it didn't do.

That's a real question. It's the right question. Nothing I'm about to say is an argument against the question. It's an argument that the way we answer it is broken before it starts.

Now, what you actually buy on Facebook and Google — where most of you buy your media.

It used to be simple. You bought impressions. You picked who to target, you picked what you'd pay, you put your ad in front of them, you watched what happened. You owned the inputs. If you wanted a clean test, you could run one, because you controlled who got the ad and who didn't.

That's not what you buy anymore. You buy outcomes.

You tell the algorithm "get me purchases at $30," and it goes and does that. But to get it, you gave up control. You don't decide who sees the ad. You don't decide when.

You don't decide what you pay to reach any given person. Smart bidding decides all of it now. You handed over the wheel in exchange for a promise about the destination.

And the algorithm has exactly one job it cannot fail: hit your number. If it stops delivering purchases at $30, your campaign dies and the budget moves on.

So it does the obvious thing. It finds the people most likely to buy and puts your ad in front of them. That's not cheating. That's precisely what you paid it to do.

Here's where people push back. Hold on — my spend clearly drives my revenue. I turn it up, sales go up. I turn it down, they drop. And depending on where you are, you're completely right.

This is what i would tell to my media buyers when i was running agency. For some clients, spend isn't incremental. It's existential.

If nobody knows your brand yet, there's no pool of people who were going to buy you anyway. Nobody's searching for you. Nobody's coming back on their own. So the algorithm has no demand to harvest — there's nothing there. Every sale is one you paid to create. Turn spend off and you don't lose a little lift, you lose the whole business. At that stage you don't need a test to tell you anything. Spend and revenue move together because they're more or less the same thing.

The problem starts once you grow.

People search for you. They come back. They buy again. A chunk of them would buy without ever seeing an ad. And now, finally, the algorithm has something to harvest and it will, because that's the cheapest way to hit your number.

Spend and revenue stop moving together cleanly, because now some of your revenue was always coming and some of it wasn't, and from the outside they look identical.

That's the exact moment someone shows up to sell you incrementality testing. Right when the clean picture breaks and you start getting nervous.

So you run a test. A test means a holdout. You show the ad to one group of people and you hold it back from another. Then you compare the two.

That only works if the ad money gets spread across people evenly — a fair slice sees it, a fair slice doesn't, and the two slices look alike going in. That's the test you think you're running.

That's not the test Meta runs.

You don't decide who the money chases. The optimizer does. And the optimizer has one instruction: find conversions at your price. So it doesn't spread anything. It goes straight for the people already closest to buying, because they convert cheapest and that's how it hits your number.

The optimizer spends your budget on people ranked by how likely they were to buy anyway. The surer the sale, the more it wants that person. It is, by design, pouring your money into the exact people who least needed to see an ad.

So of course the test says non-incremental. The money went to people who were going to buy regardless.

And you can't switch it off. There's no setting that says "stop chasing sure buyers, go find people I could actually persuade." Chasing sure buyers isn't sitting on top of the campaign. It is the campaign. Turn it off and there's nothing left to run.

That's the trap, and it doesn't care how you slice the groups. Split them fair, split them by geography, split them however you like — the optimizer still spends every dollar on whoever's closest to converting, inside whatever group you give it. The flaw was never the split. It's the machine doing the one job you pay it for.

This is also why the "independent" tools are a joke. They're not independent from anything. Every dollar in that test was spent inside Meta, on people Meta picked. The tool just takes Meta's numbers and puts a nicer dashboard on them. You can't check Meta's work with data that Meta produced.

I'm not saying throw it all away. Run a test and you still learn something small. Campaign A did better than campaign B this week. That's real, and it's fine to use. It's a thermometer.

But a thermometer isn't what they sold you. They sold you the truth. The real number under everything. And that number doesn't exist here. The same company runs the auction, picks who sees the ad, splits your test groups, counts the sales, and decides what even counts as a sale. You can't get an honest measurement when the thing you're measuring is also the thing handing you the sample. And it will never hand you a fair one, because a fair one means turning off the exact thing you're paying for.

So no. You're not dumb.

You did the honest thing. You asked the right question and went looking for a real answer, in the one place you were never going to get one. Nobody told you the test was broken before it started, because you never got to pick the groups.

That's the trick. They let you run a test that was never a test, then sell you back the confusion at two grand a month.

— Brat

P.P.S. Not sure Bratrax fits your stack? See exactly what you get, and how it compares to the tools you already know → bratrax.com