The Causality Gap in Marketing

1The green slide

Composite example, not a specific client: Sunday meeting at a restaurant group with eighteen branches. The marketing manager puts up a slide that is green from top to bottom. The digital campaign that ran in the two weeks before Ramadan returned 6.7 times its spend (ROAS), according to the platform's dashboard. Branch sales were up 28% on the two weeks before. The campaign's discount codes were used more than two thousand times.

Three numbers, three sources, one message: the campaign worked. The board decided to double the budget for Eid.

But each of those three numbers answers a different question from the one the decision rested on. The platform dashboard counts people who bought after seeing the ad. The before-and-after report compares an ordinary fortnight with the run-up to Ramadan, which is among the strongest weeks of the year anyway: according to Arab News reporting of Saudi Central Bank data, food and beverage spending rose 74.9% week on week in the week before Ramadan 2025.¹ And the codes count who used the discount, not who came because of it.

Nobody in the room asked the one question that matters: how much of this would have happened if we had not run the campaign?

2Two different questions

Attribution tells you what happened around a marketing action. Causal analysis asks what happened because of it.

The two sentences sound alike. But the first is computed from data you have, and the second needs a number you can never see directly: what would have happened in a world where the campaign did not run.

The idea is not new, and it has owners. In 1974 the statistician Donald Rubin framed what became known as the potential-outcomes model: every branch, and every customer, has two possible outcomes, one with the campaign and one without, and in reality we only ever see one of them.² The true effect is the difference between the two, and half of it is always missing. In 1986 Paul Holland summed it up in a motto he and Rubin had coined: no causation without manipulation.³

Then Judea Pearl, in The Book of Why (2018, with Dana Mackenzie), offered the image of a "ladder of causation": the first rung is seeing (who bought after the ad?), the second is doing (what happens if I run the ad here and stop it there?), and the third is imagining (what would have happened if I hadn't?).⁴ Every ad dashboard stands on the first rung. The decision you take in the meeting needs the second and third.

The difference between the two numbers, what the dashboard credits and what the campaign actually caused, is the causality gap. It is not an arithmetic error that a better analyst will fix. It is the natural result of how advertising, the calendar and people behave.

3Six reasons the number is bigger than the truth

Reason one: the ad goes to people who were going to buy anyway

Targeting systems are clever. Their job is to show the ad to whoever is closest to buying: the person who opened the app yesterday, who lives near the branch, who ordered last week. That is exactly what makes attribution misleading. The better the targeting, the more buyers "saw the ad" who would have bought regardless, so attributed return rises while the true effect does not move.

Across 25 large field experiments with US retailers and brokerages, economists Randall Lewis and Justin Rao found that this selection bias, which comes from the targeted nature of advertising, is a crippling problem for observational methods.⁵

Reason two: the calendar speaks louder than the ad

Ramadan, Eid, payday, school holidays, summer, rain. Each of them moves sales more than most campaigns do. Saudi Arabia adds a layer most Western playbooks never meet: the Hijri calendar moves. A Hijri year is 354 or 355 days, so Ramadan arrives about 11 days earlier in the Gregorian calendar each year.⁶ In Saudi Arabia Ramadan began on 1 March 2025 and on 18 February 2026, and is expected around 8 February 2027 on the Umm al-Qura calendar, with the moon sighting deciding.⁷

The practical consequence is immediate: comparing "March with March" compares two different things. March 2025 held 29 days of Ramadan. March 2026 held only 19, then Eid and the days after it. And if you set each day of Ramadan 2026 against the same Gregorian date in 2025, only 19 of the 30 days (63%) land on a Ramadan day. The other 11 land on the days before Ramadan, which are among the strongest shopping days of the year.

The rule: compare a season with its Hijri season, not with its Gregorian month. The first week of Ramadan with the first week of Ramadan, the last ten nights with the last ten nights, keeping the weekday mix in view.

Reason three: you launch at the bottom of the dip

When is marketing asked to "do something"? When a branch's sales fall. And when does one branch fall further than the rest? Often in an unlucky week: rain, an app outage, a match. The following week the branch returns to its normal level, campaign or no campaign.

Francis Galton described this in 1886, when he noticed that the sons of very tall parents were on average shorter than their parents, and the sons of short parents taller: extreme values tend to drift back toward the average.⁸ It is called regression to the mean, and it is the cheapest way to manufacture a successful campaign.

Reason four: the seller of the ad marks its own exam

An ad platform's dashboard measures the success of the product it sells you. That doesn't mean it lies; it means its definition of success is set in its own favour: long attribution windows, credit for people who "viewed" an ad without clicking, and each order credited to the last touch that happened to be its own. When you run on several platforms, each one credits the same order to itself. Add the dashboards up and the total "delivered" by the platforms can exceed your actual sales.

An earlier episode looked at delivery apps that have become ad businesses, charging commission and advertising together.⁹ The rule here is simpler: don't let the seller be the only referee.

Reason five: sales move rather than appear

A delivery-app campaign can shift a customer who was coming to the branch anyway, so you pay a commission you wouldn't have paid. An offer in one branch can pull customers from the next one. A promotional push can pull next week's orders into this week. In every case the number rises in one place and falls in a place nobody is looking. Season 2 covered the branch that eats its neighbour;¹⁰ the same logic applies to channels and to weeks.

Reason six: they were coming anyway

This is the reason that ties the other five together. eBay switched off paid search ads on its own brand name in selected cities and kept them in others. Traffic barely moved, because people typing "eBay" into a search engine reached it through the free link. Thomas Blake, Chris Nosko and Steven Tadelis estimated the return on those ads at roughly minus 63%.¹¹ The dashboards had been showing an excellent return on ads that paid for customers who were already on their way.

Try the six reasons yourself. Move the targeting, the season and the true effect, and watch the dashboard's return drift away from the true one.

Notice two things in the simulator. First, the true effect doesn't change when you move targeting or season, but the dashboard's return changes a lot. Second, even when the ad is shown completely at random, the dashboard still credits it with everyone who saw it and bought. Attribution rewards presence, not cause.

4What the big experiments found

You might say: that's theory, and today's platforms have enormous data and clever models that correct for all of it. The published research says otherwise.

Brett Gordon, Florian Zettelmeyer, Neha Bhargava and Dan Chapsky compared 15 randomised experiments at Facebook, covering 500 million user-experiment observations and 1.6 billion ad impressions, with what observational methods would have said without the experiment. Their conclusion: observational methods often fail to recover the experimental result, even after conditioning on extensive demographic and behavioural variables.¹² In 2023 Gordon, Robert Moakler and Zettelmeyer returned with 663 experiments and modern machine-learning methods, and found that those methods did not perform well either, with relative errors of 24% to 173% depending on the funnel stage.¹³

Even a proper experiment is not easy. Lewis and Rao found that the median confidence interval on return on investment was more than 100 percentage points wide, because individual sales are so volatile relative to the cost of the advertising.⁵ Put simply: the noise is louder than the signal, and you need good design to hear the signal.

A caveat: all of these are US figures from digital advertising, and they do not transfer to a restaurant in Riyadh. The lesson does. And to be plain about it: I looked for a published Saudi study comparing attribution with incrementality measured against a control group in restaurants or retail, and found none. I have no Saudi number for the size of the gap. What I have is a way to measure yours.

5The Saudi lab

Here is the good news. The hardest part of measuring causal effect is finding "a version of you that didn't run the campaign". Internet companies build one by splitting users; global brands build one by splitting cities and regions, and pay a lot to do it. A Saudi group running 10 to 50 branches already has one: similar branches, in known cities and neighbourhoods, with point-of-sale systems recording every order by week and by branch.

This is not a stretch. The most famous difference-in-differences study in economics was run on fast-food restaurants. In 1994 David Card and Alan Krueger surveyed 410 restaurants in New Jersey and Pennsylvania before and after New Jersey raised its minimum wage, and used the Pennsylvania restaurants as the comparison for what would have happened in New Jersey without the change.¹⁴ The same logic works for your branches: offer branches, branches without it, before, after.

The Saudi lab has two further advantages:

The Hijri calendar is a free natural experiment. Because Ramadan moves 11 days a year, each year it falls in a different relationship with paydays, school holidays and summer. With several years of weekly data you can separate the Ramadan effect from the payday effect from the summer effect, which a market whose biggest season sits on the same date every year cannot do.

Branch-level measurement needs no personal data. A branch test reads aggregate weekly sales per branch, not individual behaviour. Saudi Arabia's Personal Data Protection Law has been in force since 14 September 2023, and its compliance grace period ended on 14 September 2024.¹⁵ Aggregate measurement on your own first-party data (point of sale, your app, a loyalty programme with customer consent) is the simplest route to strong evidence and respected privacy at the same time. This is not legal advice; check anything that touches individuals' data with your counsel.

6The ladder of evidence

Not all evidence is equal. Here is a seven-rung ladder a multi-branch group can climb, from the simplest to the broadest. You don't need to climb all of it. You need to know which rung every number in front of you is standing on.

The ladder of evidence for multi-branch groups

Some origins and tools, because every rung has owners. Randomised geographic experiments were described by Jon Vaver and Jim Koehler at Google in 2011, and later extended to cases with a single test market against a single control.¹⁶ Synthetic control began with Alberto Abadie and Javier Gardeazabal (2003) and became a method with Abadie, Alexis Diamond and Jens Hainmueller (2010),¹⁷ and today has an open-source tool from Meta called GeoLift. The trap in staggered rollouts was set out by Andrew Goodman-Bacon in 2021: when treatment timing varies, branches treated earlier can end up as the comparison for branches treated later, and the estimate is distorted.¹⁸ Switchback design with carryover effects was worked out by Iavor Bojinov, David Simchi-Levi and Jinglong Zhao.¹⁹

On marketing mix models specifically. Meta's Robyn and Google's Meridian are open-source models, and both let you calibrate the model with experimental results.²⁰ That is their proper place: a planning layer on top of experiments, not a replacement for them. The model tells you where to put the next riyal across channels; the experiment tells the model whether it's right. And in Saudi Arabia in particular, any model that uses Gregorian months as its seasonality variable will mix Ramadan with not-Ramadan every year. Put Hijri events in as variables of their own.

For anyone building a team rather than a single test: Ron Kohavi, Diane Tang and Ya Xu's book on trustworthy online controlled experiments (2020) is the practical reference for details such as not reading results early, guardrail metrics, and checking that the randomisation didn't break.²¹

7Design your branch test

The first question before any test is not "did the campaign work?" but "can my test even see an effect of this size?". If you expect an offer to lift sales by 6%, and the natural week-to-week wobble in your sales drowns 6%, the test will come back "no effect" even when the offer worked. This is the minimum detectable effect (MDE).

The tool below computes it from four numbers you know: the number of branches, the number of weeks, the week-to-week volatility of sales, and the effect you expect. It then suggests the best-fitting design from the ladder and hands you a pre-registration card to copy.

Pre-registration card · sign before launchHypothesis: [offer/campaign] lifts [metric] by at least 6%Metric: weekly branch sales (or orders), relative to the branch's own pre-periodDesign: Holdout (matched pairs, coin flip within each pair)Branches: 20 · Test: 8 weeks · Pre-period: 8 weeks · Post-period: 8 weeks to catch pulled-forward demandMinimum detectable effect: 6.26% · Power at the expected lift: 76.5%Decision rule: roll out if the measured lift clears break-even and the 95% interval excludes zero; otherwise stop or redesign.Not allowed: changing the metric or the duration after launch, or reading the result before the end date.Calendar: [list Hijri events, paydays and holidays inside the window]Owner: [name] · Date: [    ]

Three things from the tool worth keeping:

• The pre-period is worth its weight in gold. Without measuring each branch against its own past, the natural gap between big and small branches drowns any effect. In the tool's illustrative default, the minimum detectable effect jumps from 6.26% to 37.85% when you remove the pre-period.
• Time makes up for too few branches, up to a point. Each extra week cuts the noise, but long tests take in more occasions. That's why the calendar goes on the card before you start.
• A switchback is stronger when it fits, because each branch becomes its own comparison. But it doesn't suit anything with a lasting effect.

8The decision protocol

Tools change nothing unless they change how decisions are made. Here are three rules I suggest writing into the governance manual, beside the Doctrine's rule of testing every promotion for increment.²²

Rule one: no claim of lift without a counterfactual

Any number that says "the campaign lifted sales" must answer: compared with what? If there is no control group, no matching Hijri period, and no earlier experiment used to calibrate the model, the number is shown as "attributed", not "achieved".

Rule two: the decision is written before the result

Before any campaign or offer above a set spend goes live, a one-page card is filled in and signed:

1. Hypothesis: what we expect, and how big.
2. Metric: one number, computed one way.
3. Comparison: who gets the campaign and who doesn't, and how they were chosen (a coin, not a manager).
4. Duration: a pre-period, the test period, and a post-period to catch demand pulled forward.
5. Decision rule: at what result we roll out, and at what result we stop. Written now, not after we see the number.

The point of the card is not bureaucracy. It is that the result can't change the question after it appears.

Rule three: board language

Every marketing number on every slide carries one of three words:

The small word changes the whole discussion. When a board sees "attributed" next to a big number, it asks the right question by itself.

9The Sunday-morning question

One question, one metric, one rule

10This week

1. Ask for the second number. At the next meeting, beside every attributed return, ask: what would have happened without the campaign? If there's no answer, write "attributed" on the slide.
2. Redraw your calendar in Hijri. Put the last two years of sales on an axis of Hijri days, not Gregorian months, and watch many "surprises" disappear.
3. Pick your first test. One upcoming initiative, similar pairs of branches, a coin, and a card signed before launch. First use the tool above to check it can see the effect you expect.

11The version that didn't run the campaign

Every campaign has two versions: the one you ran and the one you didn't. Every dashboard describes the first. Every budget decision needs the second. In a branch network the second version isn't imaginary: it's in the next neighbourhood, in last year's Ramadan, and in the week the offer wasn't running.

The ideas belong to their authors: Rubin's potential outcomes, Pearl's ladder of causation, the experiments of Lewis and Rao, of Blake, Nosko and Tadelis, and of Gordon and colleagues, Vaver and Koehler's geo experiments, Abadie's synthetic control, and the difference-in-differences that Card and Krueger made famous on fast-food restaurants. My share is translating them into the language of branches, the Hijri calendar and the Sunday meeting.

Marketing doesn't need to prove that it works. It needs to know how much it works, and where. That is a scientific question, and it has an answer.