Don’t ask what they’d pay

1The pricing meeting

A composite of many meetings.

The group is preparing a “family box” for delivery. In the meeting someone says: “We asked customers in the branches: would you pay 89 riyals for a box that feeds four?” Most of the people asked said yes. The launch decision leaves the room quickly.

A month later, sales are below plan. On the app, the same box sits next to two other restaurants. One has a name everyone knows and arrives sooner. The other is twenty riyals cheaper with free delivery. The customer who said “yes” at the branch chose something else at home.

Nobody lied to you. The customer answered a question that cost them nothing. On the app they were paying from their own pocket, with alternatives on the screen.

2Why the direct question misleads you

Research has a name for this: hypothetical bias. People say they will pay more than they do once paying becomes real. This isn’t a passing observation. It comes from reviews that pooled dozens of experiments:

• List and Gallet (2001), covering 29 studies: participants overstated what they would pay by a factor of about three on average.¹
• Murphy and colleagues (2005), covering 28 studies: the median ratio of stated to actual value was 1.35.²
• Schmidt and Bijmolt (2020), covering 77 studies of consumer goods: the bias averaged about 21%.³

The three numbers aren’t one series: each review has its own measure and its own kinds of goods. But they point the same way: what people say is more than what they pay.

Here is a detail most writing on this topic leaves out. The third review also found that indirect methods, including hypothetical conjoint, overstated more than direct questions on average.³ Turning the question into a choice does not automatically produce an honest riyal.

So why use it? Because its value lies elsewhere. Miller and colleagues (2011) compared four methods against real purchases. Hypothetical conjoint overstated the number, yet it could still lead to the right pricing decision, and the best methods were those that tied answers to real consequences (incentive-aligned).⁴ Earlier, Ding and colleagues (2005) found that making respondents live with their choice, by actually receiving what they picked, improved the prediction of real purchases. One of their two field studies was on restaurant dinner specials.⁵

The honest conclusion: choice doesn’t reveal “what they’d pay” to the riyal. It reveals what they give up for what. That is exactly what you need to design an offer and price it.

3The app screen on a Thursday night

Open any delivery app on a Thursday night. Three restaurants at the top of the screen. The first has a name you know, a 4.4 rating, arrives in 40 minutes, and the meal is 99 riyals plus a 9-riyal delivery fee. The second is a new kitchen at 79 with free delivery, but it takes about an hour. The third sits in between.

Your customer answers a survey like this every night. They don’t write a number; they tap a restaurant. And with every tap they tell you, without meaning to, what a quarter of an hour of waiting is worth to them, what a known name is worth, and whether the word “free” weighs more than its nine riyals.

The trouble is that you never see their answers. You see your own orders, not who chose someone else, or why. Choice-based conjoint rebuilds that screen in an organised way: it varies price, time, name and rating according to a calculated design, shows the screens to hundreds of people, and reads the weights from their choices.

4The idea isn’t new

The idea is much older than the apps. In 1927 Thurstone proposed that every judgment between two options has a fixed part and a random part. In 1959 Luce gave a mathematical rule for the probability of choosing one option among several.⁶ In 1974 Daniel McFadden built on this with the conditional logit model, which made it possible to estimate the weight of each attribute from people’s actual choices. For that he shared the 2000 Nobel memorial prize in economics, “for his development of theory and methods for analyzing discrete choice”.⁷

In marketing, Paul Green, a long-time professor of marketing at the Wharton School of the University of Pennsylvania, began measuring consumer preferences through “bundles” of attributes with Vithala Rao in 1971. With V. Srinivasan he later published the two reviews that became the field’s references (1978 and 1990).⁸ In 1983 Jordan Louviere and George Woodworth combined the two strands: bundle design from marketing, and the choice model from economics. That is where what we now call choice-based conjoint was born.⁹

Then came commercial tools, such as Sawtooth Software’s CBC and the market simulators built on it.¹⁰ Then hierarchical Bayes estimation, with Allenby, Rossi and colleagues, made it possible to estimate weights for each respondent, not just for the average.¹¹ Kamakura and Russell split the market into segments, each with its own weights, using a latent-class model.¹²

I invented none of this. What I’m doing here is carrying it from the economics and marketing journals to the app screen and the supermarket shelf in Riyadh.

5Try it on yourself

Before any explanation, try it. Twelve screens, each with three restaurants and a “none” option. At the end your own weights are estimated, in your browser, and your answers never leave your device.

What happened behind the screen is simple. We assumed each option has a “utility” for you equal to the sum of its attributes’ weights, plus a bit of mood we can’t see. Then we searched for the weights that make your actual choices the most likely. That is the conditional logit itself, estimated by Newton–Raphson, with a mathematical brake (a ridge penalty) that keeps the numbers from exploding when answers are few.

Then comes the step where you never have to ask anyone about price. If we know the weight of one riyal for you, and the weight of one minute, dividing the second by the first tells you how many riyals a minute is worth to you. Nobody asked you, but you answered.

Read the note under your result carefully: 12 answers from one person make a noisy number.

In this simulation a respondent whose preferences we know in advance answered 12 screens, and we repeated that a thousand times. Fifteen minutes is worth about 11 riyals to them. Even so, the estimate from 12 answers fell between about zero and about 24 riyals 80% of the time. With 300 respondents, each answering 12 screens, the range narrowed to roughly 10.6 to 11.8 riyals. An individual result is an entertaining mirror. A decision is built on a study.

6How to design the test

Design is half the result. Five rules, then three Saudi design examples.

First: start from the decision, not from curiosity. Write down the decision you will make after the study before you write a single question. For example: “At what price do we launch the family box, and do we fund free delivery?” Any attribute that doesn’t serve a decision goes.

Second: few attributes, realistic levels. Sawtooth warns that more than about eight attributes can confuse respondents, and that quantitative attributes such as price shouldn’t have more than about five to seven levels.¹³ Above all, the levels must be prices and times a customer could actually see. A price nobody has seen in the market teaches respondents to play, not to choose.

Third: a balanced design with no dominant option. Huber and Zwerina summarised a good design in four principles: orthogonality, level balance, minimal overlap and utility balance.¹⁴ In practice: never put an option on screen that is cheaper, faster and better rated than the others all at once. Such a screen teaches you nothing, because everyone picks it. The test on this page does this automatically: every level appears equally often, and no option dominates another.

Fourth: ask “none” separately. A “no purchase” option is essential, because the real market always has an outside alternative: the kitchen, the grocery, a restaurant you didn’t show. But if it sits among the options in a single question, respondents may escape to it whenever the choice gets hard. The known fix is the dual response: choose the best of the three first, then answer, “Would you actually buy it?”¹⁵

Fifth: pilot in Arabic before you launch. Test the instrument on twenty people from the same segment and ask each afterwards: what did you understand from this screen? A term like “cloud kitchen” may mean something different to each respondent.

Three design examples (illustrative levels, for design only)

(a) Dinner delivery (food and beverage). This is the test you just took.

Later you might add a “family size” attribute (feeds 4 or 6), or a “Ramadan box” covering iftar and suhoor, in a separate seasonal study. In Ramadan the basis of the decision changes, so don’t mix the two seasons in one design.

(b) A split air-conditioner in an electronics store (retail). Here the attributes named at the start all appear: price, features, name, warranty, delivery and configuration.

(c) A carton of tissues or a bag of rice in the supermarket (FMCG).

The last example holds a question that deserves a study of its own: is “20% extra free” worth the same to the customer as “20% off”? Arithmetically they differ, and psychologically they may differ even more. I don’t know the answer for the Saudi market, and I know of no one who has published it. That is exactly the kind of question the study answers.

7What the study gives you

Part-worths. How much each level adds to an offer’s appeal: how much a known name adds, how much each extra riyal takes away. These weights are the raw material for everything that follows.

Attribute importance. The span of each attribute’s effect, from its worst level to its best, as a share of the total. An attractive number for presentations, and a misleading one if its condition is forgotten: importance follows the range you chose in the design.

Try it in the chart: the weights didn’t change, yet widening the price range made price “more important”, and narrowing it made price matter less. So never say “price is our customers’ most important factor” without stating the range.

Willingness to pay as a ratio. An attribute’s weight divided by the weight of one riyal gives its value in riyals, as we computed the value of fifteen minutes for you. But Orme, of Sawtooth, warns against this simple division: it ignores competitors and the no-buy option, and it describes an “average respondent” who doesn’t exist. It is also unstable when the price weight is weak. His alternative is to compute value inside a market simulator.¹⁶

The market simulator. You build your offer and your rivals’ offers, and it computes each offer’s “share of preference” by the logit rule. Careful: this share is not market share. The simulator assumes every offer is equally available and equally known to everyone, which never happens. Its shares are read as relative indicators, and it is calibrated against real sales before any revenue figure is built on it.¹⁷

Price elasticity. Change your price by 5% in the simulator and see how far your share moves. This is the simulator’s elasticity, not the market’s. But it is more honest than “how much would you pay?”, because it is computed with the rivals in front of the customer.

Segments. The average customer doesn’t exist. A latent-class model splits respondents into groups with different weights, say “in a hurry, price doesn’t matter” and “a family counting every riyal”,¹² and hierarchical Bayes gives weights for each individual.¹¹ The names above are for illustration. Real segments emerge from the data; they aren’t assumed in advance.

8The market simulator and the floor test

This is where choice analysis meets what this site has argued from the start. Share alone says nothing about money. The price that wins the most orders may lose on every one of them.

In the illustrative example above, the highest share comes at the lowest tested price. But that same price fails the Doctrine’s floor test: commission, discount, and payment and packaging add up to more than gross margin minus the group floor.¹⁸ Weekly contribution peaks at a much higher price, with a smaller share. And if you fund free delivery yourself, the lowest acceptable price rises, because the nine riyals count as a brand-funded discount.

One sentence sums up the section: the price that wins share is not the price that makes money. The simulator shows you the first; the floor test shows you the second; the decision needs both.

9The traps

1. Too many attributes. Every extra attribute makes respondents take shortcuts, deciding on one or two attributes and ignoring the rest. Cut back to what serves the decision.¹³
2. A dominant option. A screen with one option better than the others on everything reveals no trade-off.¹⁴
3. Unrealistic levels. A price that doesn’t exist in the market, or delivery in 5 minutes. Respondents notice and stop taking the test seriously.
4. Forgetting what importance depends on. “Importance” follows the level range. Don’t compare it across two studies with different ranges.
5. A merged “none”. It pulls answers towards it whenever the choice gets hard. Use the dual response, and read its share in relative, not absolute, terms.¹⁵
6. A small sample. Rich Johnson’s rule, as reported by Orme (Johnson wrote the CBC system): respondents × tasks × options per task ÷ the largest number of levels of any attribute ≥ 500 as a bare minimum, ideally 1,000 or more. Orme recommends at least 300 respondents for robust work and about 200 per segment you want to compare.¹⁹ Worked example: 12 tasks × 3 options, with a largest attribute of 4 levels, needs 56 respondents for the theoretical minimum. But that minimum covers the average, not segments, and it doesn’t excuse you from the 300.
7. The red bus and the blue bus. The simple logit assumes that a new option takes share from every existing option in the same proportion. But a second restaurant of the same brand will take more from its sibling than from the others. The objection goes back to Debreu in 1960 and is now usually told with two buses.²⁰ The fix is richer models (HB, latent class, nested logit) and care when simulating similar brands. It is another name for the cannibalisation between branches we wrote about in Season 2.
8. Leaving the range. The simulator only knows what you tested. A price outside the tested levels is a guess, so the simulator on this page won’t let you leave them.
9. Hypothetical bias doesn’t vanish. Choice reduces it; it doesn’t remove it.³ Reduce it further with incentive alignment where possible,⁵ and with an explicit warning before the start that people usually overstate in surveys (cheap talk).²¹ Then calibrate the simulator against real sales.
10. Share of preference isn’t market share. Distribution, visibility and brand awareness aren’t in the model.¹⁷

10From the result to the menu

A study that doesn’t end in a written decision is a cost. This is how the results become a decision in a multi-branch group:

1. The decision before the study. Write down the options actually on the table: say three prices for the box, free delivery or not, and two sizes.
2. Simulate each option against the real rivals in each delivery zone that matters to you, not against “the market” in general.
3. Run each option through the floor test. An option that fails the floor is out, whatever its share.¹⁸ At item level, compute each item’s floor price in each door.²²
4. Don’t touch the ruler items. Items whose price customers remember change last. Channel cost is covered by items with no twin at the branch.²³
5. Test in the market against a holdout. Launch the chosen option in some branches or zones, leave comparable ones unchanged, and measure the true increment.²³
6. Calibrate the simulator with what actually happened. The gap between what it predicted and what happened corrects the next study.

The Sunday-morning question

11“How Saudi Customers Actually Choose”: method first

Status: planned. There are no results yet.

I searched for published studies of this kind in the Kingdom. I found choice experiments in health, such as patients’ preferences for preventive dental care, a study of electric-vehicle adoption in Riyadh, and a study of willingness to pay for eco-friendly packaging in retail.²⁴ But I found no published, peer-reviewed conjoint study of restaurant, delivery or FMCG choice in the Saudi market. Some may exist inside companies that never published them. The public knowledge base, though, is empty here.

So I am planning a series of studies called “How Saudi Customers Actually Choose”, under the site’s own rule: method and sample first, results after. The plan in brief:

• Three sectors: dinner delivery, a retail purchase and an FMCG pack. An independent design for each, and a separate Ramadan study.
• Sample: at least 300 respondents per study and at least 200 per segment compared, with Johnson’s rule computed before launch.¹⁹
• Instrument: Arabic first, with a comprehension pilot on twenty people, a dual-response “none”, and an explicit hypothetical-bias warning.
• Privacy: under the Personal Data Protection Law: explicit consent, minimal data, no names or mobile numbers in the analysis file, and destruction of anything not needed.²⁵
• Pre-registration: hypotheses, design and analysis plan published before data collection.
• Publication: method and sample size first, then results with their limits, then aggregated data where the law allows.

The full protocol is a separate document. Until results are published, every number on this page is one of three things: your own result, an illustrative example, or a simulation.

12This week

1. Take the test above, and ask three people on your team to take it. Your results will differ, and that is the first lesson: the “average customer” doesn’t exist.
2. Write down one pricing decision you will make this quarter, then list its attributes and realistic levels underneath, five attributes at most.
3. Run the price you’re leaning towards through the floor test before asking any customer anything.

13Credit where it’s due

Choice as a statistical model is McFadden’s; bundles are Green’s and his colleagues’; combining them is the work of Louviere and Woodworth; the tools and the warnings come from Sawtooth and Orme; hierarchical Bayes comes from Allenby and Rossi. What I did was put it on the app screen on a Thursday night and tie it to the contribution floor.

The next step isn’t an article. It is the study itself, and its method gets published first.