🌏 中文版
Today's Focus
The easiest way to lose points in a metrics interview isn't miscalculating a percentage — it's jumping to a conclusion the moment you see "a metric dropped" or "a metric spiked." A drop means you first need to locate which stage of the funnel broke, because new users getting stuck and existing users churning away call for completely different investigations. A spike can't be celebrated outright either, because the lift might be a short-term illusion bought by sacrificing user experience. Today's two frameworks practice exactly those two moves: AARRR pirate metrics locates where in the funnel a problem is happening, and OEC (Overall Evaluation Criterion) judges whether a beautiful A/B test result is actually safe to ship.
Core Frameworks
Framework 1: AARRR Pirate Metrics
Break the full user journey — from first contact with the product to generating revenue — into five stages, each pointing to a different kind of breakage and a different way to check it:
| Stage | Question | Common metric |
|---|---|---|
| Acquisition | Where are users coming from, and how many | New installs, per-channel conversion |
| Activation | Did the first use reach the core value | Signup-to-first-order conversion |
| Retention | Do they come back a second, third time | Week-two retention, repeat purchase rate |
| Referral | Do they actively bring in new users | Invite code usage, word-of-mouth conversion |
| Revenue | Can usage behavior actually be monetized | Average order value, paid conversion |
A drop in "completed orders" could mean fewer new users at the Acquisition stage, a higher checkout failure rate for new users at Activation, or existing users simply not reordering at Retention — three completely different root causes. Interviewers expect you to name which stage before digging further.
Framework 2: OEC (Overall Evaluation Criterion)
An A/B test can't be judged on a single primary metric alone. Ron Kohavi, former head of Bing's experimentation platform, built his approach around defining an OEC for every experiment: typically one primary metric (say, revenue) paired with a set of guardrail metrics (page load speed, return-visit rate, complaint volume). The primary metric answers "did this change work," while the guardrail metrics answer "did this change quietly sacrifice something" — because a search engine can always juice short-term revenue by cramming in more ads, even as it drives users away over time. Watching revenue alone will fool you.
Today's Practice Question
The Question
"A food-delivery app's completed orders dropped 12% week over week. How would you investigate?" (Source: PMMockr's practice question bank; PMMockr explicitly labels this an original hypothetical drill, not a verified employer question.)
How to Break It Down
- Clarify the question: Ask the interviewer whether "completed order" means successfully prepared or successfully delivered. Then confirm whether the 12% is relative to normal seasonal swings or a genuine deviation from trend, so you don't mistake a routine cycle for an incident.
- Validate the metric itself: Before digging into causes, rule out instrumentation or logic errors — for example, a recent release that quietly changed what counts as "completed," making the number look like it dropped when user experience didn't actually change.
- Locate the funnel stage with AARRR: Split the drop into order volume (Acquisition/Activation — how many people opened the app and attempted to order) versus completion rate (the order-to-delivery conversion). If order volume held steady but completion rate dropped, the problem is on the supply or logistics side; if order volume itself dropped, the problem is more likely at the front-door entry point or an acquisition channel.
- Segment further: Within the confirmed stage, cut by region, device, payment method, and restaurant category to see whether the drop is concentrated. A drop isolated to one city points to local capacity or a specific partner restaurant — not a site-wide product defect.
- Prioritize hypotheses and act: Rank demand-side (users not wanting to order), supply-side (not enough restaurants or drivers), and reliability (app crashes, failed payments) hypotheses by scope of impact. Investigate the broadest one first, and once you find the root cause, define both the fix and how you'll verify recovery — don't close the case the moment you find a plausible-sounding reason.
Sample Answer (how to actually say it in an interview)
Define before you localize: "I'd first confirm with the interviewer whether 'completed order' means food prepared or food delivered, and whether the 12% is versus last week or versus the same week last year — food delivery has obvious weekly and holiday cycles, so I don't want to treat seasonality as an incident. Assuming it's confirmed to be a genuine anomaly, I'd rule out instrumentation issues first — checking whether a recent backend release touched the order-status logic."
Localize with the funnel, don't investigate everything at once: "Next I'd split the drop into two stages — how many people opened the app and placed an order, and of those, how many were successfully delivered. If order volume held steady, the problem is in the completion stage, usually supply-side (slow kitchens, not enough drivers) or reliability (failed payments, system timeouts). If order volume itself dropped, the problem is more likely at the entry point or a specific acquisition channel. I'd cut by region and device first to see whether the drop is concentrated — a site-wide drop and a drop confined to one city represent completely different problem sizes."
Close on action and verification: "If the cut shows the completion rate drop is confined to one city, I'd check whether that city recently had a capacity or partner-restaurant change. Once I find the root cause, I wouldn't settle for a one-off fix — I'd set a leading indicator, like that city's average kitchen wait time, and keep watching it until the completion rate recovers, rather than declaring victory after one good day."
Self-Check Checklist
Use this table to check whether your answer missed anything important:
| Check item | Covered? |
|---|---|
| Did you clarify the metric definition first and rule out instrumentation or logic errors | |
| Did you use the funnel to pin the drop to a specific stage, instead of vaguely saying "I'd investigate" | |
| Did you cut further by region/device/segment to confirm whether the drop is concentrated | |
| Did you sort hypotheses into demand/supply/reliability and investigate by scope of impact | |
| Did you define a verification method after the fix, instead of closing the case at root cause | |
| Bonus: did you draw a line between "normal seasonal swing" and "genuine anomaly" |
Today's Case
Bing: a six-month-stale change that added over $100 million a year in revenue — and nearly got rejected by its own alert system
Ron Kohavi and Stefan Thomke tell a classic story in Harvard Business Review: in 2012, a Bing engineer proposed a small change — merging an ad's headline with its second line of text into one longer headline. The idea sat unprioritized on the backlog for six months because it looked too small to be worth the investment. When it finally ran as an A/B test, the monitoring system immediately raised a "too good to be true" alert — revenue had jumped far past the normal range. Most teams seeing an anomaly that large would want to celebrate immediately. Bing's team instead suspected a bug and investigated whether a logging or tracking error was double-counting revenue. Only after confirming there was no bug did the team move on to checking the guardrail metrics inside the OEC — user-experience numbers, such as whether users were leaving sooner because the page felt more cluttered. Only once those guardrails held did the change ship for real. The final result was a 12% revenue lift, worth more than $100 million a year — the single largest revenue-generating idea in Bing's history.
- The idea to merge the ad headline with the second line of text sat in the backlog for six months
- Finally scheduled, the team runs an A/B test
- The monitoring system raises a "too good to be true" alert — revenue jumped far past expectations
- The team suspects a bug or double-counting and investigates logging and calculation logic
- No bug found — next they check the guardrail metrics for whether user experience was sacrificed
- Guardrails hold, confirming the lift is real — the change ships, adding over $100 million a year in revenue
Case and figures drawn from Kohavi and Thomke's first-hand account in Harvard Business Review — the shipped change added over $100 million a year in revenue.
Interview tie-in: This case directly answers a common follow-up — "How do you decide whether an A/B test result can be trusted?" Many candidates answer "if it's statistically significant, I trust it," but significance doesn't mean safety; a significant revenue lift can still be bought with user experience. Bing's approach demonstrates a more complete answer: treat an unusually good result as an alert first, rule out bugs, then check whether guardrail metrics were sacrificed — both gates have to clear before it counts. Applied to today's delivery question: even after finding that some change lifted the completion rate, you'd still need to confirm complaint volume or cancellation rate didn't rise in step, or the lift is only skin-deep.
Further Reading
- The Surprising Power of Online Experiments — Ron Kohavi and Stefan Thomke's original Harvard Business Review article, the source of today's case and the OEC concept.
- Large Scale Experimentation at Bing — Bing's own engineering blog on how guardrail metrics get automatically monitored at scale.
- PM Interview Questions by Skill Area — source of today's practice question, with more metrics, strategy, and execution questions and answer outlines.
References
- The Surprising Power of Online Experiments — original source for the Bing ad-headline case, the 12% revenue lift, and the $100M+ annualized figure.
- Large Scale Experimentation at Bing — explains how OEC and guardrail metrics are implemented on Bing's experimentation platform.
- PM Interview Questions by Skill Area — source of the practice question "a food-delivery app's completed orders dropped 12% week over week," filed under the site's metrics/analytics question category, explicitly labeled an original hypothetical drill.