Table of Contents
🌏 中文版
Today's Topic
AI Product Design questions don't test whether you understand how models work — they test whether you can predict how an AI feature will break in real users' hands, and whether you've designed the recovery mechanism before it breaks. This year Meta rewrote its PM interview loop for the first time in five years, adding a round called "Product Sense with AI," where candidates solve a product problem alongside AI in real time. The bar isn't clever prompting — it's whether the candidate can notice the model is guessing, ask the right follow-up question, and make a clear product call despite incomplete information.
That's a signal of where the core AI product skill has moved: from "what can this model do" to "how will this system actually behave in the real world." That's the shift we're practicing today.
Core Framework Cheat Sheet
AI Product Sense in Three Steps: Map Failure Modes → Define MVQ → Design Guardrails
| Step | Purpose | How to use it in the interview |
|---|---|---|
| 1. Map failure modes | Deliberately push the model into messy scenarios and see how it breaks — usually "confidently invents structure when faced with ambiguity" | Cite a concrete failure, not just "the model might make mistakes" |
| 2. Define minimum viable quality (MVQ) | Set explicit "acceptable," "delight," and "do-not-ship" bars, instead of vaguely saying "quality needs to be good" | The interviewer wants to hear how you derive the bar, not a number pulled out of thin air |
| 3. Design guardrails | Block the failure mode found in step one with either a system prompt line or a UI rule | Show you know that "fixing the prompt" and "changing the interface" are two different fixes, and that you know when to use which |
This framework comes from Marily Nika, a former AI PM at Google and Meta — the point isn't memorizing the step names, it's understanding the problem it solves: an AI demo always looks great in a controlled environment; the real product risk hides in the messy data, ambiguous intent, and zero patience that real users bring.
Trust Calibration: GitHub Copilot's Ghost Text Pattern
| Design choice | Typical approach | Trust-calibration approach | Why it works |
|---|---|---|---|
| Presentation | Written directly into the user's text | Grayed-out "ghost text," visually distinct from user input | Users can tell at a glance this is a suggestion, not an established fact |
| Accepting a suggestion | Accepted by default, requires an action to reject | Requires pressing Tab to adopt | Accepting is a deliberate act; the cost of rejecting approaches zero |
| Calibration mechanism | None — users can only guess how accurate the model is | Lets users build judgment naturally across hundreds of accept/ignore interactions | Calibration comes from repeated interaction, not a one-time trust declaration |
Today's Practice Question
The Question
"You're the PM for a team-collaboration AI assistant. Its feature summarizes long Slack threads into 'decisions' and 'action items.' It worked well in early testing, but in a pre-launch re-test you find it wrote an undecided discussion as a 'committed Q4 roadmap item' and assigned an owner to someone who never agreed to anything. How would you redesign this feature so it's still trustworthy after general availability?"
(Source: original question, adapted from a real product case Marily Nika shared on Lenny's Newsletter — a failure mode she encountered helping a startup build a Slack decision-summary feature)
Breaking It Down
- Clarify the problem: Ask who this summary is actually for and what it's used for — is it a quick catch-up for team leads, or will it be cited directly as a roadmap document? That determines how much damage a wrong owner assignment does.
- Define the users: Split into two groups — people who see the summary in real time (who can spot an error but don't have time to correct every one) and people who read it later (who have no chance to catch an error and will just take the summary at face value). The second group is who the design needs to protect first.
- Structured analysis: Break it down with "map failure modes → MVQ → guardrails." Step one: recognize this is the classic failure mode of "confidently inventing structure when faced with mess." Step two: define the MVQ — set the "do-not-ship" bar on the "owner-assignment error rate," since it's the only error type that directly damages trust between colleagues, and its tolerance must be far lower than something like imprecise summary wording. Step three: design the guardrail — not a bigger model, but one rule: "only assign an owner if someone explicitly volunteers or is directly asked and confirms; otherwise, surface the discussion theme and ask the user whether to assign one."
- Propose a solution: The interface needs to visually separate "AI inference" from "facts explicitly stated in the thread" — confirmed action items in plain text, while AI-inferred themes or suggested owners get a ghost-text-like treatment, with a one-click "this is wrong" correction entry point, instead of making users re-read the original thread to catch errors themselves.
- Define success: Don't use "number of summaries produced" as the metric — that only measures usage, not quality. The core metrics should be whether the "rate at which users manually correct owner assignments" trends down over time, and whether users stop manually re-reading the original thread to double-check the summary — the latter is the real behavioral signal that trust has been built.
Sample Answer (how to actually say this in an interview)
Clarifying the problem: "I'd want to confirm who this summary is mainly for — a quick catch-up for team leads, or something that gets cited directly as a roadmap document. If it's the latter, a wrong owner assignment does far more damage than imprecise wording, so that's where I'd focus the design first."
Structured analysis: "This is the classic failure mode of a model confidently inventing structure out of messy information — it wrote an undecided discussion as a 'committed' roadmap item. I'd set 'owner-assignment error rate' as the hard do-not-ship bar, since it's the only error type that directly damages trust between colleagues, and its tolerance needs to be far lower than something like summary wording. The guardrail is simple: only assign an owner if someone in the thread explicitly volunteered, or was directly asked and confirmed — otherwise just surface the discussion theme and hand the decision of whether to assign someone back to the user."
Solution and trade-offs: "In the interface, I'd visually separate 'facts explicitly stated in the thread' from 'AI-inferred conclusions' — the inferred parts get a grayed-out suggestion style, with a one-click correction entry point, instead of making users go back and compare against the original thread to find errors. I wouldn't use 'number of summaries produced' as the success metric — that's just usage. I'd watch whether the rate of users manually correcting owner assignments trends down over time — that's the real signal that people are starting to trust this feature."
Self-Check List
Use this table to check whether your answer covers the key points:
| Check item | Covered? |
|---|---|
| Clarified the summary's actual purpose and who reads it | |
| Broke it down with "map failure modes → MVQ → guardrails," not a vague "improve accuracy" | |
| Designed the guardrail as a concrete rule (e.g. owner-assignment conditions), not a vague "add more review" | |
| The solution visually separates "AI inference" from "confirmed fact" | |
| Success metrics align with "has trust been built," not usage volume | |
| Bonus: explicitly named which error type needs the lowest tolerance, and why |
Today's Case Study
GitHub Copilot: Using Ghost Text to Drive the Cost of Distrust to Zero
Instead of writing AI-suggested code directly into the user's file, GitHub Copilot renders it as grayed-out "ghost text" — visually distinguishable at a glance from what the user actually typed. Accepting a suggestion requires the deliberate act of pressing Tab; ignoring it requires no action at all. This seemingly small interface decision reallocates how trust gets built: users don't have to decide "should I trust this AI" on the first interaction — instead, across hundreds of accept/ignore micro-interactions, they naturally calibrate which kinds of suggestions to trust and which deserve a second look.
Interview link: Use this case directly for questions like "give an example of an AI product that does trust calibration well" or "how would you design an AI feature that users will actually use without over-relying on it." The key point to emphasize: trust isn't built through a one-time disclaimer or a confidence score — it's built by designing the cost of "rejecting" a suggestion down to near zero, so users can safely trial-and-error repeatedly.
Further Reading
- Building AI product sense, part 2 — Marily Nika's full walkthrough of the "map failure modes → MVQ → guardrails" process, including details on Meta's new AI PM interview
- Designing for AI Trust (2026) — analyzes trust-design patterns in ChatGPT, GitHub Copilot, Notion AI, and Figma AI
- AI Copilot UX Design: How to Build Copilots Users Actually Trust — common design traps in onboarding and error handling for copilot-style products
References
- Building AI product sense, part 2 — source for the "Today's Topic" note on Meta's interview redesign, and for the framework and case used in "Core Framework Cheat Sheet" / "Today's Practice Question"
- Designing for AI Trust (2026) — source for the GitHub Copilot ghost text design details in "Core Framework Cheat Sheet" and "Today's Case Study"
- Designing Trust in AI Products: UX Strategies for Product Leaders — further reading on trust calibration referenced in "Today's Case Study"
Loading...