Skip to content
All tags

#experimentation

4 posts

Growth & Experimentation Interview Guide: From Growth Loops to Experiment Design

Growth interviews don't test whether you can growth hack — they test whether you have systematic growth thinking. Core skills: growth loop design (the acquisition → activation → retention → referral flywheel), experiment design (the full hypothesis → metric → experiment → analysis process), retention strategy (finding the aha moment, designing habit loops), and using data to decide what's worth continued investment.

Metrics & Analytics Interview Guide: From North Star to Experiment Design

Metrics interviews test whether you can make decisions with numbers, not how much statistics you know. Core skills: north star metric selection logic (why this one and not that one), metric tree decomposition (finding actionable levers), funnel analysis (which step's drop-off is most worth fixing), A/B testing design and pitfalls, and judgment when facing counterintuitive data.

product deep-dive

Value Validation for Digital Products: From Assumption Maps to the M3 Retention Baseline for AI

The unit of validation is an assumption, not an idea. Kohavi's data shows the industry median experiment success rate is ~10%, which means roughly 22% of 'winning' experiments at p<0.05 are false positives. Sean Ellis's 40% threshold has no publicly available dataset. AI product retention should be baselined at M3 rather than M0, and GRR splits from 23% below $50/mo to 70% above $250/mo.

RAG A/B Testing: A Scientific Approach to Comparing Pipeline Configurations

"Adding a Cross-Encoder feels better" is not a scientific evaluation. A/B testing tells you whether a change actually works, how much it helps, and which query types benefit.