Four system design questions that come up often (enterprise RAG, an LLM serving stack, a customer-support agent, online A/B testing with a data flywheel), three Python problems (dynamic batching, sampling, backoff retries plus a token bucket), and a STAR skeleton for behavioral rounds. Each design question has a speakable answer and a self-check list, and all code was actually run.
Seven LLM engineering topics that come up often in interviews: LoRA/QLoRA and fine-tuning vs RAG, RLHF/DPO/GRPO alignment, inference optimization (PagedAttention gives vLLM 2-4x throughput), sampling and decoding, LLM evaluation, the OWASP LLM Top 10 (2026 edition), and hallucination. Each section covers the concept, the mechanism, and how to answer.
Interview follow-ups usually land on mechanisms: why divide by √d_k, how AdamW differs from Adam plus L2, why a high AUC still isn't shippable. This post walks through concept, mechanism and how to answer, then ends with 3 runnable numpy exercises.
Prompt engineering shapes a single instruction, context engineering decides what the model sees right now, and harness engineering wraps the whole non-deterministic system. Bsharat's 26 principles, Breunig's four context failures, and OpenAI's roughly one-million-line, zero-handwritten-code harness experiment each map to one layer. Evaluation is the piece teams skip most, and it belongs in a CI gate that blocks silent regressions.
Answer RAG interview questions in four steps: the three-stage pipeline, where each stage fails, which of nine variants patches which hole, and the 6 kinds of problems RAG cannot solve. Self-RAG, CRAG and Adaptive-RAG add judgment; Graph RAG and KG-RAG change the index; CAG skips retrieval; Agentic RAG and Reasoning RAG hand decisions to the model.
Read each company's own interview guidance first: Meta now expects candidates to use a built-in AI assistant, while Anthropic defaults to no AI. For question banks, pick ones that cite sources and are still maintained this year, such as the alexeygrigorev field guide, ombharatiya, and alirezadir. Only 2 of the 12 popular repos link sources for their questions, and some keep answers behind a paywall.
Audio representation, temporal relationships in video, and action-conditioned world transitions are different learning tasks. The earlier combined guide checked only audio at the end. This URL now serves as a reading map to one full guide each for Lectures 24–26; the Lecture 26 guide also covers the separate alignment deck.
Text models convert text into tokens; image models also need a computational representation of pixels.
Code Generation and Agents: A Reliable Test–Repair Loop | CMU 10-423 Lecture 23
Cross-attention reads another representation, DiT is a diffusion-backbone choice, and Q-Former uses learned queries to extract or bridge information. They can coexist rather than being three competing products. The Lecture 14 lesson handles Prompt-to-Prompt and classifier-free guidance in detail.
A GAN learns from a discriminator.
A model that does not fit on one GPU has a different problem from a model that fits but decodes slowly. Distributed strategies partition data, parameters, and states. FlashAttention improves attention memory traffic; KV caching avoids recomputing historical keys and values during decoding.
An exam tests reasoning under a fixed question. A paper report tests your judgment of someone else’s evidence. A project tests your ability to produce reproducible evidence. These are distinct learning tasks, now taught separately while this URL remains an assessment map.
A classifier asks which class an image belongs to.
This series uses the public Spring 2026 materials for CMU 10-423/623/723 Generative AI.
Assignment 1 brings sequence modeling and modern attention into code.
Copying a noising equation is not enough to implement diffusion correctly.
Use an original numerical test to check gradients, data, and evaluation alongside the original assignment specification.
Use an original numerical test to check gradients, data, and evaluation alongside the original assignment specification.
A prompt with one demonstrated answer is not the same data as a prompt with a preferred and rejected response. SFT learns demonstrated tokens. A typical RLHF pipeline learns reward and then updates a constrained policy. DPO forms a loss directly from preference pairs and reference-relative ratios.