RAG: Looking Things Up Before Answering When the Model Cannot Remember or Is Out of Date | NTU Yun-Nung Chen Applied Deep Learning Lecture 8

🌏 中文版

Video status: Videos included. Source details

This is Lecture 8 in Reading NTU Yun-Nung Chen Applied Deep Learning, based on the 10/13 week of ADL Fall 2025 (114-1, 2025/09/01–12/15). Lecture 7 covered how to change the model itself. This one goes the other way and leaves the model alone, answering two questions: how can a model answer correctly about things it does not know or does not remember clearly? And if it is to look things up before answering, how should the "looking up" be done so that it finds the right things?

Official materials:

• The RAG deck (60 slides)
• Videos 8.1 What is RAG (27:32), 8.2 RAG Framework (39:52), 8.3 Advanced RAG (10:26), 8.4 RAG Roadmap (28:17), 8.5 RAG Pros & Cons (17:23), and 8.6 RAG Q&A (24:19, in-class Q&A with no matching slides)
• The HW3 briefing video (20:34)

8.1 through 8.5 have captions. 8.6 and the HW3 briefing have none, so this guide works from transcripts made from their audio. The lectures are in Mandarin. Examples marked "made up for this post" are not from the lecture.

After reading you should be able to do five things: name the four limits of relying on parametric memory alone; draw the RAG pipeline and explain the trade-off in chunk size; tell sparse from dense retrieval and bi-encoders from cross-encoders, and say which stage each is used in; place several representative methods correctly using the three questions "what to retrieve, how to use it, when to retrieve"; and, given a need, decide whether prompting, RAG, or fine-tuning fits.

Why look things up before answering

Right about famous people, inventing things about the less famous

Professor Chen opens with two questions (slides 2–3). Asked "Do you know Tsai Ing-wen?", the model answers at length and correctly. Asked "Do you know Yun-Nung Chen of NTU?", the lab's own Taiwan-LLM says she is a professor of electrical engineering, got her PhD at Stanford, and works on semiconductor devices. The slide strikes out the wrong parts one by one. She says some of it is right and a fairly large part is wrong.

The difference is how common the information is. Popular information is read again and again in pre-training and remembered clearly. Rare information may have been read and was not learned well, so when asked the model cannot produce the right content, and hallucination results.

This situation is called the long tail. Line up all information from highest frequency to lowest: a few items on the left are very high, and a very long tail trails to the right. Popular information is actually a small share, and most information sits on the tail. She uses an exam analogy: key points reviewed many times are always answered correctly; obscure questions have been seen, appear too rarely, and are missed again on the exam. And the obscure ones are the majority. The slide concludes: LLMs cannot memorize all details about long-tail information.

The fix is to change the exam from closed book to open book. The same model, with the same question, is allowed to do a web search and read the results before answering (slide 4), and the answer becomes much more accurate: a professor of computer science, a PhD from Carnegie Mellon, research on language understanding and dialogue systems.

Giving an answer external material to be checked against is called knowledge grounding. She stresses that the word on the slide is reduce hallucinations, not resolve. After reading the material the model still uses its own knowledge to judge, infer, and summarize, so it can still be wrong, only less often than when answering directly.

Knowledge goes out of date

Slide 5 asks "Who is the current president of Taiwan?", and the model answers with the one from when it did its reading. It has not forgotten; what it read is out of date. Information keeps changing, the model cannot be expected to read the latest material every minute, and the only option is to give it newer reference material at inference time.

The slide mentions another route, knowledge editing (Hsueh et al. 2024): directly modifying knowledge the model has already learned. She explains why it is hard:

• The most intuitive approach is to keep training on new data. The model's head then holds both "the current president is A" and "the current president is B," it has no notion of which came first, and its answers are sometimes right and sometimes wrong.
• Even with the fact "current president" fixed, the question "who is the current president's father" needs a linked update to be right. With only one place changed, it may answer with the former president's father.

This direction is not pursued further in the lecture.

WebGPT: teaching a model to use a search engine

Slides 6–8 are WebGPT (Nakano et al.), built by OpenAI in 2021, a little before ChatGPT. The task is answering questions that need looking up, and the slide's example is: which has more words, the Harry Potter series or The Lord of the Rings? A model can hardly have that memorized.

The method is supervised fine-tuning on GPT-3. GPT-3 only continues with the next token, so "searching" has to become part of the continuation (slide 7):

1. Query generation. Given the question "What is Trump's latest statement?", the model must output [SEARCH] Trump's latest post [END]. The extra [SEARCH] tells the system to call external search now.
2. Click the reference document. After the search results return, the model must output [CLICK] 1 [END], meaning open the first one.

The training data comes from people: record which keywords a person searched after seeing the question and which result the person clicked. The slide concludes: Supervised learning of human actions.

She says generating a search query and calling external search is now built into every vendor's model. She puts the same question to Gemini, which judges that this is changing information, runs a Google Search itself, and answers from the results (slide 8).

Priors left by pre-training

Slide 9 adds the principle. What the model does in training is raise the probability of the next token in the training data. Information that appears repeatedly has its probability pushed up repeatedly; information that appeared once is pushed up once and no more, and may not surface when asked. The slide puts one conclusion under each of its two examples: "Pre-training data is outdated" and "Generation follows the distribution in the pre-training data." They correspond to two problems:

• Outdated. Time-dependent information expires, and it is hard to filter out at pre-training time because the same article contains many facts worth learning.
• Bias. The slide's example is 「我好朋友是護士，」 ("My good friend is a nurse,"), after which "she is very patient with patients" is far more probable than "he." No one told the model the nurse's gender, and most nurses in the training data are women, so the model carries that prior.

Four limits of relying on parametric memory

The model stores what it read in its parameters, and this memory is called parametric memory. Slide 10 lists its four limitations:

1. It cannot memorize everything. Long-tail information is often answered wrongly.
2. The world changes over time. It does not know what happened after training.
3. Private documents are not on the web. Her example is a department that wants a QA system for course-withdrawal and enrollment rules, which the model never saw in pre-training. The same holds for a company's internal documents.
4. A black box is hard to verify. When the model answers directly, we do not know what it is based on. If it consults external material first, it can say which items it consulted, and we have something to go back and check.

The basic RAG framework

The figure on slide 12 has four components: the user's query, external documents (open-sourced material on the web, or a closed-source store), indexing and retrieval, and the LLM at the end. The flow is: documents are processed and indexed → a query arrives and relevant passages are retrieved → the passages and the question go into the prompt together → the LLM produces the answer.

The LLM in the figure carries a snowflake, meaning it is frozen. She gives the reason: the model is large, or you are using someone else's API, and this part usually cannot be changed. In practice the effort therefore goes into the front half: how well indexing is done and how accurate retrieval is.

She also explains indexing in passing. It is a basic search-engine technique, and library cataloguing long before was the same thing: organize the material first so that lookup is fast. The simplest form gives each document a number; for more efficiency, cluster by topic first, decide the topic at query time, and search only inside that cluster.

Chunking: how large

The unit retrieved is usually not a whole article but a cut piece, called a chunk or passage (slide 13). She explains why to cut: an article is long, covering A at the start and B, C, and D later. When the question concerns only A, the relevance of the whole article is diluted by the unrelated rest, and it is not found. Cut into small pieces, the information is more concentrated.

There is no standard way to cut, and she lists some practical details:

• A common approach sets a fixed length, such as 300 tokens per chunk, with adjacent chunks overlapping in part.
• Cut at sentence ends where possible, so that a chunk does not end with two tokens of the next sentence.
• Cut too finely and the information is fragmented, with two pieces that belong together split apart; the number of chunks also grows, and so does the number of comparisons.
• Cut too coarsely and there are fewer comparisons, and one chunk mixes several topics, so even the "most relevant" one is not that relevant.

The choice depends on the application and on what the documents look like. A document about one regulation can be one chunk; a document packed with many regulations should be cut apart.

Sparse retrieval: matching keywords

The traditional search-engine method is sparse retrieval (slide 14). Turn the query and each document into a TF-IDF vector: each dimension corresponds to a word in the vocabulary, and the value is computed from how often the word appears. Take the inner product of two vectors, and a higher value means more words in common.

The slide's example is q = "what is nlp," and the inner products with three documents d₁, d₂, and d₃ are 0.165, 0.0825, and 0.0413.

Example (the slide's numbers, recomputed here; she does not walk through this example in class, and the reading below is this post's). Setup: the five dimensions are, in order, what, candy, nlp, is, language. q = (0.33, 0, 0.33, 0.33, 0), d₁ = (0.25, 0.125, 0, 0.25, 0), d₂ = (0, 0, 0.125, 0.125, 0), and d₃ = (0, 0, 0.125, 0, 0).

• Reasoning: q·d₁ = 0.33×0.25 + 0.33×0.25 = 0.165. q·d₂ = 0.33×0.125 + 0.33×0.125 = 0.0825. q·d₃ = 0.33×0.125 ≈ 0.0413.
• Result and meaning: d₁ ranks first. Yet its content is "what is life? candy is life!", unrelated to NLP; it merely contains the words "what" and "is." d₂, "nlp is an acronym for natural language processing," is the answer and ranks second. Counting only repeated words lets common words contribute to the score.

It is called sparse because the vector has as many dimensions as the vocabulary and nearly all are 0. The benefit is fast computation. The drawback is that the same keyword must appear: if the document says 「大型語言模型」 ("large language model") and you search "LLM," it is not found. She says everyone has had the experience with search engines of needing to think up keywords that happen to match.

The advanced version of TF-IDF is BM25 (the slide includes the formula), which uses the same TF and IDF ideas and so still needs matching words.

Dense retrieval: matching meaning

The opposite approach is dense retrieval (slide 15): a neural network encodes a piece of text into a vector with few dimensions, each holding a value; the query is encoded the same way, and the cosine similarity of the two vectors is computed.

The slide divides where the encoder comes from into two kinds:

• Off-the-shelf embeddings. A pre-trained BERT, taking the vector at the first [CLS] token; or a pre-trained GPT, taking the vector after the whole input has been read. She says she finds the latter, LM-based embeddings, the most used recently, with OpenAI and Google both offering embedding APIs. Using a pre-trained embedding as it is is also what industry most often does when building RAG, since it needs no training data.
• Learned embeddings with contrastive learning. The slide lists DPR (Karpukhin et al. 2020) and Contriever (Izacard et al. 2022).

Bi-encoders and cross-encoders

She then covers two common architectures:

The two encoders of a bi-encoder can share weights or be separate. Her reason for separating them: a user's question is usually one sentence and a passage is long, so their forms differ a lot.

Why is the better-performing cross-encoder not used for retrieval? Retrieval picks from thousands or even millions of documents, each of which would have to be concatenated with the query and run through the model, which costs too much. Use a bi-encoder to pick a small batch of candidates first, say 30, then use a cross-encoder to rank those 30 carefully, and the cost is acceptable. The exception is when there are few documents to begin with.

Training with paired data

Slide 16 shows how a dense retriever is trained. The query is 「美國哪一州面積最大？」 ("Which US state is the largest?"); one English document, "The biggest state in the U.S.…", is the positive (d⁺), and two Chinese documents that also mention US states and cannot answer the question are negatives (d⁻). The training objective is:

L = −log [ e^s(q, d⁺) ÷ Σᵢ e^s(q, dᵢ) ]

s is the inner product of the query and document vectors. The denominator covers the positive and all negatives. She adds that negatives are usually sampled, without including every irrelevant document, for example five negatives per positive.

Example (made up for this post). Setup: one positive and two negatives. The current scores are s(q, d⁺) = 2 and 0 for both negatives.

• Reasoning: e² ≈ 7.39 and e⁰ = 1. The positive's probability is 7.39 ÷ (7.39 + 1 + 1) ≈ 0.79, and the loss is −ln 0.79 ≈ 0.24.
• For comparison: if the model cannot yet tell them apart and all three scores are 0, the positive's probability is 1/3 and the loss is −ln(1/3) ≈ 1.10.
• Result and meaning: there are only two ways to lower the loss, raising the positive's score or lowering the negatives' scores. This is "pull the relevant closer and push the irrelevant away" written as a formula.

This training needs paired data: for each query, which document is the positive. The slide concludes: Paired training data is difficult to collect. She says those who hold such data are companies with search engines such as Google and Microsoft, since which result a user clicks after searching is a ready-made label. Everyone else has to pay people to label, which is costly.

Without paired data: query likelihood

The alternative on slide 17 uses what a language model already does. A language model can estimate the probability of text, so it can be asked: after reading this document, what is the probability P(q | d) of producing this query? If the document is relevant to the question, this probability should be higher.

So collecting queries is enough. For each query, feed each candidate document in the pool to the language model and compute P(q | d), treat the high-probability ones as positives, and you have paired data labeled by a language model (pseudo labels). The slide credits this to Sachan et al., Improving Passage Retrieval with Zero-Shot Question Generation.

In the Q&A (8.6) a student asks: one document can correspond to countless queries, so how is the probability computed? She clarifies: it is not computed over all possible queries but over the batch of queries you collected. The computation is the same as for perplexity, with the document as input and the probabilities of the query's tokens multiplied together. The limit is that queries are needed first, and labels are not.

This approach suits multilingual settings especially well (slides 18–19, Huang and Chen 2024). Human labeling rarely crosses languages: someone who reads a Japanese question may not read English documents. Yet we want a model asked in Chinese to be able to find English articles and answer in Chinese. A multilingual language model can compute the probability for "Japanese question paired with English document" directly.

Slide 19 has the results on XOR-TyDi QA:

Added by this post (she does not explain these three metrics): EM is exact match, whether the answer is identical to the gold answer; F1 and BLEU measure how much the answer's wording overlaps with the gold answer. Higher is better for all three.

Her explanation: the orange one (CORA) is trained on human-labeled data, and the red one (UMR) uses no human labels at all, performs about the same, and is even slightly higher. The slide concludes: Training with query likelihood outperforms training with paired data.

Generation: putting passages into the prompt

The retrieved passages go into the prompt with the question and are handed to the frozen LLM (slides 20–21). The slide gives a prompt example with three points:

• The model should answer the query using the information in the context.
• If there is no relevant information in the context, it should try to answer by itself and tell the user that there was no relevant context to base the answer on.
• The answer should be concise, less than 80 tokens.

She explains the second point: retrieval may be inaccurate and bring back something unrelated. In that case the model should not be misled by unrelated content, and it must say so, which is how you know in use whether it actually drew on the retrieved content.

What goes wrong in RAG

Slide 23 divides the issues into two parts:

Retrieval

• Misaligned or irrelevant chunks mislead the generation that follows.
• Redundant information from multiple sources wastes tokens, and becomes a problem when the model's context window is not long enough.

Generation

• Hallucinations not supported by the retrieved context can still occur. Doing retrieval does not rule them out.
• The output suffers from irrelevance, toxicity, or bias in the retrieved content. She gives an attack example: one family of methods asks the model to play a character in a game and describe how a drawn item is made, as a way around safety mechanisms. If retrieved documents contain such content, the model may be led along too.
• Overly relying on retrieved information and simply echoing it. She says this is not always bad; when the answer is direct anyway, copying is fine. If you ask "how does this regulation apply in this situation" and it pastes the full text of the regulation without the reasoning that follows, it has not answered the question.

Advanced RAG: one more stage before and after retrieval

The advanced framework adds processing before and after retrieval (slides 24–25):

Pre-retrieval

• Retrieval decision: decide whether to search. For things the model already knows, searching only adds cost and delay.
• Query rewriting: make the query suitable for the retrieval task. In her words, do not paste the question into search without thinking; sometimes it has to be split into searching A first and then B.
• Context enrichment: expand chunks. A single retrieved chunk may be incomplete on its own, and the surrounding paragraphs can be brought in with it.
• Fusion retrieval / hybrid search: use sparse and dense together. The slide says the two run in parallel; in class she describes narrowing with sparse first and then searching finely with dense.

Post-retrieval

• Reranking: relocate the most relevant content to the front.
• Context compressing: select the essential information and shorten the context. Her contrast: paste five retrieved passages as they are, or summarize and condense the key points first so the model focuses better. She also says doing these does not guarantee better results.

Reranking: pointwise and pairwise

Slide 26 compares two units of reranking:

• Pointwise: look at one document at a time, ask how relevant it is to the query, and give a score from 1 to 5.
• Pairwise: look at two at a time, ask which is more relevant, and answer A or B.

The slide concludes that pairwise reranking is more accurate. Her reason (the same logic as the reward-model discussion in Lecture 7, a link added by this post): absolute scores have no fixed baseline, and the same document may get a 3 this time and a 4 next time; comparing two side by side provides a baseline, and the ranking is more precise.

Example (made up for this post). Setup: the first stage returns 30 candidates to rerank.

• Reasoning: pointwise needs 30 judgments. Pairwise, comparing every pair once, needs 30 × 29 ÷ 2 = 435.
• Result and meaning: with n candidates, pointwise grows with n and pairwise with n². She calls this a fairly large drawback of pairwise. It is also why reranking is done only on a small batch of candidates.

Having people judge which is more relevant is hard, so slides 27–28 (Huang and Chen 2024) have a language model do it: write the two question forms above directly as instructions and let the LLM answer, with no labels and no training. Results are in NDCG@10, a measure of ranking quality over the top 10 results where higher is better (this gloss is added by this post; she only says the table evaluates whether the search is right):

Her reading: the label-free pairwise approach is about on par with the reranker trained on labeled data; pointwise costs somewhat less and performs somewhat lower.

PairDistill on slide 29 (Huang and Chen 2024) goes one step further: since the reranking scores come from a language model, they can be fed back to train the retriever. The better retriever finds better candidates, which are reranked again and used for training again, repeatedly. The slide's loss is the sum of three terms: L = L_CL + λ_KD·L_KD + λ_pair·L_pair. She says that in the end no LLM reranker has to be kept, since the ability has been distilled into the retriever.

When once is not enough, retrieve again

Some questions need more than one lookup. ITER-RETGEN on slide 30 (Shao et al. 2023) concatenates the previous round's output with the question as the basis for the next round of retrieval. The slide's example: what is the height of the player who won the 2015 AFL Rising Star award?

• Round 1: retrieval finds that the award went to Jesse Hogan, with no height. The model continues on its own with "198 cm tall," which is wrong.
• Round 2: retrieving again with the previous output finds that Jesse Hogan is 1.95 m tall, and the answer is right.

She says this is a multi-hop question: you have to know who it is before you can look up his height.

How RAG developed: what to retrieve, how to use it, when to retrieve

Slides 31–48 organize the evolution of RAG with three questions, a structure taken from an ACL 2023 tutorial. Combining the figures across slides gives this table:

The "every token" entry for kNN-LM is not on the roadmap figure; it comes from the text on slide 47.

RAG (Lewis et al. 2020). The term RAG comes from this paper. She says it resembles what people now call RAG only a little: it is an end-to-end model, the retriever and generator are trained together, and retrieved documents are never turned back into text but enter the generator directly as vectors. The retriever learns to give the right documents higher weight, and the generator learns to produce the right tokens given those documents. In the Q&A a student asks why modular RAG differs so much from the original RAG, and she answers: there were no large language models then, the concept is similar and the method differs; the name RAG was well chosen and it stuck.

RETRO (Borgeaud et al. 2022). Inject retrieved information into the model's intermediate layers: insert a chunked cross-attention between the Transformer's attention and fully connected layers so the model attends to the retrieved chunks. The slide concludes that it is more efficient than injecting in the input layer, and that training is required. She says this is why it is now used less: most RAG systems have no labeled data to train this layer.

Retrieval-in-context LM (Ram et al. 2023; Shi et al. 2023). Retrieve every n tokens and put the results in front of the input. Her analogy is writing a long essay and stopping now and then to check the references. On slide 38 the horizontal axis is n, from 64 down to 1, and the vertical axis is perplexity. The smallest GPT-2 (117M) falls from 34.7 to 29.5 and the largest (1.5B) from 18.6 to 16.4. She points out two things: retrieving more often gives better performance, and small models improve more than large ones because they cannot remember as much themselves. The price is on the slide: higher cost in inference, and slower.

FLARE (Jiang et al. 2023). The model generates content first and retrieves only where it is uncertain. The example on slide 40 is a summary about Joe Biden: the model is certain about the first sentence (birth date, which president) and does not look it up. The second sentence says he attended the University of Pennsylvania, and the token probabilities in this span are low. It retrieves, finds that he graduated from the University of Delaware, and rewrites the sentence. She says this approach uses the probability of the generated tokens as a stand-in for confidence.

Search-R1 (Jin et al. 2025). Turn "whether to search" into a reinforcement-learning decision (slide 42). A small language model acts as the policy and decides for itself whether to search and what to search for, the search engine is the environment, and the reward is whether the final answer is correct. She says a 3B model reaches about the same performance as GPT-4 this way.

AdaSearch (Lin et al. 2025). She points to a problem with Search-R1: with answer correctness as the only reward, searching is always encouraged whenever searching leads to a correct answer. Some questions are answered correctly without searching, searching is then redundant, and the result is oversearch. AdaSearch adds a term to the reward for "whether this decision was right" (the slide: answer correctness + correct decision making), asking the model to assess first whether it is able to answer. The slide's phrase is knowing what you know. She adds that such decisions transfer better across models: the same fact may be known without lookup by an 8B model and need a lookup by a 3B one.

kNN-LM (Khandelwal et al. 2020). The unit of retrieval becomes the token. To continue "Obama's birthplace is," first find the k nearest contexts in the training data, such as "Obama was born in" and "Obama is a native of," see what followed them, aggregate those candidates into a distribution, and mix it with the language model's own distribution. The numbers on slide 45: the language model gives Hawaii and Illinois 0.2 each; the neighbors give Hawaii 0.8 and Illinois 0.2; after mixing, Hawaii is 0.6. The mixing formula is (the fixed λ appears in the figure on slide 45):

P(y | x) = (1 − λ) P_LM(y | x) + λ P_kNN(y | x)

The slide's comment is more fine-grained and compute-efficient, and space-expensive, since every training context and its next word has to be stored.

Efficient kNN-LM (He et al. 2021). Not every token needs a lookup. Her example is "from" after "graduated," which is a matter of grammar and unrelated to knowledge. So the decision becomes adaptive: retrieve only at positions that need knowledge and use the language model alone elsewhere. The formula on slide 47 therefore replaces λ with λ(x), varying with the input.

Retrieval sources beyond text

Slide 49 divides sources into three kinds:

• Unstructured: text, from the web or in-house.
• Semi-structured: text plus tables, such as PDFs. The issues the slide lists are data corruption and how to retrieve information from tables; transforming the table into text and using text-only RAG works and is suboptimal. Her explanation is that a table has an easily understood structure, and once it is read out row by row as prose, the structure is gone.
• Structured: knowledge graphs. Nodes are entities and edges are relations, for example a "works at" relation between "Yun-Nung Chen" and "NTU." A question like the earlier "who is the current president's father" can be reasoned step by step on the graph.

She discusses three ways to use tables: convert the question to SQL and query directly; when the table is small, convert the whole thing to text and paste it in; and the more common hybrid, which first filters a relevant sub-table out of a large table and then hands the sub-table to the language model to read closely. She says this is much like the idea of RAG: filter first, then judge finely on the filtered information.

Slide 50 is a three-column comparison for GraphRAG (Peng et al. 2024): asking the LLM directly gives a wrong answer; retrieving text passages is still wrong; retrieving triplets from the graph gives the right answer. The slide concludes: KG can better help reasoning-intensive tasks.

Three kinds of pipeline

Slide 51 (Gao et al. 2023) groups common RAG pipelines into three types:

• Iterative: retrieve → generate → judge whether the information is enough, and retrieve again if not. Her analogy is clicking "next page" when the first page of search results has no answer.
• Recursive: the difference is that each round rewrites the question being looked up. The earlier height example is one: the first round looks up who won the award, and the second switches to that person's height.
• Adaptive: a judgment before retrieval decides whether to look anything up.

She says most vendors' products are now adaptive: answer directly when there is no need, enter retrieval only when needed, and the retrieval part is mostly recursive, breaking the question down, reasoning about what to look up, and continuing until the information is sufficient.

Choosing among prompting, RAG, and fine-tuning

The figure on slide 52 (Gao et al. 2023) has two axes: horizontal for how much model adaptation is required and vertical for how much external knowledge is required. Prompt engineering is at the lower left, with no change to the model and no external knowledge. Upward is RAG: external knowledge is needed and the model is untouched. Rightward is fine-tuning: the model's parameters change.

Slide 53 uses a current event as its example. Ask a computer science student: 「NVIDIA 總部和新壽的合約牽涉哪些法律規範？」 ("Which legal regulations does the contract between the NVIDIA headquarters and Shin Kong Life involve?")

• Prompting: answer directly from existing knowledge. The domain does not match and errors come easily.
• RAG: hand over legal textbooks and answer after reading.
• Fine-tuning: take a related law course first and come back to answer.

Her rule of thumb: if the student can understand the material once given, RAG is enough. If the legal terms are unreadable to the student, textbooks do not help and fine-tuning is needed. If a prompt solves it, use a prompt; if RAG solves it, do not consider fine-tuning, since each is more trouble than the last. The benefit of fine-tuning is long-term: the knowledge is absorbed and the statutes need not be reread each time.

The summary on slide 54:

Her example for fine-tuning is speaking style: a few sentences are not enough to learn it, and it has to be internalized in the model. The slide concludes that choosing between RAG and fine-tuning depends on data dynamics, customization, and computational capabilities.

Whether to train the retriever and LLM together

Slides 55–56 list four options:

• Training-free: neither is trained, and retrieval results go straight to the LLM.
• Independent training: each is trained separately. The retriever learns to find relevant documents and the LLM learns to answer.
• Sequential training: train one first, fix it, then train the other. She says the more practical version fixes the LLM and trains the retriever, since most people cannot change the LLM. The retriever's objective is then not just finding relevant documents but finding documents that let this fixed LLM produce good answers.
• Joint training: train both together. She says the two objectives are related and do not conflict, so this works.

Applications and advantages

Slide 57 lists what RAG is used for: it suits knowledge-intensive tasks best (question answering, fact verification, dialogue), and also applies to general generation tasks (summarization, machine translation, code and proof generation) and classification tasks (NLI, sentiment analysis, commonsense reasoning). Her sentiment-analysis example: if you can retrieve that "many people use this word in this setting sarcastically," the judgment is more accurate.

Slide 58 lists six advantages: accuracy (reduces hallucinations), up-to-date, interpretability and reliability, customization, safety and privacy, and cost-efficiency. On privacy she explains: RAG does not fine-tune the model, so confidential documents do not enter the parameters. With an open-source model hosted on your own machine, there is no worry that someone else can extract the content from the model.

Slide 59 closes with one question: when to use RAG? It lists four cases: long-tail knowledge, knowledge updates, interpretability, and safety or privacy. On the last, she says that when documents must not leak, they should not be fine-tuned into the model.

Selected in-class Q&A

Video 8.6 answers questions collected in class through Slido and has no slides. Below are those related to this lecture, with the questions reworded by this post:

• When ChatGPT answers with citations, is that RAG? Yes. A citation means it searched first and used the result as reference. Clicking through and finding that the page was not actually used is also possible.
• How can one tell how many times it retrieved and how many pages it read? One cannot; it does not say, and how current products work is not fully transparent. Only when building your own system can you set this yourself.
• How are reasoning and RAG related? They are completely independent. One can reason in complex ways without looking anything up, look things up and then reason, or do neither.
• Does doing RAG through an external search engine depend on that engine's ranking? Yes. With poor candidates, no reranking afterward helps. What remains in your hands is rewriting the question, or judging that everything found is irrelevant and leaving it out. The search-engine part is beyond your control. She gives an example: Google Search used to allow 100 results per request, the parameter later stopped working, and it now returns 10 at a time, so 100 results need 10 searches and cost much more.
• Why not attach a time to every piece of data so the model learns temporal order? Some do, and attaching a time to every sentence is cumbersome, with no clear answer on how fine the time should be. It involves temporal reasoning and is not easily solved, and related research is under way.

HW3: training a retriever and a reranker

The end of 8.5 announces that HW3 is released that day and is to implement RAG, including a retriever and a reranker.

The briefing follows a homework slide deck that this guide does not have; what follows is from the TAs' spoken briefing in the video. Model names and numbers in the transcript may be misheard, and enrolled students should follow the course announcements.

• System: the retriever turns the corpus and the query into embeddings for first-stage retrieval, the reranker reorders the first-stage results, and a prompt then has the LLM answer from the retrieved passages.
• What to do: fine-tune the retriever and reranker, and design the prompt. The LLM may not be trained, which the TA says accounts for students having different compute resources.
• Models: the retriever is multilingual E5 small, the reranker is from the MS MARCO MiniLM family, and the LLM is Qwen3 1.7B (the transcript hears "QM3"; the name is this post's inference). The retriever is a bi-encoder and the reranker feeds two texts into one model together, matching the division of labor in this lecture.
• Data: a corpus (one passage per line), qrels (the positive passage for each query), and training and test data. Each item has a query (the field is named rewrite), a list of candidate passages drawn with BM25 with positive or negative labels, and the expected answer. The answer may be a span of text in the positive passage or a fixed "cannot answer" string: even with a positive passage, if its information is insufficient to answer, the model is expected to say it cannot answer.
• Training: the retriever should bring the query's vector close to positives and far from negatives, with the loss function chosen by the student; the reranker should score positives high and negatives low. Negatives can come from the supplied list or be sampled by the student.
• Evaluation: Recall@10 and MRR@10, on the top 10 passages returned by retriever plus reranker. Recall counts a hit when the correct passage is among the top 10; MRR looks at where the correct passage ranks, the higher the better. There is also the sentence-embedding similarity between the LLM's answer and the gold answer.
• Not allowed: using extra training data, training on the test data, or trying to find the private test data.
• Bonus: train a small model with reinforcement learning (the TA's example is a multilayer perceptron, rewarded by the similarity between the LLM's response and the answer) to decide dynamically how many passages to give the LLM. The default is the top three after reranking. The TA's explanation: if the top-ranked passage is the correct one, giving only one reduces noise; if the correct passage ranks lower, more passages have to be given.

The TA closes by noting that this 1.7B model may behave somewhat differently from the 4B model in HW2, so the prompt has to be tuned to the model at hand.

The briefing only says the assignment link is in the homework slides and gives no public dataset link. To practice, pick a public QA dataset, use BM25 as the baseline, fine-tune a dense retriever with the contrastive loss above, add a reranker, and compare Recall@10 and MRR@10 across the three stages.

Common misreadings

This list is my own summary, not something the lecture enumerated point by point.

• "With RAG there is no hallucination." There is only less. The model still mixes in its own knowledge and inference, and the retrieved content may itself be irrelevant or wrong.
• "The finer the chunks, the more accurate the retrieval." Too fine splits apart content that belongs together and increases the number of comparisons. The right size depends on the application and the documents.
• "Dense retrieval has to be trained by yourself." It does not. Using off-the-shelf embeddings directly is the most common practice in industry and needs no training data.
• "A cross-encoder is more accurate, so retrieval should use it." It has to concatenate the query with every document and run the model each time, which costs too much with many documents, so it is placed at reranking.
• "Training a retriever requires human-labeled query–document pairs." P(q | d) computed by a language model can serve as the label, and only queries are needed.
• "The more retrievals the better." Performance improves, and inference gets slower and more expensive; looking up what is already known is redundant.
• "RAG and reasoning are the same thing." They are independent. One is whether external material is used and the other is whether inference is done.

Self-check

The questions were written for this post, and every answer can be found above.

1. What are the four limits of relying on parametric memory alone?

It cannot memorize everything (the long tail), the world changes (outdated knowledge), private documents are out of reach, and a black box is hard to verify.

2. A document says 「大型語言模型」 and the user searches "LLM." What happens with sparse and with dense retrieval?

Sparse scores by words in common, the two share no words, and the document is not found. Dense compares meaning vectors and has a chance of finding it.

3. Which performs better, a bi-encoder or a cross-encoder? Why is the other one used at the retrieval stage?

The cross-encoder performs better because query and document attend to each other directly. It needs a model run for every document, which is too expensive with many documents, so retrieval uses a bi-encoder and the cross-encoder is used for reranking.

4. In the contrastive loss, the positive's score is 1 and all three negatives' scores are 1. What is the loss?

The four scores are equal, the positive's probability is 1/4, and the loss is −ln(1/4) ≈ 1.39.

5. Without paired query–document labels, how can a dense retriever be trained? What is needed first?

Collect a batch of queries, have a language model compute P(q | d) for each candidate document, and treat the high-probability ones as positives. Queries are needed, and human labels are not.

6. The first stage returns 20 candidates. How many judgments do pointwise and all-pairs pairwise need?

20 and 20 × 19 ÷ 2 = 190.

7. Why is one lookup not enough for "what is the height of the player who won a certain award"?

It is a multi-hop question. The first lookup only finds who won, and that result has to be used for a second lookup to get the height.

8. What goes wrong when Search-R1 uses only answer correctness as the reward, and how does AdaSearch change it?

Searching is encouraged whenever it leads to a correct answer, even for questions answerable without searching, which causes oversearch. AdaSearch adds "whether the decision was correct" to the reward, so the model first judges whether it knows.

9. How does kNN-LM differ from ordinary RAG, and what is its drawback?

The unit of retrieval is the token, the retrieved result is used at the output layer and mixed with the language model's distribution, and every token triggers a lookup. The drawback is that all training contexts must be stored, which takes much space; looking up every token is also inefficient, which is why it was later made adaptive.

10. You want a model to learn someone's speaking style: RAG or fine-tuning? And if it should answer about company rules that are updated weekly?

Style has to be internalized in the model and cannot be learned from a few sample sentences, so fine-tuning. Internal rules updated weekly are changing private data, so RAG.

Further reading

• CS224N Lecture 10: Six Components of RAG and Language Agents
• The Complete Guide to RAG System Patterns
• RAG vs Fine-tuning: It's Not Either/Or

Next lecture: Lecture 9, NLG: Decoding, Control, and Evaluation

Course video sources

These videos were checked on 2026-10-10 against the official course page and official YouTube playlist (lecture numbers and titles match); no timestamp is supplied.

Original videos: ADL 8.1: Retrieval-Augmented Generation (RAG) (YouTube, in Mandarin), ADL 8.2: RAG Framework (YouTube, in Mandarin), ADL 8.3: Advanced RAG (YouTube, in Mandarin), ADL 8.4: RAG Roadmap (YouTube, in Mandarin), ADL 8.5: RAG Pros & Cons (YouTube, in Mandarin), ADL 8.6: RAG Q&A (YouTube, in Mandarin), ADL 2025 Fall Homework 3 (YouTube)

Course and recording entries:

• Official course and recording entry

Checked: 2026-10-13.

Content check: verified against the video transcript (2026-10-13): 8.1 through 8.5 were written from their captions. 8.6 and the HW3 briefing have no captions, so this guide was written from speech-recognition transcripts of their audio, which may mishear some proper nouns and numbers. Formulas, tables, and chart numbers follow the slide pages.

Update Log

• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-10-13: Rewritten from the spoken content of the six videos, adding the content of 8.3 through 8.6 and the HW3 briefing video, three examples, common misreadings, and a self-check with answers.

References

• ADL Fall 2025 (114-1) course page — the 10/13 lecture row, videos, slides, and HW 3 link
• RAG slides (251013_RAG.pdf) — the source of all slide numbers here
• ADL 2025 Fall Homework 3 (YouTube)
• 2025 Fall NTU CSIE ADL playlist
• ACL 2023 Tutorial: Retrieval-based Language Models and Applications — the source of the roadmap
• Nakano et al., WebGPT
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering
• Izacard et al., Unsupervised Dense Information Retrieval with Contrastive Learning (Contriever)
• Borgeaud et al., Improving language models by retrieving from trillions of tokens (RETRO)
• Jiang et al., Active Retrieval Augmented Generation (FLARE)
• Jin et al., Search-R1
• Khandelwal et al., Generalization through Memorization: Nearest Neighbor Language Models
• Shao et al., Enhancing Retrieval-Augmented LLMs with Iterative Retrieval-Generation Synergy (ITER-RETGEN)
• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey
• Peng et al., Graph Retrieval-Augmented Generation: A Survey
• Fan et al., A Survey on RAG Meeting LLMs