Natural Language Processing: A Learning Path from Text Representations to Language Models | NTHU Hung-Yu Kao Natural Language Processing Course Overview

🌏 中文版

Video status: Videos included. The intro embeds the Week 1 Tuesday recording (1 h 25 min); each week's recordings are in the individual posts. Source details

Natural Language Processing, taught by Professor Hung-Yu Kao of National Tsing Hua University's computer science department, is a lead course in TAICA, Taiwan's inter-university AI program: graduate level, taught in Mandarin, and taken remotely and synchronously by students at member schools. Slides, recordings, and assignments all live in the lab's GitHub repo, so outside readers get nearly everything enrolled students get, minus solutions and grading.

The course answers one question: how did computers get from not understanding text at all to LLMs that answer questions? It opens with sentences that read two ways and runs through TF-IDF, word vectors, RNNs, Transformers, and BERT to RLHF, PEFT, and RAG. Each step patches a gap left by the previous one, so reading in order shows best why the technology looks the way it does today.

This post is the series entry point. It lays out what each of the six segments teaches, works a TF-IDF example by hand to show the first week's core idea and its limit, and ends with a prerequisite check and ways to read. Below is the Week 1 Tuesday recording, which matches the first post in the series; watch it before reading, or come back to it after the first post.

The storyline: six segments

Turning text into numbers (Posts 1–3)

Post 1 takes an information-retrieval view and strings inverted indexes, tokenization, stemming, TF-IDF, and BM25 into one pipeline: the more often a word appears in a document and the rarer it is in other documents, the higher the score. The pipeline's weak spot is synonyms and polysemy, which leads to dimensionality reduction with LSA's SVD. Post 2 switches to predicting the next word: statistical language models with bigram count tables, add-one smoothing, and perplexity; dense word vectors with Word2Vec's negative sampling; and RNNs for sentences of any length. Assignment 1 tests word vectors on nearly 20,000 Google Analogy questions to see whether "king − man + woman" really lands near queen.

Handling sequences (Posts 4–7)

When input and output lengths differ (translation, summarization), the classifier trick of adding a final layer no longer works; you need an encoder-decoder. Post 4 covers vanishing gradients in RNNs, the three gates of an LSTM, and attention, which lets the decoder look directly at the whole input. Post 6 removes the RNN: self-attention lets every word look at every other word and computes it all in one matrix multiplication, at the cost of losing word order, which positional encodings restore. Post 7 explains why vocabularies are made of word pieces: BPE repeatedly merges the most frequent adjacent pair, so even unseen words can be split. Assignment 2 treats "14*(43+20)=882" as a character sequence and has an LSTM generate the answer character by character after the "=".

Pretrained models (Posts 8–11)

Post 8 starts from "I record the record": Word2Vec gives both "record"s the same vector, and context has to decide the meaning. With Transformers, pretraining splits into three paths: encoder-only BERT (predict masked words, good at understanding), encoder-decoder T5 and BART, and decoder-only GPT (predict only the next word, which became the mainstream). Post 10 covers how to pick words during generation: greedy cannot recover from one bad step, beam search keeps several candidates at once, and top-k/top-p sample to get more variety; scoring uses BLEU (counts n-gram overlap and penalizes sentences that are too short), ROUGE, and perplexity. Assignment 3 puts two heads on one BERT to do 1–5 regression and three-way classification at once; Post 11 does Chinese summarization once with GPT-2 and once with mT5: the decoder-only model needs the article and summary joined into one sequence, left padding, and −100 labels on positions that should not count toward the loss; the encoder-decoder tooling handles this automatically.

Following instructions and fine-tuning with few parameters (Posts 12–13)

Post 12 explains the gap between continuing text and following instructions: the maximum-likelihood objective cannot tell that inventing a fact is worse than picking the wrong synonym, so InstructGPT adds three steps: SFT to learn how people write, a reward model to learn how people rate, and PPO to adjust to the ratings, with a KL penalty to keep the model from drifting too far. Post 13 starts with a calculation: full fine-tuning of Llama 2-7B needs about 56 GB of GPU memory, and training only a small set of parameters cuts it to about 17 GB, because gradients and optimizer state almost disappear. Methods fall into four kinds: adding parameters (adapters, prompt tuning), selecting parameters (BitFit), reparameterizing (LoRA), and hybrids.

Plugging in outside knowledge (Posts 14–17)

LLMs confidently state wrong facts; RAG retrieves first, then generates. Post 14 is about finding the right material: sparse vectors (TF-IDF, BM25) are cheap and reliable, dense vectors catch paraphrases; cross-encoders are accurate but slow, while bi-encoders can precompute document vectors. Post 15 connects the retriever to a generator and surveys fixes such as rewriting the query, making the generator robust to noise, and deciding when to retrieve (Self-RAG). Post 16 shows prompts, JSON output, and few-shot through APIs. The two TA sessions in Post 17 each build a RAG, the second showing hybrid BM25-plus-cosine retrieval; Assignment 4 reuses the first session's LangChain-plus-Chroma skeleton to build a RAG over cat trivia and reports recall@k and exact match.

Summary and reasoning (Posts 18–19)

Post 18 wraps up the semester in three columns (fundamentals, models, advances), then summarizes Denny Zhou's talk: pretrained models can already reason, and the key is decoding, drawing reasoning out with CoT decoding and self-consistency. Post 19 covers the final project's grading structure and the switch to a midterm exam in Fall 2026.

Work it by hand: how TF-IDF scores, and where it fails

This is a teaching example written for this post, to show the first week's core idea and why it carries all the way to word vectors and RAG.

Three already-tokenized documents:

• D1: cat　likes　fish
• D2: dog　likes　bones
• D3: kitty　hates　baths

TF-IDF weights each word by how many times it appears in the document times $\text{idf} = \ln(N / \text{df})$, where $N$ is the number of documents and df is how many contain the word. Here $N = 3$:

• "cat" appears only in D1: idf $= \ln 3 \approx 1.10$.
• "likes" appears in D1 and D2: idf $= \ln 1.5 \approx 0.41$.

For the query "likes cat", add up the query words' weights: D1 $= \ln 1.5 + \ln 3 = \ln 4.5 \approx 1.50$, D2 $= 0.41$, D3 $= 0$. D1 ranks first, as it should: "cat" is rarer than "likes", so it discriminates better and gets more weight.

The problem is D3. It is clearly about a cat, yet scores 0, because "kitty" and "cat" are different strings. This is the vocabulary mismatch Post 1 talks about.

What this means: with exact matching alone, rare words are correctly weighted up, but synonyms never match. Post 2's word vectors put "cat" and "kitty" close together, and Post 14's dense retrieval applies the same idea to RAG; but dense vectors have their own weak spots (proper nouns, exact numbers), which is why the TA session in Post 17 combines BM25 and cosine. The whole course works like this: each step patches the previous one's gap and brings a new trade-off.

Prerequisites and self-check

You need Python, basic linear algebra and probability, and a little machine learning (what a loss function and gradient descent are). For linear algebra, check yourself with Harvard CS181's linear algebra review; for probability, read the Stanford CS109 guide; for loss functions and gradient descent, see NTU ADL Lecture 2. Assignments use PyTorch and Hugging Face, which the TA sessions in Posts 5 and 9 teach from scratch. The syllabus says the course does not teach prompt usage, and that it is not for people who expect no exams and plan to get through the homework by collaborating.

After reading this post, try to answer:

1. Add a fourth document D4 "cat sleeps" to the example. What is the idf of "cat" now, and how does D1's score change? Answer: $N = 4$, df $= 2$, so idf $= \ln 2 \approx 0.69$; "likes" rises from 0.41 to $\ln 2 \approx 0.69$ (with more documents it is relatively rarer), so D1 $= 2\ln 2 \approx 1.39$, only a little below the original 1.50. As "cat" gets more common, its weight drops.
2. Why does self-attention need positional encodings? Answer: Every word looks at every other word at once and the computation ignores order, so "dog bites man" and "man bites dog" would come out the same; position has to be added separately.
3. How do BERT's and GPT's pretraining objectives differ, and what is each good at? Answer: BERT masks some words and predicts them from both sides (good at understanding); GPT predicts the next word from the left context only (good at generation).
4. Why does InstructGPT's PPO add a KL penalty? Answer: To keep the model from drifting far from the original just to please the reward model, producing odd outputs the reward model never saw but scores highly.
5. Why do RAG systems often combine BM25 with dense vectors? Answer: BM25 catches exact terms and proper nouns, dense vectors catch paraphrases; their failures complement each other.

Three ways to read

This series follows the finished Fall 2025 term in its actual teaching order, with each TA session merged into the post for its assignment.

• One quick pass (8 posts): 1 → 2 → 6 → 8 → 12 → 13 → 14 → 18. Lectures only, for readers who want to know what happened between TF-IDF and RAG.
• Follow the whole course (19 posts): read in order; the recording linked in each post is that week's live stream. About a semester's worth.
• Assignments only: 3 → 5 → 9 → 17, plus the tool posts 11 and 16. Each assignment has a problem PDF, a main.ipynb, and data files you can drop into Colab. There are no public solutions and the course platform (NTU COOL) needs an enrolled account, so set up your own checks; for Assignment 1, for example, use the pretrained GloVe accuracy as your baseline.

Two things that affect how you read: the Topics column in the 2025 README is a syllabus template that does not match the slides actually posted in each row, so just follow this series' order; and the LLM API TA session uses 2024 model names, some since retired, which Post 16 explains how to replace.

Fall 2026 is running now with 75% homework plus a 25% in-person midterm and a new Reasoning/Agent unit; only the first few weeks are public, and this series does not draw on it.

Posts in this series

What to do tonight

1. Open the 2025 README, click Week 1's slides, and find the TF-IDF pages to compare with the hand calculation above.
2. Drop Assignment 1's main.ipynb into Colab and run it up to the cell that loads GloVe.
3. Read Post 1.

Further reading

• Reading Stanford CS224N: Stanford's main NLP course, taught in English.
• Reading Stanford CS224U: natural language understanding, retrieval, and evaluation methods.
• Reading NTU Hung-yi Lee's Machine Learning 2026 Spring: also taught in Mandarin with all recordings public, but centered on AI agents and model behavior.

Course video sources

This is a course overview; the intro embeds the Week 1 Tuesday recording. For other weeks, use the week table in the official README; each post also links its recording.

Course and recording entries:

• Official course and recording entry
• Week 1 Tuesday recording (embedded in the intro, 1 h 25 min)

Checked: 2026-10-14.

Update Log

• 2026-10-14: Rewrote the body as a real guide: what each of the six segments teaches, a hand-worked TF-IDF example, self-check questions, and the Week 1 recording in the intro; the semester comparison, README week column, and access ratings were cut down to a few sentences in the reading section.
• 2026-10-10: Added explicit video status and checked recording sources and access notes.

References

• IKMLab/NTHU_Natural_Language_Processing (main README, 2026 edition)
• 2025 README: Fall 2025 weekly table
• 2025 W0_Syllabus.pdf (in Chinese and English)
• 2026 Syllabus-115.pdf (in Chinese and English)
• 2025 Assignments README
• 2026 Assignments README
• 2025 Reference notebooks
• TAICA course list, Fall AY114 (in Chinese and English)
• TAICA course list, Fall AY115 (in Chinese and English)
• IKMLab NTHU YouTube channel (in Mandarin)
• Fall 2025 Week 1 Tue. recording (in Mandarin)
• Fall 2026 Week 1 recording (in Mandarin)
• W1_NLP_brief.pdf (Fall 2025) — Week 1 slides: TF-IDF, BM25, vocabulary mismatch, and LSA
• W2_Word embeddings and Language Modeling (RNN).pdf (Fall 2025) — Week 2 slides: language models, word vectors, and RNNs