CNNs and ViT: Turning an Image into Model Inputs | CMU 10-423 Lecture 5
Text models convert text into tokens; image models also need a computational representation of pixels.
#bert
7 postsNTHU NLP Guide 8: ELMo, BERT, T5, BART, GPT and the Three Roads to Pretraining | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
A guide to the BERT and its Family unit of Prof. Hung-Yu Kao's NLP course at NTHU (Fall 2025). It starts from "I record the record": Word2Vec and GloVe give both records the same vector. ELMo fixes this with a bidirectional LSTM language model. With Transformers, pretraining splits into three roads: encoders (BERT: MLM plus NSP, strong at understanding, weak at generation), encoder-decoders (T5's span corruption, BART's five noise types), and decoders (GPT: pure next-token prediction). The last part covers GPT-3's in-context learning and scaling laws, and why decoders became today's dominant backbone.
NTHU NLP Guide 9: One BERT That Scores and Classifies at Once — The Hugging Face Tutorial and HW3 Multi-Output Learning | NTHU Hung-Yu Kao Natural Language Processing Topic Guide
A guide to the Hugging Face BERT tutorial and HW3 in NTHU Prof. Hung-Yu Kao's NLP course (Fall 2025). The tutorial walks through binary IMDb sentiment classification: AutoTokenizer, the input_ids / token_type_ids / attention_mask fields, AutoModelForSequenceClassification, and Trainer. HW3 applies the same tools to SemEval 2014 Task 1: one bert-base-uncased with two heads, one regressing a 1–5 relatedness score and one classifying entailment into three classes. You add the two losses and write the training loop yourself, because Trainer is not allowed.
Tokenization and BERT: How Words Are Split and How Representations Become Contextual | NTU Yun-Nung Chen Applied Deep Learning Lecture 5
This lecture has two halves. The first covers tokenization: with whole words as units, unseen words can only become UNK; with characters as units, meaning cannot be put back together. The compromise is subwords, and a common way to build them is BPE: start from characters and repeatedly merge the most frequent adjacent pair. The second half covers BERT: a static word vector gives "apple" one embedding and cannot tell the fruit from the company. ELMo uses a language model to produce contextual representations, and BERT replaces the LSTM with a Transformer, pre-trains with a masked LM that hides 15% of tokens plus next sentence prediction, and then fine-tunes the whole model on downstream tasks.
Chinese Extractive QA: Finding the Answer Span Among Four Paragraphs | NTU Yun-Nung Chen Applied Deep Learning Assignment 1
HW1 in ADL Fall 2025 gives a question and four Chinese paragraphs. The model first picks the relevant paragraph (paragraph selection, framed as four-way multiple choice), then marks the answer's start and end inside it (span selection), scored by Exact Match. The spec slides point you straight at Hugging Face's run_swag_no_trainer.py and run_qa_no_trainer.py. The simple baseline uses bert-base-chinese, length 512, effective batch size 2, and learning rate 3e-5, and both stages together take under three hours on an 8GB RTX 3070. The Kaggle leaderboard closed 9/29, and code plus report were due 10/1 on NTU COOL. Outside readers cannot get the Kaggle data or grading, but the task design, baseline settings, and five report questions are all usable for practice.
How One Transformer Grew into BERT, GPT, and a Zoo of Attention Variants | Stanford CME295 Lecture 2
CME295 Lecture 2 takes the original Transformer apart and refits it: position information moves from "added to the embedding" to RoPE's "rotate Q and K inside attention"; attention gets cheaper with sliding windows and MQA/GQA; models split into encoder-only, encoder-decoder, and decoder-only families; and the second half dissects BERT's MLM (15% of tokens) and NSP pretraining. The 2026 edition folds all of this into a single "Large Language Models" lecture, and BERT is no longer a syllabus item.
Language — N-grams, Parser, Attention & the Transformer | Harvard CS50 AI Week 6
Week 6 is about getting computers to handle natural language: parse sentences with hand-written context-free grammar rules, then switch to statistics, from n-grams and Markov-chain text generation to a Naive Bayes classifier for review sentiment; turn words into vectors with word2vec, process whole sequences with an RNN encoder–decoder, and finish with attention and the Transformer. Includes section-by-section time links, plus an approach, common mistakes and ways to check your work for the Parser and Attention projects.