A generative model plus its latent space works as data you can operate on: moving along a latent direction yields counterfactual images, and a handful of manual labels expands into a whole segmentation dataset (one labeled GAN image is worth about 100 ordinary labeled images). But representations trained on generated data still trail real data (42.6% against 43.9% in the numbers cited in class), and rare classes are exactly where generators are least reliable.
Test loss follows an approximate power law in data, parameters and compute, but the exponents are small: Kaplan et al. measure a data exponent of about 0.095, so doubling the data lowers the loss by only 6% and halving it takes over 1,400 times more. Two papers disagree on how to split compute: Kaplan gives parameters ∝ C^0.73 and data ∝ C^0.27, Chinchilla about half each. The public recording stops at the second finding; everything after that is written here from the slides, without the lecturer's spoken explanation.
A language model does one thing: given the preceding text, output a distribution over the next word. At large enough scale, the way to control it becomes changing the input: a few examples (in-context learning), or asking it to write its reasoning first (chain-of-thought; the example cited in class is one sentence, Let's think step by step, taking an arithmetic task from 18% to 80%). But following instructions and saying 'I don't know' do not come from next-word prediction alone; instruction tuning and RLHF add them.
Gradient descent silently assumes the weight space is Euclidean, and for neural networks it mostly is not. Bernstein's approach picks a norm for each layer (a shape-rescaled spectral norm for linear layers) and computes a norm for the whole network from two combination rules; normalize updates in it and the optimal learning rate stops drifting with width and depth. Push the same idea to duality maps and a linear layer's update becomes 'set every singular value of the gradient to 1', which is also what the nanoGPT training speed record of the time used.
A trained model is not limited to one forward pass. Greedy sampling does not guarantee the sequence the model itself considers most likely (in this post's example it gets 0.30 where the best is 0.36), so sampling several times and keeping the best already helps; swap the scoring criterion for a verifier and you can pick answers that are actually correct. Feed the good answers found by search back into training and you have the loop behind AlphaGo and STaR, which Isola speculates is also how o1 works.
Apodex is a reasoning-first Mixture-of-Experts model family built by the Apodex team for complex long-horizon research, forecasting, and verifiable reasoning tasks. An agent runtime supplies file, data, code, and tool execution; the core API accepts text only.
Cohere's North series marks its first agentic coding model family. North Mini Code (30B total / 3B active sparse MoE) is purpose-built for code generation and automated development, with 256K context and free access via OpenRouter as `north-mini-code:free`.
LFM2.5-2.6B is an on-device hybrid reasoning model; checkpoint and OpenRouter context limits differ, with license and data terms to check before deployment.
LongCat-Flash and LongCat-2.0 are Meituan open-weight MoE models. This guide explains dynamic compute, LSA, long context, tool use, and the difference between model specifications and hosted endpoints.
Nemotron covers text reasoning, multimodal understanding, and safety classification. Ultra, Super, Lightning, Nano Omni, and Content Safety differ in architecture, context, and licensing; model specifications and API limits must be checked separately.
pplx-embed is the embedding family Perplexity has been releasing since February 2026, in three lines: v1 single-vector (0.6B/4B, API at $0.004/$0.03 per 1M tokens), context embeddings (v1 plus a 9B v2 preview), and v2-late multi-vector (0.6B/9B, searches PDF page images directly). They solve different problems; v2 does not replace v1.
The public materials for the CS181 2026 final (May 9) — the final checklist, 16 second-half practice problems, and a 66-page final review — are all 2025 versions. They cover Bayes nets and EM, which the 2026 schedule never lists, and skip Transformers, VAEs, GANs, and autoregressive models. Sort the checklist against the 2026 schedule into three buckets, cover the new topics with homework and sections, then do the 2025 practical for one end-to-end project.
CS181 Spring 2026 HW2 (due Feb 27) has four problems worth 90 points: train 10 logistic models on planet observations to see bias and variance, derive the MLE of a generative classifier, implement five classifiers on 27 loan applicants, and watch ridge reshape the loss surface under SGD, momentum, and Adam. The core skill is separating what one model's probability says from how much 10 models disagree.
CS181 Spring 2026 HW3 (due Mar 23) has three problems worth 100 points: unpack polynomial and RBF kernels into feature maps and go from ridge to dual coefficients α and the support-vector intuition; hand-derive backprop for a two-layer sigmoid network; and, for half the grade, train ResNets on Fashion-MNIST, measure your own scaling law, and use C≈6ND to split a fixed compute budget between model and data.
HW4 Problem 2 has you train a convolutional autoencoder on 64×64 CelebA, sample from N(0, I), and watch it fail to produce faces. You then derive the ELBO, the reparameterization trick, and the closed-form KL, and turn the same backbone into a VAE to compare reconstructions and samples.
HW4 Problem 1 (40 pts) takes self-attention apart in five steps: a 2×2 hand calculation, why we divide by √dk, permutation equivariance without positional encoding, single-head attention in pure NumPy, and multi-head attention in PyTorch with an attention heatmap on synthetic data.
HW4 Problem 3 quantifies why voting trees get more accurate in three steps: with p=0.6 the Hoeffding bound needs B≈691 independent trees to reach 10⁻⁶; correlation ρ between trees floors ensemble variance at ρσ²; and random forests' dense ensembling is compared with MoE's sparse routing.
HW5 Problems 3–4 give handwritten digits to three methods that never see a label: K-means summarizes the data with 10 mean images, HAC builds a merge tree you can cut at any number of clusters, and PCA compresses images onto a few continuous directions. All three answer the same question — how much error do you pay to describe the data with a few objects — and the assignment makes you compare their objectives and reconstruction errors directly.
HW5 Problems 1–2 both turn learning into a classification task. SimCLR's NT-Xent loss asks the network to pick the other augmented view of the same image out of 2N−1 candidates; a GAN's discriminator classifies real versus fake. You derive the math behind each (the cross-entropy equivalence, the optimal discriminator and the JS divergence), then write both training loops on FashionMNIST and MNIST.