The primal hard-margin SVM has d̃+1 variables, so it stops being solvable once the feature dimension gets large. This lecture uses Lagrange duality to trade it for a quadratic program with only N variables and N+1 constraints, then uses the KKT conditions to recover w and b from α; only the support vectors, the points with α greater than 0, affect the answer. The dual's matrix still contains d̃-dimensional inner products, so the feature dimension is not really gone until the next lecture.
Gradient descent silently assumes the weight space is Euclidean, and for neural networks it mostly is not. Bernstein's approach picks a norm for each layer (a shape-rescaled spectral norm for linear layers) and computes a norm for the whole network from two combination rules; normalize updates in it and the optimal learning rate stops drifting with width and depth. Push the same idea to duality maps and a linear layer's update becomes 'set every singular value of the gradient to 1', which is also what the nanoGPT training speed record of the time used.
L2 gave us a score function and a loss. L3 answers how to find a good W. The first half covers regularization: add λR(W) next to the data loss so the model does not fit the training data too well. The second half is a lineage of optimizers. SGD zigzags in narrow valleys, Momentum builds up velocity, RMSProp scales each dimension's step, Adam combines the two and adds bias correction, and AdamW moves weight decay outside the moment estimates. The slides close with practical advice: Adam(W) is a good default in many cases, and SGD+Momentum can do better but needs more tuning of the learning rate and schedule.
Techniques Lecture 16 has no new model. It re-sorts the models of the first 15 lectures by three questions: where the features are hidden (kernel, aggregation, extraction, low-dimensional projection), how the error is brought down (gradients, equivalent problems, multiple steps), and how to hit the brakes when the model is too powerful (regularization, validation). This article works PCA plus AdaBoost-Stump on eight points from start to finish and checks the answer against each of the three questions; it closes by keeping the ReLU, Xavier/He initialization, momentum, RMSProp, and Adam material that the Fall 2024 on-campus course added.
Many lines classify all the data correctly; this lecture picks the one farthest from its nearest data point, the fattest one. After scaling (b, w) so that min yₙ(wᵀxₙ+b)=1, the largest margin is the same as minimizing ½wᵀw subject to yₙ(wᵀxₙ+b)≥1 for every point, which can go straight to a quadratic programming solver. Fatness helps for two reasons: it is weight decay within E_in=0, and it reduces the dichotomies that can be produced.
HW3 has two halves. The four written problems run from Newton's method for logistic regression and a convergence analysis of coordinate descent, through backprop, VJPs, and implicit differentiation, to an information-bottleneck view of what deep networks compress. The notebook has you build a BearTensor computation graph in NumPy, topological-sort backprop, and SGD/Momentum/Adam, then train a red-wine quality regressor with it, plus an optional Muon optimizer. Problems and notebook are public; official solutions and hidden tests are not.
CS189 Spring 2026 covers gradient descent in two lectures. Lec 13 derives the learning-rate limit and the condition number from the Hessian's eigenvalues, then moves through momentum, learning-rate schedules, AdaGrad/RMSProp/Adam, and mini-batch SGD. In Lec 15, Dimakis walks through the same material again, starting from a gradient computed by hand on a small data table. Both slide decks, the recordings, Lec 13's handwritten notes, and Discussions 6 and 7 (with solutions) are all publicly accessible.
HW3 has three parts. The programming part has you build an LP solver by vertex enumeration, stack branch and bound on top of it for integer programs, and formulate three word problems. The written part covers integer programming by hand, the ethics of Amazon's delivery routing, PCA via SVD, and a proof that a Laplace prior equals L1. It is due 10/1, so this guide explains structure and concepts only, with no solutions.
07-380 Lec5 turns the Diet Problem from words into min cᵀx s.t. Ax ⪯ b, then draws it: each constraint is a half-plane, the cost is a direction, and cost contours are perpendicular to c. Push a contour in the −c direction until it last touches the feasible region and you always hit a vertex, so solvers only need the intersections of constraint boundaries. Vertex enumeration checks them all; simplex walks greedily from one vertex to a better neighbor.
07-380 Lec6 adds one constraint to an LP, x ∈ ℤᴺ, and the vertex solution may no longer be an integer. Searching the integer points near the LP solution is not guaranteed to work either. The fix: drop the integer constraint (relaxation) to get an LP lower bound, split on a fractional coordinate into xᵢ ≤ floor and xᵢ ≥ ceil, and keep every subproblem in a priority queue ordered by LP objective. The first all-integer solution popped is optimal.
The course site's schedule splits 07-380's first ten lectures into Reasoning Under Certainty, Optimization and Reasoning Under Uncertainty: prove things with logic and plan with search, then write problems as constrained objectives, and finally let a prior in with MAP and turn to probabilistic models. HW1 tests logic plus search, HW2 planning plus LP graphing, HW3 writing solvers plus PCA and MAP derivations.
Week 3 covers three ways to write an optimization problem. Local search keeps a single state: hill climbing moves to a better neighbor and gets stuck at local optima, and simulated annealing accepts a worse move with probability e^(ΔE/T). Linear programming writes the problem as a linear cost function plus linear constraints and hands it to an existing solver. Constraint satisfaction problems shrink domains with node consistency and arc consistency (AC-3), then find a solution with backtracking search guided by the MRV, degree and least-constraining-value heuristics. Includes per-section time links, plus the approach, pseudocode, pitfalls and checks for the Crossword project (the reference implementation is folded at the end).
Optimization is not an isolated numerical problem: view SGD spectrally, the magnitude of weight updates determines feature learning; Maximal Update Parameterization transfers LR/init across width, and the critical batch size sets the marginal return of trading compute for convergence.
A model uses loss to know how wrong it is and gradients to know which direction to adjust. Gradient descent repeats three things: compute loss, compute gradients, update parameters. The learning rate controls step size — too large and you overshoot, too small and training takes forever.
Lecture 8 moves from a one-dimensional parabola to vector gradients and compares batch GD, SGD, and mini-batches; the learning rate determines whether updates converge, oscillate, or diverge.
Lecture 11 reads public recipes from MiniCPM, DeepSeek, Qwen, and Llama 3: hold most architectural ratios fixed, sweep learning rate and batch at small scale, then choose model/data allocation with IsoFLOPs. μP helps, but normalization, optimizers, and weight decay can break transfer.
Automatic prompt optimization (APO) has evolved from APE/OPRO to GEPA: replacing sparse rewards with linguistic reflection, winning over GRPO by ~6pp with 4-35x fewer rollouts. Meanwhile, tool descriptions are the overlooked prompt -- small wording changes can shift tool selection rates by 10x, and Anthropic's experiments show Claude self-rewriting tool descriptions outperforms human experts. These two lines are converging: eval-driven automatic optimization is eating hand-tuned prompts.