MIT 6.7960 L03: Optimization Overview — SGD, Adam, LR Schedules & Scaling Rules
From SGD to Adam: pick the right optimizer and scale LR with batch size using scaling rules
#optimization
11 postsHarvard CS50 AI Week 3: Optimization — Local Search, Simulated Annealing, CSP & Crossword Generation
Week 3 tackles optimization: hill climbing, simulated annealing escaping local optima, CSP framework with AC-3 arc consistency, backtracking with MRV/degree heuristics. Project Crossword builds a crossword puzzle generator.
Harvard CS50 AI Synthesis (1): From Search to Language — The Complete Arc of Seven Weeks
Synthesis 1: Tracing how seven weeks form a deliberate knowledge arc from symbolic search to language models, revealing the design philosophy from classical AI to modern ML.
MIT 6.7960 L07: Scaling Rules for Optimization — Spectral View, Feature Learning, Hyperparameter Transfer
Optimization is not an isolated numerical problem: view SGD spectrally, the magnitude of weight updates determines feature learning; Maximal Update Parameterization transfers LR/init across width, and the critical batch size sets the marginal return of trading compute for convergence.
How Models Improve Themselves: Gradient Descent and the Training Loop
A model uses loss to know how wrong it is and gradients to know which direction to adjust. Gradient descent repeats three things: compute loss, compute gradients, update parameters. The learning rate controls step size — too large and you overshoot, too small and training takes forever.
CMU 07-280 Lecture 8: Gradient Descent, SGD, and Learning Rate
Lecture 8 moves from a one-dimensional parabola to vector gradients and compares batch GD, SGD, and mini-batches; the learning rate determines whether updates converge, oscillate, or diverge.
CS336 Lecture 11: Scaling Laws in Practice Must Scale Learning Rate and Batch Too
Lecture 11 reads public recipes from MiniCPM, DeepSeek, Qwen, and Llama 3: hold most architectural ratios fixed, sweep learning rate and batch at small scale, then choose model/data allocation with IsoFLOPs. μP helps, but normalization, optimizers, and weight decay can break transfer.
Support Vector Machines: Margins, Duality, and SMO
Chapter 6 formalizes classification confidence as geometric margin, then builds an implementable SVM through Lagrange duality, kernels, and SMO.
CMU 10-301 HW4: Turn Logistic Regression Likelihood into a Classifier
HW4 joins probabilistic interpretation, cross-entropy gradients, and implementation into one traceable training pipeline.
Stanford CS107 Lecture 24: Profile with Callgrind, Then Read What GCC Optimized
CS107 Lecture 24 builds a measurement workflow with matrix multiplication and Callgrind, then examines GCC constant folding, common-subexpression elimination, dead-code elimination, strength reduction, code motion, and recursion-to-loop conversion. Optimization starts with bottleneck evidence.
Stop Hand-Tuning Prompts: From GEPA to Tool Descriptions, Automating Agent Behavior Optimization
Automatic prompt optimization (APO) has evolved from APE/OPRO to GEPA: replacing sparse rewards with linguistic reflection, winning over GRPO by ~6pp with 4-35x fewer rollouts. Meanwhile, tool descriptions are the overlooked prompt -- small wording changes can shift tool selection rates by 10x, and Anthropic's experiments show Claude self-rewriting tool descriptions outperforms human experts. These two lines are converging: eval-driven automatic optimization is eating hand-tuned prompts.