This lecture re-reads bagging's bootstrap as putting a weight on every example, then deliberately tunes the weights so that the previous round's gₜ has a weighted error rate of exactly 1/2, which forces the next round to learn a different hypothesis. The recipe multiplies incorrect examples by ◆ₜ = √((1−εₜ)/εₜ), divides correct ones by it, and uses αₜ = ln ◆ₜ as the vote weight in a linear vote. That is AdaBoost. As long as the base algorithm is a little better than random guessing in every round, E_in drops to 0 within O(log N) rounds; with decision stumps, three hand-computed rounds on six examples are enough to see it.
A random forest is bagging applied to fully-grown C&RT trees, with each branch looking only at a few randomly projected features. When bootstrap draws N times, each example has about a 1/e ≈ 36.8% chance of not being used by a given tree. These out-of-bag examples give E_oob directly, a validation that needs no held-out data and no re-training; add the permutation test and you get an importance for every feature. The price is that the result wobbles with the random seed, so you keep adding trees until G is stable.
The sum of AdaBoost's weights in each round is exactly the sum of the exponential error exp(−ys). From that, this lecture derives that the gₜ found by the base algorithm is an approximate functional gradient direction, and the vote αₜ is the best step size along that direction. Swap the exponential error for squared error and the same derivation becomes 'in each round, run regression on the residuals yₙ − sₙ, then set the step size with one-variable linear regression'; with a weakened C&RT as the base model, that is GBDT.
With T hypotheses gₜ in hand, you can do more than pick one by validation: give each one vote (uniform blending), learn a set of ballot counts (linear blending), or learn another model that combines them (the form the handout calls any blending, also named stacking). This lecture proves in six lines of algebra that, under squared error, the averaged G is never worse than the average performance of the gₜ, and the gap is exactly how spread out the gₜ are, which is the variance. Finally, bootstrapping re-samples T data sets from the one data set you have, trains one gₜ on each, and lets them vote; that is bagging.
A decision tree is a model that learns conditional aggregation as it goes: every path from the root to a leaf is a condition, and the leaf is a constant. C&RT splits the data in two with one decision stump at a time, picks the split with the smallest weighted impurity, and keeps going until nothing can be split. A tree grown this way almost always has E_in equal to 0, so it is pruned with "E_in plus λ times the number of leaves", with λ chosen by validation. The same recursion also handles multiclass problems, categorical features, and missing values.
The Techniques half of Fall 2024 has two homework sets and a final project, and all three PDFs are public. HW6 covers kernels, soft-margin SVM, and aggregation; its programming part uses LIBSVM on the 3-vs-7 subproblem of mnist.scale to count support vectors, compute margins, and run 128 validation rounds. HW7 covers bootstrap, impurity, AdaBoost, gradient boosting, and neural networks; its programming part is a 500-round AdaBoost-Stump on madelon. The final project is a fictional baseball league, HTMLB: predict home-team wins across two Kaggle stages and write an English report of at most seven pages that compares at least four methods. There are no official solutions. On 2026-09-30 both Kaggle pages returned 404 without login, so outside readers probably cannot get the HTMLB data and should reproduce the same splits on a public dataset instead.
HW4 Problem 3 quantifies why voting trees get more accurate in three steps: with p=0.6 the Hoeffding bound needs B≈691 independent trees to reach 10⁻⁶; correlation ρ between trees floors ensemble variance at ρσ²; and random forests' dense ensembling is compared with MoE's sparse routing.
LLM Council is a local Web App Andrej Karpathy built over a weekend. It sends one question to multiple LLMs simultaneously, has them anonymously peer-review each other, and then a Chairman model synthesizes a final answer. Positioned as a small tool for comparing models while studying — 99% vibe coded with no plans for long-term maintenance — but the architecture itself is a minimal ensemble LLM implementation worth studying.