Chapter 9 tuned the decision tree by hand — a single slider called max_depth. The deck now asks the question behind that manual work: how do we tune hyperparameters?
Hyper-parameters are parameters that are not directly learnt within estimators.
The distinction is the key of the chapter:
.fit() — the tree's thresholds, the forest's votes, the perceptron's weights (they end with an underscore, as Chapter 8 showed);.fit() never touches them.The same estimator class can produce very different models just by changing constructor arguments. DecisionTreeClassifier(max_depth=1) and DecisionTreeClassifier(max_depth=9) are the same algorithm with different hyperparameters — and, as Chapter 9 showed, one underfits while the other risks overfitting. Choosing the values is a second optimization problem on top of training, and this chapter gives it a name: hyperparameter optimization.
The deck first makes the scope precise: any parameter provided when constructing an estimator may be optimized. You can see the full list for any fitted estimator with:
estimator.get_params()Then it defines what a search is made of — five ingredients, two of which the course already has (marked with the deck's checkmarks):
The remaining sections take the three missing ingredients in order: the space (§3), the method (§4), and the validation scheme (§5–6) — then the deck runs the whole pipeline on a random forest (§7).
The deck defines the search space geometrically:
Search Space: space where each dimension represents a hyperparameter and each point represents one model configuration.
Its example uses the random forest — the model that will close the chapter. Two hyperparameters, two dimensions:
max_depth: maximum depth of a single tree, domain [1, 10];n_estimators (the deck writes #estimators): number of trees in the forest, domain [2, 20].Every pair (max_depth, n_estimators) is one forest configuration — one point in the 2D search space. With these domains there are 10 × 19 = 190 distinct configurations.
Click cells of the space to read the configuration they represent, then flip the switch to see what a 5% good region looks like inside it.
Once the space exists, something must walk it. The deck presents the two classical strategies:
The deck then quantifies random search with a striking back-of-the-envelope calculation. If good parts of the search space occupy 5% of the volume, the chances of hitting a good configuration with one draw is 5%. With repeated independent draws the probability of finding at least one good configuration grows:
1 - 0.95^60 = 0.953 > 0.95After 60 configurations, the probability of having seen at least one good one is above 95% — a guarantee grid search cannot make any cheaper, and one that stays valid even when the space is too large to enumerate (e.g. when a hyperparameter is continuous).
Grid search is exhaustive but fragile: it commits to a fixed list of values per dimension. Random search spends the same budget on 60 different configurations spread across the whole volume — and for high-dimensional spaces that spread is usually worth more than completeness. This is why the deck's practical recipe (§7) uses RandomizedSearchCV, not GridSearchCV.
Move the good-region share and the sampling budget, and watch the probability of at least one hit respond.
The search produces candidate configurations, and each one needs a score. The deck's question is blunt: how do we test the hyperparameter configurations?
The answer cannot be the test set — that would leak it into the choice of hyperparameters. And the training set alone is misleading: a deeper tree scores perfectly on training while overfitting, as Chapter 9 showed. The standard solution is cross-validation:
Three sets, three roles: the training set fits the model, the validation folds (inside cross-validation) choose the hyperparameters, and the test set is used exactly once, at the end, to report the honest performance. If a pipeline tunes on the test set, its accuracy is optimistic — this is the leak the exam asks about.
The deck illustrates the mechanism with a k-fold diagram. The essential property to see in it: every training sample is used for validation exactly once, and every model is trained on k−1 folds. With the deck's later choice of cv=5 on 120 training samples, each fold holds 24 samples and each configuration costs 5 fits.
cv=5. Five fits per configuration; the reported score is the mean of the five validation accuracies — which is exactly what best_score_ will contain in the next section.Choose k and step through the rounds: watch one fold turn vermilion while the model trains on the rest.
The deck now closes the loop on the model promised in Chapter 9 — the random forest — and runs a real random search over its hyperparameters:
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
rf = RandomForestClassifier(random_state=seed) # the estimator
param_dist = { # the parameter space
'n_estimators': randint(2, 200), # Number of trees in the forest
'max_depth': randint(2, 20), # Maximum depth of the tree
'min_samples_split': randint(2, 20), # Minimum number of samples to split a node
'min_samples_leaf': randint(1, 20), # Minimum number of samples in a leaf node
}
random_search = RandomizedSearchCV(rf, param_distributions=param_dist,
n_iter=50, scoring='accuracy',
cv=5, random_state=seed, n_jobs=-1)
random_search.fit(X_train, y_train) # the search
print("Best Parameters from Random Search:", random_search.best_params_)
best_rf = random_search.best_estimator_
test_accuracy = best_rf.score(X_test, y_test)
print("Best Cross-validation Accuracy:", random_search.best_score_,
"Test Set Accuracy with Best Parameters:", test_accuracy)
forest = plot_boundary(best_rf, "rf_cplot")The deck's printed output:
Best Parameters from Random Search: {'max_depth': 4, 'min_samples_leaf': 3, 'min_samples_split': 2, 'n_estimators': 124}
Best Cross-validation Accuracy: 0.9583333333333334 Test Set Accuracy with Best Parameters: 1.0Every ingredient of Plate 10.1 is in those lines: the estimator (rf), the score function (scoring='accuracy'), the space (param_dist, four dimensions this time — the deck's 2D drawing was just the example), the sampling method (RandomizedSearchCV, n_iter=50), and the validation scheme (cv=5).
Reproducing this exact search on the deck's setup (seed 42, 50 iterations, 5-fold CV) returns precisely the printed values: best parameters {'max_depth': 4, 'min_samples_leaf': 3, 'min_samples_split': 2, 'n_estimators': 124}, cross-validation accuracy 0.9583, test accuracy 1.0. The replay widget below walks the 50 actual sampled configurations of that run. One observation worth keeping: 18 of the 50 sampled configurations already reached ≥ 0.95 CV accuracy — on this dataset the good region is much larger than the 5% worst case of the deck's example, which is why random search finds it so reliably.
The real sampled configurations of the run above, in order. Step through them and watch the best candidate so far evolve.
Why does the winning forest (depth 4, 124 trees) draw a cleaner boundary than any single tree? This widget trains a real bagged forest on the petal plane and lets you grow it tree by tree.
Hyperparameters are parameters not directly learnt within estimators — they are set before training. In scikit-learn they are passed as arguments to the constructor of the estimator class (e.g. max_depth=2), and .fit() never changes them. The learnt parameters are instead exposed as attributes ending with an underscore after fitting.
An estimator (✓ already available), a score function (✓), a parameter space, a method for searching or sampling candidates, and a cross-validation scheme. Any parameter passed to the constructor may be optimized; get_params() lists them.
One dimension per hyperparameter, one point per configuration: max_depth ∈ [1, 10] and n_estimators ∈ [2, 20] for a random forest. The space holds 10 × 19 = 190 distinct forest configurations.
Grid search exhaustively tries every combination of the provided values; random search samples from the entirety of the space and needs no gradient (direct-search / derivative-free / black-box). If the good region is 5% of the volume, one draw hits it with probability 5%, and 60 draws find at least one good configuration with probability 1 − 0.95^60 = 0.953 > 0.95.
Using the test set for tuning would leak information into the choice of hyperparameters, making the final accuracy optimistic. Cross-validation splits only the training data into k folds, trains on k−1 of them, validates on the held-out fold, and repeats so each sample is validated exactly once; the mean over folds scores the configuration. The test set is used exactly once, at the very end.
On the random forest with n_iter=50, scoring='accuracy', cv=5 and random_state=42, the search sampled 50 configurations over 4 dimensions (n_estimators, max_depth, min_samples_split, min_samples_leaf). Best: {'max_depth': 4, 'min_samples_leaf': 3, 'min_samples_split': 2, 'n_estimators': 124}, cross-validation accuracy 0.9583, test accuracy 1.0. A real reproduction returns exactly these values; 18 of the 50 samples already scored ≥ 0.95 in CV.