With the workflow of Chapter 8 in place, the deck turns to the models: "Let's train some machine learning models." Three models are covered in this lecture:
It is important to understand the model dynamics — not only the final result. Actually, it is mandatory for the exam. This is why every model here is studied through its decision boundaries: the picture of where the model draws its regions shows how it behaves, not just how well it scores.
The first model is a decision tree with max_depth=2 — the same five-step pattern of Chapter 8:
from sklearn.tree import DecisionTreeClassifier # Import the model
from sklearn.metrics import accuracy_score
clf = DecisionTreeClassifier(max_depth=2, random_state=seed) # Instantiate the model
clf.fit(X_train, y_train) # Train the model
y_pred = clf.predict(X_test) # Predict new values
accuracy_score(y_test, y_pred) # Evaluate the model (on the test set)0.966666666666666729 of the 30 test flowers are classified correctly — and the one that is not is a versicolor that the tree confuses with virginica, as the boundary plots later show. The deck now asks the question that matters: what does this tree actually look like?
Two tools open the black box:
from sklearn.tree import plot_tree
plt.figure(figsize=(4, 3))
plot_tree(clf, feature_names=df.columns, class_names=['setosa', 'versicolor', 'virginica'], filled=True);The depth-2 tree found on Iris has exactly two splits, and both use the same feature:
Then the deck checks feature relevance with feature_importances_:
feature_importance_df = pd.DataFrame({'Feature': X.columns, 'Importance': clf.feature_importances_})
feature_importance_df = feature_importance_df.sort_values(['Importance', 'Feature'], ascending=[False, True])
feature_importance_df| Feature | Importance | |
|---|---|---|
| 2 | petal length (cm) | 1.0 |
| 3 | petal width (cm) | 0.0 |
| 0 | sepal length (cm) | 0.0 |
| 1 | sepal width (cm) | 0.0 |
All the importance is on petal length. This is the direct consequence of the two splits in Plate 9.1: a depth-2 tree has exactly two decisions, and both were made on the same feature.
Click each feature to see why it scored what it scored.
The deck now plots the data twice — the 2 most important features and the 2 least important features:
plot(df, clf, feature_importance_df['Feature'].iloc[:2].tolist()) # plot the 2 most important features
plot(df, clf, feature_importance_df['Feature'].iloc[2:].tolist()) # plot the 2 least important featuresThe deck now checks whether the features are redundant, using the Pearson correlation of the training set:
X_train.corr(method='pearson', numeric_only=True)| sepal length | sepal width | petal length | petal width | |
|---|---|---|---|---|
| sepal length | 1.000000 | -0.106926 | 0.862175 | 0.801480 |
| sepal width | -0.106926 | 1.000000 | -0.432089 | -0.369509 |
| petal length | 0.862175 | -0.432089 | 1.000000 | 0.962577 |
| petal width | 0.801480 | -0.369509 | 0.962577 | 1.000000 |
Two pairs stand out: petal length vs petal width at 0.96 and sepal length vs petal length at 0.86 — the petal features are strongly correlated with each other (and with sepal length). Highly correlated features carry largely the same information, so a model can often drop some of them with little loss. This is a feature selection argument: the tree already made the same discovery by ignoring the sepal features entirely.
Click a cell to read its meaning.
The tree has one obvious hyperparameter: max_depth. Before running anything, the deck stops and asks: what do you expect?
Expectation: one single split — a single straight, axis-aligned cut across the petal plane. It will isolate the easiest group (setosa) and lump the other two together, so the accuracy will be mediocre.
What happens: the split lands on petal length ≈ 2.45. Setosa is perfectly isolated; versicolor and virginica share the rest. This is underfitting by construction: the model is too shallow to see the versicolor/virginica boundary.
Expectation: three cuts (a depth-3 tree can use up to 7 regions). Now versicolor and virginica should get their own region — the boundary that depth 1 and 2 could not draw.
What happens: the test accuracy reaches 1.0. The tree separates the three species completely: with the petal features, three axis-aligned regions suffice. Deeper trees can only add fragments around single points.
Expectation: a much more fragmented boundary — up to 63 regions, each fit to tiny clusters of training points. The danger is overfitting: the boundary starts following individual points and noise.
What happens: the training accuracy is 1.0 and the test accuracy stays 1.0 — on this clean dataset the fragmentation is harmless. But on noisy data the same depth produces the wiggly, extrapolating shapes of the xkcd strip from Chapter 8.
The pedagogical trick: the deck asks for a prediction before showing the plots, because predicting the model's behavior is exactly the skill the exam tests. Try it on the widget in the next section: set the depth, guess the boundary, then reveal it.
The deck plots the decision boundaries for max_depth = 1, 2, 2 (seed 1), 3, 6 on the petal plane. The sequence tells the whole story of this chapter in pictures:
This widget trains a real greedy decision tree (the same CART logic as scikit-learn) on the actual 150 Iris flowers, projected on the petal plane — exactly the deck's plots. Move the depth slider and watch the boundary, the splits, and the training accuracy change.
Finally the deck makes the trade-off quantitative, plotting training and test accuracy for max_depth from 1 to 9:
max_depths = range(1, 10)
train_accuracies, test_accuracies = [], []
for max_depth in max_depths:
clf = DecisionTreeClassifier(max_depth=max_depth, random_state=seed)
clf.fit(X_train, y_train)
train_acc = accuracy_score(y_train, clf.predict(X_train))
test_acc = accuracy_score(y_test, clf.predict(X_test))
train_accuracies.append(train_acc)
test_accuracies.append(test_acc)
plt.plot(max_depths, train_accuracies, label="Training", marker='o')
plt.plot(max_depths, test_accuracies, label="Test", marker='o')
plt.xlabel("max_depth"); plt.ylabel("Accuracy"); plt.legend(); plt.grid(True)The values below are the actual numbers this code produces (seed 42, the deck's setup — the depth-2 test value reproduces the printed 0.9666666666666667 exactly):
| max_depth | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Training | 0.675 | 0.950 | 0.9583 | 0.975 | 0.9917 | 1.0 | 1.0 | 1.0 | 1.0 |
| Test | 0.6333 | 0.9667 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
The classic pattern, in miniature: at depth 1 the tree underfits (both curves low — the model cannot even separate setosa from the rest cleanly); around depth 2-3 both curves are high; from depth 6 the training curve reaches 1.0 while the test curve stops improving. On this clean dataset the plateau is flat, but the shape is the warning from Chapter 8: once the training curve keeps climbing while the test curve stalls, additional capacity is buying memory, not generalization.
The same curves, computed from the real runs. Hover the markers to read the exact values.
Decision tree, random forest, k-NN. The rule: it is important to understand the model dynamics, not only the final result — actually, it is mandatory for the exam. This is why the deck studies each model through its decision boundaries.
from sklearn.tree import DecisionTreeClassifier; clf = DecisionTreeClassifier(max_depth=2, random_state=seed); clf.fit(X_train, y_train); y_pred = clf.predict(X_test); accuracy_score(y_test, y_pred) → 0.9666666666666667 (29 of 30 test flowers correct).
The tree makes exactly two splits, both on petal length: first at 2.45 (True → setosa leaf with 50 samples), then at 4.75 on the remaining 70 samples (True → versicolor, False → virginica). Consequently the feature importances are petal length 1.0 and everything else 0.0: a depth-2 tree has only two decisions and both used the same feature.
Plotting the 2 most important features (petal length × petal width) shows the three species as three well-separated clouds — a tree can separate them with axis-aligned cuts. Plotting the 2 least important (sepal length × sepal width) shows heavy overlap — no simple rule separates them. The tree discovered this on its own by never using the sepal features.
The strong positive pairs are petal length vs petal width (0.9626) and sepal length vs petal length (0.8622); sepal width is almost uncorrelated with the others (e.g. -0.1069 with sepal length). Strongly correlated features carry largely the same information, so some can be dropped — a feature-selection argument that matches the tree's behavior.
Depth 1: one straight cut at petal length ≈ 2.45, isolating setosa. Depth 2: a second cut at ≈ 4.75 separating versicolor from virginica (test accuracy 0.9667). Depth 2 with seed 1: a different boundary for the same depth — tree learning has randomness and the seed controls it. Depth 3: test accuracy 1.0. Depth 6: many small rectangular regions hugging individual points — the fragmentation that on noisy data becomes overfitting.
Training accuracy rises from 0.675 (depth 1) to 1.0 (depth 6); test accuracy rises from 0.6333 to 1.0 by depth 3 and then plateaus. The pattern: depth 1 underfits (both curves low); around depth 2-3 both are high; beyond, training keeps climbing while test stalls — the signature of capacity buying memory, not generalization. On this clean dataset the plateau is flat; on noisy data it turns downwards (the overfitting shape).