Modeling and Evaluation

1. The modeling phase of CRISP-DM

After business understanding, data understanding and data preparation, the CRISP-DM cycle reaches its center: Modeling. This chapter and the next three follow the deck 6 - Modeling, which is where the course finally trains models — and, as the deck title announces, evaluates them. The whole deck is hands-on: it uses scikit-learn on the Iris dataset, one of the classical benchmarks of machine learning.

2. What machine learning means: two definitions

The deck opens the phase with the two classical definitions of machine learning, and both are worth memorizing verbatim:

Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed.

Arthur Samuel, 1959

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

Tom Mitchell, 1997

In the deck, this definition is followed by a single word that will become the red thread of the whole lecture: under and overfitting.

3. Under- and overfitting, from the start

Before writing a single line of code, the deck shows two xkcd strips that frame the two failure modes of every model:

• Underfitting (xkcd 605, Extrapolating): the model is too simple to capture the structure of the data — it misses the pattern entirely and keeps missing it.
• Overfitting (xkcd 2048, Curve-fitting): the model is complex enough to fit every training point — including the noise — and then produces absurd shapes between the points, fitting the past perfectly and the future terribly.

The pedagogical point is deliberately visual: a model can be wrong in two opposite directions, and both are diagnosed by looking at how the model behaves on data it has not seen during training. Chapter 9 will make the same idea quantitative with learning curves; Chapter 10 will add the cross-validation machinery to measure it properly.

4. The disclaimer: understand why the model behaves

This is why the chapters that follow spend as much time on reading models (trees, boundaries, learning curves) as on training them. The exam in this course is oral, and the questions are on all theoretical and practical aspects: being able to narrate why a model draws a given boundary is the skill being tested.

5. Types of machine learning

There are many types of machine learning algorithms, and the deck classifies them in broad categories according to three criteria:

Keep this taxonomy in mind: Chapter 11 returns to the instance-based vs model-based contrast with k-NN on one side and the perceptron on the other.

6. scikit-learn: the workbench

All the hands-on work of this lecture happens in scikit-learn, the Python machine learning library:

• it is built upon NumPy, SciPy and Matplotlib;
• it is open source and commercially usable;
• it covers many algorithms: supervised learning (linear regression, support vector machine, …), unsupervised learning (clustering, factor analysis, PCA, neural networks, …), validation (checking the accuracy of supervised models on unseen data), and feature extraction (extracting features to define attributes in image and text data).

import sklearn as sk
print(sk.__version__)

The version printed in the deck is 1.6.1.

7. The Estimator API

Every scikit-learn algorithm obeys one consistent interface, the Estimator:

• An algorithm that learns from the data (fitting the data) is an estimator — usable with any algorithm, classification, regression or clustering.
• All the parameters can be set when creating the estimator: estimator = Estimator(param1=1, param2=2); parameters are readable as attributes, estimator.param1.
• All estimator objects expose a .fit() method that performs the training: estimator.fit(X_train, y_train).
• Once fitted, all the estimated (learned) parameters become attributes ending with an underscore: estimator.estimated_param_.
• Finally, you can .predict() unseen data: y_pred = estimator.predict(X_test).

The underscore convention is the single most useful detail of the API: you set the hyperparameters before fitting, the model learns the parameters during fitting, and the trailing underscore is scikit-learn's way of telling the two apart.

Widget — annotated code: the Estimator lifecycle

8. scikit-learn in action

The deck walks the whole pipeline in five lines — choose a model, choose its hyperparameters, fit, predict, evaluate. This is the canonical scikit-learn usage pattern, and every chapter from here on is a variation of it:

Widget — annotated code: the five-step pipeline

9. Supervised learning: classification and regression

The course focuses on supervised learning tasks: the training includes the desired solutions, i.e. the labels. Within supervised learning, two tasks:

Everything trained in the next chapters is a classifier — Iris has three classes of flowers — but the same Estimator API works for regression with minimal changes.

10. The data: feature matrix and target array

For a supervised learning problem we need input data along with labels, and we must split the data between test and training set. How? That is the question the deck keeps open and answers in section 13.

scikit-learn uses data in the form of an N-dimensional matrix, split into two objects:

• Data as a feature matrix X (e.g. a Pandas DataFrame): the samples are the individual objects described by the dataset (e.g. a person); the features describe each sample in a quantitative manner (e.g. age and height).
• Data as target array y (e.g. a Pandas Series): along with the features matrix, we also have the target array (the label).

And then the deck asks the crucial question: how do we distinguish target and feature columns? The answer is conceptual, not technical: the target is the column we want to predict — everything else, the columns we are allowed to observe, becomes the features. The choice of what to predict is a modeling decision, made before any code runs.

11. Built-in datasets

scikit-learn ships toy datasets that can be loaded with one call. The deck lists the load_* functions:

import sklearn.datasets as datasets

for dataset in [func for func in dir(datasets) if 'load_' in func]:
    print(dataset)

load_breast_cancer
load_diabetes
load_digits
load_files
load_iris
load_linnerud
load_sample_image
load_sample_images
load_svmlight_file
load_svmlight_files
load_wine

Three of these will appear in this course: load_iris here, and load_wine in the final exercise of the lecture (Chapter 11).

12. The Iris dataset, loaded and profiled

The Iris dataset is the classic benchmark: 150 flowers, 4 measurements, 3 species (setosa, versicolor, virginica). The deck loads it and builds a DataFrame:

from sklearn.datasets import load_iris
import pandas as pd

iris = load_iris()                                        # Load the iris dataset
df = pd.DataFrame(data=iris.data, columns=iris.feature_names)  # DataFrame with the iris data
df['species'] = iris.target                               # Add the species column
# df['species'] = df['species'].map({0: 'setosa', 1: 'versicolor', 2: 'virginica'})

Then the deck profiles it — the df.info() output is itself a first data-understanding step:

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150 entries, 0 to 149
Data columns (total 5 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   sepal length (cm)  150 non-null    float64
 1   sepal width (cm)   150 non-null    float64
 2   petal length (cm)  150 non-null    float64
 3   petal width (cm)   150 non-null    float64
 4   species            150 non-null    int64
dtypes: float64(4), int64(1)
memory usage: 6.0 KB

150 rows, 5 columns, no missing values, 4 numeric features and 1 integer label — a clean dataset that lets the lecture focus on modeling rather than cleaning.

Widget — the Iris explorer

Pick a species and read its mean profile. Notice that the two petal features separate the species almost perfectly, while the sepal features overlap — this is exactly what the tree in Chapter 9 will exploit.

13. Splitting into training and test sets

Back to the open question: how do we split? The answer is train_test_split, with a test size and a random seed:

from sklearn.model_selection import train_test_split

X = df.drop("species", axis=1)          # features matrix
y = df["species"]                       # target array
seed = 42                               # Setup random seed. Why?
test_size = 0.2
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=seed)
print(f"X_train: {X_train.shape}")
print(f"y_train: {y_train.shape}")
print(f"X_test:  {X_test.shape}")
print(f"y_test:  {y_test.shape}")

X_train: (120, 4)
y_train: (120,)
X_test:  (30, 4)
y_test:  (30,)

The comment in the code — "Setup random seed. Why?" — is the topic of the next two sections, and the deck will answer it thoroughly. train_test_split randomly shuffles the data before the split is implemented; the random_state parameter controls that shuffle.

Widget — size the split

Move the test size and read the resulting shapes, exactly as the deck prints them.

14. Randomness, seeds and reproducibility

Why does the code set a seed at all? Because train_test_split shuffles randomly, and randomness is the lack of definite pattern or predictability in information. The deck makes the distinction precise:

• Individual random events are, by definition, unpredictable — e.g. the roll of a dice;
• … but if there is a known probability distribution, the frequency of outcomes over repeated trials is predictable.

The deck's example: when throwing two dice, the outcome of any single roll is unpredictable, but a sum of 7 will tend to occur twice as often as 4. The simulation code rolls a die 100,000 times and two dice 100,000 times, plotting both histograms: the single die is flat, the sum of two dice is triangular, peaking at 7.

Pseudorandom number generators

A Pseudorandom Number Generator (PRNG) is an algorithm for generating a sequence of numbers whose properties approximate the properties of sequences of random numbers. Two key properties:

• The PRNG-generated sequence is not truly random: it is completely determined by an initial value, called seed.
• Pseudorandom number generators are important in practice for their reproducibility.

print(np.random.randint(1, 7, size=10))
print(np.random.randint(1, 7, size=10))
np.random.seed(42)
print(np.random.randint(1, 7, size=10))
np.random.seed(42)
print(np.random.randint(1, 7, size=10))
np.random.seed(42)
print(np.random.randint(1, 7, size=10))
print(np.random.randint(1, 7, size=10))

[6 6 2 5 4 1 5 1 5 2]
[2 3 1 2 2 6 3 6 5 3]
[4 5 3 5 5 2 3 3 3 5]
[4 5 3 5 5 2 3 3 3 5]
[4 5 3 5 5 2 3 3 3 5]
[4 3 6 5 2 4 6 6 2 4]

Read the output carefully: the two unseeded rolls differ; the three seeded rolls are identical; and the roll after the third seeding differs again, because the generator advanced past the seeded point.

Widget — the dice and the seed

Simulate a seeded PRNG. Roll without a seed and the sequence changes; roll with seed 42 twice and the sequence is identical — exactly the reproducibility the deck demonstrates.

15. Why the random_state matters

Back to train_test_split(..., random_state=seed): the parameter controls the randomness of the shuffle, and that is essential for three reasons:

1. Reproducibility — by using the same random_state, you can ensure that others can replicate your results exactly.
2. Consistency in model evaluation — when comparing different models or tuning hyperparameters, the training/test split cannot change: otherwise a difference in accuracy could be caused by a different split, not by the model.
3. Debugging and testing — during development you might need to debug your code or test different configurations on a stable split.

And the famous question: why 42? The deck answers with the classic reference: The Hitchhiker's Guide to the Galaxy — 42 is "the answer to the ultimate question of life, the universe, and everything". In code, any integer would work; 42 is the community's joke, and a widely shared convention that makes examples comparable.

Check your understanding