The deck opens with a single sentence on an empty slide: without clean data, the results of a data mining analysis are in question.
The data preparation phase (also called data pre-processing) covers all activities to construct the dataset fed into the modeling tools from the initial data. It plays a key role in a data analytics process and avoids "garbage in, garbage out".
The activities involved are a broad range, from correcting errors to selecting the most relevant features. The kinds of error named:
Income: -100Exam mark: 15 and Exam result: PassedAnd a warning that returns at the end of the next chapter: data scientists cannot easily foresee the impact of pipeline prototypes — there are no pre-defined rules on the impact of pre-processing transformations. You must also outline how each quality problem reported in the earlier "Verify Data Quality" step has been addressed.
Data pre-processing includes (Shearer 2000) data:
This chapter covers selection and cleansing; Chapter 6 covers construction, integration and formatting.
Deciding on the data that will be used for the analysis is based on several criteria:
Part of the data selection process should involve explaining why certain data was included or excluded. It is also a good idea to decide if one or more attributes are more important than others.
| Example | Decision |
|---|---|
If decisions are based on the geographical region, individuals' addresses can be dropped to reduce the amount of data (Address: Via dell'Università 50 · Country: Italy · Sales: 1000) | Keep Country, drop Address |
To learn how sales are characterized by store Type, you do not need to consider the StoreId (S1 grocery 1000 · S2 supermarket 1500) | Keep Type, drop StoreId |
The worked exercise of the section: the flight delays dataset contains information about flights in the United States. The task is to predict flight delays (ARR_DELAY, arrival delay) based on 31 features. The first thirteen, as listed:
| # | Feature | # | Feature |
|---|---|---|---|
| 1 | FL_DATE — date of the flight | 8 | ORIGIN_CITY — city of origin airport |
| 2 | AIRLINE — name of the airline | 9 | DEST — destination airport code |
| 3 | AIRLINE_DOT — DOT identifier for the airline | 10 | DEST_CITY — city of destination airport |
| 4 | AIRLINE_CODE — code assigned to the airline | 11 | DEP_TIME — actual departure time |
| 5 | DOT_CODE — DOT identifier | 12 | DEP_DELAY — departure delay |
| 6 | FL_NUMBER — flight number | 13 | ARR_TIME — actual arrival time |
| 7 | ORIGIN — origin airport code | … | … |
What features would you select or drop to predict flight delays? The question is left open on the slide, and it is a good exam question. Notice the material available for an answer: four different ways of naming the same airline (AIRLINE, AIRLINE_DOT, AIRLINE_CODE, DOT_CODE) — redundancy, in the vocabulary of the correlation filter in Chapter 6; a pure identifier (FL_NUMBER) with no predictive content; and a feature, ARR_TIME, that is only known after the outcome you are trying to predict.
Selection is not only a technical decision. The criteria explicitly include GDPR and other technical constraints, and the deck immediately follows the selection slides with the list of sensitive personal data. Dropping the address in favour of the country is presented as a way to reduce the amount of data; it is also a way to reduce the amount of personal data you are holding.
The following personal data is considered sensitive and is subject to specific processing conditions:
The AI Act is a European Union regulation concerning artificial intelligence that classifies applications by their risk of causing harm.
| Risk class | Regime | Examples from the deck |
|---|---|---|
| Unacceptable risk | Banned | Applications that manipulate human behaviour; real-time remote biometric identification in public spaces; social scoring (ranking individuals based on their personal characteristics, socio-economic status, or behaviour) |
| High risk | Must comply with security, transparency and quality obligations and undergo conformity assessments, evaluated both before being placed on the market and throughout their life cycle | AI applications expected to pose significant threats to health, safety, or the fundamental rights of persons |
| Limited risk | Only transparency obligations: ensure users are informed that they are interacting with an AI system and are allowed to make informed choices | Applications that make it possible to generate or manipulate images, sound, or videos |
| Minimal risk | Not regulated | AI systems used for video games or spam filters |
The deck pairs the high-risk categories with Black Mirror episodes, as a way of memorising them:
Know the four risk classes in order and at least one example of each, and know that social scoring and real-time remote biometric identification in public spaces sit in the banned class, not in the high-risk one. Connect this back to Chapter 3: a personal gazetteer built from anonymous trajectories is a geographical fingerprint, and Chapter 4 gave you the fairness cases — the legal layer is the third side of the same triangle.
The deck poses four situations and then answers them:
| Situation | Proposed treatment |
|---|---|
| A retail company tracks daily sales, but some records are missing due to system failures. | Use historical sales trends to impute missing values. |
| A telecom company is predicting customer churn, but some customers have missing contract durations or monthly bill values. | Use median imputation for numerical features (e.g. replace missing monthly bill amounts with the median). |
| A hospital maintains records of patients' blood pressure, but 15% of entries are missing. | Use K-Nearest Neighbors (KNN) imputation to estimate missing values based on similar patients. |
| A bank evaluates loan applications, but some applicants have missing income data. | Use group-based imputation (e.g. average income for self-employed individuals). |
Four different answers to the same word, missing. The choice depends on the structure you can exploit: time (historical trends), the distribution (median), similarity between rows (KNN), or a grouping variable (self-employed vs employee). There is no default; there is a question — what do I know about why this value is absent?
Imputation is the process of replacing missing data with substituted values. The catalogue:
Listwise deletion (complete case) deletes data with missing values.
The worked example, before and after: three rows, one missing type, one missing sales; only S3 grocery 100 survives.
Pairwise deletion deletes data when it is missing a variable required for a particular analysis, but includes that data in analyses for which all required variables are present.
Less wasteful than listwise deletion, at the cost of every statistic being computed on a slightly different subset of rows.
Hot-deck imputation: the information donors come from the same dataset as the recipients.
One form of hot-deck imputation is called "last observation carried forward":
Worked example, sorted by StoreId and Date: S1 2024-10-05 inherits 1000 from S1 2024-10-04, and S2 2024-01-04 inherits 1000 as well — from the previous row, which belongs to a different store. Read that twice: the sort order decides who donates.
Cold-deck imputation replaces missing values with values from similar data in different datasets.
The donor is external, which makes the assumption explicit — and auditable: you can say exactly which other dataset your numbers came from.
Mean substitution replaces missing values with the mean of that variable for all other cases.
Worked example, averaging by StoreId: the missing S1 2024-10-05 becomes 1500, the average of 1000 and 2000; the missing S2 2024-10-04 becomes 1000.
Apply each strategy to the same small table and watch both the imputed values and the damage to the statistics.
The question posed by the case study is how do we impute?, and the table it is asked about is a portfolio with three missing years:
| Year | Portfolio Value | Year | Portfolio Value |
|---|---|---|---|
| 2008 | 1000.00 | 2017 | 1551.33 |
| 2009 | 1050.00 | 2018 | missing |
| 2010 | 1102.50 | 2019 | 1710.34 |
| 2011 | 1157.63 | 2020 | 1795.86 |
| 2012 | missing | 2021 | 1885.65 |
| 2013 | 1276.28 | 2022 | 1979.93 |
| 2014 | 1340.10 | 2023 | missing |
| 2015 | 1407.10 | 2024 | 2182.87 |
| 2016 | 1477.46 |
A portfolio has an initial value V0 = 1000 € and each year a return of X%. The first year it increases to V1 = 1000 + (1000 × X%) = 1050 €; the second year to V2 = V1 + (V1 × X%) = 1102.50 €; and so on, reaching 2406.62 € at year 18.
This is not a linear increase but a geometric sequence:
Final value = Initial value x (1 + r/n)^tHence, with annual compounding over 18 years:
Final value = Initial value x (1 + X)^18
X = (Final value / Initial value)^(1/18) - 1 = 0.05 = 5%In this case, the return is equal every year, so the average interest is 5%.
Now let the returns vary:
| Year | Value | Return |
|---|---|---|
| 0 | 1000.00 € | – |
| 1 | 1050.00 € | 5% |
| 2 | 997.50 € | -5% |
| 3 | 1047.38 € | 5% |
| 4 | 995.01 € | -5% |
| 5 | 995.01 € | 0% |
What is the average return? Applying the arithmetic mean gives (5 - 5 + 5 - 5 + 0)/5 = 0. However, this is wrong: a portfolio with a 0% return every year would still be worth 1000.00 € at year 5, and this one is worth 995.01 €.
Using the geometric formula, X = (Final/Initial)1/5 − 1, the average return is −0.1% — which reproduces the table exactly (999.00, 998.00, …, 995.01).
X% is the Compound Annual Growth Rate (CAGR): the mean annualized growth rate for compounding values over a given time period. CAGR smoothes the effect of the volatility of periodic values that can render arithmetic means less meaningful.
The case study exists to answer how do we impute the missing 2012?. Since the series is geometric at 5%, carrying 2011 forward (1157.63) is wrong, and so is the arithmetic mean of the neighbours, (1157.63 + 1276.28)/2 = 1216.96. The right imputation follows the semantics: 1157.63 × 1.05 = 1215.51, which is also the geometric mean of the two neighbours, and which multiplied by 1.05 gives back 1276.28 — the value actually recorded for 2013. The arithmetic mean is off by about 1.45 €, and the error compounds.
The case study closes with its own moral: pay attention to the semantics of the features.
Four questions open the section, and all four have the same shape:
Outlier removal is the process of eliminating data points that deviate significantly from the rest of the dataset. An outlier is a data point that differs significantly from other observations — e.g. a measurement error or a heavy-tailed distribution.
In the case of normally distributed data:
Other methods flag outliers based on measures such as the interquartile range. If Q1 and Q3 are the lower and upper quartiles, an outlier is any observation outside the range
[ Q1 - k(Q3 - Q1) , Q3 + k(Q3 - Q1) ] for some nonnegative k
k = 1.5 -> "outlier"
k = 3 -> data that is "far out"The same small sample judged by both rules. Move the multipliers and watch the two criteria disagree.
Isolation Forest (Liu, Ting, and Zhou 2008) is an algorithm for data anomaly detection using binary trees.
That is the whole idea, and it is a beautiful inversion: instead of modelling what normal looks like and measuring distance from it, you measure how hard a point is to separate from everything else with random axis-parallel cuts. An inlier sits in a crowd and needs many cuts; an outlier stands alone and falls out after two or three.
Choose which point to isolate and add random cuts one at a time. Count them.
Juvenal (55-128, Roman poet) wrote in his Satire VI of events being "a bird as rare upon the earth as a black swan". When the phrase was coined, the black swan was presumed by the Romans not to exist: all swans are white, because all records reported that swans had white feathers.
In 1697, Dutch explorers became the first Europeans to see black swans in Australia. Observing a single black swan is the undoing of the logic of any system of thought, as well as of any reasoning that followed from that underlying logic: conclusions are potentially undone once any of its fundamental postulates is disproved.
The black swan theory was developed by Nassim Nicholas Taleb (2008) to explain:
Such extreme events — outliers — collectively play vastly larger roles than regular occurrences.
LTCM was a highly leveraged hedge fund. Members of its board of directors included Myron Scholes and Robert C. Merton, who shared the Nobel Prize in Economics. It was initially successful, with annualized returns of around 21% in its first year, 43% in its second and 41% in its third. In 1998 it lost $4.6 billion in less than four months, due to an unlikely combination of the 1997 Asian and 1998 Russian financial crises.
Quoting Jorion (2000): on 21 August the portfolio lost $550 million; by 31 August the portfolio had lost $1,710 million in 1 month.
Surely this assumption was wrong.
This case study is the counterweight to the whole outlier-removal section, and the examiner may well set the two against each other. The three sigma rule and the IQR fences let you remove extreme points; the black swan argument says extreme events collectively play vastly larger roles than regular occurrences. The reconciliation is in the definition given earlier: an outlier can be a measurement error — remove it — or the tail of a heavy-tailed distribution — in which case removing it is deleting the only evidence you have that your normality assumption is false. The arithmetic makes the point unanswerable: 1710/206 = 8.3 sigma, an event a normal distribution allows once every 800 trillion years, happened.
It covers all activities to construct the dataset fed into the modeling tools from the initial data, avoiding "garbage in, garbage out". The five activities (Shearer 2000) are data selection, cleansing, construction, integration and formatting. Typical errors handled: out-of-range values (Income: -100), impossible combinations (Exam mark: 15 with Exam result: Passed), missing values.
Relevance to the data mining goals, quality, and technical constraints such as GDPR or limits on data volume or data types. You must explain why certain data was included or excluded, and it is a good idea to decide whether some attributes are more important than others. Example: if decisions are based on geographical region, drop the address and keep the country; to study sales by store Type you do not need the StoreId.
Data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs; trade-union membership; genetic data and biometric data processed solely to identify a human being; health-related data; and data concerning a person's sex life or sexual orientation. All are subject to specific processing conditions.
Unacceptable risk — banned: manipulation of human behaviour, real-time remote biometric identification in public spaces, social scoring. High risk: applications posing significant threats to health, safety or fundamental rights; they must comply with security, transparency and quality obligations and undergo conformity assessments before market entry and throughout their life cycle. Limited risk: systems that generate or manipulate images, sound or video — transparency obligations only, so users know they are interacting with an AI. Minimal risk — not regulated: video games, spam filters.
Listwise deletion (complete case) removes any row with a missing value: if data are missing at random it adds no bias but decreases the sample size; otherwise it introduces bias, because the remaining data are not representative. Pairwise deletion removes data only from the analyses that require the missing variable, and keeps it in analyses where all required variables are present.
In hot-deck imputation the information donors come from the same dataset as the recipients. The most common form is last observation carried forward: sort the dataset by any number of variables to create an ordered dataset, then impute each missing value with the value immediately before it. Cold-deck imputation, by contrast, takes the values from similar data in a different dataset.
Because mean imputation attenuates any correlations involving the imputed variable: the imputed values have no relationship with any other measured variable, so the column is quietly flattened towards its own average and the associations you are trying to discover are weakened. It can be carried out within classes (e.g. by gender, or the average income of self-employed applicants), which restores some of the structure.
Because a portfolio compounds: it follows a geometric sequence, Final = Initial x (1 + r/n)^t. With returns of +5%, −5%, +5%, −5%, 0% the arithmetic mean is 0, but the portfolio has fallen from 1000.00 to 995.01. The correct figure is the CAGR, X = (Final/Initial)^(1/t) − 1 = −0.1%, which reproduces the observed series. CAGR smoothes the effect of the volatility of periodic values that can render arithmetic means less meaningful. Takeaway of the case study: pay attention to the semantics of the features.
For normally distributed data, nearly all values — 99.7% — lie within three standard deviations of the mean; roughly 1 in 22 observations differ by twice the standard deviation or more; and 1 in 370 deviate by three times the standard deviation. With a sample size of only 100, three such outliers are already a reason for concern.
An outlier is any observation outside [Q1 − k(Q3 − Q1), Q3 + k(Q3 − Q1)] for some nonnegative k. k = 1.5 indicates an "outlier"; k = 3 indicates data that is "far out".
It detects anomalies using binary trees. Because anomalies are few and different from the other data, they can be isolated using a few partitions. Unlike decision tree algorithms, it uses only path length — the number of random splits needed to isolate a point — to output an anomaly score.
LTCM was a highly leveraged hedge fund whose board included Nobel laureates Myron Scholes and Robert C. Merton, with annualized returns around 21%, 43% and 41% in its first three years. In 1998 it lost $4.6 billion in less than four months because of the 1997 Asian and 1998 Russian crises. Per Jorion (2000), the portfolio lost $550 million on 21 August and $1,710 million in one month by 31 August: against a presumed monthly standard deviation of $206 million this is an 8.3 sigma event, which under a normal distribution would occur once every 800 trillion years — 40,000 times the age of the universe. Surely the normality assumption was wrong.
Taleb (2008) uses it to explain the disproportionate role of hard-to-predict rare events, the non-computability of their probability by scientific methods, and the psychological biases that blind people to uncertainty — with extreme events collectively playing vastly larger roles than regular occurrences. It complicates outlier removal because an outlier may be a measurement error (remove it) or the tail of a heavy-tailed distribution (the most informative point you have). Observing a single black swan undoes the logic of the whole system of thought that assumed white feathers.