Three situations open the section, all asking the same thing — is the dataset ready for machine learning?
Feature engineering refers to the manipulation — addition, deletion, combination, mutation — of your data set to improve machine learning model training. Derived attributes should be added if they ease the modeling algorithm:
Area = Length x Width
Loyalty_Score = Total_Purchases x 0.4 + Avg_Order_Value x 0.3 + Frequency x 0.3
Ocean_Proximity = distance((Ocean_Latitude, Ocean_Longitude),
(District_Latitude, District_Longitude))Encoding may be necessary to transform symbolic fields ("definitely yes", "yes", "don't know", "no") to numeric values.
You have already seen a whole chapter of feature engineering without the name: Chapter 3. NP is a derived attribute built from raw power readings; IF = NP/FTP is a normalisation; TSB = CTL − ATL is a combination of two aggregations. The three lines above are the same operation applied to a table — and the Ocean_Proximity example is exactly the housing lab of the course.
Encoding is the process of converting categorical variables into numeric features.
Categorical features can be nominal or ordinal:
| Kind | Definition | Example |
|---|---|---|
| Nominal | Do not have a defined ranking or inherent order | Colors |
| Ordinal | Have an inherent order or ranking | Size |
One-hot encoding and ordinal encoding are the most common methods to transform categorical variables into numerical features.
Ordinal encoding replaces each category with an integer value. These numbers are, in general, assigned arbitrarily. Ordinal encoding is a preferred option when the categorical variable has an inherent order.
| Before encoding | After encoding (small = 0, medium = 1, large = 2) | |||
|---|---|---|---|---|
| ProductId | Size | ProductId | Size | Size_Enc |
| P1 | small | P1 | small | 0 |
| P2 | medium | P2 | medium | 1 |
| P3 | large | P3 | large | 2 |
| P4 | small | P4 | small | 0 |
One-hot encoding (OHE) replaces categorical variables by a set of binary variables, each representing a category in the variable. The binary variable takes the value 1 if the observation shows the category, and 0 otherwise. One-hot encoding treats each category independently.
| Before encoding | After encoding | |||||
|---|---|---|---|---|---|---|
| ProductId | Color | ProductId | Color | red | green | blue |
| P1 | red | P1 | red | 1 | 0 | 0 |
| P2 | green | P2 | green | 0 | 1 | 0 |
| P3 | blue | P3 | blue | 0 | 0 | 1 |
| P4 | red | P4 | red | 1 | 0 | 0 |
OHE increases the dimensionality of the dataset, and it may not be suitable for encoding high cardinality features. To prevent a massive increase in the feature space, we can one-hot encode only the most frequent categories in the variable; less frequent values are then treated collectively and represented as 0s in all the binary variables.
The Likert scale is widely used in social work research and is commonly constructed with four to seven points — [*, **, ***, ****, *****] or [1, 2, 3, 4, 5]. And then the trap: what about averaging?
It is usually treated as an interval scale, but strictly speaking it is an ordinal scale, where arithmetic operations cannot be conducted (Wu and Leung 2017). The passage quoted on the slide is worth reading twice:
Converting responses to a Likert-type question into an average seems an obvious and intuitive step, but it doesn't necessarily constitute good methodology. One important point is that respondents are often reluctant to express a strong opinion and may distort the results by gravitating to the neutral midpoint response. It also assumes that the emotional distance between mild agreement or disagreement and strong agreement or disagreement is the same, which isn't necessarily the case. At its most fundamental level, the problem is that the numbers in a Likert scale are not numbers as such, but a means of ranking responses.
And the empirical companion: the J-shaped distribution of product reviews (Hu, Zhang, and Pavlou 2009). People tend to write reviews only when they are either extremely satisfied or extremely unsatisfied; people who feel the product is average might not bother to write a review. So the sample is not merely ordinal — it is also selected, which is the self-selection face of the biases in Chapter 4.
The Y2K22 bug: encoding the date 2022-01-01T00:01 into a signed integer 2201010001.
A signed integer is a 32-bit datum that represents an integer in the range:
[ -2^31 , 2^31 - 1 ] = [ -2147483648 , 2147483647 ]
however 2201010001 > 2147483647The date, encoded that way, does not fit. The deck points to the year 2000 problem as the ancestor of the same mistake.
Encoding decisions have a range, and the range is part of the decision. The pattern YYMMDDHHmm was a perfectly reasonable encoding until the year rolled over to 22 and the concatenation crossed a boundary nobody had checked. Same lesson as the CAGR case study of Chapter 5, from the opposite direction: there, semantics were ignored; here, the physical type was.
If data values vary widely, objective functions will not work properly without normalization in some ML algorithms. For example, many classifiers calculate the distance between two points using the Euclidean distance:
d(p, q) = sqrt( (p1-q1)^2 + (p2-q2)^2 + ... + (pn-qn)^2 )
= sqrt( SUM over i of (pi - qi)^2 )If one of the features has a broad range of values, the distance will be governed by this particular feature. The worked example, with age ∈ [0, 120] and income ∈ [0, 100000]:
| Point | age | income | Distance from p1 |
|---|---|---|---|
| p1 | 50 | 10000 | — |
| p2 | 50 | 20000 | d(p1, p2) = 10000.00 |
| p3 | 60 | 10000 | d(p1, p3) = 10.00 |
| p4 | 60 | 20000 | d(p1, p4) = 10000.00 |
Read the last row: p4 differs from p1 in both features, and yet its distance is indistinguishable from p2, which differs only in income. Age has been erased by the units of income.
| Method | Formula | What it guarantees |
|---|---|---|
| Min-max normalization | x' = a + (x - min(x))(b - a) / (max(x) - min(x)) | Rescales the feature into [a, b], typically [0, 1] |
| Standardization | x' = (x - mean(x)) / sd(x) | Zero mean and unit variance for every feature |
| Robust scaling | x' = (x - Q2(x)) / (Q3(x) - Q1(x)) | Designed to be robust to outliers: built on the median and the IQR, not the mean and the range |
On Iris the deck runs the same demonstration in the other direction: it multiplies petal_length by 10 and adds a single outlier at [petal_length=100, petal_width=100], and shows what the transformed dataset does to the plot. One point, one unit change, and the picture is unreadable.
Know which algorithms care. Anything built on distances — k-NN (Chapter 8), k-means, SVM — and anything built on gradient descent over a shared learning rate (Chapter 7) is sensitive to scale, and so is PCA (which is why standardization is its first step). Decision trees, which cut one feature at a time, are not. In Chapter 8 you will see exactly this: the k-NN decision boundary with and without min-max normalization.
Long Tail refers to the concept where a large number of niche products collectively generate more sales than a few bestsellers. E-commerce sites such as Amazon stock a vast array of products that traditional retailers would not carry due to space constraints. The Long Tail phenomenon is directly related to skewed distributions, specifically a type of right-skewed distribution.
What happens to mean values? The deck compares two distributions:
| Gaussian distribution (heights) | Skewed distribution (wealth) | |
|---|---|---|
| Mean | 173 cm | 103 158 262 914 |
| Median | 173 cm | 36 666 524 821 |
In the Gaussian case the two coincide. In the skewed case the mean is nearly three times the median — and neither number describes a typical individual. Skewed distributions can be transformed using mathematical functions such as the logarithm.
Aggregation and binning may be necessary to transform ranges to symbolic fields. The first example bins sales every 1000 €:
| StoreId | Date | sales | sales_bin |
|---|---|---|---|
| S1 | 2024-10-04 | 1000 | [1000-2000) |
| S1 | 2024-10-05 | 1500 | [1000-2000) |
| S1 | 2024-10-06 | 2000 | [2000-3000) |
A meteorologist analyzes hourly temperature readings, but the data has fluctuations due to temporary weather conditions:
| Hour | Temperature (°C) |
|---|---|
| 1 | 24.1 |
| 2 | 24.3 |
| 3 | 23.8 |
| 4 | 24.5 |
| 5 | 22.9 (sudden drop due to rain) |
| 6 | 24.2 |
| 7 | 23.9 |
Using equal-width binning — grouping every 3 hours and averaging:
| Time period | Smoothed temperature (°C) |
|---|---|
| 1-3 AM | 24.0 (avg of 24.1, 24.3, 23.8) |
| 4-6 AM | 23.9 (avg of 24.5, 22.9, 24.2) |
| 7 AM | 23.9 |
Noise from sudden drops (e.g. 22.9°C at 5 AM) is smoothed, making temperature trends more reliable.
Aggregation computes new values by summarizing information from multiple records and/or tables — for example, converting a table of product purchases, with one record per purchase, into a new table with one record per store.
Correct (sum of sums): (1 + 2) + (3 + 4 + 5) = 1 + 2 + 3 + 4 + 5 = 15.
Wrong (average of averages): avg(avg(1, 2), avg(3, 4, 5)) = avg(1.5, 4) = 2.75, whereas avg(1, 2, 3, 4, 5) = 3.
Sum is distributive over grouping; average is not, because the groups have different sizes. This is the same class of error as averaging returns in Chapter 5.
Data binning is a pre-processing technique that reduces the effects of minor observation errors: the original values that fall into a given interval (bin) are replaced by a central value representative of that interval. Histograms are an example of data binning, used to observe underlying frequency distributions.
| Equal-width | Equal-frequency | |
|---|---|---|
| Rule | Divide the range of values into equal-sized intervals | Divide the values into bins that have the same number of observations |
| Example | Values from 0 to 100 into 10 bins → each bin has width 10 | 100 observations into 10 bins → each bin has 10 observations |
| Advantage | Bin boundaries are interpretable and evenly spaced | Creates balanced bins that handle skewed data and outliers better |
| Disadvantage | Can create empty or sparse bins, especially if the data is skewed or has outliers | Can distort the distribution of the data and create irregular bin widths |
A streaming platform wants to recommend movies based on user preferences. Each movie is represented by a vector of features: genre, director, lead actor, IMDB rating, budget, user reviews, box office revenue, soundtrack style… and many more — let us assume 100+ features per movie.
If movies had only 2 features (genre and IMDB rating) we could easily visualize clusters of similar movies. With 100+ features, the data points are spread out across a vast space:
Dimensionality reduction is the transformation of data from a high-dimensional space into a low-dimensional space.
The main approaches divide into feature selection and feature extraction.
Feature selection is the process of selecting a subset of relevant features (variables, predictors) for use in model construction.
The dummy algorithm: test each subset of features to find the one that minimizes the error. This is an exhaustive search of the space, and is computationally intractable for all but the smallest of feature sets. If S is a finite set of features with cardinality |S|, the number of all subsets is
|P(S)| = 2^|S| - 1 (the empty set is not considered)
3 features -> 2^3 = 8 subsets
4 features -> 2^4 = 16 subsets
10 features -> 2^10 = 1024 subsetsFeature selection approaches are therefore characterized by two things: a search technique for proposing new feature subsets, and an evaluation measure for scoring the different subsets.
Filter strategy: select variables regardless of the model, based only on general features like the correlation with the variable to predict.
Variance threshold. With mean = (1/n) SUM xi and Var(X) = (1/n) SUM (xi - mean)^2: features with low variance do not contribute much information to a model, so a variance threshold removes any feature with little to no variation in its values. Since variance can only be calculated on numeric values, this method only works on quantitative features.
| StoreId | sales | PostalCode |
|---|---|---|
| 1 | 1000 | 47522 |
| 2 | 1500 | 47522 |
| 3 | 1000 | 47522 |
VAR(StoreId) = 0.67 (?), VAR(sales) = 55555.56, VAR(PostalCode) = 0. Selecting VAR(X) > 0.6 keeps StoreId and sales. The question mark on the slide is the point of the example: StoreId passes the numeric test while being a meaningless identifier.
Pearson's correlation measures the linear relationship between 2 numeric variables: a coefficient close to 1 is a positive correlation, -1 negative, 0 none. Two uses:
Wrapper strategy: each new feature subset is used to train a model, which is tested on a hold-out set. Counting the number of mistakes made on that hold-out set — the error rate of the model — gives the score for that subset. As wrapper methods train a new model for each subset, they are very computationally intensive but provide good results.
Methods include forward selection, backward elimination, and exhaustive search. Stepwise regression adds the best feature (or deletes the worst feature) at each round.
Backward elimination: start with the full model including all features, then incrementally remove the most insignificant feature, repeating until the final set of significant features remains.
SL = 0.05, with 95% confidence).Embedded strategy: add or remove features while building the model, based on prediction errors. A learning algorithm takes advantage of its own variable selection process and performs feature selection and classification simultaneously.
The linear regression model is y_hat_i = b1x1 + b2x2 + ... + bpxp, and the goal is minimizing the sum of squared errors between predicted and actual values:
min ( SUM over i of (yi - y_hat_i)^2 )LASSO — Least Absolute Shrinkage and Selection Operator — adds a penalty proportional to the absolute values of the coefficients:
min ( SUM over i of (yi - y_hat_i)^2 + lambda * SUM over j of |bj| )bj are the coefficients of the model;lambda is the regularization parameter controlling the penalty strength.Lasso performs automatic feature selection: by shrinking some coefficients to 0, it removes irrelevant features. The optimal lambda can be determined with cross-validation techniques (Chapter 8). See also Katrutsa and Strijov (2017) and Chan et al. (2022) on multicollinearity.
The variance-threshold table from the deck, with the threshold under your control.
Feature projection (or feature extraction) transforms the data from the high-dimensional space to a space of fewer dimensions. The transformation may be linear, as in principal component analysis (PCA), but many nonlinear dimensionality reduction techniques also exist.
PCA is a linear dimensionality reduction technique:
Iris contains 4 features, so it cannot be plotted directly. Running PCA:
| Principal component | Explained variance |
|---|---|
| PC 1 | 92.46% |
| PC 2 | 5.31% |
| PC 3 | 1.71% |
Feature relevance for 3 components:
| Feature | PC 1 | PC 2 | PC 3 |
|---|---|---|---|
| Sepal Length (cm) | 0.361 | 0.657 | -0.582 |
| Sepal Width (cm) | -0.085 | 0.730 | 0.598 |
| Petal Length (cm) | 0.857 | -0.173 | 0.076 |
| Petal Width (cm) | 0.358 | -0.075 | 0.546 |
A hospital wants to analyze patient health records by integrating data from multiple sources: electronic health records (EHRs), wearable devices, and insurance claims. That is the integration problem.
Integration involves combining information from multiple tables or records to create new records or values. With table-based data, an analyst can join two or more tables that have different information about the same objects.
The worked example: a retail chain has one table with each store's general characteristics (floor space, type of mall), another with summarized sales data (profit, percent change in sales from the previous year), and another with the demographics of the surrounding area. These tables can be merged into a new table with one record for each store.
Data integration combines data residing in different sources and provides users with a unified view of them. Two strategies:
| Aspect | Semantic integration | Primary key-based integration |
|---|---|---|
| Approach | Based on meaning and understanding of the data | Based on matching unique keys |
| Suitability | Data with heterogeneous terminologies or structures | Datasets that have common, well-defined keys |
| Complexity | Complex to interpret and align meanings | Simpler, relies on exact key matches |
| Flexibility | Integrates data with different schemas/representations | Less flexible, requires shared primary key fields |
| Challenges | Requires mapping of concepts and domain semantics | Limited to datasets that share a key |
Primary key-based integration combines multiple sources by matching unique identifiers; it works when both datasets have a well-defined and consistent schema with common key fields. Semantic integration focuses on understanding the meaning of the data from different sources: the goal is to merge data that may use different names, terminologies, or structures to describe the same concepts. It involves the use of ontologies or data dictionaries to map similar concepts across datasets, and requires understanding the context, meaning and relationships within the data — for instance, spatial data can be easily integrated into maps.
In some cases the analyst will change the format (structure) of the data: sometimes to make it suitable for a specific modeling tool, in other instances to be able to pose the necessary data mining questions. Examples:
The slide carries the 5 V's of Big Data as a reference point for this discussion.
Things are even more complex when applying sequences of transformations:
The verdict on the slide: more an art than a science… at least for now.
Pre-processing steps do not commute. Every step changes the statistics the next step reads: standardization uses the mean and the standard deviation, so anything that changes the sample changes the standardization. This is the technical restatement of the warning at the start of Chapter 5 — data scientists cannot easily foresee the impact of pipeline prototypes, and there are no pre-defined rules on the impact of pre-processing transformations.
ETL (Extract, Transform, Load) is one of the most widely used data integration techniques in data warehousing:
ELT (Extract, Load, Transform) loads data into a storage system (like a data lake) and then transforms within the storage system.
info() and describe() of Chapter 4.Feature engineering is the manipulation — addition, deletion, combination, mutation — of your data set to improve machine learning model training. Derived attributes should be added if they ease the modeling algorithm: Area = Length x Width; Loyalty_Score = Total_Purchases x 0.4 + Avg_Order_Value x 0.3 + Frequency x 0.3; Ocean_Proximity = distance((Ocean_Lat, Ocean_Lon), (District_Lat, District_Lon)).
Most machine learning algorithms — linear regression, support vector machines — require numeric input because they use numerical computations to learn the model, and are not inherently capable of interpreting categorical data. Some implementations of decision tree-based algorithms can directly handle categorical data.
Ordinal encoding replaces each category with an integer, generally assigned arbitrarily; it is preferred when the variable has an inherent order (small = 0, medium = 1, large = 2). One-hot encoding replaces the variable with a set of binary variables, one per category, each taking 1 if the observation shows that category and 0 otherwise; it treats each category independently, which suits nominal features such as colors. OHE increases dimensionality and may not suit high-cardinality features; a remedy is to encode only the most frequent categories and let the rest be represented as 0s in all binary variables.
Because the Likert scale is strictly an ordinal scale, even though it is usually treated as an interval one, and arithmetic operations cannot be conducted on it (Wu and Leung 2017). Three reasons are given: respondents are often reluctant to express a strong opinion and gravitate to the neutral midpoint; averaging assumes equal emotional distance between mild and strong (dis)agreement; and fundamentally, the numbers are not numbers as such, but a means of ranking responses. Add the J-shaped distribution of reviews (Hu, Zhang and Pavlou 2009): people write reviews only when extremely satisfied or extremely unsatisfied.
The date 2022-01-01T00:01 was encoded into the signed integer 2201010001. A signed 32-bit integer covers [-2^31, 2^31 - 1] = [-2147483648, 2147483647], and 2201010001 > 2147483647: the encoded date overflows the type. It is the modern descendant of the year 2000 problem.
Because many classifiers use the Euclidean distance, and if one feature has a broad range the distance is governed by that feature. With age ∈ [0,120] and income ∈ [0,100000]: p1 = (50, 10000), p2 = (50, 20000) at distance 10000.00, p3 = (60, 10000) at distance 10.00, and p4 = (60, 20000) at distance 10000.00 — indistinguishable from p2 although it differs in both features. Age has been erased by the units of income.
Min-max: x' = a + (x - min)(b - a)/(max - min), rescaling into [a,b], typically [0,1]. Standardization: x' = (x - mean)/sd, giving zero mean and unit variance. Robust scaling: x' = (x - Q2)/(Q3 - Q1), designed to be robust to outliers because it uses the median and the interquartile range instead of the mean and the range.
The mean is dragged into the tail and separates from the median: on the Gaussian example both are 173 cm, on the skewed one the mean is 103 158 262 914 against a median of 36 666 524 821. The Long Tail is the concept where a large number of niche products collectively generate more sales than a few bestsellers — a right-skewed distribution, made possible on sites like Amazon that are not limited by shelf space. Skewed distributions can be transformed with mathematical functions such as the logarithm.
Equal-width divides the range of values into equal-sized intervals (values 0 to 100 into 10 bins gives width 10); it can create empty or sparse bins, especially with skewed data or outliers. Equal-frequency divides the values into bins with the same number of observations (100 observations into 10 bins gives 10 each); it creates balanced bins that handle skewed data and outliers better, at the cost of distorting the distribution and creating irregular bin widths. Binning generally reduces the effects of minor observation errors by replacing the values in a bin with a representative central value.
Averaging averages. Correct: sum of sums, (1+2) + (3+4+5) = 15. Wrong: avg(avg(1,2), avg(3,4,5)) = avg(1.5, 4) = 2.75, whereas avg(1,2,3,4,5) = 3. The group sizes differ, so the second average silently re-weights the data. Pay attention to the aggregation operator.
Because the number of subsets of a feature set S is |P(S)| = 2^|S| - 1 (excluding the empty set): 8 subsets for 3 features, 16 for 4, 1024 for 10. The dummy algorithm — test each subset and keep the one minimizing the error — is therefore computationally intractable for all but the smallest feature sets. Real approaches are characterized by a search technique and an evaluation measure.
Filter: select variables regardless of the model, based on general properties such as variance or correlation with the target. Wrapper: train a model on each subset and score it by its error rate on a hold-out set — computationally intensive but good results; methods include forward selection, backward elimination and exhaustive search. Embedded: add or remove features while building the model, based on prediction errors, so that selection and classification happen simultaneously — LASSO being the standard example.
By adding to the least-squares objective a penalty proportional to the absolute values of the coefficients: min( SUM (yi - y_hat_i)^2 + lambda * SUM |bj| ). The regularization parameter lambda controls the strength of the penalty; the absolute-value penalty shrinks some coefficients to exactly 0, thereby removing irrelevant features. The optimal lambda is determined with cross-validation.
Pearson's correlation measures the linear relationship between two numeric variables: close to 1 means positive correlation, -1 negative, 0 none. Between features: two highly correlated features are redundant, so keep one — the other contributes unnecessary noise. Between feature and target: a feature whose coefficient sits between -0.3 and 0.3 may not be very predictive and can potentially be filtered out.
PCA is a linear dimensionality reduction technique that preserves as much of the variance as possible in fewer dimensions, by linearly transforming the data onto a new coordinate system whose directions — the principal components — capture the largest variation; the first component captures the highest variance, the second the next, and so on. Steps: standardize the features (PCA is sensitive to scale), compute the covariance matrix, take its eigenvectors as the directions and eigenvalues as the magnitude of variance along them; the eigenvector with the largest eigenvalue is PC1. On Iris, PC1 explains 92.46%, PC2 5.31%, PC3 1.71%, and the heaviest PC1 loading is 0.857 on petal length.
Primary key-based integration matches unique identifiers, works when both datasets have a well-defined consistent schema with common key fields, is simpler but less flexible, and is limited to datasets that share a key. Semantic integration works from the meaning of the data, merging sources that use different names, terminologies or structures for the same concepts; it uses ontologies or data dictionaries to map concepts, is flexible across schemas, and is complex because it requires domain semantics — spatial data integrated into maps being the example given.
Because each transformation changes the statistics the next one reads. Normalization should be applied before rebalancing, since rebalancing can alter average and standard deviations; and applying feature engineering before or after rebalancing produces different results depending on the dataset and the algorithm. The deck concludes: it is more an art than a science, at least for now.
ETL — Extract (pull data from multiple sources: databases, APIs, flat files), Transform (clean, standardize, transform into the desired format), Load (into a target database or data warehouse). ELT loads the data into a storage system such as a data lake first, and then transforms it within the storage system.