Скачать или смотреть An Intuitive Introduction to Data Preprocessing

An Intuitive Introduction to Data Preprocessing

Data preprocessing is one of the most important stages in machine learning: this is where we turn raw, messy inputs into a clean feature matrix that models can learn from reliably. In this lecture, we build preprocessing into a clear, end-to-end workflow – from duplicate detection to encoding, feature engineering, and scaling – while connecting math, visual intuition, and Python code (pandas + scikit-learn).

What you’ll learn:
• Why preprocessing matters and how it affects model quality
• How to think of preprocessing as operations on a feature matrix
• How to clean data: duplicates, missing values, outliers
• How to detect near-duplicates via correlations and why visualization matters
• How to spot noisy features and why "low correlation ≠ useless feature"
• How to encode features: binary, one-hot for nominal categories, ordinal encoding for ordered categories
• What polynomial features are, how they make linear models non-linear, and why this can lead to overfitting
• Why scaling is needed and how to choose StandardScaler vs MinMaxScaler
• How to combine steps into a reproducible pipeline

The code from the video can be found here: https://shorturl.at/zOZuF

00:00 Introduction: why preprocessing matters
02:09 Our goals
03:39 Duplicate detection
06:01 Near-duplicates: what they are and why they appear
07:07 Correlations as a tool for near-duplicate detection
16:12 "Correlation is not causation"
17:19 Near-duplicates in Python
18:43 When duplicates can be useful
20:14 Handling missing values
22:42 Outliers
23:40 Visual tools: histogram, scatter plot
24:52 Boxplot
28:57 Noisy features
31:06 Feature encoding
31:12 Feature types: boolean / categorical (nominal, ordinal) / numerical / text / datetime
32:57 Binary encoding
34:40 Nominal encoding: one-hot encoding
37:36 Ordinal encoding
39:32 Polynomial features
41:46 Model comparison: linear regression vs regression in polynomial feature space
46:06 Generalization: degree-2/3 terms, interactions, bias term
48:52 How many features you get: binomial coefficient (d=3, p=2 → 10; p=3 → 20)
50:29 PolynomialFeatures in Python: include_bias, degrees, interpreting generated columns
53:50 Polynomial regression in practice: training + predicting in the transformed space
54:46 Polynomial degree vs overfitting (MSE can hit 0, but that’s a warning sign)
57:20 Feature scaling
1:00:10 Standardization (Z-score)
1:00:22 Min-Max scaling
1:05:01 Pipeline: scaling → polynomial features → linear regression
1:07:30 Visual comparison: StandardScaler vs MinMaxScaler
1:08:52 Effect on linear regression
1:11:00 Full preprocessing workflow recap (clean → reduce noise → transform → scale)
1:11:42 Conclusion