ML II — Unsupervised, Feature Engineering & Model Discipline

Learning objectives

By the end you can:

1. Implement Lloyd's algorithm (k-means) from scratch (the assign/update loop) hand-trace it to convergence, explain why convergence is guaranteed only to a local optimum, describe what k-means++ initialization changes, and explain (with this module's own 21.85× figure) why standardizing features first changes the result.
2. Implement PCA from scratch via centering, covariance, and eigendecomposition; read and compute explained_variance_ratio_; explain why a component's sign is arbitrary; and apply this module's sample-size honesty check (a single month's PCA "finding" versus the same computation pooled across four years of the same panel).
3. Engineer leak-free financial features that obey the as-of rule (trailing rolling windows, correctly grouped lag features on a multi-ticker panel) and diagnose the centered-window and bare-.shift() bugs that violate it.
4. Recognize, quantify, and fix look-ahead leakage: a feature whose own construction reaches across the as-of boundary, demonstrated on real NSE and US data (a ~99% leaky score collapsing to a ~50–53% honest one).
5. Recognize, quantify, and fix target leakage (a distinct failure mode from look-ahead leakage) where a feature is definitionally circular with its label regardless of calendar timing, demonstrated on real India and US fundamentals data (a 100% leaky score falling to a 37–43% honest one).
6. Wire preprocessing through sklearn.pipeline.Pipeline (and, for mixed column types, ColumnTransformer) so that every step is fit on the training split alone; explain why fitting a scaler on combined train+test data is a genuine, error-free leak.
7. Implement walk-forward, expanding-window cross-validation (TimeSeriesSplit) by hand, and explain, and quantify, on a real factor panel, why a shuffled KFold split systematically overstates performance on chronologically ordered data.
8. Apply a four-point "too good to be true" checklist (implausible metric, one dominant feature, survival under walk-forward re-validation, a named as-of row for every feature) to any unsupervised, feature-engineering, or model-validation result before trusting it.
9. (Productivity objective: R10 duality.) Use an AI coding assistant to draft clustering, PCA, feature-engineering, or pipeline code, after you can already do each by hand, and verify, yourself, that nothing was fit on data it should never have seen; a pipeline that "runs without error" is not thereby certified leak-free.

The duality, stated once (R10). As in every module in this program, this node carries two objectives at once. The understanding objective (items 1–8) is what the mastery gate rewards; you earn the pass by explaining the mechanism yourself and by clearing the code pack's hidden tests, not by having an assistant produce code you cannot defend. The productivity objective (item 9) is the payoff you keep afterward: knowing how to point a copilot at unsupervised learning, feature engineering, and pipeline code without being quietly misled by a result that merely "ran." A tool can never buy you a pass on the gate.

Prerequisites & connections

Builds on. DA2.01 (Machine Learning I (Supervised Learning) is the direct foundation) its fit/predict/score fluency, its honest train/validation/test split, and its own P16 forward pointer (a StandardScaler fit on the full dataset before splitting) are exactly where this node picks up. DA1.02/DA1.03 (pandas I/II) supply the .rolling(), .shift(), and .groupby() fluency this node's feature-engineering sections lean on constantly, if a grouped shift or a trailing rolling mean is not already comfortable, revisit DA1.03 before this node's Core content. DA1.04 (EDA & Visualization) installed the "look at your data before you touch it" reflex this node assumes throughout, a leak silently hiding in a correlation nobody checked is exactly the kind of mistake EDA exists to catch first. DA1.05 (Statistics in Code) supplies the diagnostic instinct that a clean-looking fit can still be broken, which this node turns specifically toward look-ahead and target leakage. From the finance-plus galaxy, QM1.03 already gave you every concept this module turns into code: the one-line supervised/unsupervised distinction, a single illustrative k-means split (ten stocks into a real ~15.5%/~43.6% volatility clustering, no mechanism shown), one sentence naming PCA as dimensionality reduction, and a genuinely thorough treatment of look-ahead bias as a concept, complete with its own worked examples and mastery-check items.