Branch: Data Science & ML (quant-ds) · Node DA2.02 · ~18 focused hours · Gate: Quiz ≥ 85% + structured code-pack ≥ 85%

DA2.01 ended with a promise it did not keep on purpose. Its own closing practice problem (P16) handed you a colleague's bug (a StandardScaler fit on the full dataset, train and test combined, before ever splitting) and told you plainly that the fix belongs here. Its own Core content said, in as many words, that engineering a feature safely, guarding against a feature built from information that should not exist yet, and wiring preprocessing through sklearn.pipeline.Pipeline so it cannot leak by accident were "deliberately out of scope" for that node. This is the node where all three become real, executable code, plus two entire algorithms DA2.01 never needed, because neither one requires a label at all.

Here is the whole module in one sentence you will spend eighteen hours earning the right to believe: a model's validation score is only as honest as the boundary between what it was allowed to see and what it is being asked to predict, and the tools in this module either find real structure that already lives inside that boundary, or they exist to defend it. k-means and PCA are the "find structure" half: hand k-means a set of numbers with no label anywhere and it will partition them into groups on its own terms; hand PCA a table of correlated columns and it will tell you, honestly, how many independent directions of variation actually exist inside it. Feature engineering, leak-free Pipelines, and walk-forward cross-validation are the "defend the boundary" half, and they exist because of an uncomfortable fact this module proves to you with real numbers rather than merely asserting it: the single easiest way to produce a machine-learning result that looks spectacular and means nothing is to let one number cross that boundary in the wrong direction, and nothing about a leaked pipeline ever raises an error to tell you it happened.

That last sentence is not rhetoric. Twice in this module you will watch the identical modeling task, on real NSE and real US data, produce two wildly different scores depending on one small design decision nobody would flag in an ordinary code review. A five-day price-direction call scores 99% with a feature built from the same window as its label, and a coin-flip-adjacent 50–53% with the honest version of the identical feature. A next-year fundamentals-growth call scores a mechanically perfect 100% with a feature built from next year's own realized growth rate, and an honest, humbling 37–43% without it. By the end of these eighteen hours you will be able to build both the leaky and the honest version of a feature on purpose, explain in one sentence which of two distinct diseases (look-ahead leakage or target leakage) produced any given gap, wire every preprocessing step through a Pipeline so the fit-on-train-only rule is structural rather than remembered, and validate a chronologically ordered panel with a walk-forward split instead of a shuffled one that quietly lies about how good your model really is.

Every exercise in this module lives in its code pack, code-packs/DA2.02.json (module id DA2.02, eight exercises DA2.02-e01 through DA2.02-e08), reachable from this node's Code tab. Every one of the eight ships a structured fill-in-the-blank fallback as its definition of record, so the pack is 100% completable even with zero runtime, live execution in the browser is a bonus, never a requirement to pass the gate.

Learning objectives

By the end you can:

Implement Lloyd's algorithm (k-means) from scratch (the assign/update loop) hand-trace it to convergence, explain why convergence is guaranteed only to a local optimum, describe what k-means++ initialization changes, and explain (with this module's own 21.85× figure) why standardizing features first changes the result.
Implement PCA from scratch via centering, covariance, and eigendecomposition; read and compute explained_variance_ratio_; explain why a component's sign is arbitrary; and apply this module's sample-size honesty check (a single month's PCA "finding" versus the same computation pooled across four years of the same panel).
Engineer leak-free financial features that obey the as-of rule (trailing rolling windows, correctly grouped lag features on a multi-ticker panel) and diagnose the centered-window and bare-.shift() bugs that violate it.
Recognize, quantify, and fix look-ahead leakage: a feature whose own construction reaches across the as-of boundary, demonstrated on real NSE and US data (a ~99% leaky score collapsing to a ~50–53% honest one).
Recognize, quantify, and fix target leakage (a distinct failure mode from look-ahead leakage) where a feature is definitionally circular with its label regardless of calendar timing, demonstrated on real India and US fundamentals data (a 100% leaky score falling to a 37–43% honest one).
Wire preprocessing through sklearn.pipeline.Pipeline (and, for mixed column types, ColumnTransformer) so that every step is fit on the training split alone; explain why fitting a scaler on combined train+test data is a genuine, error-free leak.
Implement walk-forward, expanding-window cross-validation (TimeSeriesSplit) by hand, and explain, and quantify, on a real factor panel, why a shuffled KFold split systematically overstates performance on chronologically ordered data.
Apply a four-point "too good to be true" checklist (implausible metric, one dominant feature, survival under walk-forward re-validation, a named as-of row for every feature) to any unsupervised, feature-engineering, or model-validation result before trusting it.
(Productivity objective: R10 duality.) Use an AI coding assistant to draft clustering, PCA, feature-engineering, or pipeline code, after you can already do each by hand, and verify, yourself, that nothing was fit on data it should never have seen; a pipeline that "runs without error" is not thereby certified leak-free.

The duality, stated once (R10). As in every module in this program, this node carries two objectives at once. The understanding objective (items 1–8) is what the mastery gate rewards; you earn the pass by explaining the mechanism yourself and by clearing the code pack's hidden tests, not by having an assistant produce code you cannot defend. The productivity objective (item 9) is the payoff you keep afterward: knowing how to point a copilot at unsupervised learning, feature engineering, and pipeline code without being quietly misled by a result that merely "ran." A tool can never buy you a pass on the gate.

Prerequisites & connections

Builds on. DA2.01 (Machine Learning I (Supervised Learning) is the direct foundation) its fit/predict/score fluency, its honest train/validation/test split, and its own P16 forward pointer (a StandardScaler fit on the full dataset before splitting) are exactly where this node picks up. DA1.02/DA1.03 (pandas I/II) supply the .rolling(), .shift(), and .groupby() fluency this node's feature-engineering sections lean on constantly, if a grouped shift or a trailing rolling mean is not already comfortable, revisit DA1.03 before this node's Core content. DA1.04 (EDA & Visualization) installed the "look at your data before you touch it" reflex this node assumes throughout, a leak silently hiding in a correlation nobody checked is exactly the kind of mistake EDA exists to catch first. DA1.05 (Statistics in Code) supplies the diagnostic instinct that a clean-looking fit can still be broken, which this node turns specifically toward look-ahead and target leakage. From the finance-plus galaxy, QM1.03 already gave you every concept this module turns into code: the one-line supervised/unsupervised distinction, a single illustrative k-means split (ten stocks into a real ~15.5%/~43.6% volatility clustering, no mechanism shown), one sentence naming PCA as dimensionality reduction, and a genuinely thorough treatment of look-ahead bias as a concept, complete with its own worked examples and mastery-check items.

The C-3 binding boundary, stated once. QM1.03's coverage above is exactly what it should be (concept, intuition, failure mode) and this node does not re-derive any of it. This node's job is different: implementation, validation, pipelines, and evaluation. Concretely, this node shows you Lloyd's algorithm's actual assign/update mechanism and why it converges only to a local optimum; PCA's actual covariance-eigendecomposition machinery, its sign ambiguity, and its sample-size honesty check; the actual pandas code that keeps a feature on the safe side of the as-of boundary; and an entire second leakage disease (target leakage) that QM1.03 never names, because it has nothing to do with chronological order and everything to do with a feature being definitionally circular with its label. Cross-link QM1.03 for why overfitting happens and why a non-chronological split is wrong in the first place; this node is where you get the exact line of code that fixes it, and the real accuracy number, before and after.

Feeds forward. DA2.03 (the branch's crown: Deep Learning, NLP & Time-Series Forecasting) explicitly inherits "pipelines, feature engineering and (critically) leakage" from this node, extending every idea here from the tabular case into text and time; its own Part III opens by naming the same exchangeability assumption this node's walk-forward section exists to correct. Beyond quant-ds, QD1.02 (Signal & Factor Construction and Evaluation) evaluates predictive signals on the same kind of factor panel this node's walk-forward worked examples use, with its own specialized metrics (IC, IR, decay) that are cousins of, but not identical to, the R² this node grades with. QD1.03 (Building a Backtesting Engine) leans on this node explicitly for "leak-free ML pipelines in general, target leakage, fitting a scaler on the full sample, the Pipeline object that makes it hard to get wrong", and builds one backtesting-specific extension on top of this node's general walk-forward split: purging and embargoing, the extra gap needed when labels overlap (a 20-day-forward return evaluated daily), which is QD1.03's job to teach in full, not this node's. QD2.04 (a reproducible research protocol) will assume every pipeline you build from here forward is already fit-on-train-only and walk-forward-validated by default, and layers logging, seeding, and configuration discipline on top of that assumption.

Core teaching content

The one sentence that explains almost everything

Every technique in this module sits on one thesis, and it is worth stating before a single line of code: a model's validation score is only as honest as the boundary between what it was allowed to see and what it is being asked to predict. k-means and PCA are the two "no label needed" tools in this module, they earn their keep by finding real structure that already exists inside that boundary, never by peeking past it. Feature engineering, leak-free pipelines, and walk-forward validation are the three "defend the boundary" tools, they exist because the single easiest way to produce an ML result that looks spectacular and means nothing is to let so much as one number cross that boundary in the wrong direction, whether by one row of calendar time (look-ahead leakage) or by one column that is secretly the label wearing a disguise (target leakage). DA2.01 gave you the boundary itself, the honest train/test/CV split. This module's entire job is teaching you what can go wrong on either side of it, and how to build code that makes going wrong structurally difficult rather than merely "something to remember."

Unsupervised learning: what changes when there is no label

QM1.03 already gave you the one-line distinction: supervised learning has a known target for every row; unsupervised learning does not, and the task shifts from "predict this known quantity" to "discover a structure that was never labeled as such." That single illustration QM1.03 ran, ten stocks clustered by trailing annualized volatility into two natural groups, a ~15.5% low-volatility cluster and a ~43.6% high-volatility cluster, no label supplied anywhere, is completely correct as a concept, and it is exactly where this module starts building the machine underneath it. Two tools cover almost everything you will do with unlabeled data in practice: clustering (k-means, this module's first algorithm) groups similar rows together; dimensionality reduction (PCA, this module's second) collapses many correlated columns into a handful of uncorrelated ones that carry almost all of the original information. Both tools share one property that makes them fundamentally different from everything in DA2.01: there is no y, so there is no "right answer" to score against, which means the ordinary bias-variance, train/test-split machinery from DA2.01 does not directly apply, and evaluating whether a clustering or a projection is any good becomes its own, separate skill (the elbow method and the sample-size honesty check, both below, are exactly that skill).

k-means and Lloyd's algorithm: assign, update, repeat

k-means partitions n points into k groups using an algorithm (Lloyd's algorithm) short enough to hold in your head completely. Start with k initial centroids (more on choosing them below). Repeat two steps until nothing changes: (1) Assign. Put every point with whichever centroid it is closest to, by squared Euclidean distance. (2) Update. Recompute each centroid as the plain mean of every point currently assigned to it. Stop the moment step (1) produces the exact same assignment as the previous round, nothing will change again if you kept going, because the update step (a mean) is deterministic given a fixed assignment, and the next assignment is deterministic given a fixed set of centroids.

Trace it on six real tickers so the mechanism is not an abstraction. Take each ticker's own annualized return (mean daily return × 252, computed the same way every worked example and code-pack hidden test in this module computes it) for three India and three US names on this branch's nse-eod/us-eod panels: BLUSTM (US, −20.45%), PIPPAL (India, −17.07%), UDAYAN (India, −15.49%), KOKILA (India, +39.66%), WLBROK (US, +42.15%), NRTHBY (US, +59.41%). Initialize two centroids at the two extremes, −20.45% and +59.41%. Round 1, assign: the three negative-return names (BLUSTM, PIPPAL, UDAYAN) are closer to −20.45%; the three positive-return names (KOKILA, WLBROK, NRTHBY) are closer to +59.41%. Round 1, update: the low centroid becomes the mean of (−0.2045, −0.1707, −0.1549) = −0.176733; the high centroid becomes the mean of (0.3966, 0.4215, 0.5941) = 0.470733. Round 2, assign: re-checking every point against the new centroids changes nothing, the assignment is identical to round 1. Converged, after exactly one update, matching DA2.02-e01's own hidden test on this exact six-ticker set. Read it: notice what the algorithm did not do, it never once consulted the IN/US market label. It clustered purely on the number, and the number happened to split three-and-three, mixing both markets into each group, a small, concrete illustration that k-means finds structure in whatever numbers you hand it, market label or no market label, and it is entirely your job to decide whether that structure means anything.

Convergence is guaranteed, to a local optimum, not the global one. k-means's objective (the total within-cluster sum of squared distances, sometimes called inertia) can only fall or stay flat at both steps, the assign step, by construction, can only ever move a point to a closer centroid or leave it where it is; the update step, by construction, sets each centroid to the mean, which is the single point that minimizes squared distance to a fixed set of members. An objective that can only fall or hold, bounded below by zero, must eventually stop falling, that is the entire convergence guarantee, and it is real. What it does not guarantee is that the point where it stops is the best possible clustering of the data; a different starting set of centroids can, and often does, converge to a different, worse local optimum (see this module's own Practice set for a hand-verified example where a bad initialization lands on a clustering with roughly 67 times the inertia of a good one). This is precisely why scikit-learn's KMeans does not use plain random initialization by default, it uses k-means++, which chooses each new centroid with probability proportional to its squared distance from the nearest centroid already chosen, deliberately spreading the starting points out across the data rather than risking two initial centroids landing close together and permanently splitting one true cluster in half.

Why k-means needs standardization (and how badly it fails without it)

k-means's assign step is entirely a squared-Euclidean-distance computation, and Euclidean distance has no opinion about units, it will treat one unit of "annualized return" as exactly as important as one unit of "annualized volatility," even though those two quantities can live on wildly different scales. Compute both, honestly, across this branch's full 30-name India-plus-US universe (fifteen NSE tickers, fifteen US tickers, each ticker's own daily closing-price history): the standard deviation of annualized return across all 30 names comes out to 20.22 percentage points; the standard deviation of annualized volatility across the same 30 names comes out to 0.925 percentage points, a ratio of essentially 21.85-to-1. Hand that raw, unstandardized pair of columns to k-means, and the assign step's squared-distance arithmetic is decided almost entirely by the return column; volatility barely moves the needle, no matter how genuinely different two tickers' risk profiles are. sklearn.preprocessing.StandardScaler (subtract each column's mean, divide by its standard deviation) is the fix, and it is not optional cosmetics: it is what makes "distance" mean the same thing along every axis you hand the algorithm. Standardize before k-means, every time, with no exceptions for "this feature already looks kind of standardized", check it with the actual numbers, the way this section just did, not by eye.

Choosing k: the elbow method and its limits

Nothing about Lloyd's algorithm tells you what k should be, that decision sits entirely outside the algorithm and needs its own honest method. The elbow method is the standard first move: fit k-means at k=1, 2, 3, ... up to some reasonable ceiling, record the inertia (total within-cluster sum of squared distances) at each k, and plot inertia against k. Inertia always falls as k grows (more clusters can only ever fit the data at least as well, down to inertia of exactly zero at k=n, one cluster per point), so the question is never "which k minimizes inertia," it is "where does adding another cluster stop buying much improvement," a bend in the curve you find by eye, not a single formula that returns one correct integer. This is a genuinely soft judgment call, not a weakness to apologize for, a silhouette score (how much closer, on average, is a point to its own cluster than to the next-nearest one) is a common complement that gives a second, independent signal when the elbow itself is ambiguous, though this module's code pack does not grade a silhouette implementation directly (a natural extension for your own practice, not a graded requirement here). The discipline that matters most is smaller than either formula: look at the actual inertia numbers, and be honest about how blunt "eyeball the bend" really is, a real dataset's elbow is rarely as clean as a diagram in a textbook.

{
  "skill": "Using an AI coding assistant to scaffold a k-means/PCA exploratory analysis on a real return panel without accepting an unverified choice of k or component count",
  "use": "After you can already hand-trace Lloyd's algorithm and compute explained_variance_ratio_ yourself (this section), ask an assistant to draft the boilerplate for a clustering or PCA exploration -- reading a CSV, standardizing, fitting KMeans/PCA, plotting an elbow curve -- so you spend your time reading the actual result, not re-typing sklearn's constructor signatures.",
  "tools": ["AI coding assistants -- Claude, GitHub Copilot, Cursor, ChatGPT"],
  "prompt": "Write a Python function that reads a ticker return/volatility table, standardizes both columns, fits sklearn.cluster.KMeans for k=2..6, and returns the inertia for each k as a list, so I can read the elbow myself.",
  "verify": "Never accept an assistant's own suggested 'best k' or 'best number of components' as the answer -- read the actual inertia/explained_variance_ratio_ numbers yourself and make the elbow/cutoff call by eye, exactly as this section teaches; also confirm it standardized BEFORE fitting KMeans (a frequent, silent omission that changes every downstream cluster), and check any PCA component signs against your own eigendecomposition rather than assuming a flipped sign is a bug.",
  "diy": "You must already be able to compute explained_variance_ratio_ and hand-trace at least one iteration of Lloyd's algorithm yourself before an assistant's clustering/PCA code is something you can sanity-check rather than something you are simply trusting.",
  "market": "IN"
}

PCA: covariance, eigendecomposition, and explained variance

Principal Component Analysis answers a different question than clustering: not "which group does this row belong to," but "how many of these columns do I actually need, if I am willing to replace them with a smaller set of uncorrelated combinations of the originals?" The mechanism is four steps, and each one is worth naming explicitly rather than trusting as a black box behind PCA().fit(). (1) Center. Subtract each column's own mean, so every column has mean zero: PCA is only meaningful relative to how columns vary around their own center, not their absolute level. (2) Covariance. Compute the covariance matrix of the centered data, a square, symmetric matrix whose diagonal holds each column's own variance and whose off-diagonal entries hold how each pair of columns co-moves. (3) Eigendecomposition. Find this matrix's eigenvalues and eigenvectors (numpy.linalg.eigvalsh is the right tool for a real, symmetric matrix like a covariance matrix, and it returns eigenvalues you can trust to be real numbers rather than needing complex-number handling). Each eigenvector is one principal component (a direction in the original column space) and its corresponding eigenvalue measures how much variance the data has along that direction. (4) Rank and normalize. Sort the eigenvalues descending, and divide each by the sum of all of them: that is explained_variance_ratio_, the fraction of the data's total variance each component carries, and by construction the whole list sums to exactly 1.0.

Verify the mechanism on a case simple enough to check by hand before trusting it on real data: two columns that are perfectly collinear (the second is always exactly double the first) carry, definitionally, only one independent direction of variation, feed [[1,2],[2,4],[3,6],[4,8]] through the four steps above and the first component's explained variance ratio comes out to exactly 1.0, the second exactly 0.0, because every bit of variance already lives along the one direction where the two columns move together. Real financial data is never this clean, which is exactly why the honesty checks in the next section matter.

PCA's honesty checks: sign ambiguity and sample size

Two properties of PCA look, the first time you meet them, like bugs. They are not, and knowing the difference between "this looks wrong" and "this is a known, harmless property" is itself part of what this module is teaching. Sign ambiguity, first. If a vector v satisfies the eigenvector equation Av = λv, then so does −v, since A(−v) = −Av = λ(−v), the same equation, satisfied by the exact negation. Two entirely correct implementations of PCA can therefore return a component with every sign flipped relative to each other, describing the identical direction in space, and neither is wrong. What is not ambiguous, and what you should actually compare across implementations or across runs, is the explained_variance_ratio_ itself and the relative magnitudes of a component's loadings, never raw signs.

Sample size, second, and this is the one that has genuinely misled real research. Run PCA on this module's synthetic-factor-panel dataset (four factor columns (value, momentum, quality, low-volatility) for 60 tickers across 48 monthly snapshots) restricted to a single month, 2022-07-01, and the explained-variance ratios come out 35.56%, 26.94%, 19.11%, 18.40%, a real, visible skew, with the first component alone carrying more than a third of the variance, suggesting genuine, exploitable structure (perhaps two of the four factors are quietly correlated with each other in a way you could compress). Run the identical four columns and the identical PCA mechanism, but pooled across the entire panel, all 48 months, 2,880 rows total, and the ratios come out 26.41%, 24.84%, 24.68%, 24.07%: nearly perfectly even, essentially indistinguishable from four genuinely uncorrelated factors. The single month's "finding" was never real structure; it was sampling noise from a sample of only 60 observations, and the only thing that exposed it was checking a much larger sample of the same data. The honesty check this module installs is simple to state and easy to skip under deadline pressure: before you trust a PCA result (or a clustering result) as a genuine, exploitable finding, check whether it survives being recomputed on more data than the one convenient slice you first looked at.

Financial feature engineering and the as-of rule

Every feature you will ever hand a model is computed "as of" some specific row, and the single rule every feature must obey, without exception, is this: a feature computed as of row t may use row t and everything strictly before it, and nothing after. A label, by contrast, is required to reference the future, because predicting the future correctly is the entire point of having a label in the first place. Confusing which of your two columns is playing which role (letting a feature quietly behave like a label) is the single most common way a financial ML pipeline produces a number that looks excellent and means nothing, and the next two sections show you exactly what that confusion looks like in real, executable code.

pandas's .rolling(w) is, by default, exactly the safe direction: the window ending at row t covers rows t-w+1 through t, that row and the w-1 rows strictly before it, nothing later. .rolling(w, center=True) is a separate, explicit opt-in that reaches into future rows around the window's center, which is legitimate for smoothing a chart you already hold in full (nothing about drawing a finished picture violates the as-of rule) and actively dangerous the moment the same computation is handed to a model as a feature meant to inform a decision made at row t, because now the model is being trained on information from rows that, at prediction time, do not exist yet. Exercise DA2.02-e03 builds exactly the safe direction, a trailing rolling-mean-return feature over an OHLCV panel, verified against both an NSE ticker (NILAM) and a US ticker (BRPEAK): NILAM's row 24 carries a daily return of −3.614% and a 5-day trailing momentum of −0.171% (the average of the five most recent daily returns up to and including that row); BRPEAK's row 40 carries a daily return of +2.425% and a 5-day trailing momentum of +0.642%, both entirely mechanical, both computable by hand from the five preceding rows, both provably reaching no further forward than the row itself.

One more failure mode belongs here, distinct from the centering trap but just as silent: a multi-ticker panel sorted by ticker then date, lag-featured with a bare df['x'].shift(1) and no .groupby('ticker') first. The shift operates on the raw, concatenated row order, meaning the first row of the second ticker in the sort order silently inherits the last row of the first ticker as its "lag," splicing two completely unrelated companies together at the ticker boundary, with no error, no warning, and a plausible-looking number sitting in the column the whole time. out.groupby('ticker')['net_income'].shift(1) (exactly as exercise DA2.02-e05's engineer_growth_features does) resets the lag at every ticker boundary, and it is the only version of this operation that is actually correct on panel data: treat a bare .shift() on a multi-ticker frame as a bug until proven otherwise.

Look-ahead leakage: when a feature's own construction cheats

Look-ahead leakage is a time-ordering violation: nothing is wrong with the definition of the feature in the abstract, but the specific window used to compute it reaches across the as-of boundary into rows that, at the moment the feature claims to describe, would not yet exist. See the entire failure, and its fix, in one deliberately dramatic real-data comparison. Define a 5-day-ahead direction label (did the close price rise over the next five trading days, yes or no) on both NILAM (NSE) and BRPEAK (US). Now build two candidate features and grade each one's raw directional accuracy (predict "up" whenever the feature is positive) against that label. The leaky feature computes the mean of the same next five days' daily returns the label itself is built from (a near-tautology dressed as a feature) and it scores 99.19% accuracy on NILAM and 98.99% on BRPEAK: nearly mechanically perfect, because it is, numerically, almost the same quantity as the label it is "predicting." The honest feature computes the mean of the five days strictly before the row, the trailing-momentum feature from the previous section, and on the identical data, the identical horizon, the identical label, its accuracy collapses to 49.90% on NILAM and 52.73% on BRPEAK, statistically indistinguishable from a coin flip. The entire 46-to-49-percentage-point gap traced to one single design decision: which direction, relative to row t, the feature's window looked. Exercise DA2.02-e04 builds and grades the honest version; hold the leaky number in your head as the "too good to be true" smell this whole module is training you to recognize on sight.

Target leakage: when the feature IS the label in disguise

Target leakage is a different disease from look-ahead leakage, and confusing the two costs you the ability to diagnose either correctly. Look-ahead leakage is about when a feature's window reaches, relative to a chronological boundary; target leakage is about what a feature actually is, regardless of any calendar, a feature can be built entirely from data that, chronologically, belongs to a genuinely later period (which a label is allowed to do too) and still be leakage, because the feature is not really an independent predictor at all, it is a rearrangement of the label's own defining calculation.

See it concretely on this module's real fundamentals panels (india-fundamentals, sec-fundamentals; fifteen tickers each, four fiscal years, 2022 through 2025). Build a label: did net income grow from this fiscal year to the next one, yes or no. Build an honest feature: this fiscal year's own year-over-year growth rate, using only fiscal years already on record (DA2.02-e05's engineer_growth_features, a grouped, prior-year shift). Predicting next year's growth direction from this year's already-realized growth rate is a real, legitimate, if imperfect, modeling question, and the honest answer is honestly mediocre: 43.33% accuracy on the India panel, 36.67% on the US panel, both worse than a coin flip, a genuinely humbling, genuinely real result about how hard fundamentals-based growth prediction actually is. Now build the leaky version instead: compute next fiscal year's own year-over-year growth rate directly ((next_net_income - net_income) / net_income) and hand that to the identical accuracy check. It scores a mechanically perfect 100% on both panels, because it is not a feature, it is the label's own defining arithmetic, computed one column over and given a different name. No chronological rule was broken (the leaky "feature" is honestly built from next_net_income, which the label also legitimately uses), the mistake is purely definitional, and it is exactly the trap a learner who has only ever been warned about "using future data" will walk straight into, because nothing about the calendar looks wrong. The tell, once you know to look for it: a feature this good on a task this hard for a model is a red flag, not a breakthrough, and the two facts this module trains you to name on sight, a suspiciously perfect score, and a feature that is definitionally close to the label, apply regardless of which of the two leakage diseases produced it.

Leak-free pipelines: fit on train, transform everywhere else

DA2.01 left one specific bug unfixed on purpose (its own P16, a direct forward pointer here): computing a feature's mean and standard deviation across the full dataset (train and test combined) before ever splitting, then using those combined statistics to standardize every row. Run that mistake and nothing about it looks broken: no feature is obviously wrong, no error is raised, the code runs cleanly, and the resulting number is simply, quietly optimistic, because the test rows' own values already shaped the statistics the training rows were scaled with. This is the third leakage disease this module names, distinct from both look-ahead and target leakage: a preprocessing-statistics leak, and it is, per this branch's own backtesting module (QD1.03), "the leak that survives careful people, because it looks like data cleaning rather than strategy logic."

sklearn.pipeline.Pipeline is the structural fix, not merely a style preference. Build Pipeline([('scale', StandardScaler())]), call .fit(X_train) exactly once, and every contained step is fit on X_train alone; call .transform(...) afterward on X_train or on X_test and the pipeline reuses those already-fitted statistics without ever recomputing them. Exercise DA2.02-e06's leak_free_pipeline_scale builds this literally: on a toy three-row train set with values [1,10], [2,20], [3,30] and a two-row test set [10,100], [20,200], the fitted scaler's mean comes out to [2.0, 20.0] and its standard deviation to [0.8165, 8.165], numbers computable entirely from the three training rows, and provably unmoved by whatever appears in X_test, because .fit() was never called on it. When your feature set mixes numeric and categorical columns (a ticker's market alongside its trailing return, say), sklearn.compose.ColumnTransformer extends the identical discipline column-by-column, route numeric columns to a StandardScaler, categorical columns to a OneHotEncoder, wrap the whole assembly in one Pipeline, and the fit-on-train-only guarantee holds across every column type at once, with no step left exposed to a manual, easy-to-forget call order.

{
  "skill": "Using an AI coding assistant to draft a scikit-learn feature-engineering or preprocessing pipeline without shipping a silent leak",
  "use": "After you can already state the fit-on-train-only rule and diagnose look-ahead vs. target leakage yourself (this module's Core content), ask an assistant to draft the first pass of a Pipeline/ColumnTransformer or a rolling-feature function -- then read every fit()/transform() call and every .shift()/.rolling() direction before trusting it.",
  "tools": ["AI coding assistants -- Claude, GitHub Copilot, Cursor, ChatGPT"],
  "prompt": "Write a leak_free_scale(X_train, X_test) function using sklearn.pipeline.Pipeline and StandardScaler that fits only on X_train and transforms both X_train and X_test, returning the scaled arrays plus the fitted mean_/scale_.",
  "verify": "A leaky pipeline runs cleanly and returns a plausible number -- 'it ran without error' certifies nothing. Explicitly check: (1) is .fit() ever called on X_test, or on a concatenation of X_train and X_test? (2) does any .shift()/.rolling() call reach in the wrong direction, or omit a groupby('ticker') on a multi-ticker panel? (3) does the reported score survive being re-checked under TimeSeriesSplit instead of shuffled KFold? Re-run this module's own numbers (the 21.85x scaling ratio, the 49.90%/52.73% honest look-ahead accuracies, the 43.33%/36.67% honest target-leakage accuracies) as a sanity check that the assistant's code reproduces figures you already trust before pointing it at new data.",
  "diy": "You must already be able to name, unaided, the exact as-of row every feature and the label are allowed to use, and explain why fitting a scaler on combined train+test data is a leak, before an AI-generated pipeline is something you can actually certify rather than merely hope is correct.",
  "market": "US"
}

Walk-forward validation: why shuffled cross-validation lies to panels

DA2.01's k-fold cross-validation silently assumed something that is true for a cross-sectional snapshot and false for a chronologically ordered panel: that rows are exchangeable, that shuffling them before splitting into folds produces a fair test. A shuffled split on time-ordered data can, and routinely does, place training rows that sit after some test rows in real calendar time, training a model partly on the future relative to what it is later scored against. sklearn.model_selection.TimeSeriesSplit fixes this with an expanding-window walk-forward scheme: fold i's training set is every row before some cutoff, and its test set is the block of rows immediately after that cutoff, training only ever grows, and it always ends strictly before testing begins. Exercise DA2.02-e07's walk_forward_splits reproduces TimeSeriesSplit's exact fold boundaries by hand, and exercise DA2.02-e08 puts the discipline to work end to end, fitting the identical LinearRegression under both schemes on the real synthetic-factor-panel (four factor columns predicting forward return, 2,880 rows across 48 months): a shuffled 5-fold split reports a mean R² of 0.0262; the identical model, identical data, under a walk-forward TimeSeriesSplit instead, reports 0.0153, the shuffled number overstates the honest one by roughly 1.72×, with no feature-level bug anywhere in sight, purely from the choice of which rows were allowed to train on which. Both numbers are honestly modest (real single-factor-panel R² routinely lives in this range), which is itself worth sitting with: the walk-forward number is smaller because it is the only one of the two that could not have secretly trained on the future. This module's walk-forward split is the general discipline; when overlapping, multi-day-forward labels are involved (a 20-day-forward return re-evaluated every single day, so adjacent labels share most of their underlying data), an extra gap, purging and embargoing, is needed between train and test to stop information leaking through that overlap, and that backtesting-specific extension is QD1.03's job to teach in full, building on the general split taught here.

Where this leaves you

Six tools, one underlying discipline. k-means and PCA find real structure inside the boundary of what a model is honestly allowed to see; feature engineering, leak-free pipelines, and walk-forward validation defend that same boundary against three genuinely distinct ways it silently breaks, a feature's own window reaching into the future (look-ahead leakage), a feature that is secretly the label's own arithmetic (target leakage), and a preprocessing step quietly fit on data it was never supposed to see (the pipeline leak). None of the three produces an error message. All three produce a number that looks better than it has any right to, and the entire discipline this module installs is a single reflex: when a result looks too good, go find the exact row, at the exact moment in time, where a piece of information that should not have crossed the boundary crossed it anyway, and only report the number that survives once you have looked.

Common mistakes & how experts think differently

"I ran k-means on raw return-and-volatility numbers without scaling, the algorithm didn't error, so it must be fine." Silence is not correctness. This module's own numbers show annualized return's standard deviation is roughly 21.85× annualized volatility's across a real 30-ticker universe, an unscaled Euclidean distance is decided almost entirely by whichever column happens to have the larger absolute scale, and k-means will converge to something every time, scaled or not, so it never tells you it was fed the wrong units. The expert standardizes every numeric feature before any distance-based method, checks the actual scale ratio the way this module did, and never trusts "it ran" as evidence of correctness.
"k-means found 4 clusters, so there are 4 real groups in this data." k-means will return exactly as many clusters as you ask k to be, whether or not that many genuinely distinct groups exist, running it at k=4 on data with two real groups still produces four non-empty clusters, it simply splits one or both real groups arbitrarily. The expert treats a chosen k as a hypothesis to justify (via the elbow method, a silhouette check, or domain reasoning), never as a fact the algorithm discovered on its own.
"PCA's first two components are flipped in sign compared to my colleague's run, one of us made an error." Neither of you did. An eigenvector and its exact negation satisfy the identical eigenvector equation and represent the identical direction in space; different implementations, different library versions, or even the same code run twice with a different internal tie-breaking order can legitimately produce opposite signs. The expert compares explained_variance_ratio_ and relative loading magnitudes across runs or implementations, and never treats a sign flip alone as a bug.
"PCA on this one month of data shows two of my four factors are basically redundant, I can drop one." This module's own comparison is the direct counter-example: a single month's PCA on a real factor panel showed a skewed 35.6/26.9/19.1/18.4% split suggesting genuine redundancy, but the identical computation pooled across all 48 months of the same panel came out nearly perfectly even (26.4/24.8/24.7/24.1%), the single month's "finding" was sampling noise from only 60 observations, not a real relationship. The expert re-checks any PCA or clustering "finding" against a meaningfully larger sample before trusting it as structure rather than noise.
"My rolling feature uses .rolling(w, center=True), it's smoother, so it must be a better feature." A centered rolling window at row t explicitly incorporates rows after t, which is fine for a chart you are drawing over data you already hold in full and a direct violation of the as-of rule the moment the same column becomes a feature meant to inform a decision made at row t. The expert treats center=True as legitimate only for finished, retrospective visualization, and never for a column that will be handed to a model as an input.
"My lag feature on this multi-ticker panel looks fine, the numbers are all reasonable-looking floats." A bare .shift(1) with no .groupby('ticker') first silently splices the last row of one ticker onto the first row of the next in sort order, and the resulting number is a perfectly ordinary-looking float that happens to describe a relationship between two unrelated companies, "looks reasonable" is not a check, because a wrong-but-plausible number is exactly what this bug produces. The expert always groups by entity before shifting or rolling on any panel with more than one ticker, and spot-checks the first row of every entity boundary by hand.
"My model scored 99% accuracy predicting 5-day price direction, this is a great signal, let's size a strategy around it." This module's own worked example produced that exact number, and it was pure look-ahead leakage: a feature built from the same forward-looking window as the label. The expert treats any accuracy far above what the task plausibly allows (a 5-day equity direction call has no business clearing the high 90s) as a leakage alarm to investigate first, a strategy idea a distant second.
"My fundamentals growth-prediction model scored 100%, I finally cracked it." A feature built from next fiscal year's own realized growth rate will always score 100% predicting whether growth happened, because it is not a feature, it is the label's own arithmetic under a different column name, no model "cracked" anything. The expert asks, for any suspiciously perfect score, whether the feature could be recomputed directly from the label's own definition, not merely whether it was collected before or after some date.
"I fit my StandardScaler on the whole dataset before splitting, scaling is just data cleaning, it can't leak." Scaling parameters (a mean, a standard deviation) computed across train and test combined let the test rows' own values quietly influence how the training rows are scaled, a real leak that produces no error and no obviously broken feature, exactly why it survives careful people. The expert fits every preprocessing step, with no exceptions, inside a Pipeline called on the training split alone.
"My AI assistant wrote this whole feature-engineering-and-cross-validation pipeline and it ran without any errors, so it should be trustworthy." A leaky pipeline, a scaler fit on the full dataset, a rolling window built the wrong direction, a shuffled cross-validation on a chronological panel, executes perfectly cleanly and returns a plausible, confidently wrong number; "it ran" is evidence about syntax, not about which rows any .fit() call actually saw. The expert reads every .fit()/.shift()/.rolling() call in AI-generated pipeline code line by line, and re-checks the result against this module's own touchstone numbers (a 21.85× scaling ratio, a ~50% honest look-ahead accuracy, a ~40% honest target-leakage accuracy, a walk-forward R² smaller than a shuffled one) before trusting it on new data.

The thread through all ten: the expert never treats a suspiciously good number as a discovery to celebrate before treating it as a leak to rule out. Every mistake above is silent by construction, no error message, no obviously broken feature, often a perfectly plausible-looking float sitting exactly where you would expect one, which is exactly why the checklist later in this module (and the habit of naming the exact as-of row for every feature and the label) matters more than any single rule of thumb.

Worked examples

Worked example 1: k-means by hand: six real tickers, one converged split (India + US)

Six tickers from this branch's own nse-eod/us-eod panels, each one's annualized return (mean daily return × 252): BLUSTM (US, −20.45%), PIPPAL (India, −17.07%), UDAYAN (India, −15.49%), KOKILA (India, +39.66%), WLBROK (US, +42.15%), NRTHBY (US, +59.41%). Run k-means with k=2, initial centroids at the two extreme values, −20.45% and +59.41%. Trace it to convergence by hand.

Solution. Step 1: first assignment. Against centroids (−0.2045, 0.5941): BLUSTM, PIPPAL, and UDAYAN (all negative) are closer to −0.2045; KOKILA, WLBROK, and NRTHBY (all positive) are closer to 0.5941. Step 2: first update. Low centroid = mean(−0.2045, −0.1707, −0.1549) = −0.176733; high centroid = mean(0.3966, 0.4215, 0.5941) = 0.470733. Step 3: re-check. Re-assigning against the new centroids changes nothing, every point's nearest centroid is identical to Step 1. Converged, after exactly one update, matching DA2.02-e01's own hidden test on this exact six-ticker set. Read it. Notice the algorithm never once looked at the IN/US label, it split purely on the number, and the split happened to land three India-or-US names in each group with no relationship to market at all. This is the entire content of "k-means finds structure in the numbers you hand it", the structure is real (a genuine low-return/high-return split), but whether that structure has any economic meaning is a judgment call the algorithm never makes for you.

Worked example 2: PCA's sample-size trap: one month vs. four years of the same panel

This branch's synthetic-factor-panel dataset carries four factor columns (value, momentum, quality, low-volatility) for 60 tickers across 48 monthly snapshots, 2,880 rows total. Run PCA's explained-variance-ratio computation on the single month of 2022-07-01 (60 rows), then on the full pooled panel (all 2,880 rows), and compare.

Solution. Step 1: single month. Centering, covariance, and eigendecomposition on just the 60 tickers observed in July 2022 gives explained-variance ratios of 35.56%, 26.94%, 19.11%, 18.40% for the four components in descending order, a visible skew, with the leading component alone carrying more than a third of the total variance. Step 2: full panel. The identical four columns, the identical PCA mechanism, run on all 2,880 pooled rows instead, gives 26.41%, 24.84%, 24.68%, 24.07%, essentially even, indistinguishable from four genuinely uncorrelated factors. Step 3: name what changed. Nothing about the factors changed between Step 1 and Step 2, only the sample size did, from 60 observations to 2,880. Read it. A learner who ran only Step 1 would have a plausible, false story ready to tell a risk committee: "two of our four factors are quietly redundant, we can compress to three." The pooled computation shows that story was sampling noise from a single month, not a real relationship, exactly why DA2.02-e02's hidden tests grade both the small and the pooled case, and exactly why this module's honesty check is "recompute on more data before you believe a compression finding," not a one-line formula.

Worked example 3: Trailing momentum on a real NSE ticker (India)

Using NILAM's real daily closing-price history from nse-eod, compute the row-24 daily return, the 5-day trailing momentum, and the 20-day trailing momentum, using only trailing (never centered) rolling windows.

Solution. Step 1, the daily return. Row 24's close, divided by row 23's close, minus one, gives a daily return of −3.6141%, an ordinary, single-day pct_change() value. Step 2, the 5-day trailing momentum. The mean of the five most recent daily returns up to and including row 24 (rows 20 through 24) gives −0.17087%, a small, noisy, entirely plausible trailing average. Step 3, the 20-day trailing momentum. The mean of the 20 most recent daily returns up to and including row 24 gives −0.43131%. Step 4, check the NaN boundary. The first 5 rows of the mom_5 column are NaN (there are not yet 5 trailing returns to average), exactly as many as the window size, a useful, mechanical sanity check any trailing-window feature should pass. Read it. Every one of these three numbers is computable using only rows 24 and earlier, nothing about row 25 or beyond entered the arithmetic anywhere, which is the entire content of the as-of rule made concrete on real data, and exactly what DA2.02-e03's hidden tests check to the ninth decimal place.

Worked example 4: Look-ahead leakage, exposed: 99% vs. 50–53% on a real NSE and a real US ticker

Build a 5-day-ahead "did price rise" label on both NILAM (NSE) and BRPEAK (US). Build two candidate features for each: a leaky one (the mean of the same next five days' returns the label is built from) and an honest one (the trailing 5-day momentum from Worked Example 3's family, using only past returns). Grade raw directional accuracy for both.

Solution. Step 1, the leaky feature, both tickers. On NILAM, predicting "up" whenever the leaky feature is positive scores 99.19% accuracy against the label; on BRPEAK, 98.99%, both nearly mechanically perfect. Step 2, the honest feature, both tickers. The identical accuracy check, with the identical label, using only the honest trailing feature instead: 49.90% on NILAM, 52.73% on BRPEAK, both statistically indistinguishable from a coin flip. Step 3, name the single design decision that separates them. Both features are five-day means of daily returns; the only difference is which five days, relative to row t, they average, the next five, or the previous five. Read it. A roughly 46-to-49-percentage-point accuracy gap, on the exact same ticker, the exact same label, the exact same horizon, traced to one shift-direction decision, this is why "our model scores 99% on 5-day direction" should trigger an immediate leakage check, never a celebration, on any real desk. DA2.02-e04's hidden tests grade only the honest, ~50% version; the leaky number above exists purely so you recognize the smell instantly if you ever produce it by accident.

Worked example 5: Target leakage, exposed: 100% vs. 43%/37% on real fundamentals, India and US

Build a "did net income grow next fiscal year" label on both india-fundamentals and sec-fundamentals (fifteen tickers each, fiscal years 2022–2025). Build an honest feature (this year's own year-over-year growth, using only prior fiscal years already on record) and a leaky feature (next fiscal year's own year-over-year growth, computed directly). Grade both.

Solution. Step 1, the honest feature, both panels. Predicting "grows next year" whenever this year's own YoY growth is positive scores 43.33% on the India panel and 36.67% on the US panel, across the 30 fiscal-year rows per panel where both quantities are defined (each ticker loses its first fiscal year, with no prior year to compute a growth rate from, and its last, with no next year to form a label from). Step 2, the leaky feature, both panels. The same accuracy check, using next fiscal year's own realized YoY growth rate as the "feature" instead, scores a mechanically perfect 100.00% on both panels, and notice this version even has more gradeable rows (45 per panel, since it only loses each ticker's last fiscal year, not its first). Step 3, name why 100% is not a finding. grew_next_year is defined as next_net_income > net_income; the leaky feature's sign is (next_net_income - net_income) > 0, the identical comparison, computed in a different but numerically equivalent form, whenever net_income itself is positive (true for every row in both panels here). Read it. Nothing about this leak violates a chronological rule, the "feature" is honestly built from data belonging to a genuinely later fiscal year, exactly as the label is. The mistake is purely definitional: a feature that can be derived directly from the label's own arithmetic is not a predictor of the label, it is the label. DA2.02-e05's hidden tests grade only the honest ~40% version; hold the 100% number in your head as target leakage's specific signature, distinct from look-ahead leakage's signature in Worked Example 4.

Worked example 6: Leak-free, end to end: a Pipeline and a walk-forward split on a real factor panel

Fit the identical LinearRegression on the real synthetic-factor-panel (four factor columns predicting fwd_return, 2,880 rows, 48 months) two ways: once under a shuffled 5-fold KFold, once under a walk-forward TimeSeriesSplit with the same number of folds. Compare the mean R² each reports.

Solution. Step 1: shuffled. KFold(n_splits=5, shuffle=True, random_state=42) reports a mean R² across its five folds of 0.0262. Step 2: walk-forward. TimeSeriesSplit(n_splits=5) on the identical data, identical model, reports a mean R² of 0.0153. Step 3, the ratio. 0.0262 / 0.0153 ≈ 1.72×, the shuffled scheme overstates the walk-forward one by roughly seventy percent, on a completely honest feature set with no leakage anywhere in the feature-engineering step. Step 4: name the mechanism. A shuffled split can, and here does, place some training rows chronologically after some test rows; the model is being scored, in part, on its ability to predict a month it was partly trained on data from after. TimeSeriesSplit cannot do this by construction, training only ever grows forward in time, and testing always sits strictly after it. Read it. Both R² values are honestly small (this module never manufactures a spuriously huge walk-forward number to make a point), the lesson is not "walk-forward always gives a dramatically different number," it is that the direction of the gap is always the same: shuffled cross-validation on time-ordered data is optimistically biased, never pessimistically biased, which is exactly why it is the wrong default the moment your rows have a chronological order at all. DA2.02-e07 builds the split mechanism by hand; DA2.02-e08 is this exact comparison, graded.

Practice set

Work each problem fully before reading its solution. Numeric answers are exact unless stated otherwise.

P1 (guided). k-means, one round by hand. Trace k-means with k=2 on points {2, 3, 20, 21}, initial centroids {2, 21}. State the converged centroids.

Solution. Round 1 assign: {2,3} are closer to centroid 2; {20,21} are closer to centroid 21. Update: low centroid = mean(2,3) = 2.5; high centroid = mean(20,21) = 20.5. Re-assign against the new centroids: unchanged. Converged after one update: (2.5, 20.5).

P2. k-means can get stuck, a genuine local optimum. Six points {0, 1, 10, 11, 20, 21} form three obvious pairs. Run k=3 from two different initializations: (a) {0, 1, 2} (three centroids clustered near the first pair) and (b) {0, 10, 20} (one centroid near each true pair). State the converged clusters for each and compare their inertia (sum of squared distances to each point's own centroid).

Solution. (a) Converges to {0}, {1}, {10,11,20,21}: centroids (0, 1, 15.5); inertia = 0 + 0 + (30.25+20.25+20.25+30.25) = 101. (b) Converges to the natural grouping {0,1}, {10,11}, {20,21}: centroids (0.5, 10.5, 20.5); inertia = 0.25×6 = 1.5. Both are genuinely converged, stable fixed points of the same algorithm on the same data, initialization (a) simply reaches a dramatically worse local optimum, roughly 67 times the inertia of initialization (b), which is exactly why scikit-learn's k-means++ default spreads its starting centroids out rather than risking two or three landing close together.

P3. Why standardize? Using this module's own real 30-ticker figures: std(annualized return) ≈ 20.22 percentage points, std(annualized volatility) ≈ 0.925 percentage points, what is the approximate scale ratio, and what does it mean for unscaled k-means?

Solution. 20.22 / 0.925 ≈ 21.85×. An unscaled Euclidean distance between two tickers is decided almost entirely by their return difference; volatility differences, being roughly 22 times smaller in absolute scale, barely register, meaning "clusters" found without standardizing first are really just return-sorted groups wearing a clustering label.

P4. Elbow method, by eye. A k-means run reports inertia of 840 at k=1, 210 at k=2, 90 at k=3, 78 at k=4, 71 at k=5, 68 at k=6. Where is the elbow, and what k would you choose?

Solution. The drop from k=1→2 (840→210) and k=2→3 (210→90) is large; the drop from k=3→4 onward (90→78→71→68) is small and flattening. The elbow sits at k=3, additional clusters beyond that buy only marginal inertia reduction, the visual/numeric signature the elbow method is built to catch.

P5. PCA on a toy 3-column set. Compute explained_variance_ratio_ for [[1,0,0],[-1,0,0],[0,2,0],[0,-2,0],[0,0,0.5],[0,0,-0.5]].

Solution. Matches DA2.02-e02's own hidden test: [0.7619, 0.1905, 0.0476] (76.19%, 19.05%, 4.76%), the three columns' variances scale as 1 : 4 : 0.25, and PCA (which for already axis-aligned, uncorrelated columns simply re-sorts by variance) recovers exactly that ratio.

P6. PCA sample-size honesty check, restated numerically. This module's single-month PCA on the factor panel gave ratios 35.56/26.94/19.11/18.40%; the full pooled panel gave 26.41/24.84/24.68/24.07%. What is the largest single ratio's change between the two, and what does that change mean?

Solution. The first component's ratio fell from 35.56% to 26.41%, a drop of about 9.15 percentage points, nearly a third of its original value. That drop is the signature of sampling noise evaporating: the smaller sample (60 observations) manufactured an apparent skew that the larger, more representative sample (2,880 observations) does not support.

P7. The as-of rule, applied. A feature is defined as df['close'].rolling(5, center=True).mean(). Is this feature safe to use for a model predicting a decision made at row t? Why or why not?

Solution. No. A centered window at row t includes rows on both sides of t, meaning roughly half its inputs come from after row t, a direct as-of-rule violation the moment this column is handed to a model as a feature, even though the identical computation is entirely legitimate for smoothing a finished chart you already hold in full.

P8. Groupby-shift, spot the bug. A panel sorted by (ticker, date) computes df['lag_close'] = df['close'].shift(1) with no groupby. Ticker A's last row has close=150; ticker B's first row (immediately after, in sort order) gets what lag_close value, and why is this wrong?

Solution. Ticker B's first row's lag_close becomes 150 (ticker A's last close) because the bare .shift(1) operates on the raw concatenated row order with no awareness of the ticker boundary. It is wrong because ticker B's "lag" should describe ticker B's own prior value (or be NaN, since it has none yet), not an unrelated company's most recent price. The fix is df.groupby('ticker')['close'].shift(1).

P9. Look-ahead leakage, numerically. This module's real-data comparison found a leaky 5-day-direction feature scoring 99.19% (NILAM) and an honest one scoring 49.90%. Express the honest score's distance from a coin flip, and state what a "genuinely predictive" honest score would need to clear to be worth investigating further.

Solution. 49.90% is 0.10 percentage points below the 50% coin-flip baseline, statistically indistinguishable from chance on a single ticker's worth of data. A genuinely interesting honest score would need to clear 50% by a margin that survives its own statistical significance check (and, per this module's discipline, a walk-forward re-validation) before being treated as real signal rather than noise.

P10. Target leakage, spot it in a new scenario. A colleague builds a feature called revenue_growth_this_year defined as (next_year_revenue - this_year_revenue) / this_year_revenue, to predict whether revenue grows next year. Is this feature safe? Why or why not?

Solution. No, this is target leakage, structurally identical to this module's fundamentals worked example. The feature's numerator and denominator are built entirely from next_year_revenue, the exact same quantity the label (did revenue grow next year) is defined from, the "feature" is a renamed copy of the label's own arithmetic, not an independent predictor, regardless of the fact that both genuinely reference a later fiscal year (which is not, by itself, what makes it leakage).

P11. Target vs. look-ahead, classify. For each scenario, name the leakage type (if any): (a) a feature computed as df.rolling(10, center=True).mean() used to predict a same-day event; (b) a feature that is literally label * 0.99 + noise; (c) a feature computed as df.groupby('ticker')['ret'].shift(1).rolling(5).mean() predicting next week's return; (d) a hospital-readmission model using "was the patient prescribed post-discharge medication" as a feature, when that prescription is only ever written for patients the hospital already expects to keep monitoring closely.

Solution. (a) Look-ahead leakage: the centered window reaches into future rows relative to the prediction point. (b) Target leakage, the feature is algebraically almost the label itself, no time-ordering violation required. (c) Not leakage, a correctly grouped, trailing rolling feature, exactly the safe pattern this module teaches. (d) Target leakage, the prescription decision is itself downstream of (informed by) the very risk the model is trying to predict, making the feature circular with the label in substance even though it is not calendar-future information.

P12. Preprocessing leak, quantify the mechanism. A learner fits StandardScaler on 100 rows (80 train + 20 test combined), getting a mean of 50.0. Fit only on the 80 train rows, the mean is 48.5. If the 20 test rows have a true mean of 56.0, explain numerically why the combined-fit mean differs from the train-only mean.

Solution. The combined mean is a weighted average: (80 × 48.5 + 20 × 56.0) / 100 = (3880 + 1120) / 100 = 50.0, exactly the reported combined figure. Every training row is then standardized using a mean that is pulled 1.5 points toward the test set's own value, meaning the test set's own data literally shaped the numbers the model trains on, the mechanical definition of a preprocessing-statistics leak, quantified.

P13. Pipeline discipline, spot the leak in code. A colleague writes: scaler = StandardScaler(); scaler.fit(np.vstack([X_train, X_test])); X_train_s = scaler.transform(X_train); X_test_s = scaler.transform(X_test). Is this leak-free? Rewrite it so it is.

Solution. Not leak-free: scaler.fit(...) is called on the vertical stack of train and test combined. The fix: scaler.fit(X_train) (train alone), then scaler.transform(X_train) and scaler.transform(X_test) separately, or, more robustly, wrap it in Pipeline([('scale', StandardScaler())]).fit(X_train) exactly as DA2.02-e06 does, so the correct call order is structural rather than a rule to remember.

P14. Walk-forward split, by hand. Using TimeSeriesSplit's expanding-window logic, compute the train/test index split for n_samples=12, n_splits=3.

Solution. Matches DA2.02-e07's own hidden test exactly: ([0,1,2],[3,4,5]), ([0,1,2,3,4,5],[6,7,8]), ([0,1,2,3,4,5,6,7,8],[9,10,11]), each fold's training set is every earlier index, its test set the next equal-sized block, and training only ever grows.

P15. Shuffled vs. walk-forward, quantify the direction of the bias. This module's factor-panel comparison found shuffled 5-fold mean R² = 0.0262 and walk-forward mean R² = 0.0153. If you only had time to run one of the two schemes before reporting a result to a risk committee, which would you run, and why?

Solution. Walk-forward. Not because it is more conservative for its own sake, but because it is the only one of the two that cannot have secretly trained on data chronologically after what it is tested on, the shuffled number's optimism (here, roughly 1.72× the walk-forward figure) is a structural property of the split, not evidence the model is actually 1.72× better. A number a committee will act on should be the one that cannot lie in this specific, well-understood direction.

P16. The four-point checklist, applied. A junior analyst reports: "my fundamentals model predicting next-year growth scores 94% accuracy, and one feature (next_period_indicator) drives almost the entire result; I haven't re-checked it under walk-forward CV yet, and I can't immediately say the exact as-of row every feature uses." Apply this module's four-point "too good to be true" checklist and state your recommendation.

Solution. Running the checklist: (1) Is the metric implausibly high? Yes: 94% on a fundamentals growth call, when this module's own honest, real-data ceiling on the identical task type was 43.33%/36.67%, is a strong alarm. (2) Does one feature dominate? Yes, explicitly stated, a second alarm. (3) Does it survive walk-forward re-checking? Unknown, not yet done. (4) Can every feature's as-of row be named? Unknown, not yet done. Recommendation: do not report or act on this result yet. Two of four checklist items are already tripped, and the other two are simply unverified rather than passed, the single feature name, next_period_indicator, is itself a strong hint of target leakage (a feature that references "next period" directly, exactly this module's Worked Example 5 pattern), and it should be the very first thing investigated before any further modeling work continues.

Applied mini-project

Build a leak-free unsupervised-plus-feature-engineering research note, end to end.

This project turns the module's five load-bearing ideas (clustering, dimensionality reduction, safe feature engineering, leak-free pipelines, walk-forward validation) into one hands-on artifact, using functions from code-packs/DA2.02.json (module id DA2.02) as your toolkit. It is a self-scored worksheet: the code pack's own exercises are what the mastery gate grades; this project is where you assemble them into a single, realistic mini-decision.

Do this, in order:

Build or select a panel. Use one of this branch's own datasets (nse-eod, us-eod, synthetic-factor-panel, india-fundamentals, sec-fundamentals) or construct your own seeded synthetic panel of at least 500 rows across at least 10 tickers/entities spanning at least 2 fiscal periods or 60 trading days, note your dataset and, if synthetic, your seed (R3).
Cluster and reduce. Compute at least two candidate numeric features (e.g., annualized return and volatility, or two factor columns), standardize them, run k-means across k=2..6 and report the elbow-chosen k with your inertia numbers; separately run PCA on at least 3 numeric columns and report explained_variance_ratio_, re-checking it against a larger sample if your first slice is small (per this module's PCA honesty check).
Engineer at least one trailing, leak-free feature (a rolling momentum, a grouped lag, or a fiscal-year-over-year growth rate) obeying the as-of rule, and demonstrate (deliberately, then fixed) one look-ahead OR one target leakage on your own data: build the leaky version, report its suspiciously high score, then report the honest version's real score.
Build a leak-free Pipeline (StandardScaler, or a ColumnTransformer if you have mixed column types) fit on a train split only, and evaluate a simple model under both a shuffled KFold and a walk-forward TimeSeriesSplit, reporting both mean scores and their ratio.
Write the memo. In 200–350 words, report: your chosen k and why, your PCA honesty-check result, your leaky-vs-honest numbers for whichever leakage type you demonstrated, and your shuffled-vs-walk-forward gap, and state, in one sentence, which of your own numbers would have made you suspicious first if you had not already known to look.

Deliverable and self-check. One worksheet: your dataset/seed, the clustering and PCA numbers from step 2, the leaky-vs-honest comparison from step 3 with the leakage type named, both CV mean scores and their ratio from step 4, and the memo. Score yourself 2 points each for: (a) the clustering/PCA step reports real inertia and explained_variance_ratio_ numbers with an honest k choice and, where applicable, a sample-size honesty re-check; (b) the leaky-vs-honest demonstration shows a real, numerically dramatic gap on your own data, with the leaky mechanism named precisely (look-ahead or target); (c) the Pipeline is genuinely fit on a train split alone (spot-check: does your .fit() call ever see a test row?); (d) both CV schemes are run and their gap is reported and correctly attributed to which rows trained on which; (e) the memo's one-sentence "what would have made me suspicious first" is specific to your own numbers, not generic advice. Pass at 8/10, no zero on any item.

Lab: open the DA2.02 code pack in this node's Code tab (code-packs/DA2.02.json) and clear all eight exercises (DA2.02-e01 through DA2.02-e08), each ships a structured fallback, so the lab stays completable fully offline, with zero runtime.

Reading & resources

Everything below is an anchor to build genuine understanding from and verify claims against, never a substitute for hand-tracing this module's own worked examples and code pack. Free unless marked; level tags reflect the ramp from intuition to mechanism.

The core spine (start here):

scikit-learn's own User Guide. "Clustering" (sklearn.cluster), "Decomposition" (sklearn.decomposition, PCA), and "Pipelines and composite estimators", the primary, authoritative source for every API this module wraps; read the actual parameter lists for KMeans, PCA, and Pipeline before trusting any secondary summary, including this one. [Free ✓] [Beginner–Intermediate ▲]
Gareth James et al., An Introduction to Statistical Learning (ISLR, free PDF plus R/Python labs), its unsupervised-learning chapter (clustering, PCA) and resampling chapter (cross-validation) are the standard, rigorous companion to this module's mechanism-first treatment. [Free ✓] [Beginner–Intermediate ▲]
scikit-learn's own User Guide. "Cross-validation: evaluating estimator performance," specifically the TimeSeriesSplit section, the primary source for the exact walk-forward mechanism DA2.02-e07/e08 grade. [Free ✓] [Intermediate ▲]

Foundational and practical:

Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, its clustering, dimensionality-reduction, and pipeline chapters are an excellent, code-first complement to this module, with more worked API detail than this document has room for. [Paid $] [Beginner–Intermediate ▲]
Kaggle's own "Data Leakage" micro-lesson, a short, free, widely cited treatment of exactly the two leakage diseases this module names (there called "target leakage" and "train-test contamination"), from a different angle and with different datasets. [Free ✓] [Beginner ▲]
scikit-learn's own User Guide ("Preprocessing data" and "Compose" (ColumnTransformer)) the primary reference for extending this module's single-column Pipeline example to a real, mixed-type feature table. [Free ✓] [Intermediate ▲]

Practical and perishable (treat as dated):

sklearn.cluster.KMeans's own current docstring/changelog for its default init and n_init behavior, scikit-learn has changed k-means's own defaults across versions (n_init='auto' is comparatively recent); verify against your installed version rather than assuming this module's description of "k-means++ by default" holds for every release. [Free ✓] [Intermediate ▲] (implementation detail: check against your installed version)
Your own re-run of this module's own headline numbers (the 21.85× scaling ratio, the single-month-vs-pooled PCA ratios, the 99%-vs-50% look-ahead gap, the 100%-vs-43%/37% target-leakage gap, the shuffled-vs-walk-forward R² gap) against the shipped datasets, to build the habit of verifying a module's own claims rather than accepting them, the code pack's hidden tests already do this for you mechanically; doing it once yourself by hand is worth the hour. [Free ✓] [Beginner ▲]

Do this, not just read: on your own machine (or this node's Code tab), take any panel you have (even a single ticker's daily prices) and deliberately build one leaky feature and one honest feature for the same prediction task, the way Worked Examples 4 and 5 do, and watch the accuracy gap appear on data nobody chose for you. A leak you found yourself, once, is worth more than reading about ten leaks other people found.

Teach it back & journal

Feynman prompt. Explain to a colleague who has only ever fit one model on one clean train/test split why the exact same code, run on data with a "feature" removed and replaced by an honest one, can see its accuracy fall from 99% to 50% with nothing else changing, using the idea that a feature's own construction can secretly reach across the same boundary a train/test split is supposed to protect, and no error message will ever tell you it happened. Then explain, in the same breath, why a fundamentals model that scores a perfect 100% predicting growth is not a breakthrough but an alarm, using the idea that a feature built directly from the label's own defining arithmetic is not a predictor, it is the label wearing a different column name, and a model cannot "learn" what has simply been handed back to it. If your colleague can, unprompted, name a suspiciously perfect score as the first thing to investigate rather than the first thing to celebrate, and can describe, in their own words, the difference between a leak that crosses a chronological boundary and a leak that is definitional regardless of timing, you have installed the mechanism this module exists to teach.

Journal prompt. For the next real modeling task you touch (at work, in a personal project, or in this program's own code pack) before you look at a single accuracy or R² number, write down: for every feature, what is its exact as-of row, and can you prove nothing in its construction reaches past that row? Is any feature computable directly from the label's own definition, even approximately? Was every preprocessing step (a scaler, an encoder, a PCA) fit on the training split alone? And if the data has any natural order (time, or a sequence of related events), did you validate with a scheme that respects that order, or a shuffled one that assumes it does not matter? Then look at the number. If it is not close to what this module's own honest numbers suggest is realistic for a task this hard, a coin flip on 5-day direction, a sub-50% ceiling on fundamentals-based growth calls, a low-single-digit-percent R² on cross-sectional factor panels, treat the gap as a question to answer, not a result to report. Close with a paragraph on this: the analyst who has personally watched a 99% accuracy collapse to 50% by changing one shift's direction does not get fooled by the next one, and that immunity, not any single rule memorized from this document, is the actual, durable skill this module was built to install.

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.

ML II: Unsupervised, Feature Engineering & Model Discipline