ML I: Supervised Learning

Branch: Data Science & ML (quant-ds) · Node DA2.01 · ~22 focused hours · Understanding gate ≥ 85%

QM1.03 already gave you the whole shape of this problem, in words and in one unforgettable table: fit a model of growing flexibility, watch its training error fall in a straight line while its honest, held-out error falls, bottoms out, and then rises, and know that the number at the bottom of that second curve, not the number at the bottom of the first, is the only one that means anything. That table is correct, it is durable, and by itself it will not predict a single stock's return, flag a single loan's default risk, or ship a single line of code. Somebody has to actually fit the model, actually hold out the data honestly, actually compute the numbers that go into that table, and actually decide which of a dozen candidate models and a dozen candidate metrics is the right one for the actual question being asked. That somebody, for the rest of this program, is you, and this node is where you start.

Here is the whole module in one sentence you will spend twenty-two hours earning the right to believe: a supervised model is a function you fit on data with known answers, and the only honest measure of whether it is any good is how it performs on data it never saw during fitting, every technique in this node exists to build that function, to guard that honesty, and to read the number it produces correctly. Everything else, which model family, which hyperparameter, which metric, is detail sitting on top of that one sentence. Get the sentence wrong (grade a model on the same rows it was trained on, or grade it with a metric that flatters a useless model) and every downstream number is decoration on a lie. Get it right and the rest of this node, and the two nodes after it, are a sequence of increasingly capable tools built on a foundation that never moves.

Concretely, by the end of these twenty-two hours you will fit and honestly score a regression model and a classification model using scikit-learn's one shared fit/predict/score contract; split data into train, validation, and test sets the way a real research pipeline does, with zero index leaking between them; run k-fold cross-validation to pick a model's complexity without ever touching the test set to do it; fit a decision tree and watch, in numbers you compute yourself, the exact bias-variance shape QM1.03 described, a rigid tree doing badly everywhere, an unconstrained one memorizing training noise and doing worse than a shallower sibling on data it hasn't seen; fit a random forest and watch an ensemble of noisy, overfit trees average into something that generalizes far better than any one of them alone; and, the skill most confidently misapplied in the industry, choose the right metric for a task, knowing exactly when accuracy is a trap, when RMSE beats MAE, and when a bare "it's 95% accurate" is actually the least informative sentence anyone in the room could say. Your code pack, DA2.01 (nine exercises, DA2.01-e01 through DA2.01-e09), is where every one of these becomes a function you write, a hidden test defeats a shortcut on, and a hidden test from the other market defeats a coincidence on.

Learning objectives

By the end you can:

1. State the one-sentence discipline that governs everything in this node (fit on data with known answers, judge only on data the model never touched during fitting) and explain, in mechanism, why grading a model on its own training rows is not merely optimistic but actively meaningless.
2. Fit and score a linear regression using scikit-learn's fit/predict contract, and compute its two standard regression metrics (RMSE and R²) correctly, on the test set only, by hand from sklearn.metrics (not by re-deriving least squares, which is QM1.02's job).
3. Build a leakage-free three-way train/validation/test split in code (sklearn.model_selection.train_test_split, applied correctly twice) that is deterministic under a fixed seed and exactly partitions every row exactly once.
4. Run k-fold cross-validation (KFold + cross_val_score) to score a candidate model's complexity using only the training set, and correctly read neg_mean_squared_error's sign convention.
5. Fit a decision tree for both regression and classification, control its capacity with max_depth, and connect a max_depth sweep's train/test score table directly to QM1.03's bias-variance shape, a concept you already hold, now a number you compute.
6. Fit a random forest, explain the bagging mechanism (bootstrap resampling + averaging) that lets an ensemble of individually overfit trees generalize better than any single one, and demonstrate this numerically on held-out data.
7. Fit a logistic-regression or tree-based classifier and compute the full metric set that matters for it (accuracy, precision, recall, F1, ROC-AUC) and explain precisely why accuracy alone is actively misleading on an imbalanced target.
8. Choose the right primary metric for a stated task (regression versus classification, and (within classification) balanced versus imbalanced classes), and defend the choice from first principles rather than habit.
9. Diagnose underfitting versus overfitting from a train/test (or train/CV) score table alone, identifying the best-generalizing complexity level and the point at which the train-test gap signals a model has started memorizing noise.
10. (Productivity objective: R10 duality.) Use an AI copilot to accelerate the unglamorous parts of a supervised-learning workflow, drafting a first-pass sklearn pipeline, or explaining a cryptic warning, while applying the three-law Copilot Discipline: understand the by-hand version first, then accelerate, then always review every line and verify no API was hallucinated before trusting a single number it produced.

The duality, stated once (R10). This module carries two objectives at once, as every module in this program does. The understanding objective (items 1–9) is what the mastery gate rewards; you earn the pass by fitting real models, computing real metrics, and reading them correctly, not by having watched someone else do it. The productivity objective (item 10) is the payoff you keep afterward: knowing when a copilot can safely draft the unglamorous scaffolding of a modeling script, and knowing exactly what "review it first" has to mean before a metric it computed enters a report. A tool can never buy you a pass on the gate.

Prerequisites & connections

Builds on. DA1.01 (NumPy) and DA1.02 (pandas I), every function in this node takes plain lists or arrays of features and targets, and in real work those arrays come out of a pandas DataFrame you already know how to build and slice. DA1.04 (EDA & Visualization) is where you would have first plotted a feature against a target and felt whether a relationship looked linear, clustered, or noisy, the instinct this node turns into a number. DA1.05 (Statistics in Code) implements the descriptive and inferential statistics that sit conceptually one level below everything here.

The three-way boundary this node sits inside (read this once, carefully). Three modules touch "regression," and each owns a genuinely different question about it, so that none of them re-teaches the other:

Cross-link every derivation back to its owner; this node re-derives no theory from either QM module, it builds the code that operationalizes QM1.03's concepts and complements, never replaces, QM1.02's inferential lens (C-3 binding boundary).

Feeds forward. DA2.02 (ML II. Unsupervised, Feature Engineering & Model Discipline) is the very next node and assumes this one's fit/predict/score fluency as a given; it is also where the deeper leakage story lives, target leakage, look-ahead leakage in engineered features, and the fully leak-free Pipeline/ColumnTransformer machinery this node only gestures at. If you finish this node wanting to ask "but what if my feature itself was built using future information?", that question is correct, and DA2.02 is where it gets answered; this node's job stops at the honest train/validate/test split and the honest metric. DA2.03 (the branch's crown: Deep Learning, NLP & Time-Series Forecasting) reuses every evaluation habit built here, the train/test discipline, the metric-choice discipline, and layers a from-scratch neural network and honest time-series backtesting on top. Beyond quant-ds, QD1.02 (Signal & Factor Construction) evaluates predictive signals with its own specialized metrics (IC, IR, decay) that are cousins of, but not identical to, the RMSE/accuracy families taught here; and QD1.03's no_lookahead harness is the production-grade descendant of the chronological-split warning QM1.03 raised and this node's train/test split makes literal.

Core teaching content

The one discipline everything else in this node serves

Before a single line of scikit-learn, hold the sentence that makes every technique below make sense: a model's score on the data it was fit to tells you almost nothing about how it will perform on data it hasn't seen, and the entire craft of supervised learning is building the machinery that measures the second thing instead of accidentally reporting the first. A sufficiently flexible model (a deep enough tree, a high enough polynomial degree, a large enough neural network) can, in the limit, fit its training data perfectly: zero error, 100% accuracy, R² of 1.0. That number is real and it is worthless, because a model that has memorized its training rows has not learned the pattern those rows came from; it has learned the rows. The only way to find out whether it learned the pattern is to ask it about rows it never saw, which is exactly what a held-out test set is for. Every section below (the train/validate/test split, cross-validation, the tree-depth sweep, the tree-versus-forest comparison) is a different angle on this one idea, made mechanical.

The shared contract: fit, predict, score

Every supervised model in scikit-learn, regression or classification, linear or tree-based or ensemble, exposes the identical three-method shape, and learning it once means you already know the surface of every model this node touches. model = SomeModel(...) constructs an unfit model holding only its hyperparameters. model.fit(X_train, y_train) is where the actual learning happens, X_train is a 2-D structure, one row per observation and one column per feature (a plain list-of-lists, a NumPy array, or a pandas DataFrame all work identically), y_train a 1-D structure of targets, and after this call the model's internal parameters (a line's slope and intercept, a tree's splits, a forest's hundred trees) are set and frozen. model.predict(X_new) takes any 2-D structure shaped like X_train (same number of columns) and returns predictions, a continuous number per row for a regressor, a class label per row for a classifier. model.score(X, y) is a convenience that predicts internally and returns one default number: R² for a regressor, accuracy for a classifier, useful for a first look, and, as you're about to learn in depth, frequently the wrong number to actually trust once you understand what it hides.

One more shared method matters for classification specifically: model.predict_proba(X) returns, for each row, the model's estimated probability of each class, not just its final label. predict is predict_proba with a threshold (0.5, by default) already applied and thrown away, and, as you'll see in the metrics section, throwing that probability away before it's needed is exactly what makes a metric like ROC-AUC (which needs the probability) impossible to compute from predict output alone. Keep both outputs whenever you can; you never know in advance which one your evaluation will need.

Regression: fitting a line, scoring it honestly

sklearn.linear_model.LinearRegression fits ordinary least squares, the same line QM1.02 derives the statistical theory of, but here the question is purely predictive: given X_test, how close do the predictions land to y_test? Two metrics answer that, and you should carry both, because they answer subtly different questions. RMSE (root-mean-squared-error: sqrt(mean_squared_error(y_test, preds))) is in the same units as the target, a revenue-prediction RMSE of 1.30 (₹ crore, say) means "on average, off by about 1.30 crore, with large errors penalized extra hard by the squaring." R² (r2_score(y_test, preds)) is unit-free: it is 1 minus the ratio of your model's squared error to the squared error of the dumbest possible baseline (always predicting the mean of y_test). An R² of 1.0 is a perfect fit; an R² of 0.0 means your model is exactly as good as guessing the mean every time; and, the fact every learner is surprised by exactly once: R² can be negative, meaning your model is worse than that dumb baseline, which happens more often in the real world than beginners expect (Worked Example 4 shows it happening to an entire model family). DA2.01-e01 is exactly this: fit on train, predict and score on test, return (rmse, r2), and its hidden tests include a negative-slope relationship and a genuine two-feature regression, specifically so that a solution which silently assumes "slope is always positive" or "there's always exactly one feature" fails loudly.

Classification: predicting a label, and the probability hiding behind it

sklearn.linear_model.LogisticRegression and the tree-based classifiers below solve a structurally different problem: the target is discrete (most often binary: 0/1, down/up, no-default/default), and the model's real output is a probability of the positive class, with predict collapsing that probability to a hard label at a threshold. This matters immediately for finance: an "up-move probability of 0.51" and an "up-move probability of 0.98" both round to the same predict output (class 1), but they are wildly different statements about confidence, and a metric computed only from hard labels (accuracy, precision, recall, F1) cannot see the difference, only a probability-aware metric like ROC-AUC can. Keep predict_proba's output on hand whenever the downstream question might care how sure the model was, not just what it guessed.

Trees: splitting on the feature that separates the target best

An sklearn.tree.DecisionTreeRegressor or DecisionTreeClassifier builds a model by repeatedly asking "which single feature, and which single threshold on it, best separates the data into two groups with more similar targets than the group had before the split?", then recursing on each half. The single hyperparameter you will spend the most time reasoning about is max_depth: how many times the tree is allowed to split before it must stop and predict a leaf's average (regression) or majority class (classification). max_depth=1 (a "stump") can only ask one question of the data, total, often too rigid to capture a real pattern. max_depth=None lets the tree keep splitting until every leaf is pure (or as pure as it can get), which, given enough depth relative to the data, means the tree can eventually carve out a leaf for nearly every individual training row, a memorization machine, not a generalizer. max_depth is, mechanically, the exact same complexity knob that polynomial degree was in QM1.03's Worked Example 7; it is a different model family expressing the identical bias-variance trade-off. DA2.01-e06 fits the same tree at three depths on one small, deliberately noisy dataset and has you watch the actual numbers move: depth 1 trains at 73.3% and tests at 60.0% (both mediocre, underfit); depth 2 trains at 86.7% and tests at 73.3% (the best-generalizing depth on this data, despite not having the highest training score); depth 3 and beyond train at a perfect 100% while test accuracy falls to 63.3%, worse than depth 2's test score, even though depth 3's training score is obviously "better." That gap, a rising train score paired with a falling test score, is what overfitting looks like as a number, and it is the exact shape QM1.03 asked you to recognize conceptually, now something you produced yourself.

Ensembles: averaging away the noise a single tree memorizes

A single unconstrained decision tree, given enough depth, will fit training noise as eagerly as it fits real signal, it has no way to tell the two apart. An sklearn.ensemble.RandomForestRegressor (or RandomForestClassifier) fixes this not by constraining any one tree, but by fitting many trees, fifty, a hundred, more, each on an independent bootstrap resample of the training rows (sampling with replacement) and, at each split, considering only a random subset of the features. Each individual tree in the forest is still an overfit, noisy model; the trick is that each tree memorizes different noise (different resample, different feature subsets), so when you average all their predictions together, the genuine signal, which every tree, however noisy, is still picking up, reinforces across trees, while the noise, being different and uncorrelated from tree to tree, washes out. DA2.01-e07 puts this head-to-head: on a noisy, many-weak-feature panel shaped like a real cross-sectional factor dataset, a single unconstrained tree scores an R² of −0.10 on held-out data, worse than just predicting the mean, while a fifty-tree forest on the identical data scores +0.49. Same features, same target, same training rows; the only difference is whether you trust one memorizing tree or average a hundred of them. This result holds, with different exact numbers, across every random seed the exercise tries, it is not a lucky draw.

{
  "skill": "Drafting a first-pass scikit-learn model-comparison script, verified before trusting a single number",
  "use": "Ask a coding assistant to scaffold boilerplate you already know how to write by hand -- importing the right sklearn classes, looping over a few candidate models or hyperparameters, and printing a comparison table -- so you spend your own time on the modeling decision, not re-typing import statements.",
  "tools": ["Coding assistants -- Claude, ChatGPT, GitHub Copilot", "IDE copilots integrated into a notebook or editor"],
  "prompt": "Write a short Python script using scikit-learn that fits a DecisionTreeRegressor and a RandomForestRegressor with n_estimators=50 on the same X_train/y_train, scores both on X_test/y_test with r2_score, and prints a small comparison table. Use random_state=<SEED> for both models so the result is reproducible.",
  "verify": "Read every import and every constructor argument before running it -- a hallucinated argument name (e.g. a parameter that doesn't exist in your installed sklearn version) fails loudly, but a SUBTLY wrong one (fitting on X_test by mistake, or scoring train instead of test) fails silently and just hands you an inflated, meaningless number. Re-derive the expected direction of the result yourself first (an ensemble should generalize at least as well as, usually better than, one unconstrained tree) and treat a result that contradicts that expectation as a bug to find, not a surprising discovery to report.",
  "diy": "You must already be able to write fit/predict/score by hand, unaided, for both a tree and a forest -- exactly what this section and DA2.01-e06/e07 build -- before a generated script is something you can actually review rather than merely run.",
  "market": "US"
}

Train, validate, test: the discipline in code

QM1.03 named the standard discipline; here is the code that enforces it. A training set is what a model is fit on. A validation set is used to tune choices (which max_depth, which model family) without ever touching the test set to make that decision. A test set is touched exactly once, at the very end, for a single honest read of how the finished, fully-decided model actually performs, and the moment you adjust anything in response to a test-set number and re-check it, that test set has become a second validation set and its number no longer means what you think it means (QM1.03's Common Mistake #10, worth re-reading before you touch a real test set). In code, sklearn.model_selection.train_test_split(indices, test_size=..., random_state=seed) performs one split; to get three sets, you call it twice, once to peel off the test set, then again on what remains to separate train from validation, and the second call's test_size must be rescaled relative to what's left, not the original total, or your validation set silently ends up the wrong size. DA2.01-e02 builds exactly this, and its hidden tests check something beginners rarely think to check themselves: that the three returned sets are a true partition (every original index appears in exactly one of the three, none missing, none duplicated) and that the same seed reproduces a byte-identical split every time, the same reproducibility discipline CS2.01 built for a shell environment, now applied to a data split.

k-fold cross-validation: squeezing more honest signal from a limited sample

A single train/validation split is a noisy read. QM1.03's own Worked Example 7 flagged this directly: the single-split test-MSE minimum landed at polynomial degree 4, while 5-fold cross-validation, run entirely inside the training data, correctly recovered the true degree, 2. K-fold cross-validation rotates which slice of the training data plays "held out" k times (commonly 5 or 10) and averages the result, using every row for both training and validation across the k rotations without ever letting a row validate the same fold it trained in. In code: sklearn.model_selection.KFold(n_splits=k, shuffle=True, random_state=seed) defines the rotation, and sklearn.model_selection.cross_val_score(model, X, y, cv=kf, scoring='neg_mean_squared_error') runs it, returning k scores to average. Read the sign convention carefully, because it trips up almost everyone the first time: scikit-learn's cross-validation scoring convention is "higher is always better," so an error metric like MSE is reported negated, you must flip the sign back (-scores.mean()) to get an interpretable positive MSE. DA2.01-e03 rebuilds QM1.03's Worked Example 7 with a decision tree instead of a polynomial (sweeping max_depth instead of degree) on a small, deliberately noisy factor-style panel (two genuinely predictive features carrying a mild interaction, three pure-noise features alongside them), and the result echoes the concept exactly: max_depth=1 (too rigid to capture the interaction) cross-validates worse than max_depth=2; max_depth=2 is the winner; and letting the tree grow unbounded cross-validates worse again, because by then it is fitting the noise features along with the real signal.

Diagnosing overfitting from numbers, not a feeling

You now have everything you need to turn "I have a bad feeling this model is overfit" into an actual, defensible number. Given a table of training error and test (or CV) error at increasing complexity levels, two facts read off it directly: (1) the complexity level with the lowest test/CV error is the one you should actually use, never the one with the lowest training error, which by construction favors the most flexible model every time; and (2) the gap between test error and training error at each level (test_error − train_error) is a direct measurement of overfitting, small and stable at a well-chosen complexity, and growing sharply once the model starts memorizing. DA2.01-e04's diagnose_complexity(train_errors, test_errors) computes exactly these two things, and its hidden tests reuse QM1.03's own Worked Example 7 numbers verbatim: fed the single-split train/test MSE table, it correctly identifies degree 4 (index 3) as the split-minimum, with the single largest train-test gap sitting at degree 10, the same "blow-up" region QM1.03's prose described; fed the train/CV-MSE table instead, it correctly identifies degree 2 (index 1), cross-validation recovering the true generating degree, exactly as QM1.03 reported. Nothing here re-derives why that happens (QM1.03's job); this node just builds the function that reads the table.

Choosing the right metric: the decision most often made on autopilot

Every technique above assumed you already knew which number to look at. You don't, automatically, and getting this wrong is one of the most common, most consequential mistakes in applied machine learning. For regression: RMSE and MAE (mean absolute error) both report error in the target's own units, but RMSE squares each error before averaging, so it penalizes a few large misses far more than many small ones, the right choice when a single catastrophic miss is much worse than being consistently a little off (most risk and P&L contexts); MAE treats every unit of error identically and is more robust when your data has a few extreme outliers you don't want to dominate the score. For classification, the far more dangerous trap: accuracy is only informative when the classes are roughly balanced. The moment one class is rare, a market-crash day, a loan default, a genuine up-move on a mostly-flat name, a model that never once predicts the rare class can still score a deceptively high accuracy, simply because being right about the common class most of the time is enough. DA2.01-e05 makes this unmissable with real numbers: on an illustrative 20-day window where a genuine up-move happens on exactly 1 day, a model that predicts "no up-move" every single day scores 95% accuracy, precision, recall, and F1 are all 0.0, and its ROC-AUC is exactly 0.5 (no better than a coin flip). A second model on the identical data that actually catches the real up-move (with one false alarm along the way) scores the same 95% accuracy, and yet has recall 1.0 and ROC-AUC 1.0. Two models, identical accuracy, one useless and one excellent, accuracy alone cannot tell them apart; you must reach for precision/recall/F1 (which need a hard threshold) or ROC-AUC (which needs the probability, threshold-free) the moment a class is imbalanced. DA2.01-e08 turns the decision itself into a callable: given a task and, for classification, the positive class's rate, it returns the metric you should actually be looking at.

{
  "skill": "Interpreting a cryptic scikit-learn warning or an unexpected metric value",
  "use": "Paste a confusing warning (e.g. an UndefinedMetricWarning from precision_score on an all-negative prediction) or a surprising number (a negative R2) and ask what it means mechanically, before assuming your code is broken.",
  "tools": ["Chat / coding assistants -- Claude, ChatGPT, Gemini"],
  "prompt": "I got this scikit-learn warning/output: <PASTE VERBATIM>. My code calls <FUNCTION NAME> with roughly this shape of input: <DESCRIBE, do not paste real data>. Explain what specifically triggers this, whether my result is still usable, and what a correct fix looks like -- without inventing an API that doesn't exist in scikit-learn.",
  "verify": "Check every function name and parameter the explanation references against the actual scikit-learn documentation for your installed version -- an assistant explaining an error can still invent a parameter or a function that sounds plausible but does not exist. Re-derive the mechanism yourself (why would THIS input produce THIS warning) rather than accepting the explanation on authority; if you can't re-derive it, you don't understand it yet, you've just been told something.",
  "diy": "You must already know what accuracy, precision, recall, F1, and R2 individually measure -- and that R2 can legitimately be negative -- before you can judge whether an AI's explanation of a strange number is actually correct or merely plausible-sounding.",
  "market": "IN"
}

A first, honest look at the full workflow

Put every piece from this node together and you get the actual shape of a real modeling task, end to end: split the data into train and test (never touch test again until the very end); using only the training data, cross-validate across a handful of candidate hyperparameters (a max_depth sweep, say) and pick the winner by CV score, never by a peek at test; refit the winning configuration on the full training set; and touch the test set exactly once, for the one number that goes in the report. DA2.01-e09 builds this exact sequence as a single function, and its hidden tests check the property that matters most: the winning hyperparameter must be identical regardless of what order the candidates were listed in, and calling the function twice with identical inputs must return identical output, the same determinism discipline CS2.01 built for a shell environment now built for a model-selection loop. What this node's pipeline does not yet do, engineer features safely, guard against a feature that was itself built using future information, or wire this into sklearn.pipeline.Pipeline so preprocessing itself can never leak, is deliberately out of scope here; DA2.02 picks up exactly there.

Common mistakes & how experts think differently

1. "My model got 98% accuracy on the training data, so it's great." A training-set score measures memorization capacity, not predictive skill, a sufficiently flexible model can hit 100% on training data it has simply learned by rote, and tell you nothing about tomorrow's row. The expert's reflex on hearing any score is to ask "trained on, or tested on?" before reacting to the number at all.
2. "Accuracy is 95%, ship it." Whenever the target class is imbalanced, a model that never predicts the rare class can hit that number while being completely useless. DA2.01-e05's naive always-predict-0 classifier proves it with real numbers (95% accuracy, 0% recall). The expert checks the base rate of the target before trusting a bare accuracy figure, and reaches for precision/recall/F1 or ROC-AUC the moment a class is rare.
3. "I tuned my hyperparameter until the test-set score looked best." The instant you adjust anything based on a test-set number, that test set has become a second training set and its score no longer estimates true out-of-sample performance. QM1.03's Common Mistake #10, and just as fatal here. The expert tunes exclusively against cross-validation on the training data and touches the test set exactly once, at the very end.
4. "My cross-validation score came out negative, so my model is broken." scikit-learn's cross_val_score negates error-type scoring functions (neg_mean_squared_error) by convention, so a "higher is better" comparison still works uniformly across metrics, a negative number here is the correct, expected sign, not a bug; you flip it back yourself before reporting it. The expert always checks a scoring function's name for the neg_ prefix before reacting to its sign.
5. "A deeper tree is always a better tree." Depth is capacity, not quality. DA2.01-e06 shows a depth-3 tree hitting 100% training accuracy while scoring worse on the test set than a shallower depth-2 tree that never got a perfect training score at all. The expert reads the test/CV column, never the training column, when choosing complexity.
6. "One good tree is basically as good as a forest of them, just simpler." A single unconstrained tree memorizes whatever noise its specific training sample happens to contain; a forest of many such trees, each on an independent resample, cancels that noise out by averaging while the shared real signal reinforces. DA2.01-e07 shows a single tree scoring a negative R² where a fifty-tree forest on the identical data scores +0.49. The expert defaults to an ensemble the moment a single tree's variance is a concern, which is most of the time on noisy financial data.
7. "R² can't be negative, that must be a bug in my code." R² is 1 minus your model's error relative to a naive always-predict-the-mean baseline; when your model does worse than that baseline (a real, common outcome for an overfit or badly-specified model on new data), R² is mathematically, correctly negative. The expert treats a negative R² as a real, informative result, "this model is actively worse than a trivial guess", not a sign the metric itself is broken.
8. "I can eyeball the confusion matrix and skip computing precision/recall properly." A confusion matrix's raw counts don't communicate rate information at a glance the way a computed ratio does, and eyeballing gets actively misleading the moment class sizes are unequal. The expert always computes the actual ratios (sklearn.metrics's functions, not mental arithmetic) before comparing two models or reporting a result.
9. "I don't need to set random_state, my result is still valid without it." A model or split with no fixed seed produces a genuinely different result on every run, which means a report describing "the model's performance" is describing one arbitrary draw, not a reproducible fact, exactly the reproducibility leg CS2.01 built for an environment, now required for a model too. The expert seeds every stochastic step (train_test_split, KFold, any model with internal randomness) and records the seed, so "run it again" means "get the same answer."

The thread through all nine: the expert never lets a training-set number, a bare accuracy figure, or an un-cross-validated hunch stand in for the one thing that actually matters (honest, held-out, correctly-metric'd performance) and treats every shortcut around that discipline as a decision to actively distrust the result, not a convenience to accept.

Worked examples

Worked example 1: Fitting and honestly scoring a regression (India, ₹)

Ananya, an analyst reviewing a small panel of mid-cap manufacturers, has eight companies' revenue and net income for training and three more held out for testing (all figures illustrative, ₹ crore). Train: revenue [200, 400, 600, 800, 1000, 1200, 1400, 1600], net income [24.5, 56.0, 84.0, 116.5, 144.0, 176.5, 204.0, 236.5]. Test: revenue [300, 900, 1500], true net income [41.0, 128.5, 221.5]. She fits a linear regression on the eight training companies and wants an honest read on the three she held out.

Solution. Step 1: fit on train only. LinearRegression().fit(X_train, y_train) learns a slope and intercept from the eight training pairs alone; the three test companies have not been seen by the model at all yet.

Step 2: predict on test, then score. Calling .predict on the three held-out revenue figures and comparing against their true net income gives RMSE ≈ 1.3003 (₹ crore) and R² ≈ 0.99969. The relationship in this illustrative panel is close to perfectly linear, so both scores are excellent, but notice the order of operations: fit happened first, on train alone; scoring happened second, on test alone. Reversing that order (even scoring on the training rows "just to check" and reporting that number instead) would silently inflate the read.

Step 3, connect it to the code pack. This is exactly DA2.01-e01's function, fit_linear_and_score(X_train, y_train, X_test, y_test), and its hidden tests include this exact India-flavored panel, an analogous US panel (Worked Example 2), a negative-slope relationship (so a solution that silently assumes revenue and net income always move together fails), and a genuine two-feature case (so a solution that assumes exactly one feature fails too).

Read it. Nothing about the fitting required anything beyond QM1.02's least-squares theory. What this node adds is the discipline around it: which rows the model saw, which rows it didn't, and computing the metric only from the second group.

Worked example 2: The same discipline, a different market (US, $)

Miguel, working the US side of the same kind of panel, has eight companies' revenue and net income for training ($ million, illustrative): revenue [50, 100, 150, 200, 250, 300, 350, 400], net income [8.5, 13.5, 20.5, 25.0, 32.5, 37.0, 44.5, 49.0], and three held out for test: revenue [75, 225, 375], true net income [10.5, 29.5, 46.0].

Solution. Identical procedure, different market, different currency, and (deliberately) a different underlying slope and intercept than Ananya's panel, the same function must get both right. Fitting on the eight US training companies and scoring on the three held-out ones gives RMSE ≈ 0.6086 ($ million) and R² ≈ 0.99824. Read it. The code that produced Worked Example 1's numbers and this one is the exact same function, unmodified, proof that a correctly-written fit_linear_and_score generalizes across markets, currencies, and underlying relationships, while a solution that hardcoded Ananya's specific slope to pass one visible test would fail this hidden one instantly.

Worked example 3: A tree's capacity, watched as a number (neutral, mechanism)

A small, deliberately noisy classification panel, two genuinely informative features (illustrative factor-style scores) buried among eight pure-noise features, thirty rows for training and thirty fresh rows for test, is fit with a decision tree at three depths, holding everything else fixed.

Solution. Step 1, the too-rigid extreme. At max_depth=1, the tree can ask exactly one yes/no question of the data. Train accuracy: 73.3%. Test accuracy: 60.0%. Both mediocre, the model is too simple to capture the real pattern, so it does badly even on the data it was fit to.

Step 2, the sweet spot. At max_depth=2, train accuracy rises to 86.7% and, this is the number that matters, test accuracy rises further, to 73.3%, the best test score across every depth tried.

Step 3, the overfit extreme. At max_depth=3 (and beyond, the tree is already fully pure by depth 3 on this small dataset, so depth 5 and unbounded depth give identical results), train accuracy hits a perfect 100%, and test accuracy falls to 63.3%, worse than depth 2's 73.3%, despite depth 3's obviously "better-looking" training score.

Read it. This is QM1.03's bias-variance curve, produced from a real, small, random_state-seeded dataset rather than described in prose: too rigid (depth 1) does badly everywhere; too flexible (depth 3+) memorizes the training rows and generalizes worse than the depth in between. DA2.01-e06 is exactly this experiment, and its hidden tests check the gap itself (train_accuracy − test_accuracy) grows strictly larger for the unconstrained tree than for the shallow one, the mechanical signature of overfitting, made numeric.

Worked example 4: When one tree isn't enough: the forest that rescues a negative R² (neutral, mechanism)

The same style of noisy, many-weak-feature panel as Worked Example 3, but now a regression target (three genuinely informative factor-style features, five pure-noise ones, eighty training rows, forty fresh test rows), fit two ways: one unconstrained DecisionTreeRegressor, and one fifty-tree RandomForestRegressor, both with random_state=42.

Solution. Step 1, the single tree. Left completely unconstrained, the tree memorizes the training rows' noise along with their signal. On the held-out test rows, it scores R² ≈ −0.10, worse than the trivial baseline of predicting the mean net income every time.

Step 2, the forest, on the identical data. Fifty trees, each grown on an independent bootstrap resample with a random feature subset considered at each split, then averaged. Test R²: ≈ +0.49, a swing of roughly 0.59 in R², from "worse than a naive guess" to "explains about half the held-out variance," using the exact same training rows and the exact same target.

Step 3, confirm it isn't a lucky seed. Repeating the identical comparison at two more random seeds gives tree/forest R² pairs of (−0.0985, 0.5543) and (+0.0934, 0.5156), the forest wins clearly every time, even though the exact numbers shift from seed to seed (a further reminder that a single result without a fixed, reported seed is not yet a fact. Common Mistake #9).

Read it. Nothing about any individual tree changed between Step 1 and Step 2, same maximum flexibility, same lack of a depth limit. What changed is that fifty differently-noisy trees, averaged, cancel each other's memorized noise while their shared real signal reinforces. DA2.01-e07 is this exact experiment; its hidden tests require the forest to beat the tree across three separate seeds, so a solution that got lucky once cannot pass by accident.

Worked example 5: The accuracy paradox, made concrete (India, ₹)

Over a twenty-trading-day illustrative window on an NSE-style name, a genuine "large up-move" happens on exactly one day out of twenty. Two candidate classifiers are evaluated against this same twenty-day truth: Classifier A always predicts "no large up-move." Classifier B flags the one real up-move correctly, with one false alarm along the way.

Solution. Step 1: score Classifier A. Nineteen of twenty days are correctly called "no up-move," and the one real up-move is missed. Accuracy: 95%. Precision, recall, and F1 are all 0.0 (it never once predicts the positive class, so precision and recall are undefined-as-zero), and ROC-AUC is exactly 0.5, mathematically indistinguishable from a coin flip, because its predicted probabilities carry no information that separates the one true up-day from the nineteen flat ones.

Step 2, score Classifier B on the identical twenty days. It correctly flags the real up-move and raises one false alarm elsewhere. Nineteen of twenty predictions are again correct (18 true negatives + 1 true positive, against 1 false positive): accuracy: also 95%. But now precision is 0.5, recall is 1.0, F1 is ≈0.667, and ROC-AUC is a perfect 1.0.

Step 3, the point. Both classifiers score the identical 95% accuracy on the identical data. One is worthless (never catches the event it exists to catch); one is excellent (catches every real event, with only one false alarm). Accuracy alone cannot tell you which is which, you must look at recall, precision, F1, or ROC-AUC to see the difference at all.

Read it. This is QM1.03's warning about a "deceptively high accuracy" turned into an exact, side-by-side number: not just "accuracy can mislead," but two specific numbers, identical to three significant figures, hiding two models a real desk would treat completely differently. DA2.01-e05 is this exact pair of scenarios (plus a matched US-market pair using the same construction), and its hidden tests check both the accuracy-tie and the metrics that break it.

Worked example 6: The full discipline, end to end (US, $)

Miguel has a training panel (sixty rows, the same noisy factor-style construction as Worked Example 4's regression target) and a separate, freshly-drawn thirty-row test panel he has not looked at yet. He wants the best max_depth for a decision tree, chosen honestly, and one final, trustworthy test score.

Solution. Step 1: cross-validate on training data alone. Sweeping candidate depths [1, 2, 3, 5, None] with 5-fold cross-validation, using only the sixty training rows, gives mean CV-MSE of 1.6318 (depth 1), 1.6212 (depth 2, the winner), 1.7015 (depth 3), 2.0884 (depth 5), and 1.7432 (unbounded). Depth 2 wins.

Step 2, refit on the full training set. A fresh DecisionTreeRegressor(max_depth=2) is fit on all sixty training rows (not just the folds used during CV), cross-validation's job was choosing the hyperparameter, not producing the final model.

Step 3, touch the test set exactly once. Scoring this single, already-chosen model against the thirty held-out rows gives test MSE ≈ 1.4228, a number computed exactly once, after every decision was already made, with nothing left to tune in response to it.

Read it. Reorder the candidate depth list, or run the entire procedure twice with identical inputs, and the winning depth (2) and every downstream number come out identically: determinism, the same discipline CS2.01 built for a reproducible shell environment, now built into a model-selection loop. DA2.01-e09 is this exact function, select_and_evaluate, and it is the node's capstone: every earlier exercise is a piece of this one workflow.

Practice set

Work each problem fully before reading its solution. All data below is illustrative/synthetic (R3) unless stated otherwise.

P1 (guided). The contract, in order. Put these four steps in the order a correct supervised-learning workflow actually executes them: (a) model.predict(X_test), (b) model.score or a metric function on the test predictions, (c) model = SomeModel(...), (d) model.fit(X_train, y_train).

Solution. (c) → (d) → (a) → (b): construct the unfit model, fit it on training data only, predict on test data, then score those test predictions. Any reordering that fits or predicts before construction is nonsensical; any that scores before predicting has nothing to score; and (the version that actually happens in real bugs) fitting on test data, or fitting after scoring, silently produces a number that looks like a result but measures nothing real.

P2 (guided). RMSE and R² by hand. A tiny model predicts [10, 20, 30] against true values [12, 18, 33]. Compute RMSE and R² by hand. (Mean of true values = 21; total sum of squares uses that mean.)

Solution. Errors: -2, 2, -3. Squared errors: 4, 4, 9, sum 17, mean 17/3 ≈ 5.667. RMSE = √5.667 ≈ 2.38. For R²: total sum of squares against the mean (21) is (12-21)² + (18-21)² + (33-21)² = 81 + 9 + 144 = 234. R² = 1 − 17/234 ≈ 1 − 0.0726 = 0.927.

P3. A negative R², explained. A model scores R² = −0.15 on a test set. Is this a bug? What does it mean, precisely, and what is the one-sentence diagnosis?

Solution. Not a bug. R² = −0.15 means the model's squared test error is 115% of the squared error a naive "always predict the mean" baseline would produce, the model is actively worse than guessing the average every time. This is a real, if unflattering, measurement, most often produced by an overfit model whose training-time performance does not transfer, or a model applied to data that violates an assumption it relied on.

P4. Three-way split arithmetic. n = 100, val_frac = 0.15, test_frac = 0.25. How many rows go in each of train, validation, and test, and what is the correctly-rescaled test_size for the SECOND train_test_split call (the one separating train from validation)?

Solution. Test: 100 × 0.25 = 25 rows. Remaining after peeling off test: 75 rows, of which validation should be 100 × 0.15 = 15 rows and train the remaining 60. The second call's test_size must be expressed as a fraction of the 75 remaining rows: 15 / 75 = 0.20, not the original 0.15: passing 0.15 directly to the second call would give 75 × 0.15 ≈ 11 validation rows, silently wrong.

P5. Why the test set is not for tuning. In one or two sentences, explain why choosing a hyperparameter by re-checking the test-set score after each change invalidates that test set, using the exact mechanism (not just "it's not allowed").

Solution. Every time you adjust a choice in response to the test-set number and re-measure, you are using that number as feedback to steer a decision (which is precisely what a validation set is for) so the test set has functionally become a second validation set, "peeked at" as many times as you checked it. Its final number then reflects the best of however many looks you took, not a single honest, un-optimized read of true out-of-sample performance.

P6 (guided). Averaging k-fold scores by hand, and reading the sign. A 4-fold cross-validation with scoring='neg_mean_squared_error' returns fold scores [-2.1, -1.8, -2.4, -1.9]. What is the reported mean CV-MSE (as a positive, interpretable number)?

Solution. Mean of the four raw (negative) scores: (-2.1 - 1.8 - 2.4 - 1.9) / 4 = -8.2 / 4 = -2.05. Flip the sign back: mean CV-MSE = 2.05. Forgetting the flip and reporting "−2.05 MSE" is meaningless: MSE cannot be negative by construction, and the negative sign is purely scikit-learn's "higher score is better" convention leaking through.

P7. Diagnose complexity from a table. Given train errors [0.90, 0.60, 0.30, 0.10] and test errors [1.20, 0.75, 0.55, 0.95] at four increasing complexity levels (indices 0–3), which index should you choose, and which index shows the clearest sign of overfitting?

Solution. Choose index 2 (test error 0.55, the lowest in the test column), never index 3, whose test error (0.95) is actually worse despite having the lowest training error (0.10) of all four; that gap (0.95 − 0.10 = 0.85, by far the largest of the four) is the clear overfitting signature, exactly the pattern DA2.01-e04's diagnose_complexity is built to surface.

P8. The sign-convention trap, applied. A colleague reports "my cross-validated model scored −4.2, so it's broken; a positive score would mean it works." What is wrong with this statement?

Solution. Nothing is broken, scoring='neg_mean_squared_error' (and every other neg_* scorer) is defined to return a negative number for a real, valid model, by scikit-learn's own "higher is always better" convention. The colleague should report the flipped, positive value (4.2) as the actual MSE and judge it by whether 4.2 is small relative to the target's own scale, not by whether the raw returned number happens to be positive or negative.

P9. Predicting the winning depth before computing it. You are told a dataset has a genuine but modest nonlinear pattern buried in noisy features. Before running anything, predict, in one sentence, what shape you expect a max_depth vs. test-score sweep to have, and why.

Solution. A U-shaped (for error) or inverted-U-shaped (for accuracy/R²) curve: very shallow depths underfit (too rigid to capture the nonlinearity, doing badly on both train and test), some middle depth generalizes best, and very deep/unbounded settings overfit (near-perfect training score, degraded test score) as the tree starts fitting the noise features instead of just the real pattern.

P10. Why averaging noisy trees helps (intuition, not proof). In your own words, explain why fifty trees, each individually overfit to a different bootstrap resample, can average into a model that generalizes far better than any one of them, without invoking a formal proof.

Solution. Each tree's overfitting is fitting to different noise (different resample of rows, different random feature subsets at each split), so the trees' errors are only weakly correlated with each other, while every tree (however noisy) is still at least partly capturing the same real, shared signal. Averaging predictions cancels out errors that point in different, uncorrelated directions across trees while reinforcing the signal that every tree agrees on, which is exactly why the forest's test score can land far above any individual tree's.

P11. Precision, recall, and F1 by hand. A classifier's predictions against 10 true labels give: 3 true positives, 1 false positive, 2 false negatives, 4 true negatives. Compute precision, recall, and F1.

Solution. Precision = TP/(TP+FP) = 3/(3+1) = 0.75. Recall = TP/(TP+FN) = 3/(3+2) = 0.60. F1 = 2×(precision×recall)/(precision+recall) = 2×(0.75×0.60)/(1.35) = 0.90/1.35 ≈ 0.667.

P12. ROC-AUC intuition. A model assigns these up-move probabilities to four days, in true-label order (0=no up-move, 1=up-move): day A (true 0) → 0.2, day B (true 1) → 0.9, day C (true 0) → 0.4, day D (true 1) → 0.7. Without computing the exact AUC, does this model rank the two true up-move days above the two true no-move days? What would a ROC-AUC of exactly 1.0 mean here?

Solution. Yes, both true up-move days (B: 0.9, D: 0.7) received higher probabilities than both true no-move days (A: 0.2, C: 0.4), so every positive-negative pair is correctly ordered. A ROC-AUC of exactly 1.0 would mean precisely this: every true-positive example ranks above every true-negative example, regardless of what specific threshold you'd pick to convert probabilities into hard labels.

P13. The accuracy paradox, generalized. If a rare event's true base rate is 2%, what accuracy does a model that never predicts the event achieve, and what does that number tell you about the model's actual skill?

Solution. 98% accuracy: automatically, by construction, for a model that does zero work. It tells you nothing at all about skill: the number is entirely determined by the base rate, not by anything the model learned, which is exactly why accuracy must never be the primary metric for a rare-event target.

P14. Choosing the metric before touching the model. For each task, name the single metric you would report as primary, and why: (i) forecasting next quarter's revenue (continuous, no extreme outliers expected); (ii) flagging a rare loan default (positive rate ≈ 3%); (iii) predicting which of two roughly equally common outcomes will occur.

Solution. (i) RMSE: a continuous target with no reason to specially guard against outliers; MAE is an acceptable alternative if you specifically distrust a few extreme errors dominating the score. (ii) F1 (or ROC-AUC), a 3% positive rate makes accuracy nearly meaningless (a do-nothing model already scores ~97%), so precision/recall/F1 or a threshold-free ROC-AUC are required. (iii) Accuracy, roughly balanced classes are exactly the case where accuracy is not misleading and remains the simplest honest choice.

P15. Reproducibility of a split, connected back to CS2.01. Two teammates run train_test_split(idx, test_size=0.2, random_state=42) on the identical idx list, on two different machines, a week apart. Should their resulting splits be identical? What CS2.01 concept does this rely on?

Solution. Yes, identical, a fixed random_state makes the split fully deterministic given the same input, exactly the reproducibility CS2.01 built for an entire environment (the same seed always produces the same result, regardless of machine or when it's run). If the two splits differed, the most likely cause is that idx itself, or the installed scikit-learn version's internal shuffling algorithm, silently differed between the two runs, the same "what exactly changed?" audit CS2.01 taught you to run.

P16. Spot the leak (a forward pointer to DA2.02). A colleague computes the mean and standard deviation of a feature across the entire dataset (train and test combined), uses those to standardize every row, and only afterward splits into train and test. What is wrong with this, in one sentence, and which node's job is it to teach the full fix?

Solution. The test set's own values quietly influenced the standardization statistics (the mean/standard deviation), meaning information from data the model is supposed to have never seen leaked into the preprocessing applied to the training set, a real, if subtle, form of leakage. The fix (fitting any preprocessing step on training data only, then applying it unchanged to validation/test, ideally wired through sklearn.pipeline.Pipeline so it can never happen by accident) is DA2.02's job in full; this node's split discipline is the foundation that fix builds on.

Applied mini-project

Build, validate, and defend one small model, honestly.

This project is the whole node compressed into one deliverable: pick a supervised task, hold out data correctly, cross-validate a hyperparameter without peeking at test, fit the final model, score it with the metric you can actually defend as the right one, and write down (before you look at your own numbers) what you expect to see.

The task. Using any of this module's dataset ids (sec-fundamentals / india-fundamentals for a fundamentals-based regression, or a factor-style panel of your own construction for either regression or classification), pick one predictive question: either a continuous target (e.g., predict a company's net income from its revenue and one or two other fields) or a binary target (e.g., predict whether next period's growth exceeds a threshold). Include at least one India-flavored and one US-flavored instance of your chosen task (R6).

Do this, in order:

1. Split honestly. Build a train/validation/test partition (or a train/test partition plus k-fold CV inside train) with a fixed, recorded seed. State the exact row counts in each set before doing anything else.
2. Predict, before computing. In one paragraph, write down which model (linear/logistic, a single tree, or a forest) you expect to perform best on your specific data, and why, in terms of this node's bias-variance vocabulary: before you fit anything.
3. Cross-validate a real choice. Using only the training data, cross-validate across at least three candidate configurations (e.g., three max_depth values, or linear regression versus a shallow tree versus a forest) and record every CV score, not just the winner.
4. Fit the winner once, touch test once. Refit the CV-selected configuration on the full training set, then compute your test-set score (exactly once) using the metric you can defend as correct for your task (name it and justify it in one sentence using this node's metric-choice logic).
5. Diagnose. State whether your final model shows any sign of underfitting or overfitting (the train/test or train/CV gap), and whether your prediction from Step 2 was right.

Deliverable and self-check. One short write-up containing: the exact split sizes and seed; your pre-registered prediction from Step 2 (dated before you had results, not adjusted afterward); the full CV comparison table; the single final test-set score with its metric named and justified; and the overfitting diagnosis. Score yourself 2 points each for: (a) both an India-flavored and a US-flavored instance of the task are present; (b) the split is provably leak-free (state the row counts and confirm they partition your data with no overlap); (c) the prediction was genuinely written before the results were computed; (d) the metric choice is defended from first principles, not habit (accuracy is never the silent default for an imbalanced target); (e) the test set was touched exactly once, with the CV comparison shown as the actual basis for the final choice. Pass at 8/10, with no zero on any item.

Code Lab: open the DA2.01 code pack in the Practice tab: nine exercises, DA2.01-e01 through DA2.01-e09, covering honest regression scoring, a leak-free three-way split, cross-validated depth selection, reading a train/test error table, imbalance-aware classification metrics, a tree's depth-vs-accuracy curve, an ensemble beating a single tree, choosing the right primary metric, and the full CV-select-then-evaluate-once pipeline, and clear every one of its hidden tests, not merely the single visible example shown for each.

Reading & resources

Everything below is an anchor to build fluency from and verify your own code against, never a substitute for fitting a model and reading its own numbers yourself. Free unless marked; level tags run from a first pass to full reference depth.

The official reference (read these first, and keep them open while you code):

• The scikit-learn User Guide (Supervised learning section, and the dedicated pages for LinearRegression, LogisticRegression, DecisionTreeClassifier/Regressor, and RandomForestClassifier/Regressor) the authoritative, always-current statement of every constructor argument and method this node uses. [Free] [Beginner–Intermediate]
• The scikit-learn User Guide's "Model evaluation: quantifying the quality of predictions" page, the canonical reference for every metric and every scoring= string this node uses, including the neg_* sign convention. [Free] [Beginner–Intermediate]
• The scikit-learn User Guide's "Cross-validation" page, the full account of KFold, cross_val_score, and the family of splitting strategies beyond what this node covers. [Free] [Intermediate]

Building intuition (a friendlier second pass):

• Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, chapters on training models, decision trees, and ensembles are the closest print companion to this exact node, with more worked code than fits here. [$ Paid] [Beginner–Intermediate]
• StatQuest (Josh Starmer), the video series on decision trees, random forests, and ROC/AUC, the friendliest visual walkthroughs of exactly the mechanisms this node describes in prose. [Free] [Beginner]

Going deeper (for when you want the full statistical grounding):

• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning (freely available PDF from the authors), the standard rigorous-but-readable textbook covering bias-variance, cross-validation, trees, and ensembles with the mathematics this node deliberately keeps out of scope. [Free] [Intermediate–Advanced]

Do this, not just read: take any small tabular dataset you can find or construct (even a handful of rows you type by hand), split it with a fixed seed, fit a LinearRegression and a DecisionTreeRegressor on the identical split, and compute RMSE and R² for both on the test set only. Then change max_depth on the tree from 1 up to unbounded and watch the test score rise, peak, and fall, on data you generated yourself, not a number this module told you to expect. Ten minutes of watching your own overfitting curve appear, unprompted, will install this node's central habit faster than any amount of additional reading.

Teach it back & journal

Feynman prompt. Explain to a smart friend who has just heard the phrase "our model is 97% accurate" and is impressed why that single number might be telling them almost nothing, using the idea that a model which never predicts the rare, important outcome can still score high accuracy purely from how common the ordinary outcome is. Then, in the same breath, explain why a model scoring perfectly on the data it was trained on is not evidence of anything useful, using the idea that a flexible enough model can memorize its training rows without having learned the pattern those rows came from. If your friend can, unprompted, ask "accurate on WHAT data, the data it was trained on, or data it's never seen?" the next time they hear a model's score quoted, you have installed the mechanism this module exists to teach.

Journal prompt. For one working session, before you compute or read any model's score (your own code, a report, or an AI assistant's claimed number) pause and write down: (1) was this score computed on training data or held-out data, and how do you know; (2) is the metric being reported actually the right one for this task, or just the default one somebody reached for; (3) if the target is a classification task, what is the base rate of the class that matters, and does that change how impressive the reported number actually is? Log what you find. Then write a paragraph on this: the difference between someone who can call .fit() and someone who can be trusted with a real model is not who can produce a high-looking number (a flexible enough model can always produce one) it is who insists on knowing exactly which data produced that number and exactly why that metric was the right one to look at. Which single habit from this module, checking train-versus-test before reacting to a score, checking the base rate before trusting accuracy, or refusing to tune against a test set more than once, would most change how a reviewer trusts your modeling work, and what will you do this week to make it automatic?

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.