Machine learning in finance

Every technique in this module (an AR model, an ADF test, a Monte Carlo VaR) is, at bottom, a way of using data to make a disciplined statement about the future. Machine learning is the modern, more flexible extension of that same idea, and this section's entire job is to give you the vocabulary, the failure modes, and the correct mental discipline, never a coded model.

Supervised vs unsupervised learning

Supervised versus unsupervised, the one-line distinction. In supervised learning, every training observation carries a known label or target, and the task is to learn a function predicting that target from the input features: regression for a continuous target (next quarter's revenue growth, a bond's yield), classification for a discrete one (default / no default, an up or down next-day move). In unsupervised learning there is no target at all, the task is to discover structure on its own terms: clustering groups similar observations together (a tiny illustration: cluster ten stocks by trailing annualized volatility with a simple k-means split, and the algorithm finds two natural groups on its own, a low-volatility cluster centered near 15.5% and a high-volatility cluster near 43.6%, with no label ever supplied), and dimensionality reduction (principal component analysis and its relatives) collapses many correlated factors into a handful of uncorrelated ones, precisely what turns "thirty positions, one hidden common factor" into a number a risk committee can read, a theme AA1.01's frontier work and the quant-code region's factor construction both depend on.

How do you avoid overfitting?

The discipline that makes any of this trustworthy: train, validate, test, never grade on the homework you handed out the answer key for. A model's fit to its own training data (in-sample fit) tells you almost nothing about out-of-sample skill, a sufficiently flexible model can fit training data perfectly while being useless on anything new. The standard discipline splits data into a training set (fit), a validation set (tune complexity), and a test set (touched exactly once, for an honest final read), and k-fold cross-validation stretches a single split further by rotating which slice plays "held-out" across k folds and averaging, squeezing more honest signal from a limited sample without ever touching the true test set.

Overfitting is a bias-variance tradeoff, directly visible in a train/test error curve. An overly rigid model underfits, high bias, missing real signal, doing badly on both train and test. An overly flexible model overfits, low bias but high variance, memorizing training-set noise as though it were signal, so training error keeps falling while test error, after an initial improvement, turns around and gets worse. Worked Example 7 fits polynomials of degree 1–12 to a genuinely quadratic relationship and shows exactly this: training error falls monotonically from 0.733 (degree 1) to 0.523 (degree 12) (the flexible model looks unambiguously "better" if you only look at its own training data) while the honest test-set error bottoms out around degree 4 before rising sharply. A single split is itself a noisy read; k-fold cross-validation, run entirely inside the training set, is the more trustworthy tool, and in the same example it correctly recovers the true degree (2) that generated the data, with a dramatic blow-up in cross-validated error at high degree that the single split only hinted at.

What is look-ahead bias?

The finance-specific trap that breaks all of the above: look-ahead bias. Everything above silently assumes any split order is fine, because a cross-sectional dataset (many companies, one snapshot) has no natural order to violate. Time-series data does not have that luxury. Shuffle a chronologically ordered dataset before splitting, and some "training" observations will sit after some "test" observations in real time, the model learned from the future before being asked to predict it. This is the single most common way a backtested finance ML result turns out worthless against live data. The fix is a chronological split (train on the earlier period, test strictly on the later one) or, more robustly, a walk-forward / expanding-window scheme, validating only on data not yet seen in time, never merely data not yet seen in the shuffle. This is exactly what QD1.03's no_lookahead harness enforces automatically.

How do you score a machine learning model?

Evaluation, named at a survey level. Regression tasks are typically scored by RMSE or MAE (how far off, on average, in the target's own units); classification tasks by accuracy, precision, recall, and the area under an ROC curve (AUC), and a bare accuracy number is dangerous on its own: a model that never predicts a rare event (a default, a market crash) can score a deceptively high accuracy simply because the rare event is, definitionally, rare. None of these metrics are derived here in depth; that derivation, and the leak-free pipeline that computes them correctly, is DA2.01's and DA2.02's job.

{
  "skill": "Using an AI tutor to build intuition for bias-variance and overfitting, verified against your own computed numbers",
  "use": "Have a chat assistant explain why a training-error curve and a test-error curve diverge as model flexibility rises, using your own fitted-degree numbers as the concrete example, and generate a few quiz questions to test whether you actually internalized the idea.",
  "tools": ["Chat assistants — Claude, ChatGPT, Gemini", "Learning/notebook AI — grounded on your own train/test MSE table, not a generic textbook example"],
  "prompt": "Here is my train-MSE and test-MSE table across polynomial degrees 1-12: <paste your table here>. Explain, using these specific numbers, why train MSE keeps falling while test MSE bottoms out and then rises. Then explain why a 5-fold cross-validation number computed only on the training set is more trustworthy than reading the single test-MSE column for choosing a degree. Quiz me with three questions.",
  "verify": "The explanation is only as good as the numbers you fed it — recompute your own train/test/CV MSE table before trusting any explanation built on it, and check that the assistant's stated 'best degree' actually matches the minimum in YOUR table, not a generic rule of thumb. An AI explaining overfitting can itself overfit its explanation to a plausible-sounding but wrong reading of your specific numbers.",
  "diy": "You must be able to compute a train/test MSE table and a k-fold CV table yourself, and read the bias-variance story off them unaided, before treating any AI-generated explanation as confirmation rather than a first draft.",
  "market": "IN"
}

Where is each technique built in code?

The explicit hand-off (the binding C-3 boundary, restated in full). Every idea in this section has a real, hidden-tested, auto-graded build waiting for you, and this module's job stops exactly at the concept:

No worked content is duplicated across that boundary, not one line of code, not one dataset. If you find yourself wanting to build rather than understand, that want is correct, and it belongs in one of the rows above.