Set deployment aside for a moment and consider a purely statistical fact with an uncomfortable consequence. Suppose you evaluate a strategy that has genuinely zero skill (its returns are pure noise) against the ordinary significance threshold of 5%. By the definition of that threshold, there is a 5% chance, on any single such test, that noise alone produces a result that looks "significant." Now suppose, instead of testing one skill-less strategy, you test forty of them (forty parameter tweaks on the same underlying noise, forty tickers, forty lookback windows, it does not matter which) each independently, each against the same 5% bar. The expected number that clear the bar by pure chance is simply forty times five percent: two. Not zero. Two strategies with no real edge whatsoever, expected to look statistically significant, purely from trying forty times. This is the multiple-testing problem, also called the multiple-comparisons problem or, in its most damning everyday phrasing, p-hacking: judging each of many trials against a threshold calibrated for a single trial guarantees false discoveries as the trial count grows, and the guarantee holds regardless of how careful, honest, or well-intentioned the researcher running each individual test happens to be. Nobody needs to cheat for this to happen. A researcher who genuinely, honestly tries forty reasonable variations of an idea, reports only the one that cleared the bar, and never mentions the other thirty-nine, has produced exactly the failure mode this section names, not through dishonesty in the reporting of the winning trial, but through silence about the trials that did not win.
Advertisement
What is backtest overfitting?
Marcos López de Prado's contribution (developed across a body of work culminating in Advances in Financial Machine Learning (2018)) was to take this from a well-known statistical caveat into a rigorously quantified, correctable risk specific to quantitative finance, where the temptation to keep tweaking a backtest until it looks good is close to universal and the cost of being fooled by it is measured in real capital. His vocabulary is now the field's vocabulary: backtest overfitting (a strategy fit, deliberately or not, to the noise in one particular historical sample rather than to real, persistent structure), the deflated Sharpe ratio (a Sharpe ratio adjusted downward to account for how many trials produced it. QD2.03's territory to compute and report, not this module's), and the Probability of Backtest Overfitting, estimated via a resampling technique called combinatorially symmetric cross-validation, a further refinement QD1.03 already gestures at in its own selection-validation step ("walk-forward or purged CV, count trials honestly, deflate") and this module does not re-derive. What this module owns, and what none of those does, is the layer underneath all of them: making the trial count that any of those corrections depends on actually true, by building the infrastructure (the ledger) that a memory or a spreadsheet cannot provide.