Backtest overfitting

This is the pitfall that survives all the others. Fix your alignment, use point-in-time data, model your costs, then run a parameter sweep and keep the best cell, and you have re-introduced everything you just removed, in a form no harness can detect by looking at one backtest, because each individual backtest is now correct.

Worked example: the best of 1,000 noise strategies

Start with the demonstration that makes it undeniable. Take a seeded price path with zero drift (seed 90210, 1,260 bars, 1.25% daily volatility). Generate 1,000 strategies that are pure noise: each holds ±1 and flips with probability 0.20 each bar, independently of the price. By construction there is nothing to find. Split into 755 in-sample bars and 504 out-of-sample bars.

The winner posted a 2.23 Sharpe on an edge that is exactly zero. Its out-of-sample Sharpe is −0.12. The top ten averaged 1.65 in-sample and 0.01 out-of-sample, which is the honest number, because the honest number is zero.

And this is predictable, which is what turns a cautionary tale into a tool. The maximum of N draws from a mean-zero distribution with standard deviation σ has expectation approximately

E[max] ≈ σ · [ (1 − γ)·Z⁻¹(1 − 1/N) + γ·Z⁻¹(1 − 1/(N·e)) ],   γ = 0.5772156649…

With N = 1,000 the bracket evaluates to z = 3.255; times the observed σ of 0.567 gives an expected winner of 1.85 against an observed 2.23. The theory called it. Note also that the observed σ of 0.567 is almost exactly the theoretical standard error of an annualised Sharpe over three years, 1/√3 = 0.578, the dispersion is not a quirk of the simulation, it is what three years of data can resolve.

What is the deflated Sharpe ratio?

So the correct response to "my best parameter set has a Sharpe of 2.2" is not admiration; it is arithmetic. How many did you try, and what is the best of that many worth under the null? That is the deflated Sharpe ratio (Bailey and López de Prado): set the bar at SR₀ = E[max], then ask for the probability the observed Sharpe exceeds it, given the sample length and the shape of the return distribution:

DSR = Z( (SR̂ − SR₀)·√(T − 1) / √(1 − γ₃·SR̂ + (γ₄ − 1)/4·SR̂²) )

with γ₃ skewness and γ₄ kurtosis of the returns; under normality the denominator is √(1 + SR̂²/2). Every Sharpe in that formula is per-observation, not annualised (divide an annualised Sharpe by √252 for daily data) and mixing the two is by far the most common way this statistic is reported wrong. For our winner: SR₀ = 1.847 annualised (0.1163/day), SR̂ = 2.231 annualised (0.1405/day), T = 755, giving

DSR = 0.746: the winner of 1,000 trials clears its selection bar with 74.6% confidence, well short of a 95% bar. It fails.

That is the whole discipline in one number, and it is exercise QD1.03-e06.

How many trials did you really run?

The hard part: N is not the number of backtests you ran. Grid cells are correlated (a 10/50 crossover and an 11/50 crossover are almost the same strategy) so 1,000 grid cells are not 1,000 independent trials. Run the same seeded series through an actual grid: fast moving average 2–41 crossed with slow 42–66, exactly 1,000 cells.

A thousand correlated grid cells were worth about seventy-three independent bets. Inverting the expected-maximum formula to recover an effective N is a rough but genuinely useful estimate, and it is far better than either of the two things people actually do (use N = 1,000, which over-deflates; or use N = 1, which is the status quo). Note too how much lower the grid's maximum is than the independent-trials maximum: correlated searches explore less, so they find less, but they also feel more thorough, which is the trap.

Counting trials honestly is a discipline, not a calculation. The number includes every variant you ran and discarded, every "let me just try it with a 20-day window", every earlier version of the idea on the same data, and every parameter your colleague tried on the same dataset last quarter. It is not the number in your final notebook. Keep a research log with one line per configuration tested and the count is free; reconstruct it afterwards and it will be wrong low, always in the direction that flatters you.

How do you defend against backtest overfitting?

The other defences, in the order you should reach for them:

• Fewer parameters. A rule with two parameters and an economic reason to exist is worth more than a rule with nine and a good backtest. Complexity is a cost paid in degrees of freedom, and you can see it in the deflation.
• Parameter plateaus, not peaks. If 18/52 works and 17/51 and 19/53 do not, you have found a feature of this sample, not of the market. Plot the surface; a broad plateau is evidence, a spike is noise.
• Walk-forward (exercise QD1.03-e07): choose on a rolling in-sample window, evaluate on the immediately following out-of-sample window, stitch the out-of-sample pieces. The selection never sees the evaluation. It is not free of the problem (you can still overfit by tuning the walk-forward protocol itself) but it is honest about a much larger fraction of the pipeline.
• Purged, embargoed cross-validation (exercise QD1.03-e10) when labels overlap. If observation i's label is "the return over the next 21 bars", then i and i+1 share 20 of their 21 bars, and a random train/test split trains on the answer. Purge the training observations whose label window overlaps the test block, then embargo a few observations after it to handle serial correlation. KFold on financial data is a leak with a respectable name.
• A holdout you genuinely do not touch. Not "do not touch much". The value of a holdout is destroyed by looking, because you cannot unsee it; if you have checked it twice, it is a validation set and you no longer have a holdout.