Backtest

Simulating a strategy on historical data to estimate how it would have performed.

Every backtest is optimistic, and the reasons are structural rather than accidental. The data usually excludes companies that failed, the rules were chosen after seeing the data, transaction costs and market impact are underestimated, and information is often used a little earlier than it was actually available.

The defences are known: out-of-sample testing, a held-out period never examined during development, realistic costs, and a strong prior against complexity.

A strategy with many parameters that works beautifully in-sample has usually fitted the noise.