Backtest

Simulating a strategy on historical data to estimate how it would have performed.

Every backtest is optimistic, and the reasons are structural. Its data usually excludes companies that failed, and its rules were chosen after seeing the data. Transaction costs and market impact are underestimated, and information is often used a little earlier than it was actually available.

Known defences exist. Test out of sample, and keep a held-out period that is never examined during development. Use realistic costs and hold a strong prior against complexity.

A strategy with many parameters that works beautifully in-sample has usually fitted the noise.