p-value

The probability of seeing a result at least as extreme as the one observed if there were in fact no effect, the null hypothesis. A small p-value says the data would be surprising if nothing were going on; it is not the probability that the finding is true.

By convention p below 0.05 counts as significant, and the convention breaks down when many things are tested. Test 20 trading rules that have no real edge, and at a 5% threshold about one of them will look significant by chance. A backtest that reports only the winner of a large search is reporting that chance.

A tiny effect can be significant in a large sample and a large one insignificant in a small one, so read the size of the effect and its confidence interval as well. In finance, where the same history is mined again and again, the bar for a new signal should be far stricter than 5%.

Count the tests before you trust the winner.