A t-statistic means what it means only if the hypothesis was specified before the data were seen. Take the maximum of 400 t-statistics and you have a different object, distributed as a maximum rather than as a single draw, and the conventional table no longer applies.
The winner of Worked example 9's screen has a monthly Sharpe of 0.20 over 120 months, hence t = 0.20 × √120 = 2.190890 and a two-sided p-value of 0.028460. On its own that is a publishable result. As the best of 400, it is unremarkable: with 400 true nulls tested at 5%, twenty are expected to clear the bar by luck alone.
How do Bonferroni and Holm control the family-wise error rate?
Two families of correction exist, and they control different things.
Family-wise error rate control targets the probability of even one false positive. Bonferroni divides the significance level by the number of tests: 0.05/400 = 0.000125. Sidak solves 1 − (1 − α)^(1/N) = 0.000128225 exactly, assuming independence, which buys 2.58% more room and is essentially never worth the extra assumption. Holm is a step-down procedure that compares the smallest p-value to α/N, the next to α/(N−1), and so on, stopping at the first failure; it controls the family-wise rate exactly as Bonferroni does while rejecting at least as much. On the eight-p-value table in the practice set, Bonferroni rejects two and Holm rejects three, from identical data.
The cost is power, and it is severe. To clear Bonferroni at N = 400 with 120 monthly observations, a candidate needs a monthly Sharpe of 0.350187, which annualises to 1.213084. Very little in public markets clears that bar honestly, which means family-wise control applied to a factor screen rejects almost everything, true findings included.
Advertisement
How does Benjamini-Hochberg control the false discovery rate?
False discovery rate control targets a different and usually more sensible quantity: the expected proportion of the rejections that are false. Benjamini-Hochberg sorts the p-values ascending, finds the largest rank i for which p₍ᵢ₎ ≤ (i/N)q, and rejects everything up to that rank. On the 400-candidate screen at q = 0.10, BH rejects 5, of which 4 are genuine signals and one is a null, while Bonferroni rejects 1 and an uncorrected 5% screen rejects 33 with only 18 genuine among them.
Two honest notes on that result. The realised false-discovery proportion in that single run is 1 in 5, above the q of 0.10, because BH controls the expected proportion across repetitions rather than the realised one in any particular run. And the step-up rule takes everything up to the last passing rank, so a null at rank 5 is rejected while genuine signals at ranks 6 through 12 are not. Ranking by p-value sorts evidence from noise. That is weaker than sorting truth from noise, and the two are easy to confuse.
Which correction should you use?
Pick the target before you pick the procedure. A context where a single false claim is unacceptable wants family-wise control. A factor screen, whose output is a shortlist and whose false leads cost only time, wants false discovery rate control. Choosing the strict one and quietly relaxing it after seeing the results is the failure mode both were designed to prevent.