Once you accept that multiple testing inflates false discoveries, the next question is mechanical: given a family of p-values from several independent trials, which ones should still be trusted after correcting for how many you ran? The simplest correction, plain Bonferroni, divides the significance level by the number of trials and applies that one flat, stricter threshold to every trial, simple, always valid, and needlessly conservative, because it treats every trial as if it were the last one being checked rather than crediting the ones that were extraordinarily significant with a little more room. Holm's method (Holm, 1979) fixes exactly this without giving up any rigor: sort the p-values from smallest to largest, and check them in that order against a threshold that loosens as you go, at sorted rank k out of m total trials, the threshold is alpha / (m − k + 1), tightest at rank one and equal to the full, uncorrected alpha at the very last rank. Walk down the sorted list applying that widening threshold, and the moment one trial fails it, stop, that trial and every later (larger p-value) trial in the sorted order are marked not-significant too, even if a later trial's own looser threshold would, in isolation, have let it through. That stopping rule is not a technicality; it is the entire reason Holm's method is a valid correction rather than an ad hoc improvement, and it is also the single most common place a hand-rolled or AI-generated implementation gets subtly wrong, checking each rank's threshold independently and forgetting to stop, which silently resurrects exactly the false discoveries the correction exists to catch. Code pack exercise QD2.04-e06 (holm_bonferroni) hands you four candidate implementations and asks you to identify the one correct one against three that fail in exactly this family of ways, plain Bonferroni's flat threshold, the correct widening threshold with the stopping rule removed, and an off-by-one in the threshold's denominator, because recognizing why each wrong option is wrong is worth more than typing the right one from a template.
Advertisement
How do you keep an honest count of trials?
None of this arithmetic means anything if the "N" you feed it (the number of trials in the family) is not true. This is where the experiment ledger from the scheduling discussion returns, doing research-integrity work instead of scheduling work: every trial, however small the tweak, gets one append-only entry (its own stable identifier, its parameters, its result, its p-value) written the moment the trial runs, using the identical append_if_new pattern that kept your scheduler honest about duplicate firings. "Know your N" is not a slogan; it is a query against that ledger, and the discipline that makes it trustworthy is that the ledger is append-only and sits outside any single researcher's power to quietly delete an unfavorable run after the fact. A researcher who can edit their own trial history can always report N as "the one trial that worked," and no correction formula, however rigorous, survives contact with a ledger that lies about its own row count.
{
"skill": "Drafting an experiment-ledger schema or a Holm–Bonferroni implementation's boilerplate",
"use": "Have an assistant propose a first-draft ledger record schema (which fields a trial entry needs) or a first-draft step-down multiple-testing correction — then spend the time saved doing the one thing that actually matters here: checking the STOPPING RULE by hand against a case specifically built to expose it, because that is exactly where this class of implementation most often breaks.",
"tools": ["Chat assistants — Claude, ChatGPT, Gemini", "IDE copilots — GitHub Copilot, Cursor"],
"prompt": "Implement the Holm–Bonferroni step-down correction: sort p-values ascending, threshold at rank k of m is alpha/(m-k+1), and STOP marking further ranks significant the instant one rank fails its threshold — do not check remaining ranks independently. Then list, in your own words, exactly what happens if the stopping rule is omitted.",
"verify": "Build a test case with a p-value that would pass its OWN per-rank threshold in isolation, but sits after an earlier rank that already failed — QD2.04-e06's own hidden tests are built around exactly this case. If the generated code marks that later p-value significant anyway, the stopping rule is missing or wrong, however plausible the code reads. Never accept a multiple-testing correction on the strength of its docstring; recompute one case by hand (Primary-Source Guardrail: Recompute).",
"diy": "You must be able to trace the step-down algorithm by hand on a small p-value list — sorting, computing each widening threshold, and applying the stop — before you can tell a correct implementation from one of the three specific wrong ones this module's own code pack builds as distractors. If you cannot explain why the stopping rule matters, you cannot review code that implements it.",
"market": "US"
}