Reproducible Research, Deployment & the Quant Stack

Branch: Quant Developer (quant-dev) · Node QD2.04 · ~16 focused hours · Understanding gate ≥ 85%

Somewhere in every quant shop there is a spreadsheet, a Slack thread, or a researcher's memory holding a fact nobody wrote down: how many variations of "the momentum idea" got tried before this one made it into a memo. Nobody hid anything on purpose, a promising idea gets tweaked twenty times over a month, most tweaks quietly die, and the one that finally looks good becomes "the strategy." Six months later a portfolio manager asks a colleague to rerun the backtest and gets a different Sharpe ratio, and nobody can say why: which pandas version, which random seed, which exact slice of data, and how many other ideas were tried and discarded before this one, have all evaporated. Those two failures. I cannot get the same number twice and I cannot tell you how many times I looked before I found this number: feel like different problems. They are not. Both are a claim standing with no evidence behind it. This module's whole argument, the sentence you will spend sixteen hours earning the right to believe, is this: a number is not a result until someone else, or you, in six months, can rerun it and get the same answer, and it is not a finding until you can prove it wasn't just the best of many tries. Reproducibility and research honesty are the same discipline wearing two hats, and the fix for both is identical: replace trust with evidence, a logged, versioned, seeded pipeline as evidence a number reproduces; a logged, uneditable trial ledger as evidence a number wasn't cherry-picked.

You already have real machinery for the first half of that sentence. CS2.01 gave you the four legs of reproducibility, pinned code and dependencies, a fixed seed for every random draw, an identified exact data snapshot, and a documented run sequence, and showed you a container as the tool that automates them, gesturing forward, by name, to this exact node for "reproducibility at production scale." CS2.02 gave you the version control underneath leg one. CS2.06 gave you logging and the discipline of never leaking a secret into a log line. QD1.03 went further still: it shipped two working code harnesses (no_lookahead and cost_sensitivity) and taught you, as prose and as a checklist, all four gates a real strategy must clear before anyone trusts it: no_lookahead, cost_sensitivity, determinism, oos_split. QD1.03 says this about the two of you, in so many words: this node ships two of the four harnesses; QD2.04 industrialises all of them. That is your job in the hours ahead, not to reteach what a harness checks, but to build the surrounding machinery that makes running all four, honestly, on a schedule, at scale, the only way a strategy ever reaches a portfolio manager's desk. And you have new machinery to build for the second half of the sentence, because reproducibility alone does not save you from the oldest trap in empirical research: if you try enough variations, one will look good by pure chance, and a perfectly reproducible, perfectly seeded, perfectly logged run of a worthless strategy is still worthless. Marcos López de Prado's name belongs on this half of the module for the same reason Fama and French's belongs on a factor model, he is the person who took "we tried a lot of things and this one worked" from an embarrassing admission to a rigorously quantifiable, correctable risk, and gave the field a vocabulary (backtest overfitting, the multiple-testing problem, deflated performance metrics) for exactly the failure this node exists to close.

By the end of these sixteen hours you will be able to hand a colleague, or your future self, a research pipeline with four properties none of your earlier modules demanded together: it reads its settings from a single resolved configuration built from a fixed, auditable precedence order; it never has a real secret sitting in its source or its logs; it runs on a schedule without double-counting a retried job; and it keeps an append-only ledger of every trial anyone has ever run against an idea, so that when a Sharpe ratio finally does look good, the honest question, out of how many tries?, has a true, checkable answer instead of a shrug. That combination is what "the quant stack" means in this module's title: not one tool, but the small set of disciplines (configuration, secrets, seeds, scheduling, logging, and an honest count of your own trials) that turn a script on a researcher's laptop into a production research process a firm can actually stand behind.

Code pack. This node ships app/public/code-packs/QD2.04.json: nine graded exercises, QD2.04-e01 through QD2.04-e09, spanning layered configuration, secret-reference resolution, run-manifest drift detection, a minimal cron scheduler, seeded exponential backoff, the Holm–Bonferroni multiple-testing correction, an idempotent experiment ledger, walk-forward split generation, and one SQL query against a trial ledger. Every exercise ships a structured (fill-in-the-blank / order-the-lines / choose-the-correct-implementation) fallback, so the entire node is completable offline with zero runtime, exactly as the region's authoring law requires; live execution in your browser is progressive enhancement on top of that guarantee, never a requirement.

Learning objectives

By the end you can:

1. State the module's thesis from memory, a number is not a result until it reproduces, and not a finding until you know how many tries it took, and explain why reproducibility and multiple-testing discipline are one habit, not two.
2. Resolve a layered production configuration (defaults, file, environment, invocation) into one auditable config, in a fixed, stated precedence order, without mutating any input layer.
3. Distinguish a .env secret (CS2.01) and a logging redaction filter (CS2.06's redact) from a secret-reference indirection system, and explain why a pipeline should fail loudly, not silently, on a missing secret.
4. Build and use a run manifest (the single artifact that ties a run's seed, resolved config, code version, and data snapshot together) and use it to detect drift between two runs that should have been identical.
5. Explain why "I set the seed" is necessary but not sufficient for determinism once a pipeline has more than one source of randomness, and connect this to QD1.03's determinism harness as the test your own infrastructure must be built to pass.
6. Design a schedule (cron-style), and explain idempotency, why a scheduler that fires at least once, not exactly once, requires your pipeline to tolerate being invoked twice for "the same" run without corrupting its own record of what happened.
7. Implement exponential backoff with full jitter for a retrying scheduled job, from your own seeded generator, and explain why a retry policy is part of a pipeline's reproducible behavior, not an implementation detail beneath notice.
8. State the multiple-testing problem in one sentence, compute the expected number of false "discoveries" from N independent skill-less trials at a stated significance level, and explain why judging each trial against the ordinary threshold guarantees false positives as your trial count grows.
9. Implement the Holm–Bonferroni step-down correction by hand and in code, explain why it dominates plain Bonferroni without sacrificing rigor, and apply it to a family of backtest trial p-values.
10. Explain why an experiment ledger must be append-only and outside any single researcher's power to quietly edit, and connect this directly to why QD2.05's capstone scores honesty about limitations as its own graded line, separate from a strategy's raw performance.
11. State, precisely and without re-deriving any of it, the division of labor across QM1.02/QM1.03 (the statistics), QD1.03 (a single backtest's own parameter-search deflation and the four harnesses), QD2.03 (deflated Sharpe as a reported performance statistic), and this module (the logged, ledgered, scheduled infrastructure that makes every one of those numbers honest).
12. (Productivity objective: R10 duality.) Use an AI copilot to draft the unglamorous scaffolding of deployment infrastructure, a scheduler config, a logging skeleton, a ledger schema, while keeping the decision-bearing logic, and every subtle correctness question a reviewer must check, for yourself.

The duality, stated once (R10). Objectives 1–11 are the understanding gate; you clear it by building the infrastructure and reasoning about it correctly, with or without an AI tool in the room. Objective 12 is the productivity payoff you keep afterward. No tool buys you a pass on the gate, this region's own rule, R10, holds here exactly as it does everywhere else in the app: the app itself never calls a model at runtime, so every AI-flavored exercise is "generate it in your own tool, paste it back, our hidden tests grade it locally."

Prerequisites & connections

Builds on. CS2.01's four legs of reproducibility (pinned code and dependencies, a fixed seed, an identified data snapshot, a documented run sequence) are the foundation this entire module stands on, this node does not re-teach them, it operationalizes them at a scale one script on a laptop never has to face: many config sources, a scheduler that can misfire, secrets that must never touch a log, and a team, not a single researcher, generating trials. CS2.02's Git discipline is the silent precondition for "the code is version-controlled", a pipeline is not reproducible if the code that ran it cannot be checked out again byte-for-byte, and that fact is CS2.02's territory, referenced and never re-derived here. CS2.06 gave you logging at the module level and the redact pattern for masking a secret-looking key before it reaches a log line; this module assumes both and extends them to a full run manifest and a secret-reference resolution system, which solves a different problem than redaction (redaction hides a secret from a log after it is already in your config; resolution keeps the real secret out of your config in the first place, pulling it in only at the moment of use). QD1.01's point-in-time, survivorship-controlled data panel is the "exact data snapshot" this module's manifest records an identifier for: QD1.01 builds the panel; this module is content merely to know, and log, which snapshot version a run touched. QD1.02 already named this module as "the direct forward reference" for evaluating many candidate signals without fooling yourself, that promise is redeemed below. QD2.02's seeded Monte Carlo VaR engine (QD2.02-e03, drawing from numpy.random.default_rng(seed), never the shared global state) is this module's thesis in miniature, one module early: a fixed, logged seed is what turns a random draw into a checkable fact, and you will meet it again as a worked example. QM1.03 (Time-Series, Monte Carlo & Intro ML-for-Finance) taught the concept of overfitting as a bias-variance tradeoff and named look-ahead bias as the finance-specific trap of a naive train/test split, a concept this module assumes and never re-derives (C-3 boundary: QM owns the statistics and the math; this branch owns the code, the pipeline, and the process).

The QD1.03 hand-off, stated precisely. QD1.03 shipped two working code harnesses: no_lookahead (QD1.03-e02) and cost_sensitivity (QD1.03-e09), plus a complete conceptual account of all four harnesses a strategy must clear: no_lookahead, cost_sensitivity, determinism, oos_split. In QD1.03's own words: this node ships two of the four harnesses; QD2.04 industrialises all of them. This module honors that hand-off directly, not by rebuilding a harness's internal logic (QD1.03 already owns that, correctly, and this module does not re-derive it), but by building the deployment machinery that turns "run the four harnesses if you remember to" into "a promotion pipeline that cannot skip them." You will meet this explicitly in Core Teaching Content, and it is the thread that ties this module's scheduling and logging content directly back to QD1.03's harness table.

Feeds forward. QD2.03 (Performance & Attribution Analytics in Code) computes and reports the deflated Sharpe ratio as a performance statistic, this module does not compute that formula and does not need to; it builds the ledger that makes QD2.03's reported trial count true. QD2.05, the branch's capstone, requires four things this module installs directly: a determinism harness that actually passes because the whole pipeline underneath it, config, secrets, scheduler, retries, is reproducible by construction; an oos_split harness whose "how many trials" figure comes from a ledger, not a memory; a reproducibility section in the final write-up; and, the line worth reading twice, honesty scored as its own graded dimension: a strategy that only works in-sample still passes if its limitations are reported honestly, and one that hides them fails. Every discipline in this module exists to make that honesty checkable rather than a matter of the researcher's word.

Core teaching content

The quant stack, and the two failures this module closes

Hold the thesis sentence steady, because everything below is a consequence of it: a number is not a result until it reproduces, and not a finding until you know how many tries it took. Split it in half and each half names one failure this module exists to close. "Until it reproduces" is the reproducibility failure, the same script, the same inputs, run again, gives a different answer, and nobody can say why. "Until you know how many tries it took" is the research-integrity failure, a Sharpe ratio that looks real but is actually the lucky winner of fifty quiet, unlogged attempts. Notice what the two failures share: in both cases, evidence that would settle the question was never captured. A seed that was never logged cannot be reproduced, not because reproducing it is hard, but because the fact of what was run is gone. A trial that was never logged cannot be corrected for, not because the correction formula is hard, but because the count is gone. The fix, in both halves, is the same kind of artifact: something written down, outside the researcher's ability to quietly edit after the fact, that turns "trust me" into "here is the log."

"The quant stack" in this module's title names the small set of disciplines that make that logging real rather than aspirational: a configuration system with one resolved truth instead of four competing guesses about what setting is actually in effect; a secrets system that never puts a real credential where a config file, a log line, or a git commit can capture it; a seed and manifest discipline that makes "run it again" a testable claim instead of a hope; a scheduler that survives being invoked twice for the same logical run without corrupting its own record; logging that makes a run legible without leaking it; and an experiment ledger that makes "how many times did we try this" an auditable fact. None of these are exotic. All six are things a well-run engineering team does as a matter of course and a research desk skips as a matter of course, and the gap between those two teams, at scale, is this entire module.

Configuration at production scale: precedence, validation, one source of truth

A script on a laptop hardcodes a file path and calls it a day. A production pipeline reads its settings from at least four places, in a fixed order, because each layer answers a different question. Defaults live in the code itself and answer "what should happen if nobody says otherwise." A file config, checked into version control alongside the code (CS2.01's project layout), answers "what does this deployment usually want", a different default for a staging environment than for production, say. Environment variables, set by whatever system actually launches the process (a scheduler, a container orchestrator, a CI runner), answer "what does this specific machine or environment need right now", a database host that differs between a laptop and a cloud box. And a CLI argument or invocation-time override answers "what does this specific run, right now need to be different", a one-off backtest with a different date range than the daily job. The rule that makes four answers into one is precedence: each layer, applied in a fixed order, overrides only the keys it actually mentions, leaving every other key exactly as the previous layer left it. Get the order backwards even once, read the environment before the file, say, and a deploy-time override silently loses to a stale checked-in default, and the bug will not throw an exception; it will just quietly run with the wrong setting, which is worse.

Code pack exercise QD2.04-e01 (resolve_config) asks you to implement exactly this merge (defaults, then file, then environment, then CLI, each later layer overriding the earlier one key by key) and to do it without a single mutation of any input, because a pipeline that silently rewrote its own defaults dict in place would corrupt every later call in the same process, a bug that would only show up on the second run of the day, which is exactly the kind of bug nobody catches in a demo. The discipline generalizes past any one config format: whether your real system reads YAML, TOML, or a Python dict, the four-layer precedence order and the "never mutate an earlier layer" rule are the same two facts worth memorizing.

The run manifest: the artifact that ties seed, config, code, and data together

CS2.01's Worked Example 6 built a fingerprint: a short, stable string combining pinned dependency versions with a dataset snapshot's own id, seed, and date, engineered so that two teammates who installed packages in a different order still land on the identical fingerprint if their environments truly match. QD1.03 restates the same instinct as a bullet-point discipline for a single backtest: seed everything and record the seed, freeze the data snapshot and hash it, version the configuration with the result rather than trusting memory. This module's job is to turn that instinct into a standing artifact that survives across many scheduled runs over the pipeline's whole lifetime, not just inside one script's single invocation: a run manifest, a small, flat record, written at the end of every run, holding exactly the facts that determine whether two runs are the same experiment: the resolved config (or its hash), the seed, the code version (a Git commit hash, CS2.02's territory), and the data snapshot identifier. Alongside those load-bearing facts sit volatile fields that are expected to differ on every single run and mean nothing on their own, a run id, a start timestamp, a wall-clock duration.

The manifest earns its keep the moment you use it to answer a question a bare "byte-compare the output" cannot: why did two runs differ? QD1.03's determinism harness runs a whole pipeline twice and byte-compares the output, the right test, cheap to run, and this module does not replace it. But at production scale, byte-comparing the output only tells you that something moved, not what. Code pack exercise QD2.04-e03 (manifest_diff) closes that gap: given two manifests and the set of keys understood to be volatile, it reports every non-volatile key that differs, and only those, a code version bump, a changed dataset snapshot, a config key that quietly appeared between runs, while silently ignoring the run id and the timestamp that were never supposed to match in the first place. Two runs that really are the same experiment produce an empty diff; anything else is a specific, named fact about what changed, not a shrug.

Recall QD2.02's Monte Carlo VaR engine (QD2.02-e03) as this module's thesis in miniature: it draws every random number from numpy.random.default_rng(seed), never the shared global generator, for exactly one reason, so that its own hidden tests could grade a stochastic simulation deterministically, calling the same function twice with the same seed and demanding the identical answer. A logged, fixed seed turned an inherently random process into a checkable fact one module before this one even opened. This module's manifest is what makes that same guarantee hold not just for one function's random draws, but for an entire pipeline's config, code, and data, run after run, on a schedule, indefinitely.

Logging a run so it can be debugged and reproduced, never leaked

CS2.06 already gave you the mechanics: a module-level logger = logging.getLogger(__name__), the five severity levels used as designed, logger.exception inside an except block for the traceback it attaches for free, and the redact pattern that masks any secret-shaped key before a record is ever written. This module adds nothing to that mechanism and everything to what you point it at. CS2.06 stated the rule this module exists to satisfy in one line: a quant run whose seed was not logged cannot be reproduced. Read that literally, the two things worth logging, at INFO level, on every single run without exception, are (1) the run manifest itself (seed, resolved-config hash, code version, data snapshot id, everything manifest_diff needs later to explain a discrepancy) and (2) enough shape-of-the-data facts (row counts loaded, date range covered, tickers processed) that a human skimming the log can sanity-check the run without re-executing it. Neither of those is optional decoration; both are the evidence half of this module's thesis, made concrete as text on disk. And the redaction rule from CS2.06 applies with extra force here, because a resolved config (the output of resolve_secrets, the very thing you are about to log for reproducibility's sake) is, by construction, a structure that now contains the real secret values. The order matters and is easy to get backwards under deadline pressure: resolve secrets to run the pipeline, but redact the config before it ever reaches a log call. Logging the pre-resolution config (still full of ${secret:...} placeholders) is safe but useless for debugging a connection failure; logging the post-resolution config unredacted is a live credential in a log aggregator, a support ticket, and a screenshot, forever. Redacting the resolved config is the only option that is both useful and safe.

Shipping it: containers and the promotion checklist

CS2.01 named the tool this whole module's infrastructure is heading toward without building it, because building and running one needs a real container engine this sandbox cannot provide: a container (Docker is the dominant example) packages the code, the interpreter, the pinned dependencies, and often the operating system itself into one portable image that runs identically wherever a container engine is installed. Nothing above changes: a container automates the same four legs CS2.01 taught, plus this module's scheduling and secrets discipline layered on top, the same way a calculator automates arithmetic you must already understand, and the first time a containerized pipeline does not behave the way you expect, you will have nothing to reason from if you skipped learning the four legs, the manifest, and the ledger by hand first. Treat a container as the packaging format for everything this module teaches, never a substitute for understanding any of it: a container that runs an unseeded random draw, or reads a config from an untracked file mounted at runtime, is exactly as unreproducible as the un-containerized version, it just fails more expensively, because now three people believe it must be reproducible, since "it's containerized."

The practical question a research pipeline eventually has to answer is which deployment shape it needs. A batch pipeline runs on a schedule, processes a bounded amount of data, and finishes, a nightly ingestion job, a weekly factor recompute, the shape almost everything in this module has assumed so far. A pipeline promoted to something closer to real time, reacting to each new tick or order-book update rather than waiting for a nightly window, inherits every discipline above (a config, secrets, a manifest, logging) but adds a much less forgiving latency budget and a much less forgiving idempotency requirement, since "at least once" delivery under a tight time budget is a genuinely harder problem than "at least once, sometime in the next hour" is. Most research infrastructure lives in the batch shape for good reason, it is dramatically easier to make reproducible, and the honest engineering judgment is to stay there until a specific, named requirement forces the harder shape, not to reach for real-time architecture because it sounds more serious.

Industrializing the four harnesses: from a script you remember to run, to a gate nothing can bypass

QD1.03 put a name on your job in one sentence: this node ships two of the four harnesses; QD2.04 industrialises all of them. Read "industrialise" precisely, it does not mean rebuilding no_lookahead's future-perturbation logic or cost_sensitivity's cost-doubling check; both stay exactly where QD1.03 built them, and re-deriving either here would violate this region's own R1 rule. It means: a research pipeline built with everything above, a resolved config, secrets kept out of the ledger, a run manifest, a schedule, an idempotent write path, redacted logs, is precisely the machinery that turns "the four harnesses" from a checklist a tired researcher might skip under deadline pressure into a gate a promotion step cannot bypass. Picture the promotion step itself as one more scheduled job, built with this module's own tools: it has a config (which strategy version, which harness thresholds); it reads a registry credential through the same secret-resolution pattern as any other job; its own dry run is seeded and manifested, so promoting a strategy is itself reproducible; and every promotion decision, pass or fail, and on which harness, is one more append-only entry in the same kind of ledger this module builds for research trials. Run no_lookahead and cost_sensitivity (QD1.03's own code) and determinism (a byte-compare of two full pipeline runs, made possible because every source of randomness anywhere in the pipeline, including a retry policy's jitter, now comes from a logged seed) and oos_split (this module's walk-forward machinery, below) as automated steps in that one promotion job, and refuse to promote on any failure. Nothing about any individual harness has changed. What has changed is that "did anyone actually run all four" stopped being a question about a person's diligence and became a question about whether a pipeline step exists, which is precisely what "industrialise" means.

Deflated metrics: four modules, one instinct, no duplication

Four different modules in this branch touch the same underlying instinct (a good-looking number produced by trying many things is not automatically a real number) and each owns a genuinely different layer of it, on purpose, with no overlap:

Read the table as a single sentence: the statistics tell you how to correct, QD1.03 applies one specific correction inside a single backtest's own parameter search, QD2.03 reports the result, and this module makes sure the count everyone is correcting for was never a guess in the first place.

The reproducibility + research-integrity checklist before you ship a strategy

Pull every thread in this module into one checklist, because that is the shape a real promotion decision takes. Configuration: is there one resolved config, built from a stated, fixed precedence order, with no setting living only in someone's memory? Secrets: does every credential reach the pipeline through resolution at the point of use, never as a literal in code, a checked-in file, or an unredacted log line? Determinism: does every source of randomness anywhere in the pipeline (including a retry policy's jitter) trace to a seed that is itself logged in the run manifest, such that running the whole thing twice is byte-identical (QD1.03's determinism harness, passing because the infrastructure underneath it was built to let it pass)? Scheduling: is the pipeline idempotent against being invoked twice for the same logical run, and does its retry policy avoid a thundering herd? Logging: does the run manifest get written and logged on every run, redacted, before anything else touches a log line? The ledger: is every trial (every parameter tweak, every variant, every "quick check" that a researcher was tempted to consider off the books) logged append-only, the moment it runs, where nobody can quietly remove the ones that didn't work? The correction: given the ledger's true trial count for this family of ideas, has the appropriate multiple-testing correction actually been applied, and is the survivor honestly reported whether or not it is the answer anyone wanted? And last, the line QD2.05's capstone grades as its own scored dimension, separate from raw performance: is the limitations memo honest, does it say, in plain language, how many trials this result survived, which corrections were applied, and what would have to be true for the result to be noise rather than skill? A strategy that clears every line above and turns out, honestly reported, to be marginal has passed this module's discipline. A strategy with a beautiful Sharpe ratio and a blank answer to "out of how many tries" has not, no matter how good the number looks.

Common mistakes & how experts think differently

1. "I set the random seed, so my pipeline is reproducible." A single random.seed(42) call only covers whichever generator that specific call touches. NumPy, Python's random module, and scikit-learn each maintain their own independent generator state; a pipeline that seeds one and forgets the others gets identical results from the seeded part and different results from everything else, and the failure looks like flaky code rather than an incomplete seed. The expert seeds every generator the pipeline actually uses, explicitly, and records every seed in the run manifest, never relies on "I set a seed" as a synonym for "this pipeline is deterministic."
2. "I pinned my dependencies, so I don't need to bother logging the seed or the data snapshot." CS2.01's four legs are independent and all required; pinning one does not substitute for the others. A perfectly pinned environment with an unlogged data snapshot still cannot tell you, six months later, whether last Tuesday's numbers used this week's data or a stale copy from three weeks prior.
3. "My cron job only fires once, so I don't need to worry about idempotency." Almost no real scheduler guarantees exactly-once delivery; at-least-once is the practical norm, and a network hiccup, a restart, or an overzealous retry policy will eventually invoke your job twice for the same logical run. The expert builds every ledger write and every side-effecting action to tolerate being called twice with the same identifier, as a default habit, not a response to an incident that already happened.
4. "I found a strategy that backtests beautifully after trying two hundred variations, the Sharpe ratio proves it." This is the multiple-testing problem in its purest, most common form. Trying two hundred variations of a skill-less idea against a 5% significance bar produces an expected ten "significant" results from chance alone; a beautiful Sharpe ratio, with no correction applied and no honest trial count attached, proves nothing about real skill. The expert's first question on seeing an impressive backtest is never "how good is the number", it is "out of how many tries, and where is the ledger."
5. "I used a walk-forward split, so I'm safe from overfitting." A walk-forward or purged split defends against look-ahead bias, training on data that, in real time, comes after the period being predicted. It says nothing about how many walk-forward configurations, lookback windows, or purge widths you tried before settling on this one; that is a separate multiple-testing exposure requiring its own correction, on top of, not instead of, the walk-forward discipline.
6. "A secrets manager means I can log the resolved config safely." Resolving a secret reference produces a structure that now contains the real credential value, a secrets manager keeps the credential out of your checked-in files, but does nothing to protect a log line that prints the resolved config afterward. The expert redacts the resolved config, every time, immediately before any log call, treating "I used a secrets manager" and "this config is safe to log" as two entirely separate claims.
7. "If the pipeline runs inside a container, it's reproducible." A container freezes the operating system, the interpreter, and the pinned dependencies, exactly CS2.01's leg one, automated. It does nothing, by itself, about an unseeded random draw, a config read from a file mounted fresh at each run, or a data snapshot silently re-fetched from a live source with no version pin. Containerizing an unreproducible pipeline produces an unreproducible pipeline that three people now incorrectly believe must be reproducible, which is a worse failure mode than the honest, visibly-fragile script it replaced.
8. "Applying Holm–Bonferroni can only ever be safer than not correcting at all, so the choice of correction barely matters." Skipping any correction is the dangerous default, but the choice between corrections is not cosmetic either: plain Bonferroni's flat threshold is needlessly conservative and can reject genuine discoveries that Holm's widening, step-down threshold would correctly retain, on one of this module's own worked examples, plain Bonferroni credits a single survivor where Holm, applied to the identical data, correctly credits all four. Under-correcting invites false discoveries; over-correcting throws away real findings. Both are mistakes, and "more conservative" is not automatically "more correct."
9. "I log every run, so my research process is honest." Logging is necessary and not sufficient. A ledger that a researcher can quietly edit or delete unfavorable rows from provides exactly the same false confidence as no ledger at all, the discipline that makes a ledger meaningful is that it is append-only and sits outside the researcher's own power to rewrite history, not merely that entries get written somewhere at the time.

Worked examples

Worked example 1: Config precedence resolved by hand, and the bug when the order is reversed (mechanism, illustrative)

A research pipeline's settings live in four layers: defaults = {"universe": "sp500", "cost_bps": 10, "seed": 1}, a checked-in file_cfg = {"cost_bps": 8}, an environment layer env_cfg = {"seed": 2024}, and a CLI override for one specific run, cli_cfg = {"universe": "nifty50"}.

Solution. Step 1, apply the layers in the CORRECT order (defaults → file → env → cli), one key at a time. Start from defaults: {"universe": "sp500", "cost_bps": 10, "seed": 1}. Apply file_cfg: only cost_bps is mentioned, so it becomes 8; the rest is untouched: {"universe": "sp500", "cost_bps": 8, "seed": 1}. Apply env_cfg: only seed is mentioned, updating to 2024: {"universe": "sp500", "cost_bps": 8, "seed": 2024}. Apply cli_cfg: only universe is mentioned, updating to "nifty50": final resolved config = {"universe": "nifty50", "cost_bps": 8, "seed": 2024}. Step 2, now apply the layers in the WRONG order (cli before env, env before file) to see the bug. Start from defaults, apply cli_cfg early: {"universe": "nifty50", "cost_bps": 10, "seed": 1}. Apply env_cfg: {"universe": "nifty50", "cost_bps": 10, "seed": 2024}. Apply file_cfg LAST: {"universe": "nifty50", "cost_bps": 8, "seed": 2024}, in this particular example the final answer happens to come out identical, because no two layers touched the same key in a way the reordering could expose. Step 3, construct the case that DOES expose it. Suppose instead file_cfg = {"cost_bps": 8} and cli_cfg = {"cost_bps": 12}, a researcher's explicit one-off override, meant to win. Correct order (file, then cli, last): final cost_bps = 12, exactly the override the researcher asked for. Wrong order (cli applied before file): file is applied last and silently overwrites the researcher's explicit override back down to 8, the run proceeds with the wrong cost assumption, no error, no warning, and the researcher has no way to discover this from the output alone. Read it. The bug is invisible in code review unless someone specifically checks the order of the four .update() calls; it produces a plausible number every time; and it will not be caught by any single-layer test, only by a test that deliberately sets conflicting values across two layers and asserts on which one wins, precisely what QD2.04-e01's hidden tests do.

Worked example 2: Konkan Quant Capital's pre-market job fires at the wrong hour (India, ₹)

Konkan Quant Capital (synthetic) (a Mumbai-based systematic desk) schedules its NSE ingestion job to run once daily at 9:15 AM IST, expressed in this module's simplified minute-of-day model as minute_set={15}, hour_set={9}. A junior engineer, checking whether the job is due, calls the scheduler's search function at 8:00 AM (minute-of-day 480) the morning after deployment.

Solution. Step 1: run the search. next_run_minute({15}, {9}, 480): walk forward from candidate 481 onward, checking each candidate's minute and hour. The search finds candidate 555 first (555 = 9×60 + 15), where minute = 555 mod 60 = 15 and hour = 555 // 60 = 9, both match. So the next run is minute-of-day 555, i.e., 9:15 AM the same day, exactly as intended. Step 2, the engineer now checks what happens if the job's OWN process asks "am I due?" at exactly 9:15 AM, the instant it is scheduled to fire, a realistic check, since a long-running process might poll its own schedule. next_run_minute({15}, {9}, 555): the search must find a candidate strictly greater than 555. Walking forward, no other minute-of-day within the same 24-hour cycle has both minute 15 and hour 9, the very next match is exactly one day later, at candidate 1995 (1995 = 1440 + 555, i.e., 24:00 + 9:15). Step 3, the trap this exposes. If the search function had been built with a non-strict comparison (accepting candidate == after_minute_of_day as a valid match), it would report that the job is "due right now, at 9:15," every single time the polling check itself happens to run at exactly 9:15, potentially firing the ingestion job twice in the same minute, once from the intended schedule and once from a poll that mistook "now" for "next." Read it. The strict "must be later, never equal" rule is not a stylistic nicety; without it, a scheduler's own status check can trigger a duplicate run, which is exactly the failure QD2.04-e07's idempotent ledger exists to absorb even when a scheduling bug like this one slips through.

Worked example 3: Meridian Systematic Partners' duplicate trial, caught by the ledger (US, $)

Meridian Systematic Partners (synthetic) (a Chicago-based systematic fund) logs every backtest trial to an append-only experiment ledger, keyed by a run_id. Their scheduler retries a job automatically if it does not receive a completion signal within ninety seconds; on a particular Tuesday, a trial legitimately takes ninety-four seconds to complete, the scheduler concludes it failed, and fires an identical retry with the same run_id, which finishes normally four seconds later.

Solution. Step 1, trace what a naive ledger (plain append, no duplicate check) would do. Both the original 94-second run and the retry write their trial result to the ledger with run_id = "mtm-2024-118". The ledger now has two rows for the same logical trial, an accidental duplicate, indistinguishable at a glance from two genuinely different, independent trials of the same strategy family. Step 2, see the downstream damage. Whatever queries the ledger next, a Holm–Bonferroni correction across the family, or simply a headcount of "how many times have we tried this", now silently double-counts one trial as two, inflating the apparent trial count for a correction that should have run on one fewer data point, and, worse, inflating the apparent support for whichever family that duplicate belonged to. Step 3: apply append_if_new instead. The retry's write calls append_if_new(ledger, entry) with entry["run_id"] == "mtm-2024-118"; the function scans the existing ledger, finds a row with that exact run_id already present, and returns the ledger completely unchanged, no duplicate row, no silent inflation. Read it. The fix required no change to the scheduler's retry policy, which was behaving exactly as designed (ninety seconds is a defensible timeout; a slow run four seconds over it is a normal, expected event), the fix lives entirely in making the ledger itself tolerant of being told about the same trial twice, which is the correct place to put it, since a scheduler cannot promise exactly-once delivery but a ledger absolutely can promise exactly-once recording.

Worked example 4: Determinism, byte for byte, and the one-line bug that silently breaks it (mechanism, illustrative)

A pipeline computes a summary statistic by iterating over a Python dictionary of ticker-level results and summing their values, then logs a manifest and finishes. Run twice from the identical seed, config, and data snapshot, its author expects, and, on Python 3.7 and later, correctly gets, byte-identical output, because dictionary iteration order is guaranteed to match insertion order.

Solution. Step 1: confirm the happy path. Two runs, same inputs: the dict is built by inserting tickers in the same order both times (the order they appear in the source data file), so iterating and summing produces the identical intermediate sequence and the identical final float, both times. Manifests match; manifest_diff on the two runs reports an empty dict; QD1.03's determinism harness, run as a byte-compare of the full output, passes. Step 2, introduce one innocuous-looking change: a maintenance pass that "cleans up" the code by building the dict from a set of tickers instead of a list, for what the author believes is identical behavior with tidier code. A set's iteration order is not guaranteed to match insertion order and, critically, is not guaranteed to be stable across two separate runs of the same process even with the identical elements, because it can depend on Python's string-hash randomization (seeded per-process by default, specifically as a security measure against hash-collision attacks). Step 3: trace the consequence. Summing floating-point numbers is not perfectly associative, (a + b) + c and (b + a) + c can differ in their last bit or two due to floating-point rounding, so a different iteration order can produce a summary statistic that differs from the previous run in the fifteenth decimal place. determinism's byte-compare now fails, apparently at random, on some runs and not others, and nothing about the config changed between runs, manifest_diff reports an empty diff, because the config genuinely didn't move, which is itself the clue: a determinism failure with an empty manifest diff means the non-determinism lives inside the code's own logic, not in any input the manifest tracks. Read it. "The seed is set" was never the actual guarantee anyone needed, the guarantee needed is "every step of this pipeline, including implementation details as small as which built-in collection type holds an intermediate value, produces the same sequence of operations on the same inputs, every time." A set where a list belonged is invisible in a code review and only surfaces as an intermittent, maddening determinism-harness failure, exactly why QD1.03 insists on running that harness for real, on the actual pipeline, rather than reasoning about determinism from reading the code.

Worked example 5: Twenty NSE momentum variants, and what survives correction (India, ₹)

Konkan Quant Capital tries six independent momentum-signal variants against fifteen years of NSE data (illustrative counts and p-values, not a claim about any real fund's actual research), obtaining these p-values against the null of "no real predictive skill," sorted for convenience: [0.008, 0.012, 0.02, 0.03, 0.04, 0.09].

Solution. Step 1, the naive check. Testing each variant individually against the ordinary 5% bar: 0.008 < 0.05 ✓, 0.012 < 0.05 ✓, 0.02 < 0.05 ✓, 0.03 < 0.05 ✓, 0.04 < 0.05 ✓, 0.09 < 0.05 ✗, five of the six variants look "significant." A researcher stopping here would report five apparently-real signals. Step 2, apply Holm–Bonferroni, m = 6, alpha = 0.05. The per-rank thresholds, from rank 1 to rank 6, are 0.05/6 ≈ 0.00833, 0.05/5 = 0.01, 0.05/4 = 0.0125, 0.05/3 ≈ 0.01667, 0.05/2 = 0.025, 0.05/1 = 0.05. Compare in order: rank 1 (p = 0.008) against 0.00833: passes, survives. Rank 2 (p = 0.012) against 0.01: 0.012 > 0.01, fails. The stopping rule now applies: every remaining rank (0.02, 0.03, 0.04, 0.09) is marked not-significant, regardless of the fact that rank 3's own threshold (0.0125) would, checked in isolation, have let 0.02 through, because it is not checked in isolation once an earlier rank has already failed. Corrected result: only the first variant survives. Step 3: quantify the gap. The naive check credited five variants with apparent skill; the corrected check credits one. Four of the five "discoveries" a researcher stopping at Step 1 would have reported and possibly funded were, after correcting for having tried six variants, indistinguishable from noise. Read it. This is the multiple-testing problem made concrete with real, hand-computed thresholds rather than an abstract warning, and it is exactly the shape of question QD2.04-e06's hidden tests grade, with different p-value families designed to exercise the same stopping rule.

Worked example 6: Meridian's five-day label, and why the split boundary alone isn't enough (US, $)

Meridian Systematic Partners builds a signal whose label is "the security's return over the next five trading days," observed and computed at each day i using prices through day i + 5. Their scheduled retraining pipeline uses a walk-forward split with a training window ending at index 99 (observations 0 through 99) and a test window beginning at index 100.

Solution. Step 1, confirm the split satisfies the observation-level, look-ahead-free property this module's walk_forward_splits guarantees by construction. Training uses observation indices 0–99; test uses indices 100 and later; the training range's end (100, exclusive) is not later than the test range's start (100), the invariant QD2.04-e08's hidden tests check holds exactly. Step 2, now examine the LABELS attached to the last few training observations, not just their indices. The observation at index 99 carries a label built from prices at indices 99 through 104, four of those five days (100, 101, 102, 103, 104) fall inside the test period. The observation at index 95 carries a label built from prices at indices 95 through 100, day 100, the very first test-period day, is already inside its label window. Every training observation from index 95 through 99 has a label reaching at least one day into the test period; the model would be trained, in part, on information about test-period prices, smuggled in through the label rather than through the observation itself. Step 3: apply the purge. Drop training observations 95 through 99 (five observations, exactly the label horizon) from the training set entirely, training instead on indices 0 through 94; the remaining training labels (built from indices i through i + 5 for i up to 94) reach at most to index 99, strictly before the test period begins at 100. Read it. The walk-forward split's index-level invariant (train ends at or before test begins) is necessary and was never violated, the leak lived one layer deeper, in what the label depended on, not in what the observation was. This is precisely why López de Prado's purging refinement exists on top of an ordinary walk-forward or k-fold split, and precisely why this module frames it as a concept to carry forward rather than a formula to memorize here: QD1.03's oos_split harness and QM1.03's treatment are where the full, general machinery, purging AND embargo, for an arbitrary label horizon and an arbitrary split scheme: actually lives.

Practice set

Work each problem before reading its solution. Numeric answers are exact unless stated otherwise; all entities are explicitly synthetic.

P1 (guided). Config precedence, by hand. defaults = {"env": "prod", "batch_size": 100, "retries": 3}, file_cfg = {"batch_size": 50}, env_cfg = {"retries": 5}, cli_cfg = {"env": "staging", "retries": 0}. Resolve the final config in the correct precedence order.

Solution. Start from defaults: {"env": "prod", "batch_size": 100, "retries": 3}. Apply file_cfg: {"env": "prod", "batch_size": 50, "retries": 3}. Apply env_cfg: {"env": "prod", "batch_size": 50, "retries": 5}. Apply cli_cfg: {"env": "staging", "batch_size": 50, "retries": 0}. Final: {"env": "staging", "batch_size": 50, "retries": 0}, note that cli_cfg's explicit retries: 0 correctly wins over env_cfg's retries: 5, even though zero looks like "no value set"; precedence is about which layer mentioned a key, never about whether the value looks meaningful.

P2 (guided). Secret resolution, by hand. config = {"api": {"key": "${secret:API_KEY}", "endpoint": "https://prod.internal"}}, secret_store = {"API_KEY": "sk-9f2..."}. What does resolve_secrets(config, secret_store) return, and what happens if secret_store is empty instead?

Solution. With the key present: {"api": {"key": "sk-9f2...", "endpoint": "https://prod.internal"}}, the placeholder is replaced, the plain URL string is untouched because it does not match the ${secret:NAME} pattern at all. With secret_store = {}: the function must raise KeyError("API_KEY") immediately, rather than returning None or leaving the placeholder string in place, a pipeline should fail loudly, at the exact point a required secret is discovered missing, not silently proceed with a broken credential.

P3 (guided). Manifest diff, by hand. prev = {"seed": 7, "code_version": "1.4.0", "dataset": "nse-eod@2024-06-01", "run_id": "a", "duration_s": 40}, curr = {"seed": 7, "code_version": "1.4.1", "dataset": "nse-eod@2024-06-01", "run_id": "b", "duration_s": 33}, volatile_keys = {"run_id", "duration_s"}. Compute manifest_diff(prev, curr, volatile_keys).

Solution. Compare every key not in volatile_keys: seed (7 vs 7, equal, skip), code_version ("1.4.0" vs "1.4.1", differs), dataset (equal, skip). Result: {"code_version": ("1.4.0", "1.4.1")}, a genuine, load-bearing difference (someone shipped a code change between these two runs), correctly isolated from the two volatile fields that were never expected to match.

P4. Cron next-run. Schedule minute_set = {0, 30}, hour_set = None (any hour). What is next_run_minute(minute_set, hour_set, 45) (minute-of-day 45, i.e., 12:45 AM)?

Solution. Search forward from 46: minute 46 → 46 mod 60 = 46, not in {0, 30}. Continue until candidate 60: 60 mod 60 = 0, which is in the set; hour is unconstrained. Next run = 60 (i.e., 1:00 AM).

P5. Cron next-run, rolling to the next day. Schedule minute_set = {0}, hour_set = {14} (daily at 2:00 PM). Asked for the next run at minute-of-day 14×60 = 840, exactly 2:00 PM, the instant the job is scheduled to fire. What is the result, and why can't it be 840 itself?

Solution. The search requires a candidate strictly greater than 840. No later minute-of-day within the same 24-hour cycle has both minute 0 and hour 14, the next match is exactly one cycle later. Result = 840 + 1440 = 2280 (2:00 PM the following day). The strict inequality is what prevents a status check running at exactly the scheduled instant from mistaking "now" for "the next occurrence."

P6. Backoff cap. base = 5, cap = 20. Before any jitter is applied, what is the uncapped exponential ceiling for retry attempts i = 0 through 4, and at which attempt does the cap first bind?

Solution. Uncapped values: 5 × 2^0 = 5, 5 × 2^1 = 10, 5 × 2^2 = 20, 5 × 2^3 = 40, 5 × 2^4 = 80. Capped ceilings (min(cap, uncapped)): 5, 10, 20, 20, 20. The cap first binds at attempt i = 3 (uncapped 40 > cap 20), and every attempt from there on draws its jitter from the same fixed [0, 20] range rather than continuing to grow.

P7. Holm–Bonferroni, by hand. Five p-values [0.01, 0.03, 0.04, 0.06, 0.08], alpha = 0.10. Which survive?

Solution. m = 5; thresholds by rank: 0.10/5 = 0.02, 0.10/4 = 0.025, 0.10/3 ≈ 0.0333, 0.10/2 = 0.05, 0.10/1 = 0.10. Rank 1 (0.01) vs 0.02: passes. Rank 2 (0.03) vs 0.025: 0.03 > 0.025, fails; stop. All remaining ranks (0.04, 0.06, 0.08) are marked not-significant. Only the smallest p-value (0.01) survives.

P8. Holm vs. plain Bonferroni. Four p-values [0.008, 0.014, 0.018, 0.024], alpha = 0.05. Compute both corrections and explain the difference in one sentence.

Solution. Plain Bonferroni: flat threshold 0.05/4 = 0.0125 for every trial → survivors: 0.008 ✓, 0.014 ✗, 0.018 ✗, 0.024 ✗: only one survives. Holm: thresholds 0.0125, 0.01667, 0.025, 0.05 by rank → 0.008 ≤ 0.0125 ✓, 0.014 ≤ 0.01667 ✓, 0.018 ≤ 0.025 ✓, 0.024 ≤ 0.05 ✓: all four survive. In one sentence: Holm's widening, rank-dependent threshold credits later trials with the room their smaller "remaining family size" earns them, so it never rejects more than plain Bonferroni's flat threshold does and, as here, can credit substantially more genuine discoveries for the identical data.

P9. Idempotent ledger trace. A scheduler fires four times for what is logically two distinct trials: run "t1" (genuine), a retry of "t1" (duplicate, same identifier), run "t2" (genuine), a retry of "t2" (duplicate). Trace append_if_new across all four firings, starting from an empty ledger, and give the final length.

Solution. Firing 1 ("t1", new): ledger = [{"run_id": "t1", ...}]. Firing 2 ("t1" again): existing entry found, ledger unchanged, still length 1. Firing 3 ("t2", new): ledger = [{"run_id": "t1", ...}, {"run_id": "t2", ...}], length 2. Firing 4 ("t2" again): existing entry found, ledger unchanged. Final ledger length = 2, exactly the number of genuinely distinct trials, regardless of the two duplicate firings.

P10. Walk-forward enumeration. n = 15, train_size = 5, test_size = 3, step = 3. Enumerate every (train, test) window.

Solution. Start = 0: train (0,5), test (5,8), test end 8 ≤ 15, keep. Start = 3: train (3,8), test (8,11): keep. Start = 6: train (6,11), test (11,14): keep. Start = 9: train (9,14), test (14,17), test end 17 > 15, stop. Windows: [((0,5),(5,8)), ((3,8),(8,11)), ((6,11),(11,14))]: three windows.

P11. Purge count. A label is built from "the return over the next 4 trading days" (i.e., an observation at index i uses prices through i + 4). A walk-forward split ends training at index 199 (training = indices 0–199) with test starting at index 200. How many trailing training observations must be purged, and why?

Solution. An observation at index i must be purged if i + 4 ≥ 200, i.e., i ≥ 196. That is indices 196, 197, 198, 199, four observations, exactly equal to the label horizon: each of their labels reaches into the first four days of the test period (indices 200–203 respectively for i=196..199, with i=199's label reaching to 203). Purging exactly the label-horizon-many trailing observations removes every training label that overlaps the test period, without discarding any training data unnecessarily.

P12. Multiple-testing arithmetic. Forty independent, genuinely skill-less strategies are each tested at alpha = 0.05. What is the expected number of false "significant" results from chance alone, and what would the flat Bonferroni-adjusted per-test threshold be to hold the familywise error rate at 0.05?

Solution. Expected false positives: 40 × 0.05 = 2. Bonferroni-adjusted per-test threshold: 0.05 / 40 = 0.00125 (0.125%), each individual trial would need a p-value under roughly one-eighth of one percent to be called significant once the fact that forty were tried is accounted for.

P13. Spot the leak. A pipeline calls resolve_secrets(config, secret_store) to get resolved_config, then calls logger.info("run manifest: %s", resolved_config) to record the run for reproducibility. What is wrong, and what is the fix?

Solution. resolved_config now contains every real secret value in plain text, logging it directly writes live credentials into a log file, a log aggregator, and potentially a support ticket or screenshot, permanently. The fix: call redact(resolved_config) (CS2.06) immediately before the log call, logging the redacted structure instead, never the raw resolved config, no matter how useful the unredacted version would be for debugging.

P14. Diagnose from a manifest diff. Two team members run "the same" backtest and get different terminal Sharpe ratios. manifest_diff on their two run manifests returns {"dataset": ("nse-eod@2024-06-01", "nse-eod@2024-07-01"), "seed": (None, 2024)}. Diagnose both discrepancies.

Solution. The dataset entry shows they ran against two different data snapshots a month apart, not a code or logic problem at all, simply different underlying data, which alone is sufficient to explain a different Sharpe ratio. The seed entry shows one run's manifest never recorded a seed at all (None) while the other used 2024, the first run's randomness (if the strategy has any stochastic component) was never fixed or logged, meaning that run cannot itself be reproduced even by the same person who ran it, independent of the dataset mismatch. Two separate, independently sufficient causes; fixing only one would still leave the results divergent.

P15. Does containerizing guarantee reproducibility? Justify your answer using the four-legs framework (CS2.01) plus this module's additions.

Solution. No. A container automates leg one (pinned code, interpreter, and dependencies, plus the OS), a genuine, valuable automation. It does nothing by itself about leg two (a fixed, logged seed for every random draw), leg three (an identified, versioned data snapshot rather than a live re-fetch), or this module's additions (a resolved config with a stated precedence order, secrets reached only through resolution, an idempotent scheduler, and an honest experiment ledger). A container running unseeded code, reading a freshly-mounted config file at each invocation, is exactly as unreproducible as the same code run bare, containerizing does not add any of the properties it did not already have.

P16. Why append-only? Explain, in two or three sentences, why an experiment ledger must be append-only and outside a researcher's own power to edit, and connect this to why QD2.05's capstone scores honesty as its own separate graded dimension.

Solution. If a researcher can quietly delete or edit an unfavorable trial after the fact, the ledger's row count for a given strategy family can always be made to say "one trial, which worked" regardless of how many were actually run, which defeats every multiple-testing correction downstream, since none of them can distinguish an honest N from a curated one. An append-only ledger, by construction, cannot be edited into a more flattering story after the result is known. This is exactly why QD2.05 grades honesty about a strategy's limitations as its own scored line, separate from raw performance: a strategy that is marginal, but whose write-up honestly states the true trial count and the correction applied, has demonstrated the discipline this entire module teaches; a strategy with an impressive but unverifiable Sharpe ratio and no ledger behind it has not, no matter how good the headline number looks.

Applied mini-project

Build one reproducible, scheduled, logged, honestly-corrected research trial family.

Pick one simple, checkable idea you can compute by hand or in a spreadsheet across at least five variants (a moving-average crossover with five different window lengths, a threshold rule with five different thresholds, the idea itself does not need to be sophisticated; the infrastructure around it is what this project grades). Using the concepts in this module, code is optional; a careful worksheet is entirely acceptable, exactly as the region's "run in your own tool, verify by hand" pattern permits for infrastructure-flavored work:

1. Configuration. Write out your four config layers (defaults, file, environment, CLI-equivalent) for this trial family, and resolve them by hand into one final config, showing your precedence work exactly as Worked Example 1 and Practice P1 do.
2. The run manifest. For each of your five (or more) variants, write a small manifest: a stable trial identifier, the variant's parameters, a "code version" stand-in (a version number you assign yourself), the data snapshot you used (a date range, clearly labeled if synthetic), and the seed if your variant involves any randomness at all.
3. The ledger. Log all five (or more) trials, in the order you ran them, into one append-only table, every trial that ran, including any that you personally suspected in advance would fail. No trial gets left off the ledger because it looked unpromising before you ran it; that is precisely the discipline this module exists to enforce.
4. The correction. Compute a p-value (or a comparably rigorous significance measure appropriate to your setup) for each trial against the null of "no real effect," then apply the Holm–Bonferroni correction by hand across your full, honestly-logged trial count, showing your sorted ranks and thresholds exactly as Worked Example 5 does.
5. The honesty memo. Write one paragraph, addressed to a hypothetical portfolio manager, stating: how many trials you ran, which (if any) survived correction, and whether you would recommend proceeding, and if none survived correction, say so plainly rather than reporting the best-looking uncorrected number instead.

Deliverable and self-check. One worksheet containing your resolved config, your full ledger (every trial, no omissions), your correction computation with every threshold shown, and your honesty memo. Score yourself, 2 points each: (a) the config precedence is resolved correctly and shown step by step; (b) every trial you actually ran appears in the ledger, including any you were tempted to leave out; (c) the Holm–Bonferroni thresholds and stopping point are computed correctly, matching a hand-check against the widening-threshold formula; (d) the honesty memo states the true trial count and the correction's actual outcome, even if that outcome is "nothing survived"; (e) the memo would not embarrass you if a colleague independently recomputed your correction from your own logged ledger. Pass at 8/10, with no zero on any item. The point of item (e) is not rhetorical: if your own ledger, handed to someone else, would let them reach a different conclusion than your memo states, the memo is the thing that is wrong, not the ledger.

Reading & resources

Everything below is a primary source or an authoritative reference to verify claims against, never a substitute for building the infrastructure yourself.

The anti-overfitting canon:

• Marcos López de Prado, Advances in Financial Machine Learning (Wiley, 2018), the primary source for backtest overfitting, the deflated Sharpe ratio, and combinatorially symmetric cross-validation; QD1.02 already named this the direct forward reference for this module. [Paid] [Advanced]
• Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J., "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance," Notices of the American Mathematical Society, the widely-cited, freely accessible paper that introduced the probability-of-backtest-overfitting idea to a broad audience. [Free] [Advanced]
• Bailey, D. H., & López de Prado, M., "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality," Journal of Portfolio Management, the specific correction QD2.03 implements and reports; read here for the reasoning, cross-reference QD2.03 for the applied computation. [Paid] [Advanced]

Multiple-testing statistics (the general tool this module builds):

• Holm, S., "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian Journal of Statistics (1979), the original paper behind QD2.04-e06's step-down correction. [Paid] [Advanced]
• Any standard statistics reference's chapter on the multiple-comparisons problem and familywise error rate: QM1.02 derives the hypothesis-testing foundation this section applies; read there first if p-values and significance levels are not yet second nature. [Free–Paid] [Intermediate]

Production configuration and secrets (free, code-first):

• The Twelve-Factor App, factor III ("Config"), the widely-adopted statement of the "config lives in the environment, never in code" principle this module's layered precedence system extends. [Free] [Beginner–Intermediate]
• AWS Secrets Manager and HashiCorp Vault official documentation, read for the concepts (rotation, least privilege, audit) named in this module; neither is used hands-on here. [Free] [Intermediate]

Scheduling, containers, and deployment (free, conceptual pointers):

• The cron(5) manual page, or an interactive tool such as crontab.guru, for the real five-field syntax this module's two-field model simplifies. [Free] [Beginner] (perishable)
• Apache Airflow or Prefect official documentation, "Concepts" section, for how a real workflow scheduler represents idempotency, retries, and backfills at a scale this module's exercises only sketch. [Free] [Intermediate]
• Docker's official "Get Started" guide, the container mechanics CS2.01 named and this module builds on top of. [Free] [Beginner]

Do this, not just read: take one script you have already written for this course (a backtest, a data-cleaning step, anything with at least one number that could plausibly change between runs) and apply this module's checklist to it end to end: resolve its settings through a stated precedence order, move any credential behind a resolution step, write and log a run manifest, and run it twice to confirm manifest_diff reports an empty result. The first time a real script of yours fails that test, you will have found a genuine gap this module exists to close, and that is worth more than any worked example.

Teach it back & journal

Feynman prompt. Explain to a colleague who says "our backtest is solid, we set a seed and it's version-controlled" why that is not the same claim as "our research process is trustworthy." Use this module's thesis sentence, a number is not a result until it reproduces, and not a finding until you know how many tries it took, and walk them through both halves concretely: first, name one specific thing beyond "a seed" that reproducibility actually requires (the manifest, the config precedence, the idempotent scheduler); second, ask them the question this module argues matters more than any Sharpe ratio (out of how many tries did this result survive, and where is the ledger that proves it) and explain why "we didn't write down the ones that failed" is functionally the same admission as "we have no seed."

Journal prompt. Pick one piece of research work you have done for this course (a backtest, a signal, any result with a number attached) and audit it honestly against this module's checklist: was your config resolved from a stated precedence order, or assembled ad hoc? Did every credential (real or simulated) stay out of your notes and logs? Could you rerun it today and get the identical number? And the harder question: how many variations did you actually try before landing on the version you wrote up, and is that count written down anywhere a stranger could check it? Write one honest paragraph naming the single gap you found, and one concrete step you will take this week to close it, a manifest you will start writing, a ledger you will start keeping, or a correction you will start applying before reporting a result as a "finding" rather than a "trial."

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.