Capstone: Build, Backtest & Risk-Analyze a Strategy End-to-End

Branch: Quant Developer (Quant & Code galaxy) · Node QD2.05 · ~40 focused hours · CROWN node · Mastery gate ≥ 90% · Completes the region and awards the Quant Developer title

Everything in this branch has been rehearsal. QD1.01 taught you where data comes from and how it lies; QD1.02 taught you to build a signal and measure it honestly; QD1.03 taught you to build an engine that refuses to cheat; QD1.04, QD2.01, QD2.02, QD2.03 and QD2.04 taught you optimisation, execution, risk, attribution and reproducibility one at a time, in isolation, with the rest of the pipeline held still. This node removes the scaffolding. You will state a hypothesis, build a point-in-time dataset, construct a signal, backtest it against realistic frictions, measure its risk, attribute its performance, and then (the part that is actually being graded) write down, in one page, everything that is wrong with what you just built.

That last sentence is not a flourish. It is the grading rule, and you should read it before you read anything else in this document.

The grading rule, stated once and enforced everywhere. Honesty is scored on its own axis. A strategy that only works in-sample still passes this node if its limitations are reported honestly and completely. A strategy that looks wonderful and hides its overfitting fails. There is no combination of Sharpe ratio, equity curve and confident prose that recovers a submission whose limitations memo is silent about a defect the harnesses can find. There is no penalty (none) for submitting a pipeline that concludes there is no tradable edge here, and here is the evidence. That conclusion, properly evidenced, is a first-class result and it earns full marks. The reason is not moral. It is economic. A researcher who reports an honest negative saves the desk the cost of trading it. A researcher who reports a dishonest positive costs the desk the capital, the drawdown, the unwind and the two months it takes to work out what happened. The second researcher is more expensive than no researcher at all, and every serious firm has learned this the same way. So the thing this node certifies is not "can you find alpha." It is "can your numbers be believed."

There is a second reason the rule has to be structural rather than a matter of good intentions, and it is the most important idea in this node: you cannot detect your own overfitting by being careful. Overfitting is not a mistake you make; it is a thing that happens to you while you make perfectly reasonable decisions one at a time. You try a 126-day lookback and a 252-day lookback. You notice that skipping the last week helps. You drop the two worst names because "their data looks wrong." Every one of those choices was defensible in isolation, and collectively they fit your strategy to the noise in one particular sample of one particular history. Nobody in that story did anything dishonest, and the result is still worthless. Which is why the answer is not vigilance. The answer is harnesses: mechanical, automated, adversarial checks that run on every submission, that you did not write specially for this strategy, and that fail loudly.

Code pack: this node ships with QD2.05 (app/public/code-packs/QD2.05.json): ten graded exercises, QD2.05-e01 through QD2.05-e10, worth 465 XP in total. Six of them are the pipeline stages, in order. Four of them are the anti-overfitting harnesses themselves, graded as first-class code: no_lookahead (e04), cost_sensitivity (e05), determinism (e06) and oos_split (e07). e10 is the capstone integration, one function, raw rows in, signed verdict out. The hidden tests for the four harnesses do not merely check that your harness passes a clean pipeline; they hand it deliberately broken pipelines and require it to catch them. A harness that always returns True is not a harness, and the pack knows it.

On the datasets, said plainly at the top. The bundled panels (us-eod, nse-eod) are seeded synthetic look-alikes, geometric random walks, documented as such in app/public/datasets/MANIFEST.json. They contain no cross-sectional signal, because none was put there. That is deliberate: they are your negative control. Any strategy that appears to work on them is overfitted, by construction, and the honest capstone submission on this data is a working pipeline that reports no edge. Your positive control is synthetic-factor-panel, the one bundled dataset with a deliberate seeded tilt in value, momentum and quality, a pipeline that cannot recover a signal that is known to be there is broken, and QD2.05-e09 tests exactly that pair. Build against both. A researcher who has only ever seen their pipeline find nothing cannot distinguish a disciplined pipeline from a dead one.

Learning objectives

By the end you can:

1. Write a falsifiable hypothesis before touching the data, an economic mechanism, the population it should hold in, the horizon, the sign, and in advance the observation that would kill it.
2. Assemble a point-in-time, survivorship-controlled dataset: reconstruct what was knowable on a given date, gate fundamentals on filed_date rather than fiscal period, recompute eligibility at every rebalance rather than fixing a universe once, and state what your data cannot tell you.
3. Construct a signal that is tradable, not merely predictive, with an explicit decision lag, a skip that protects you from the closes you could not have acted on, cross-sectional neutralisation, and winsorisation before scaling.
4. Backtest with the lag and the frictions made explicit: next-period execution, turnover-proportional commission and slippage, and a documented capacity limit, and explain why the vectorised one-liner hides the two bugs that matter.
5. Produce a complete risk page: historical and parametric VaR and Expected Shortfall, maximum drawdown and time under water, factor/market exposure with its R², and a scenario stress table, and read the gap between the historical and Gaussian numbers as information about your tails.
6. Report performance honestly: Sharpe with its convention stated, the deflated Sharpe ratio with the number of trials and the trial dispersion you actually used, in-sample versus out-of-sample with a purged embargo, and turnover and cost drag alongside the headline.
7. Build and run the four harnesses as code (no_lookahead, cost_sensitivity at 2× costs, determinism across two runs, oos_split with an embargo) and demonstrate that each one catches a pipeline you deliberately broke.
8. Make a research pipeline reproducible: one config object, seeds fixed inside the run, no wall-clock and no network, and a content digest that lets a third party verify they got your number rather than a number.
9. Write the one-page limitations memo to the six required headings (capacity, overfitting risk, regime dependence, data-snooping, data limitations, and what would break it) with the specific, quantified admissions that make it useful rather than ritual.
10. Judge someone else's backtest in ten minutes, by asking the six questions that separate a result from a rendering, and say precisely which harness would have caught which defect.
11. (Productivity objective: R10 duality.) Use a coding copilot the way a working quant researcher does, after you can build the pipeline by hand, and review its output for the failure modes specific to this domain: silently vectorised look-ahead, a resampling call that quietly reorders your index, a "cleaned" dataset with the failures dropped, and a confidently hallucinated statistical formula.

The duality, stated once (R10). The gated skill is objectives 1–10, and the gate is unusually hard to fake: the code pack's hidden tests attack your harnesses with broken pipelines, check point-in-time invariance by deleting rows you were not allowed to see, and require the same fingerprint from two independent runs. Objective 11 is the payoff you keep. The app never calls an AI at runtime, the callout below asks you to generate something in your own tool, paste it back, and let the local hidden tests judge it. The machine drafts; the harnesses decide; you are the one who has to know why it failed.

Prerequisites & connections

Builds on, and this node re-derives none of it. QD1.01 owns data pipelines, corporate actions, and the definition of a point-in-time panel; this node uses that definition and tests it mechanically. QD1.02 owns signal and factor evaluation: IC, IR, decay, turnover, neutralisation; if the phrase "information coefficient" is not automatic, go back. QD1.03 owns backtest-engine design, event-driven versus vectorised, and the classic pitfalls; it ships the first no_lookahead harness, and this node hardens and extends it. QD1.04 owns portfolio optimisation and why naive optimisers blow up. QD2.01 owns execution and market impact, which is where your capacity number comes from. QD2.02 owns risk systems (VaR, ES, VaR backtesting, stress) and QD2.03 owns performance and attribution including the deflated Sharpe ratio's derivation. QD2.04 owns reproducible research, config and secrets management, and López de Prado's anti-overfitting protocols.

On the statistics. The QM branch owns the concepts, what a sampling distribution is, why multiple testing inflates a maximum, what a p-value does and does not say, the intuition and failure modes of every estimator used here. This node owns the same statistics as code: implementation, validation, pipeline placement, and evaluation. Where you need the derivation of the probabilistic Sharpe ratio, cross to QM; what you get here is the function, the convention it assumes, the argument you have to supply honestly, and the hidden tests that check you supplied it.

Also assumed. CS1.02 for clean Python and idiomatic data structures. CS1.04/CS1.05 for cost reasoning, the capstone pipeline is written as an explicit loop precisely so its cost is visible. CS2.06 for testing discipline and, in particular, its unit on reviewing AI-generated code, which is the exact skill objective 11 asks for. DA1.02/DA1.03 own pandas; the graded code here is deliberately dependency-free (see the note below), and the desk version is the pandas one.

Why the graded code is standard-library only. Every solution in the QD2.05 pack uses nothing but csv, math, hashlib and json. This is a teaching choice with three payoffs. First, the lag is visible: held[i] = target[i-1] is a line you have to write, where weights.shift(1) is a line you can forget. Second, a harness with no dependencies runs anywhere (in CI, in a container, on a colleague's laptop) which is the whole point of a harness. Third, it removes the single most common source of silent nondeterminism in research code, which is a library upgrade changing a default. On the desk you will write this in pandas and NumPy and you should; DA1.02 and QD1.03 show you how. Build it once by hand so you know what the library is doing on your behalf.

Feeds forward. Nowhere, inside the program. This is the last node of the Quant Developer branch and clearing it awards the Quant Developer title and the quant-black-belt. What it feeds forward into is your own work: the six-heading memo, the four harnesses and the digest habit are portable, and they are the artefacts that make a junior researcher's output usable by a senior one.

Core teaching content

What a capstone actually certifies

A backtest is a claim about a counterfactual: if I had run this, I would have made that. Counterfactual claims cannot be checked by inspection, which means the only thing that makes one credible is the process that produced it. So the deliverable of this node is not a number and not a chart. It is two things:

1. A reproducible pipeline that runs deterministically from raw rows to final metrics with no human in the loop, and
2. A one-page thesis memo that states the hypothesis, the result, and (at equal length) what is wrong with it.

Read those two together and notice that neither is "a profitable strategy." Profit is an outcome; the pipeline and the memo are the evidence. On a desk, the pipeline is what a colleague re-runs to check you, and the memo is what a risk officer reads to decide how much capital your claim deserves. A researcher who produces both is employable at any level. A researcher who produces a spectacular equity curve and cannot reproduce it is not employable at all, and this is not an exaggeration: the single most common reason a promising quant candidate fails a final-round technical interview is that they cannot answer "how many variants did you try before this one?"

The seven stages below are the pipeline. Each one has a specific way of lying to you, and each one has a specific defence.

Stage 0: the hypothesis, written down before the data

Write it first, in one paragraph, and timestamp it. It must contain five things:

• A mechanism. Not "momentum works" but why it would: an economic reason someone is on the other side of your trade and is content to be. Under-reaction to news, a risk premium someone is paid to bear, a structural flow (index rebalancing, tax-year effects), a constraint on other participants (leverage limits, mandate restrictions). If you cannot name who is losing and why they are willing to, you have a pattern, not a hypothesis.
• A population. Which securities, which market, which capitalisation band, which period. "US large caps, 2010–2024" is a population; "stocks" is not.
• A horizon and a sign. Held for how long, and does the signal predict positively or negatively? Deciding the sign after you look at the data is the cheapest possible way to double your effective number of trials.
• A magnitude you would find plausible. An honest guess, before the fact, at the gross annualised spread. This is your single best defence against the "wow, 4.2 Sharpe" moment, because you wrote down in advance that anything above about 1.0 net in a liquid market should make you suspicious rather than happy.
• A falsifier. The specific observation that would make you abandon it. "If the effect disappears when I exclude the smallest quintile, the hypothesis is really a liquidity story and I withdraw it."

That paragraph costs you fifteen minutes and it is the highest-leverage fifteen minutes in the whole project, because everything downstream is a test of it rather than a search for something. The difference between research and data-mining is not the tools. It is whether the hypothesis existed before the result.

Stage 1: the dataset: point-in-time and survivorship-controlled

Almost every catastrophic backtest failure is a data failure wearing a modelling costume. Two of them do most of the damage.

Look-ahead through publication lag. A company's fiscal 2025 results describe a period that ended months before anyone could read them. If you join revenue onto every date in fiscal 2025, you have given your strategy a time machine, and the leak is invisible because nothing errors and the dates all look sensible. The fix is one rule: the only fundamental data available on date t is the most recent filing whose filed_date is on or before t. In the bundled panels this is not hypothetical. FY2025's filing lands on 2025-05-18. On 2025-05-17 the newest thing you may read is FY2024; one session later it is FY2025. QD2.05-e01 tests exactly that boundary, in both markets, because it is the cheapest possible demonstration that your gate is real.

Survivorship. You download today's index constituents and run history against them. Every company that was delisted, acquired at a discount, or went to zero has been silently deleted from your past, and the surviving names are, by construction, the ones that did well. The measured effect is not small: the classic estimates put the survivorship premium on a naively reconstructed US equity universe at roughly 1–4 percentage points per year, and the effect is worse in the small-cap and emerging-market corners where a backtest most often finds "alpha."

The defence is eligibility recomputed at every decision date, from information available then. Not one universe for the whole sample: a universe function, universe(t), evaluated fresh. That is what pit_universe in QD2.05-e01 is, and it produces genuine entries and exits, on the bundled US tape a top-four-by-turnover book changes composition at three of four sampled dates.

Be honest about what the panel cannot tell you. The bundled panels are balanced: fifteen tickers, five hundred sessions each, no entries, no exits, no delistings. That is itself the tell. A real universe has births and deaths; a perfectly rectangular one has had its deaths removed, or never had any. So the correct sentence in your memo is not "survivorship is controlled", it is: "the universe function is point-in-time, but the underlying panel contains no delistings, so the strategy has never been tested against a name going to zero and the survivorship premium in this result is unmeasurable rather than zero." That sentence is worth more than a green tick, and writing it is what this node is for.

Stage 2: the signal, and the gap that makes it tradable

A signal is a number you compute at time t and act on afterwards. Three disciplines separate a tradable signal from a plotted correlation.

The decision lag is not optional. Whatever you compute from the close of day t, you trade into day t+1. This sounds obvious and is violated constantly, because the violation is one character: weights * returns instead of weights.shift(1) * returns. It is worth internalising just how much free money that character prints. On the bundled US tape the same weight book scored under the same-day convention differs materially from the lagged one, and in a real momentum panel the same-day version is not a small improvement, it is the difference between a strategy and a fantasy, because you are paying yesterday's price for information that includes today's move.

The skip protects you from the closes you cannot trust. Cross-sectional momentum is conventionally measured to a point some sessions before the decision date, the classic 12–1 construction (Jegadeesh & Titman, 1993; Asness, 1994). Part of the reason is economic: very short-horizon returns tend to reverse. But the operational reason is the one that matters here, the most recent closes are precisely the ones whose availability, cleanliness and tradability you are least sure of. If deleting the last five closes changes your signal, your signal is standing on the exact data you trust least. QD2.05-e02's hidden tests triple the last five closes and require the signal not to move by a single bit, then nudge the session just outside the skip and require it to move, because a skip parameter that is quietly ignored passes every other test you would think to write.

The cross-section strips out what you are not being paid for. A raw return is a statement about a stock and about the market that day. Z-scoring within the date removes the common move and leaves the relative call, which is the only thing a dollar-neutral book can express. Then winsorise before you scale, not after: one +9σ outlier in a cross-section of eight names, scaled to a unit gross budget, becomes most of your portfolio. Clip to ±2σ, then demean, then scale so sum(|w|) = gross. The order is load-bearing, demeaning before clipping leaves a book that is not dollar-neutral, and QD2.05-e02 pins the arithmetic exactly.

Stage 3: the backtest, with the frictions where you can see them

Costs are proportional to what you trade, not what you hold, and the first entry into a position is itself a trade. The whole cost model reduces to:

turnover[i] = sum over names of |held[i] - held[i-1]|
cost[i]     = turnover[i] * (commission_bps + slippage_bps) / 10000

with held[i] = target_weights[i-1] and nothing held on the first day. Everything interesting follows from those three lines.

Turnover is the number that decides whether you have a strategy. The monthly-rebalanced dollar-neutral momentum book in QD2.05-e10 runs at an annualised turnover of 6.75× on the US tape and 6.30× on the Indian one. At a combined 15 bps round trip that is a drag of 1.01% and 0.94% per year respectively. Gross of costs the US book returns −3.32% annualised; net of 15 bps it returns −4.30%; at 30 bps, −5.27%. A strategy is a claim about the gross spread minus that drag, and a great many published effects have gross spreads smaller than their own turnover cost. Report turnover next to Sharpe, always, and treat any result that omits it as incomplete.

Slippage is not commission. Commission is a known, quoted, roughly linear fee. Slippage is the difference between the price you modelled and the price you got, and it grows with your size relative to the market's, which means it is not a constant, it is a function of the capital you deploy. Modelling it as a flat basis-point number, as the pack does, is a simplification you must declare, and the declaration is where your capacity estimate comes from. QD2.01 owns the impact models; what you owe here is the honest sentence: "slippage is modelled as a flat x bps, which is defensible up to about y of daily volume and wrong above it."

Capacity is a number, not a caveat. Take your annualised turnover, your target book size, and the median daily traded value of the names you hold. If a rebalance requires you to trade more than a few percent of a name's daily volume, your slippage assumption is fiction. Do the arithmetic once, in the memo, in currency: "at ₹50 crore of gross book, a rebalance trades about ₹X of a name whose median daily value is ₹Y, i.e. Z% of ADV, beyond that the flat-slippage assumption fails." Worked example 7 does exactly this.

Stage 4: risk: the page, not the number

A single risk number is a number chosen to be quoted. A risk page is one designed to be read. Report all of it:

• Historical VaR and ES. VaR at α is the α-quantile of your realised returns, reported as a positive loss; ES is the average of everything at or beyond it. ES is the more useful of the two because VaR tells you where the tail begins and ES tells you what is in it, and unlike VaR, ES is coherent (it respects diversification; VaR does not, in general).
• Parametric (Gaussian) VaR and ES, from your sample mean and sigma. Not because returns are normal (they are not) but because the gap between the two methods is information about your tails. On the bundled US book, 95% historical VaR is 1.243% against a Gaussian 1.167%, and historical ES is 1.665% against a Gaussian 1.459%. The realised sample kurtosis is 4.92 against a normal's 3.0: the tail is fatter than the parametric model believes, so the parametric model understates your loss and the gap quantifies by how much. Quote both and let the reader see the discrepancy.
• Maximum drawdown and time under water. These are path statistics, and the second is the one that gets people fired. The US book's worst drawdown is 17.19%, but it spends 350 of 500 sessions below its previous peak. A strategy that loses 17% quickly and recovers is a different animal from one that is under water for 70% of its life, and only the duration number tells them apart.
• Factor and market exposure. Regress your returns on the benchmark's. The US book's beta is −0.037 with an R² of 0.0034; the Indian book's is −0.022 with R² 0.0017. Those are the numbers that let you say "this is genuinely dollar-neutral, not a disguised index tracker", and if your R² came back at 0.7, you would have discovered that your "alpha" is beta with extra steps.
• Stress and scenario. Beta times a shock is the crudest possible scenario and it is still worth reporting, because it is falsifiable: "a −10% index day implies roughly +0.37% for this book, given its measured beta." Add at least one historical-analogue scenario (the worst realised five-day window in your own sample) because the crude linear estimate and the realised worst case rarely agree, and where they disagree you have found the non-linearity in your book.

Stage 5: performance and attribution, deflated

State your Sharpe convention. Excess of what, at what frequency, annualised by what factor, sample or population sigma. A Sharpe ratio quoted without its convention is a rumour. This node uses: per-period excess returns, sample sigma (ddof=1), annualised by √252.

Then deflate it. Here is the idea that makes the deflated Sharpe ratio the single most important number in this node. If you try N strategy variants and keep the best, the Sharpe you report is not a Sharpe, it is the maximum of N draws, and the maximum of many draws is large even when every draw is pure noise. Bailey & López de Prado's correction has two parts:

Part one, the bar. The expected maximum Sharpe over N independent skill-less trials whose Sharpes have variance V is approximately

E[max SR] ≈ sqrt(V) * ( (1 - γ) * Φ⁻¹(1 - 1/N) + γ * Φ⁻¹(1 - 1/(N·e)) )

with γ the Euler–Mascheroni constant, 0.5772156649. It grows with the number of trials, slowly, which is exactly right, since the maximum of a normal sample grows like √(2 ln N).

Part two, the confidence. The probabilistic Sharpe ratio asks how confident you can be that the true Sharpe exceeds a threshold, given your sample length and your higher moments:

PSR(SR*) = Φ[ (SR − SR*) · sqrt(T − 1) / sqrt(1 − γ₃·SR + ((γ₄ − 1)/4)·SR²) ]

where SR and SR* are per-period, γ₃ is skew and γ₄ is non-excess kurtosis. Negative skew and fat tails both reduce your confidence in the same headline Sharpe, which is the correct behaviour and the reason this formula is better than a t-test.

The deflated Sharpe ratio is simply PSR(E[max SR]), your confidence measured against the bar that a lucky search would have cleared.

Now the demonstration that should change how you read every backtest you are ever shown. Take the momentum long–short on the bundled synthetic-factor-panel, which has a real, deliberately embedded tilt. Over 48 months it earns a monthly Sharpe of 0.751, an annualised 2.60, a number that would get a strategy funded. Its PSR against a zero threshold is 0.999998: as close to certain as statistics gets, that the true Sharpe is above zero.

And then you ask the second question. How many variants did you try, and how spread out were their Sharpes?

The track record never changed. Every row is the same 48 monthly returns, the same 2.60 annualised Sharpe. What changed is the honesty of the search description. With a genuinely narrow search (twenty variants whose Sharpes clustered tightly) the result is overwhelming. With twenty wildly dispersed variants, the best of them would have been expected to score 0.95 by luck alone, and a 0.75 is below the bar. Same evidence, opposite verdict, and the only input that moved was the one you supply from your own memory.

Which is why n_trials and sr_variance must appear in your memo, next to the DSR, every time. They are unverifiable by the reader, which makes reporting them a matter of professional integrity rather than mechanics, and it is precisely the number people quietly shrink. Count honestly: every parameter value you tried is a trial. Every lookback. Every universe size. Every "let me just check without the outliers." Realistic counts for a few weeks of exploration are in the hundreds, not the tens. Holding V at 0.01, the bar rises from 0 at one trial to 0.119 at five, 0.190 at twenty, 0.253 at a hundred, 0.326 at a thousand and 0.386 at ten thousand.

And in-sample versus out-of-sample, with an embargo. Split chronologically, never randomly, a random split lets tomorrow train on today. Then throw away a gap between the two windows, because a 126-day lookback computed on the last in-sample day overlaps the first out-of-sample days. The bundled US book scores −0.150 in sample and −0.500 out; the Indian book +0.003 in sample and −0.720 out. Note what that Indian pair looks like: a flat, unremarkable in-sample number and a substantially worse out-of-sample one, the ordinary, undramatic shape of a strategy with nothing in it. Decay of that size is the normal case, not the alarming one, and a submission reporting near-identical in-sample and out-of-sample Sharpes should make you check the split before you celebrate it.

Stage 6: the four harnesses

Everything above is a discipline you could forget. These four are code, they run on every submission, and they are graded as first-class deliverables in QD2.05-e04 through e07. Each has a single job, and each has a matching thing it cannot see, which you must state.

no_lookahead (e04). Look-ahead has exactly one signature: your decision at time t changes when data after t is present. That is directly testable without reading a line of the strategy. Run the decision function twice (once on the full frame, once on the frame truncated to rows on or before asof) and compare. Any difference above tolerance is a leak, and the harness raises a named LookaheadError identifying the date and the ticker. What it catches: full-sample normalisation, forward-looking returns, a global mean or standard deviation, a fitted scaler applied to the whole history, sub-basis-point drift you would never see by eye. What it cannot catch: survivorship, because deleting future rows does not restore names that were removed from the file before you ever loaded it. The pack tests this explicitly, a survivorship-leaking strategy passes no_lookahead on a balanced panel, and the harness must not misreport it. A harness that claims coverage it does not have is worse than no harness.

cost_sensitivity (e05). Every backtest is profitable at zero cost. Re-price the same strategy across a grid of cost multipliers (0×, 1×, 2×, 4×) and report the whole curve, plus the one number that ends arguments: the breakeven cost, the round-trip basis points at which the annualised net return crosses zero. Because turnover does not depend on the cost rate, net return is monotone decreasing in cost, so bisection finds it exactly. A strategy whose breakeven is 300 bps in a market where you pay 15 is robust; one whose breakeven is 18 bps is a rounding error with a research report attached. What it catches: every strategy whose gross edge is smaller than its own friction. What it cannot catch: the fact that your slippage model itself may be wrong at size, doubling a wrong number gives you twice a wrong number.

determinism (e06). Run the pipeline twice, fingerprint each result with SHA-256 over a canonicalised form, compare. Fixed-point rounding to ten decimals before hashing is not sloppiness; it is what separates a harness that flags real nondeterminism from one that cries wolf over the last bit of a float. What it catches: unseeded shuffles, set iteration order leaking into results, state that survives between calls, timestamps, anything sampled without a seed. What it cannot catch: a pipeline that is deterministically wrong. Determinism is necessary and nowhere near sufficient, and a green determinism check on a leaking pipeline is a reproducible lie.

oos_split (e07). One chronological split, a purged and embargoed gap, both sides scored, the degradation reported. What it catches: the crudest form of overfitting, where a strategy tuned on the whole sample collapses on data it did not see. What it cannot catch, and this is the important one, the out-of-sample window stops being out-of-sample the moment you look at it and go back to change something. After the third look, your "OOS" number is an in-sample number with extra ceremony, and only your own trial count records that it happened. This is exactly why oos_split and the deflated Sharpe are partners: the split guards the data, the DSR prices the searching.

Stage 7: reproducibility, and the fingerprint

A result nobody can reproduce is an anecdote. Four rules make a research pipeline reproducible, and all four are cheap:

1. One config object. Every parameter in one dict, passed in, and echoed back in the output as config_used. No magic numbers in the body, no parameters read from the environment.
2. Seeds fixed inside the run. A function that seeds its own RNG is deterministic; one that relies on a seed set somewhere else is deterministic until someone imports it differently.
3. No wall-clock, no network. datetime.now() in a research pipeline is a nondeterminism generator. A live API call is that plus an availability dependency plus a silent-schema-change risk.
4. A digest. Hash the canonicalised metrics. Publish the hash with the result. Now a colleague who re-runs your code can say "I got your number" rather than "I got a number," and the difference is the entire value of the exercise.

Reading a backtest in ten minutes

You will be handed other people's results far more often than you produce your own. Six questions, in this order, will resolve almost every one:

1. What was the hypothesis, and when was it written? After the result is not a hypothesis.
2. How many variants did you try? If the answer is "a few," ask for the list. If there is no list, the DSR cannot be computed and the Sharpe cannot be interpreted.
3. What is the turnover, and what is the breakeven cost? Two numbers, thirty seconds, and they kill most candidates outright.
4. Show me the universe function. If it is a list rather than a function of the date, you are looking at survivorship.
5. What is the out-of-sample Sharpe, and how many times did you look at it?
6. Run it twice in front of me. Astonishingly often, this fails.

Notice that none of the six is about the strategy's idea. The idea is the cheap part.

{
  "skill": "Reviewing a quant research pipeline for look-ahead, leakage and non-reproducibility",
  "use": "After you have built the pipeline by hand and your four harnesses run green, paste one stage at a time into a chat assistant and ask it to attack that stage as an adversarial reviewer would — then verify every claim it makes by writing a test that either fails on the current code or does not.",
  "tools": ["Chat assistants — Claude, ChatGPT, Gemini", "IDE copilots — GitHub Copilot, Cursor", "Local static analysis — ruff, mypy, pytest"],
  "prompt": "Here is one stage of a backtest pipeline. Acting as a hostile reviewer, list every way this code could consume information that would not have been available at decision time, every way it could produce a different answer on a second run, and every place a cost or capacity assumption is buried as a constant. For each item give me a failing test I could write to prove it, and mark any item you are not confident about.",
  "verify": "Treat every finding as a hypothesis, never a fact. A copilot is fluent about backtesting and routinely invents plausible-but-wrong statistics (a mis-stated deflated-Sharpe denominator, a resampling call that silently reorders the index, an ES definition that averages the wrong tail). The test IS the verification: write it, watch it fail on the broken version, watch it pass on the fixed one. Never accept a formula without checking it against a named source (Bailey & Lopez de Prado for PSR/DSR, the QM branch for the derivation), and never accept a 'cleaned' dataset it produces without re-running no_lookahead against it.",
  "diy": "Gated: you must build all four harnesses yourself first (QD2.05-e04 through e07) and be able to state, unprompted, what each one cannot see. Only then does an adversarial reviewer add anything, because you need to be able to tell a real finding from a confident hallucination — and on this material roughly a third of what you get back is the latter.",
  "market": "IN"
}

The limitations memo: six headings, one page

This is the deliverable that carries the honesty score. One page, six headings, specific and quantified.

1. Capacity. At what book size does this stop working, in currency, and by what arithmetic? Name the ADV constraint and the assumption that breaks first.

2. Overfitting risk. How many variants did you try, in total, honestly counted? What is the deflated Sharpe, and with which n_trials and sr_variance? Which parameters are you least confident in, and how much does the result move when you perturb them?

3. Regime dependence. What kind of market was your sample? What regime is absent from it entirely? A sample with no crisis in it has not tested your drawdown behaviour, and saying so is not weakness.

4. Data-snooping and data limitations. What is the provenance of every dataset, and is it real or synthetic? What does the panel structurally not contain, delistings, corporate actions, halts, currency effects, an accurate bid-ask? What is the survivorship exposure you cannot measure?

5. Statistical honesty. In-sample and out-of-sample results side by side; how many times you looked at the out-of-sample window; which harnesses pass and which fail, verbatim, including the ones that fail.

6. What would break it. The falsifier from Stage 0, revisited: the specific observation that would make you withdraw the claim, and whether you looked for it.

The two memos. Here, compressed, are the two ends of the grading scale on identical underlying work.

Memo A (passes, full marks). "Hypothesis: 12–1 cross-sectional momentum in the bundled fifteen-name US panel, monthly rebalanced, dollar-neutral. Result: no tradable edge. Gross of costs the book returns −3.32% annualised; net of 15 bps round trip, −4.30%. Annualised turnover is 6.75×, a cost drag of 1.01%/yr, so costs explain roughly a third of the shortfall and the remaining two-thirds is the absence of a gross spread. Deflated Sharpe is 0.000 against 100 trials at V = 0.01. In-sample Sharpe −0.150, out-of-sample −0.500. All four harnesses run: no_lookahead passes, determinism passes, oos_split passes, cost_sensitivity fails (the 2× cost run returns −5.27%). This is the expected result: MANIFEST.json documents these panels as seeded geometric random walks with no cross-sectional tilt, so the correct finding is a null one, and the pipeline's ability to find signal is evidenced separately on synthetic-factor-panel, where the same code recovers a monthly Sharpe of 0.751. Capacity was not estimated because there is nothing to size. Recommendation: do not trade; the pipeline is reusable on a real panel."

Memo B (fails, zero on honesty). "Momentum works. After tuning the lookback and excluding two illiquid names the strategy returns 14% annualised with a Sharpe of 1.8 and a maximum drawdown of 6%. Costs were assumed at 5 bps. Out-of-sample performance is in line with in-sample."

Memo B is not wrong because its numbers are false, you could produce those numbers from this data by tuning. It fails on four counts, each of which one of your own harnesses would have surfaced: the trial count is missing (so the Sharpe cannot be interpreted), the exclusions are unexplained and were chosen after seeing the data, the cost assumption is asserted rather than stress-tested, and "in line with in-sample" is the claim that should always be checked first because it is the signature of a leaking split. Memo A reports a loss and passes. Memo B reports a profit and fails. If nothing else from this node survives the year, let it be that pair.

Common mistakes & how experts think differently

"I'll check for look-ahead by reading the code." You will not. Look-ahead in vectorised code hides inside a default: a .rolling() window that is centred, a .resample() that labels the right edge, a scaler fitted before the split. Experts do not read for it; they test for it, by re-running the decision on a truncated frame. The test takes ten lines and finds what six careful readings miss.

Treating the backtest as the experiment. The backtest is the measuring instrument. The experiment is the hypothesis you wrote in Stage 0. A novice tunes the instrument until it reports something pleasing; an expert fixes the instrument, runs the experiment once, and reports what came out, including nothing.

Counting trials as "the ones I wrote down." Every parameter value you tried is a trial. Every universe filter. Every "let me just look without 2020." Experts keep a research log with one line per variant precisely because the honest count is embarrassing and the dishonest count is worthless. If your recalled count is under twenty after two weeks of work, you are not remembering, you are estimating downward.

Quoting a Sharpe without a convention or a trial count. These are the two questions a senior researcher asks first, and "about 1.8" followed by silence is a complete answer to both, the wrong one. Experts write SR = 1.8 (daily excess, sample sigma, ×√252; n_trials ≈ 340, V ≈ 0.02, DSR = 0.31) and let the reader do their own arithmetic.

Assuming costs instead of solving for them. A novice picks a cost number and defends it. An expert reports the breakeven cost and lets the reader supply their own: "this dies at 18 bps" needs no argument about whether 12 or 15 was the right assumption.

Confusing "reproducible" with "correct." A green determinism check on a leaking pipeline is a lie you can reproduce exactly. Determinism is a hygiene property. Experts run all four harnesses and know precisely which blind spot each one has, which is why the fourth question in any review is "and what does that check not see?"

Treating a negative result as a failed project. This is the single largest cultural gap between students and practitioners. On a desk, "we tested it properly and there is nothing there" is a delivered result: it closes a line of enquiry, it saves the next person three weeks, and the pipeline is now reusable. Experts write negatives up as carefully as positives, because the reputational asset they are building is their numbers can be believed, and that asset is destroyed by exactly one enthusiastic overstatement.

Reporting the in-sample number as if the out-of-sample number were confirmation. After you have looked at the out-of-sample window three times and adjusted, it is training data. Experts touch it once, at the end, and record how many times they touched it, because the count is the only thing distinguishing an honest OOS from a laundered IS.

Not building a positive control. A pipeline that has only ever reported "no signal" might be disciplined or might be broken, and you cannot tell from the inside. Experts keep a dataset with a known planted effect and check that the pipeline recovers it, the same instinct as a lab running a known-positive sample alongside the unknowns.

Believing a risk number computed on a sample with no crisis in it. Your VaR is a description of the volatility regime you happened to sample. Experts state the regime explicitly and treat the absence of a stress event in the window as a limitation, not as evidence of safety.

Worked examples

Worked example 1: The publication-lag leak, priced (India, ₹)

An analyst builds a value signal on the Indian panel using net_income / total_assets, joining each fiscal year's figures onto every session in that fiscal year. On 2025-05-17 their pipeline is already using FY2025 numbers.

What the correct rule gives. The gate is filed_date <= asof. FY2025 was filed on 2025-05-18. So on 2025-05-17 the newest readable filing is FY2024, and on 2025-05-18 it is FY2025, a one-session step, verifiable directly: pit_universe(px, fu, "2025-05-17")[1] returns all 2024 and pit_universe(px, fu, "2025-05-18")[1] returns all 2025, in both markets.

Why the leak is invisible. Nothing errors. Every date is plausible. The signal is well-behaved. The only symptom is that the strategy is unusually good at the turn of each fiscal year, which the analyst will read as seasonality and write a paragraph about.

The mechanical detection. Truncate the fundamentals to filed_date <= asof and re-run. If the weights move, you leaked; assert_no_lookahead reports the date and the name. The leak's size depends on how much the market moved between period end and filing, which for a March-year-end Indian company filing in May is roughly six weeks of price action you were handed for free.

Worked example 2: Survivorship, and why a rectangular panel is a warning (US, $)

The bundled us-eod panel has fifteen tickers, each with exactly 500 sessions: 7,500 rows, no gaps, no entries, no exits.

The reasoning. A real 15-name universe over two years has at least one corporate event. A perfectly rectangular panel means one of two things: it is synthetic (true here, and documented in MANIFEST.json), or the failures were removed before you got it. Either way, the strategy has never been tested against a position going to zero.

Quantifying what you cannot measure. You cannot compute the survivorship premium in this sample, because the deaths are not in it. What you can do is bound the exposure: a dollar-neutral book at 12.5% gross per name would take a −12.5% hit to the whole book from a single long position going to zero between rebalances, against an annualised volatility of 11.12%, that is more than one full year's volatility in one event. State it as: "one delisting-to-zero in the long book is a ~1.1σ annual event for this strategy, and the panel contains none, so drawdown is understated by an unmeasurable amount."

The habit. Whenever a panel is rectangular, write the sentence. Whenever a universe is a list rather than a function of the date, refuse the result.

Worked example 3: Cost sensitivity and breakeven, by hand (US, $)

Take the same US weight book and hypothetically lift its daily gross return by 6 bps (a modest, plausible edge) then price it across the cost grid.

The arithmetic behind the drag. Annualised turnover is 6.75×. At 15 bps round trip the drag is 6.75 × 0.0015 = 1.01%/yr, which is exactly the 12.46 → 11.32 gap. The back-of-envelope breakeven divides the lifted gross return by that same turnover: 12.4566% / 6.7546 = 184.4 bps. The bisection finds 172.49 bps: 6.5% lower, and the reason is worth knowing. The linear estimate finds the cost at which the arithmetic mean daily return hits zero; at exactly 182.94 bps that mean is zero to machine precision, but the geometric annualised return there is still −0.71%, because compounding a stream with 11.12% annualised volatility costs you roughly σ²/2 ≈ 0.62 percentage points. Since the number you report is geometric, the zero crossing arrives at a lower cost than the arithmetic calculation suggests. Linear cost arithmetic is optimistic about breakeven; use it for intuition and bisect for the number you publish.

How to read it. A breakeven of 172 bps against a realistic 15 bps is a comfortable margin, this hypothetical strategy is not a cost story. Compare with the actual bundled book, whose gross return is negative: cost_sensitivity correctly returns breakeven_bps = None, because there was never anything for costs to eat. Reporting None is the honest output, and QD2.05-e05's hidden tests require it.

Worked example 4: Where historical and Gaussian VaR disagree (India, ₹)

The Indian book's 95% risk numbers:

Reading the gap. The historical numbers are larger in both cases, the realised left tail is heavier than a normal distribution with the same mean and sigma. The sample kurtosis confirms it at 4.65 against a normal's 3.0. So the parametric model understates the loss, by 8.0% on VaR and 10.9% on ES.

Why report both anyway. The parametric number is the one that scales cleanly under aggregation and the one a risk system can compute on a portfolio it has no history for. The historical number is the one that actually happened. Quoting only the parametric one hides your tails; quoting only the historical one leaves you unable to say anything about a position you have not held before. Quote both, and quote the gap, because the gap is the estimate of your model error.

The corresponding US pair: historical VaR 1.243% against Gaussian 1.167% (6.1% understated), ES 1.665% against 1.459% (12.4%), kurtosis 4.92, tells the same story slightly louder. Two independent seeded panels, the same qualitative finding: heavy tails are the default, not the exception.

Worked example 5: The deflated Sharpe: one track record, two verdicts (illustrative)

The synthetic-factor-panel momentum long–short: 48 monthly observations, monthly Sharpe 0.751, annualised 2.601, skew −0.061, kurtosis 2.400.

Step 1, the naive read. PSR against a zero threshold: 0.999998. Read alone, that is a certainty, and it is the number most people would put in the deck.

Step 2, the bar. Suppose twenty variants were tried and their Sharpes had variance V. Then E[max SR] = √V × ((1−γ)Φ⁻¹(0.95) + γΦ⁻¹(1 − 1/(20e))) with γ = 0.5772156649. At V = 0.01 that is 0.190; at V = 0.25 it is 0.950.

Step 3, the verdict. DSR = PSR(E[max SR]). At V = 0.01: 0.9997: significant. At V = 0.25: 0.110, not remotely significant, because the observed 0.751 is below the 0.950 that twenty dispersed noise trials would have been expected to produce.

The lesson. The evidence never moved. The only thing that changed is the honesty of the search description, and it flipped the conclusion from "certain" to "nothing." This is why n_trials and sr_variance belong in the memo next to the DSR, and why a DSR quoted without them is not a statistic.

And the counterpoint that keeps you calibrated. The bundled EOD momentum book, run through the identical machinery at 100 trials and V = 0.01, gives sr* = 0.253 and DSR = 0.000. Same code, same conventions, one dataset has a planted signal and clears the bar, the other does not and does not. That pair is the positive and negative control, and having both is what lets you trust either.

Worked example 6: In-sample, out-of-sample, and what the embargo costs (India, ₹)

Split the Indian book's 500 sessions at 60% with a 10-session embargo: 300 in-sample, 10 discarded, 190 out-of-sample.

Why discard the ten. The signal uses a 126-session lookback plus a 5-session skip. The decision made on the first out-of-sample day draws on prices from deep inside the in-sample window; the two are not independent. Ten sessions is a compromise, a fully purged embargo would be 131 sessions, which on a 500-session sample would cost you a quarter of the data. State the compromise: "embargo of 10 sessions against a 131-session information horizon; the split is partially rather than fully purged, and the OOS number is optimistic to that extent."

Reading the decay. From +0.003 to −0.720 looks dramatic as a ratio, but the in-sample number was never distinguishable from zero. This is the ordinary shape of a strategy with nothing in it: not a crash from greatness, just noise on one side and noise on the other. The alarming pattern is the opposite one, a strong in-sample Sharpe with an out-of-sample number that matches it too well, which usually means the split leaked.

Cross-check the boundary. QD2.05-e07's hidden tests verify that scoring the out-of-sample window from index 300 rather than 310 gives a different answer, the cheapest possible proof that the embargo was actually dropped rather than merely stored.

Worked example 7: Capacity, in rupees (India, ₹)

You want to run this book at ₹50 crore (₹500 million) gross.

The inputs. Eight names, dollar-neutral, so roughly ₹6.25 crore per name at 12.5% gross weight each. Annualised turnover 6.30×, monthly rebalancing, so roughly 0.525 of the book turns over per rebalance, about ₹26 crore traded per rebalance, or ₹3.3 crore per name.

The constraint. Take a mid-liquidity name in the bundled Indian panel: NILAM, whose median daily traded value over the sample is ₹33.5 crore (pit_universe computes exactly this quantity as its ranking score). Trading ₹3.28 crore of it in one day is 9.8% of ADV, far beyond the 1–3% band inside which a flat slippage assumption is defensible. The panel's least liquid name, UDAYAN at ₹7.4 crore, would be at 44% of ADV, which is not a trade at all.

The verdict, stated as a number. At 3% of ADV the tradable size in NILAM is about ₹1.00 crore per rebalance against the ₹3.28 crore required, so the book scales by 1.00/3.28 to roughly ₹15 crore. So: "capacity is approximately ₹15 crore at a 3%-of-ADV constraint against the median universe name, and materially less if the least liquid names are held; above that the flat 8 bps slippage assumption fails and the realistic cost curve is convex, not linear."

Why this belongs in the memo. A strategy that works at ₹15 crore and dies at ₹50 crore is not a strategy for a fund that needs to deploy ₹500 crore, and that is a decision the reader must be allowed to make. Hiding capacity is the most common way an otherwise honest backtest misleads, because everything in it is true and the one omitted number is the one that matters.

Worked example 8: The same result, written twice (US, $)

Take the identical US pipeline output: gross −3.32%, net −4.30%, turnover 6.75×, DSR 0.000, IS −0.150, OOS −0.500, cost_sensitivity failing.

Version A (passes). "No tradable edge. Costs explain 1.01% of the 4.30% annual loss; the remaining 3.29% is the absence of any gross spread, which is the expected result on a documented random-walk panel. All four harnesses ran; three pass, cost_sensitivity fails and that failure is the finding rather than a bug. The pipeline's ability to detect signal is evidenced on synthetic-factor-panel (monthly SR 0.751). Recommendation: do not trade; reuse the pipeline on a real panel."

Version B (fails). "Momentum shows promise. Excluding the two most volatile names and rebalancing weekly instead of monthly, the strategy is roughly flat gross of costs, suggesting the effect is present but obscured by friction. With optimised execution the edge should be realisable."

What separates them. Version B contains no false sentence. Each clause is defensible on its own. But "excluding the two most volatile names" is a post-hoc choice presented as a design decision; "rebalancing weekly" increases turnover and therefore costs, so the claim that friction is the obstacle is inverted; and "should be realisable" converts a negative result into a positive one using no evidence at all. Version B is how a good researcher becomes an expensive one, one reasonable-sounding sentence at a time, and every single move in it would have been caught by writing the trial count down first.

Practice set

Work each problem fully before reading its solution. Numeric answers are to about ±2%. All figures are either computed from the bundled seeded datasets (documented in app/public/datasets/MANIFEST.json) or explicitly illustrative.

P1 (guided). Price the publication lag. A US company's fiscal 2025 accounts are filed on 2025-05-18. Your pipeline joins FY2025 fundamentals onto every session from 2025-01-01. On how many sessions does the strategy hold information it could not have had, and what is the mechanical test that finds it?

Solution. From 2025-01-01 to 2025-05-17 inclusive (about 95 trading sessions) the pipeline reads a filing that does not yet exist. The test is the point-in-time invariant: recompute the decision with the fundamentals truncated to filed_date <= asof and compare. Any difference is the leak. This is precisely assert_no_lookahead, and it names the first offending date and ticker rather than merely returning False, because a harness that says "something is wrong" without saying where does not get used twice.

P2 (guided). Turnover to cost drag, and back. A book has annualised turnover of 6.75× and pays 15 bps round trip. What is the annual cost drag? If the gross annualised return is −3.32%, what is the net? And what turnover would you need for costs to be under 25 bps a year?

Solution. Drag = 6.75 × 0.0015 = 1.01%/yr. Net = −3.32% − 1.01% = −4.33%, matching the pipeline's reported −4.30% to rounding (the small gap is compounding, not error). For a drag under 25 bps you would need turnover below 0.0025 / 0.0015 = 1.67× per year, i.e. roughly annual rather than monthly rebalancing, which is why low-turnover value strategies survive costs that destroy high-turnover momentum ones.

P3. Why the skip must bite. Your momentum signal uses lookback=126, skip=5. You triple the last five closes before asof and the signal does not move. You then multiply the close at exactly t−5 by 1.5 and it does. Is the implementation right, and what would each individual failure have meant?

Solution. Yes, that is exactly the pair of behaviours required. The first check proves the last five sessions are discarded; if the signal had moved, skip was being ignored. The second proves the session at t−skip is the one actually used; if it had not moved, the lookback window was misaligned (probably off by one, or anchored to the end of the series rather than to t−skip). Either check alone passes for a wrong implementation, which is why QD2.05-e02 ships both.

P4. The one-character look-ahead. A colleague's engine computes pnl = (weights * returns).sum(axis=1). What is wrong, what is the fix, and why will their equity curve look excellent?

Solution. weights are decided at the close of day i and are being paid day i's return, a full period of look-ahead. The fix is weights.shift(1), with the first row zero. The curve looks excellent because the strategy is effectively buying at a price it already knows moved: for any signal correlated with the same day's return, this converts noise into a persistent, smooth-looking gain. It is the most common serious bug in backtesting code and it is invisible on inspection, which is precisely why no_lookahead exists rather than a code-review checklist.

P5. Breakeven arithmetic. A book's gross returns average +4.90 bps per day and it turns over 0.0268 of itself per day, at 11.12% annualised volatility. What is the breakeven round-trip cost on the linear approximation, and why does an exact bisection on annualised return give a lower number?

Solution. Linear: 0.000490 / 0.0268 = 0.018295 → 182.9 bps, which is the cost at which the arithmetic mean daily net return is exactly zero. The bisection on the bundled US book returns 172.49 bps, about 5.7% lower. The gap is volatility drag: a geometric annualised return sits roughly σ²/2 = 0.1112²/2 ≈ 0.62 percentage points below what the arithmetic mean implies, so a stream whose daily mean is exactly zero still compounds to −0.71% a year, and the geometric zero crossing is reached at a lower cost. Linear cost arithmetic is optimistic about breakeven, use it for intuition and bisect for the number you publish.

P6. VaR versus ES, and which to quote. Historical 95% VaR is 1.191% and ES is 1.539%. What does each mean, why is ES the more useful of the two, and which property does VaR lack?

Solution. VaR(95%) says: on 5% of days the loss was at least 1.191%. ES(95%) says: on those days, the average loss was 1.539%. ES is more useful because VaR tells you only where the tail begins while ES tells you what is in it, two books with identical VaR can have completely different ES. VaR also lacks subadditivity: the VaR of a combined portfolio can exceed the sum of the parts' VaRs, so it can penalise diversification. ES is coherent and does not. Quote both; act on ES.

P7. Read the tail from the gap. Historical VaR is 1.243% and Gaussian VaR is 1.167%; historical ES is 1.665% against a Gaussian 1.459%. Sample kurtosis is 4.92. What do you conclude, and what should the memo say?

Solution. The realised tail is heavier than a normal with the same mean and sigma: confirmed by kurtosis 4.92 against 3.0. The parametric model understates the loss by 6.1% on VaR ((1.243 − 1.167)/1.243) and by 12.4% on ES ((1.665 − 1.459)/1.665), and the understatement grows further out in the tail, which is why the ES gap is the larger of the two. The memo sentence: "returns are leptokurtic (kurtosis 4.92); parametric VaR understates the realised 95% loss by ~6% and parametric ES by ~12%, so parametric limits set on this book should carry an explicit tail multiplier."

P8. Drawdown depth versus duration. One book has a 17.19% maximum drawdown and spends 350 of 500 sessions below its previous peak. Another has a 25% maximum drawdown and spends 40 sessions under water. Which is worse, and for whom?

Solution. They are worse for different people. The 25%/40-session book is worse for a leveraged holder or anyone with a hard stop, because the depth is what triggers a margin call. The 17%/350-session book is worse for almost every human being who has to live with it: 70% of the sample under water means the strategy essentially never felt like it was working, which is what causes an allocator to redeem and a researcher to be reassigned. Depth is a solvency question; duration is a behavioural one, and only the second is routinely omitted from reports.

P9. Count your trials honestly. Over two weeks you tried lookbacks of 63, 126 and 252; skips of 0, 5 and 10; universe sizes 5, 8 and 10; two winsorisation caps; and once excluded the two most volatile names. What is the minimum honest n_trials, and what does that do to the bar at V = 0.01?

Solution. The grid alone is 3 × 3 × 3 × 2 = 54 combinations, plus the exclusion variant applied across at least some of them, so at least 54, realistically over 100. At V = 0.01 the bar E[max SR] rises from 0.190 at 20 trials to 0.253 at 100. Note the shape: the bar grows roughly with √(ln N), so the difference between 100 and 1,000 trials (0.253 → 0.326) is smaller than most people fear, but the difference between "I tried a few" (bar ≈ 0.12) and the truth is exactly the difference that makes a marginal result disappear.

P10. Same record, two verdicts. A 48-month track record has a monthly Sharpe of 0.751. At 20 trials, DSR is 0.9997 when V = 0.01 and 0.110 when V = 0.25. Explain the inversion in one sentence, and say what the reader must be given.

Solution. At V = 0.25 the expected best of twenty noise trials is a monthly Sharpe of 0.950, which is above the observed 0.751, so the result is worse than luck would have predicted, while at V = 0.01 the bar is only 0.190 and the result clears it easily. The reader must be given n_trials and sr_variance alongside the DSR, because the DSR is a function of both and is uninterpretable: indeed unfalsifiable, without them.

P11. Size the embargo. Your signal uses a 126-session lookback and a 5-session skip, on 500 sessions split 60/40. What is the fully purged embargo, what does it cost, and what do you do?

Solution. The information horizon is 126 + 5 = 131 sessions, so a fully purged embargo discards 131 of the 200 out-of-sample sessions, leaving 69, a Sharpe estimate on 69 observations is nearly useless. The practical answer is to use a shorter embargo (10 sessions, as the pack does), and state the compromise explicitly: the split is partially purged, so the out-of-sample number is optimistic. The alternative, a longer sample, is the right fix and usually unavailable, which is itself worth saying.

P12. Which harness catches what? For each defect, name the harness that catches it and the one that will not: (a) a scaler fitted on the full history; (b) a strategy whose gross edge is 8 bps against 15 bps costs; (c) a universe built from today's index members; (d) set iteration leaking into weights; (e) a parameter tuned by looking at the out-of-sample window six times.

Solution. (a) no_lookahead catches it; determinism will not, a leak is perfectly reproducible. (b) cost_sensitivity catches it; no_lookahead will not, the strategy is honest, just unprofitable. (c) None of the four catches it. Deleting future rows does not restore deleted names; survivorship is a dataset property and needs a universe function plus a stated limitation. (d) determinism catches it; the others will not. (e) None of the four catches it, only your own honest trial count and the DSR price it. Knowing (c) and (e) are uncovered is the point of the question.

P13. Capacity in currency. A ₹50 crore book, eight names, annualised turnover 6.30×, monthly rebalancing, median universe-name ADV ≈ ₹33.5 crore. What fraction of ADV does a rebalance trade, and what is the capacity at a 3%-of-ADV limit?

Solution. Turnover per rebalance ≈ 6.30 / 12 = 0.525 of the book, so ₹26.25 crore traded across eight names ≈ ₹3.28 crore per name, about 9.8% of ADV, more than three times a defensible limit. At 3% of ADV the per-name trade is ₹1.00 crore, so the book scales to 1.00 / 3.28 × 50 ≈ ₹15 crore. The memo line: "capacity ≈ ₹15 crore at a 3%-of-ADV constraint against the median name, and less if the least liquid names are held; the flat slippage assumption fails above it."

P14. Diagnose the too-good split. A submission reports an in-sample Sharpe of 1.9 and an out-of-sample Sharpe of 1.87 on a random 70/30 split. What is your first question, and what is the most likely explanation?

Solution. First question: was the split chronological? It was not, "random 70/30" means test rows are interleaved with training rows, so a 126-day lookback on a test row overlaps training rows on both sides, and the two windows are not independent at all. The near-identical Sharpes are the symptom, not the reassurance: genuine out-of-sample decay is the norm and its absence is the anomaly. Second question, if the split really was chronological: how many times did you look at it?

P15. Fingerprint a pipeline. You canonicalise results by rounding floats to ten decimals before hashing. A colleague says this is sloppy and you should hash the raw floats. Who is right, and what breaks in each direction?

Solution. You are, for a research harness. Hashing raw floats makes the digest sensitive to the last bit, so an irrelevant difference (a different BLAS build, a reordered summation, a compiler flag) reports as nondeterminism and the harness is disabled within a week for crying wolf. Rounding to ten decimals is far tighter than any economically meaningful quantity and still catches every real source of nondeterminism (unseeded shuffles, hash-order leaks, timestamps). Round too coarsely, though (three decimals, say) and you start hiding genuine differences. Ten is a deliberate, documented middle, and QD2.05-e06's hidden tests check both directions: 1e-15 of dust must not change the digest and 1e-6 of real difference must.

P16. Write the verdict. Your pipeline reports: gross −3.32%, net −4.30%, turnover 6.75×, DSR 0.000, IS −0.150, OOS −0.500, cost_sensitivity failing, other three harnesses passing. Write the two-sentence verdict that earns full marks, and name the trap.

Solution. "No tradable edge: the book loses 3.32% annualised before costs and 4.30% after, with 1.01% of that attributable to a 6.75× turnover at 15 bps and the remainder to the absence of any gross spread. Three harnesses pass and cost_sensitivity fails (that failure is the result, not a defect) and the pipeline's ability to detect a real signal is evidenced separately on synthetic-factor-panel." The trap is the urge to add a third sentence beginning "however, with optimised execution…" Every word after the second sentence is where honest research turns into advocacy, and it is the exact move that fails Memo B.

Applied mini-project

Build the pipeline, run it against both controls, and write the memo.

This is the node's deliverable and it is graded on six axes, one of which is honesty and carries independent weight.

What you build. One module, standard library only, exposing a single entry point run_capstone(prices, fundamentals, benchmark, config=None) that returns {'config_used', 'metrics', 'harnesses', 'digest'}, the shape QD2.05-e10 grades. Internally it composes the six stages (e01 universe, e02 signal, e03 backtest, e08 risk, e09 performance) and runs all four harnesses (e04–e07). No wall-clock, no network, no unseeded randomness, every parameter in config.

Do this, in order.

1. Write the hypothesis first, in one paragraph, with all five elements from Stage 0, and put it at the top of the memo file before you load any data. Timestamp it.
2. Start a research log. One line per variant tried, appended as you go, never reconstructed afterwards. This file is your n_trials, and it is the single artefact that most distinguishes a professional submission.
3. Build the pipeline stage by stage, running the pack exercises as you go. Do not proceed past a stage whose exercise is not green.
4. Run the negative control (us-eod + sec-fundamentals + us-index, and again on nse-eod + india-fundamentals + nifty-index). Record everything, including the fact that you expect a null.
5. Run the positive control (synthetic-factor-panel). Your pipeline must recover the planted tilt. If it does not, the pipeline is broken, not the data, fix it before you interpret a single negative-control number.
6. Break your own pipeline four times, on purpose, and confirm each harness catches its own defect: introduce a full-sample z-score (no_lookahead must fire), an unseeded shuffle (determinism must fire), a zero-cost assumption (cost_sensitivity must change its verdict), a random rather than chronological split (oos_split's degradation must vanish suspiciously). A harness you have never seen fail is a harness you have not tested.
7. Compute the deflated Sharpe using the honest trial count from your log and an sr_variance measured from the Sharpes actually in that log, not assumed.
8. Write the one-page memo to the six headings. One page. If it runs to two, you are explaining rather than reporting.
9. Publish the digest at the top of the memo, and state the exact command that reproduces it.

Deliverables. (a) the module; (b) memo.md, one page, six headings, digest at the top; (c) research_log.txt; (d) a harness_report showing all four verdicts on both controls, including failures.

Rubric: 100 points.

How honesty is scored, the 25 points, explicitly.

• 25/25: Every one of the six memo headings is present and specific. The trial count comes from the log, not from memory. Every failing harness is reported verbatim with its verdict. Every limitation that a reviewer could find is already in the memo. A submission reporting "no tradable edge" with this evidence scores 25/25.
• 18/25: All six headings present, but one is generic (e.g. "regime dependence: the sample may not be representative" with no statement of which regime is missing).
• 10/25: A limitations section exists but omits either the trial count or a failing harness.
• 0/25: Any of: a harness failure not reported; a post-hoc exclusion presented as a design choice; a trial count that the research log contradicts; a negative result rewritten as a positive one ("promising", "should be realisable", "with optimised execution"). A submission that hides its overfitting fails this node regardless of every other axis, because 75 points cannot clear a 90% crown gate.

The pass condition, stated once more. A strategy that only works in-sample passes if you say so. A strategy that hides it does not pass at all.

Reading & resources

Everything below is an anchor to learn from and verify against, never a source to quote a number from without re-deriving it (R3). Tags: ✓ free / $ paid, and ▲ beginner–intermediate / ▼ advanced.

The anti-overfitting canon (read these two first):

• Bailey, Borwein, López de Prado & Zhu, "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance" (Notices of the AMS, 2014), the paper that made backtest overfitting a named problem, and the source of the minimum-backtest-length idea. [✓ open access] [▼]
• Bailey & López de Prado, "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality" (Journal of Portfolio Management, 2014), the derivation behind QD2.05-e09. Read it with the code open. [✓ SSRN preprint] [▼]
• López de Prado, Advances in Financial Machine Learning (2018), purging, embargoing, combinatorial purged cross-validation, and the protocols QD2.04 formalises. Chapters 7 and 11 are the ones this node leans on. [$] [▼]

Backtesting practice and its failures:

• Harvey, Liu & Zhu, "…and the Cross-Section of Expected Returns" (Review of Financial Studies, 2016), the multiple-testing argument applied to the published factor zoo; the origin of the "raise your t-statistic hurdle" discipline. [✓ preprint] [▼]
• Arnott, Harvey & Markowitz, "A Backtesting Protocol in the Era of Machine Learning" (Journal of Financial Data Science, 2019), a short, practical checklist that pairs directly with this node's memo headings. [✓ SSRN] [▲]
• Chan, Algorithmic Trading: Winning Strategies and Their Rationale, the clearest practitioner treatment of survivorship, look-ahead and data-snooping bias with worked cases. [$] [▲]

Risk, performance and attribution:

• Acerbi & Tasche, "On the Coherence of Expected Shortfall" (2002), why ES is coherent and VaR is not. [✓] [▼]
• Basel Committee, Minimum Capital Requirements for Market Risk, the regulatory move from VaR to Expected Shortfall, and the backtesting regime around it. Skim for the reasoning, not the calibration. [✓] [▼]
• Bacon, Practical Portfolio Performance Measurement and Attribution, the reference for every ratio you will be asked to justify. [$] [▲–▼]

The signals themselves (so you know what you are re-testing):

• Jegadeesh & Titman, "Returns to Buying Winners and Selling Losers" (Journal of Finance, 1993), the momentum result, and the source of the 12–1 convention. [$ / library] [▼]
• Asness, Moskowitz & Pedersen, "Value and Momentum Everywhere" (Journal of Finance, 2013), cross-market evidence, including India, and a model of how to report a factor honestly. [✓ preprint] [▼]
• Novy-Marx & Velikov, "A Taxonomy of Anomalies and Their Trading Costs" (RFS, 2016), the paper that measures how many published anomalies survive their own turnover. Read it immediately after computing your first breakeven. [✓ preprint] [▼]

Tooling and reproducibility:

• zipline-reloaded, vectorbt, backtrader: production backtest frameworks. Read their source for how they handle the decision lag; do not adopt one until you can state what it does about it. [✓] [▲–▼]
• Hyndman & Athanasopoulos, Forecasting: Principles and Practice, ch. 5, the cleanest free treatment of time-series cross-validation. [✓ online] [▲]
• The bundled datasets' own app/public/datasets/MANIFEST.json, read it before you interpret any result. Provenance, seeds and the honest note about what is real and what is a labelled synthetic look-alike. [✓] [▲]

Teach it back & journal

Teach it back (20 minutes, out loud, to someone who does not trade).

Explain, without notes: (1) why a backtest is a claim about a counterfactual and what that implies about evidence; (2) the two ways a dataset lies to you before any modelling happens, and the one-line test for each; (3) why the same 48-month track record can be "certain" and "insignificant" depending only on a number you supply from memory; (4) the four harnesses, what each catches, and (the part that shows you understood) what each one cannot see; (5) why "we tested it properly and there is nothing there" is a delivered result. If your listener cannot repeat back the grading rule (honesty is scored on its own) start again at (5).

Then teach it back to a practitioner. Hand them your memo.md and your research_log.txt, and ask them the six ten-minute questions about your own work. Write down which one you were least comfortable answering. That is your real weak point, and it is almost never the code.

Journal prompts.

1. The count. Before you looked at your research log, what did you think your trial count was? What was it actually? Compute your DSR at both numbers and write down the two verdicts side by side. Nothing else in this node will teach you as much about yourself as that pair.
2. The moment. Identify the first time in this project you were tempted to change something after seeing a result. Describe the change, the justification you constructed for it, and whether you made it. Then say which harness would or would not have caught it.
3. The sentence you deleted. Every honest memo has one, a hopeful clause that survived three drafts before you cut it. Write it here, and write why you cut it.
4. Your blind spot. Of the four harnesses' known blind spots (survivorship, a wrong slippage model, a deterministically wrong pipeline, repeated looks at the out-of-sample window) which one is your project most exposed to, and what would you build to cover it if you had another week?
5. The negative. If your result is a null: write, in three sentences, the case for why it is worth someone else's time. If your result is positive: write, in three sentences, the strongest case against it that a hostile reviewer could make. Whichever you wrote, the other one is the harder exercise: do it too.
6. Forward. You now hold the Quant Developer title. Name the one thing from this node you will apply to work that has nothing to do with markets, and say why it transfers.