Building a Backtesting Engine: Design & the Classic Pitfalls

Branch: Quant Developer (Quant-Code galaxy) · Node QD1.03 · ~24 focused hours · CROWN node · Mastery gate ≥ 90%

A backtest is a machine for producing a number you want to believe. That is not a cynical description; it is a structural one. Every discretionary choice inside it, which bar you fill on, which names are in the universe, which parameter you kept, what you assumed a trade costs, has an obvious version that flatters the result and a careful version that does not, and the flattering version is almost always easier to write. The gravity of the exercise points one way. Left alone, a backtest drifts upward.

So this node is not "how to loop over prices." You can write that in eleven lines and you will, in the first exercise. This node is about the four ways the eleven lines lie to you, how to measure each lie in currency rather than argue about it, and how to build the automated defences that catch the lie next time, when you are tired and the deadline is real and you will not be looking.

Those four lies have names, and each gets the same treatment here: a seductive backtest that is wrong, then the corrected one, then the difference in money.

• Look-ahead bias. A moving-average rule on a seeded three-year price path (seed 20240301) turns ₹/$1,000,000 into $97,451,579 when the signal is used on the bar that produced it, and $747,299 when it is used on the next bar. The phantom $96.7 million is one missing .shift(1).
• Survivorship bias. A sixty-name Indian universe over ten years (seed 515151) compounds to 3.4834× using point-in-time membership and 4.9866× if you quietly run only the names still listed today: 4.14 percentage points a year of return that never existed, ₹1.50 crore of phantom wealth on a ₹1 crore start.
• Overfitting. One thousand strategies with provably zero edge, run on a zero-drift series (seed 90210): the best of them posts an in-sample Sharpe of 2.23 and an out-of-sample Sharpe of −0.12. The theory says the winner of 1,000 coin-flippers should score about 1.85, and it did.
• Unrealistic fills. One signal, five fill assumptions, an Indian mid-cap path (seed 1975): terminal wealth falls from 29.5642× to 1.7949×. That is a 1,547% overstatement, and 92% of it is destroyed before a single basis point of cost is charged, purely by refusing to be filled at the best price of the bar.

None of those gaps is subtle once measured. All of them are invisible until measured. That is the entire problem, and the discipline that solves it is not carefulness (carefulness fails at 11pm) it is executable defences: harnesses that fail your build when the leak returns.

What this node ships. The centrepiece is a graded harness, no_lookahead, that finds look-ahead bias mechanically, without reading your code: an honest strategy is by construction invariant to any change you make to the future, so perturb the future and watch which decisions move. It is exercise QD1.03-e02, it is worth more XP than anything else in the pack, and its hidden tests hand it seven strategies (two honest, five leaking in ways that look completely reasonable in a notebook) and require it to separate them exactly. Alongside it you build cost_sensitivity (exercise QD1.03-e09), the harness that doubles your cost assumption and reports who is still standing. Those two, plus determinism and an out-of-sample split, are the four gates the capstone QD2.05 will hold your strategy against. You are building the referee here, before you play the game there.

Code pack: this node ships with QD1.03 (app/public/code-packs/QD1.03.json): ten graded exercises, QD1.03-e01 through QD1.03-e10, whose hidden tests run on both a ₹-denominated and a $-denominated fixture so that a solution tuned to one market fails the other. The exercises are the assessment; this document is the argument that makes them passable.

A note on every number in this node (R3). There is no proprietary data here and no real ticker. Every figure quoted above and below is generated by a seeded, reproducible process (stated seed, stated parameters, stdlib Python) so you can recompute all of it from scratch and get the same digits. Where a figure is a market convention rather than a computation (Indian statutory charges, typical spreads) it is labelled as an order-of-magnitude convention to be re-verified against your own broker's live schedule, never as a measured constant. Synthetic data is the right choice for this node in particular: to prove that a thousand strategies with zero edge produce a 2.23 Sharpe winner, you need a series where the edge is known to be zero, and only synthetic data can give you that.

Learning objectives

By the end you can:

1. State what a backtest is and what it is not, a simulation of a decision rule against a historical data-generating process, whose output is a distribution of possible histories and whose single realised path is one draw from it, and name the four assumption classes (data, timing, execution, selection) that every backtest silently makes.
2. Choose between vectorised and event-driven architectures on the merits, prove they are numerically identical in the frictionless case, and name precisely what breaks that equality: path dependence (stops, targets, margin), state (position, cash, inventory), and any rule that reads inside a bar.
3. Build an event loop with correct state discipline, the decide on t−1, act on t rule, the single ordering that makes look-ahead structurally impossible rather than merely avoided.
4. Detect look-ahead bias mechanically, by implementing and running the no_lookahead future-perturbation harness, and explain why invariance-to-the-future is a complete test for the class of leaks it covers and what it cannot see.
5. Name and quantify the six mechanisms of look-ahead, the missing shift, the full-sample normalisation, the centred window, the restated or revised data point, the publication-lag leak, and the survivorship-selected universe, and price each in terminal wealth on a worked case.
6. Build a point-in-time universe from listing and delisting records, explain why "the current index constituents" is the most expensive free dataset in finance, and measure the bias it induces.
7. Treat the best of N backtests as an order statistic. Compute the expected maximum Sharpe of N mean-zero trials, deflate an observed Sharpe against that bar, estimate the effective number of independent trials behind a correlated grid, and state why the deflated Sharpe is a floor on your scepticism rather than a certificate.
8. Model execution honestly: the fill-price ladder from fantasy to defensible, the half-spread you cross, the fee and statutory stack in both India and the US, and market impact as k·√participation, and derive the net-alpha identity net = alpha − turnover × cost and the break-even cost c* = alpha / turnover it implies.
9. Size capacity two ways (the participation ceiling and the impact-erosion ceiling) say which binds, and explain why an unrealistic fill assumption and an unstated capacity are the same error in different clothing.
10. Design a leak-free validation protocol: walk-forward selection, purged and embargoed cross-validation for overlapping labels, and the discipline of a genuinely untouched holdout.
11. Ship the four harnesses (no_lookahead, cost_sensitivity, determinism, oos_split) as executable gates on your own research, and read a backtest report the way a risk officer does: hunting for what is absent.
12. (Productivity objective: R10 duality.) Use a coding copilot on backtesting code the way a working quant developer does, after you can do it by hand, and review its output for the failure modes specific to this domain: silently vectorised look-ahead, cost models that charge one-way for a round trip, and confidently hallucinated pandas and vendor APIs.

The duality, stated once (R10). The gated skill is objectives 1–11. The gate is deliberately hostile to shortcuts: the pack's hidden tests check both a ₹ and a $ fixture, they assert structural properties (that the frictionless event loop equals the vectorised product to 1e-12; that a harness flags exactly the leaking bars and no others) as well as values, and several of them fail the most plausible wrong answer rather than only the lazy one. Objective 12 is the payoff you keep afterwards. The app never calls an AI at runtime, the callouts below ask you to generate something in your own tool, paste it back, and let the local hidden tests judge it. The machine drafts; the harness decides; you are the one who has to know why it failed.

Prerequisites & connections

Builds on. CS1.01 and CS1.02 for Python itself, CS1.04 for the cost of a loop, and CS1.05 for the structures an event-driven engine is made of, a backtester's event queue is a priority queue keyed on timestamp, and its order book is the tree-plus-queue from that node. DA1.02 and DA1.03 for pandas, because the vectorised backtest is a shift, a where and a cumprod, and because the single most dangerous pandas operation in this node, a join that silently reindexes: lives there. QD1.01 owns the data layer this node consumes: corporate actions, adjusted prices, and the construction of point-in-time, survivorship-controlled panels. QD1.02 owns signal and factor construction, the content of the signal that this node merely schedules and pays for.

Cross-links, not duplications (R1). This is a heavily bordered node; respect the borders and it stays sharp.

• QM1.03 / QM1.04 own the statistics. The sampling distribution of a Sharpe ratio, the multiple-comparisons problem, false discovery rate, and why a t-statistic from a selected maximum is not a t-statistic, all derived there. QM1.04 is the specific address: its section 17, "Multiple testing: the arithmetic of looking many times", carries the Bonferroni and false-discovery-rate corrections, and its section 18, "The sampling distribution of a Sharpe ratio, and how long a backtest must be", derives the variance you are about to use. Here we apply them: state the assumption, compute the number, act on it. If Var(SR̂) ≈ (1 + SR²/2)/T is not a familiar object, go and read it once there; it is the whole reason the deflated Sharpe has the denominator it has.
• DA2.02 owns leak-free ML pipelines in general, target leakage, fitting a scaler on the full sample, the Pipeline object that makes it hard to get wrong. Exercise QD1.03-e10 implements the split (purging and embargoing for overlapping labels) because it is a backtesting-specific split; the general principle is theirs.
• QD2.01 owns microstructure. The limit order book, queue position, order types, and the empirical estimation of impact coefficients. Here impact is a one-parameter model, k·√participation, used to price a decision, not derived.
• QD2.02 owns risk (VaR, ES, stress) and QD2.03 owns performance and attribution (Sharpe, Sortino, drawdown, the deflated Sharpe as a reporting statistic, Brinson attribution). This node computes a deflated Sharpe because it is the correct response to a search over parameters; QD2.03 is where it takes its place in a report.
• QD2.04 owns reproducible research infrastructure (configuration, secrets, seeds, scheduling, logging) and the wider anti-overfitting protocol. This node ships two of the four harnesses; that node industrialises all of them.
• QD1.04 owns portfolio construction. Here, position sizing is deliberately trivial (a scalar in {−1, 0, +1}) so that nothing in the measured gaps can be attributed to a clever optimiser.

Feeds forward. Directly into QD2.05, the capstone, which requires all four harnesses to be green and scores honesty on its own line: a strategy that only works in-sample still passes if its limitations are reported honestly, and one that hides its overfitting fails. Everything you build here is the machinery that lets you be honest cheaply.

Core teaching content

What a backtest actually is

Strip away the code and a backtest is a claim of this shape: if I had followed this rule, using only what I could have known at the time, and had been filled at prices I could plausibly have got, my wealth would have followed this path. Four clauses, and each is an assumption that can fail:

1. "This rule": was the rule fixed before you saw the result, or chosen because of it? (Selection.)
2. "Only what I could have known": did the data available to the simulation at time t match the data available to a person at time t? (Timing and data.)
3. "Prices I could plausibly have got": where in the bar were you filled, what did you pay to cross the spread, what did your own order do to the price? (Execution.)
4. "My wealth": at what size? A path is not a strategy until it has a capacity attached. (Capacity.)

The output of a backtest is not a number, it is one draw from a distribution of histories the same rule could have produced. This reframing is the single most useful mental correction available in this node, and it fixes an entire class of errors at once. A 1.4 Sharpe over three years is not "a 1.4 Sharpe"; it is a point estimate whose standard error, for an annualised Sharpe over T years, is roughly √((1 + SR²/2)/T), about 0.58 over three years at low Sharpe. Two strategies at 1.4 and 1.1 are not ranked by that data. The derivation belongs to QM1.04; the habit belongs here: always carry the error bar next to the estimate, and never rank inside it.

Survivorship bias: the graveyard you deleted

Download today's index constituents. Pull ten years of history for each. Backtest. Congratulations: you have restricted your universe to companies that did not fail, using knowledge you did not have in year one, and you have done it without writing a single line of strategy code.

The mechanism is precise. In any real universe, names leave (acquisition, merger, bankruptcy, delisting, suspension, a fall out of the index) and the ones that leave for bad reasons leave after performing badly. Deleting them removes the left tail of the cross-section. Your mean rises, your volatility falls, your drawdowns get shallower, and your Sharpe improves twice over.

Quantified on a seeded ten-year Indian panel (seed 515151): 60 names, 120 monthly periods, 12 of them delisting (20% of the universe) at months spread from 31 to 99, each taking a −80% return in its final month.

4.14 percentage points a year, a 1.432× terminal ratio, and on ₹1,00,00,000 (one crore) invested: ₹3.48 crore honest against ₹4.99 crore fantasy: ₹1.50 crore of phantom wealth. Notice the drawdown fell (25.7% → 24.2%) even though the deleted names were the disasters: removing the failures removes the periods that hurt, so the bias flatters your risk numbers as well as your return numbers. Every statistic moves in the flattering direction at once. That is the signature of a data problem rather than a modelling one.

Three practical notes, in descending order of how often they bite:

• 20% attrition over ten years is not pessimistic, it is an ordinary order of magnitude for a broad equity universe, and it is higher for small-caps and higher still in a stressed decade. If your universe file has the same number of names on day one and day 2,520, it is not a universe file, it is a screenshot.
• The delisting return is a modelling decision you must make explicitly. −80% is a defensible convention for an equity that stops trading for the usual reasons; 0% is a common and badly wrong default; −100% is right for genuine liquidations. Whatever you choose, state it, and re-run with one of the others to see how much your answer depends on it.
• Survivorship has quiet siblings. Backfill bias: a database that adds a fund or an index member and populates its history retroactively, so the record only ever contains entities that lived long enough to be added. Selection bias in vendor coverage: databases start covering a name when it becomes interesting. Both push the same way, and neither is fixed by careful code.

The fix is structural: keep membership as a table of (name, start, end) and ask it which names existed at time t. That is universe_at in exercise QD1.03-e03, and the exercise makes you implement the bias deliberately (survivors_only=True) so that you can price it rather than merely deplore it. Deliberately implementing the wrong version is a recurring device in this node, and it is not perverse: you cannot measure a bias you cannot reproduce.

Unrealistic fills: one signal, five P&Ls

Now the pitfall that is easiest to fix and most often skipped, because fixing it requires admitting how much of the result was fill assumption rather than signal.

Same signal, same data, five fill and cost assumptions, stacked from fantasy to defensible. A seeded Indian mid-cap path (seed 1975, 504 bars, deliberate positive return autocorrelation so that there is a genuine gross edge to find), a hyperactive rule (long above the 5-bar mean, short below, 113 position changes in 503 bars) and a ₹ cost stack.

From 29.5642× to 1.7949×: a 1,547% overstatement of terminal wealth. On ₹1,00,00,000 that is ₹29.56 crore against ₹1.79 crore: ₹27.77 crore of phantom money.

Read the shape of that ladder, because the shape is the lesson. Rungs 1 to 2 destroy 92.0% of the claimed wealth and cost nothing, no fees, no spread, no impact, just refusing to be filled at the single best price of the bar. The entire cost stack (rungs 2 to 5) destroys a further 24.2%. In other words: the fill assumption was four times more expensive than every cost combined, and it is the one that never appears in the assumptions table of a backtest report.

The fill ladder, ranked, is worth committing to memory:

Three refinements that separate a plausible model from a defensible one:

• Gaps. "Filled at the next open" is not conservative on a day that opens 6% away from you, because that is exactly the day your signal wanted to trade. Model gap risk as an extra slippage term on high-volatility days, or at minimum report the strategy's performance excluding the ten largest gap days and see how much of the edge lived there.
• The stop-loss fill. Exercise QD1.03-e05 fills a stop at exactly the trigger price. That is already optimistic: on a gapping day, price passes through the trigger without trading there. Every stop-based backtest you ever read carries this optimism, usually unstated.
• Volume feasibility. If your position implies trading 40% of the day's volume, "filled at the open" is not an assumption, it is a wish. This is capacity, and it is below.

Transaction costs, the net-alpha identity, and the break-even that ends arguments

Costs come in three layers, and confusing them is the most common cost-modelling error after the round-trip factor of two.

1. Explicit and statutory: brokerage, exchange and clearing fees, and in India the securities transaction tax, stamp duty and GST; in the US, SEC Section 31 fees and FINRA TAF on sales. Known in advance, asymmetric between buy and sell in India, and small relative to what follows.
2. The spread: you cross a half-spread on each side. Wide in small-caps, wider at the open and close, and widest exactly when your signal fires, because signals fire on news and news widens spreads.
3. Impact: your own order moves the price against you. The standard empirical shape is impact ≈ k·√participation, sublinear because liquidity replenishes: doubling your size costs about 1.41× the impact per share, not 2×. The coefficient k is name-, regime- and venue-specific; QD2.01 owns estimating it, and here it is a parameter you must state and stress.

Treat any single number as an order of magnitude to re-verify against your own broker and venue. As conventions, not measurements: a liquid US large-cap round trip might be 1–3 bps all-in at institutional size; an Indian mid-cap round trip closer to 15–25 bps once statutory charges are included; a small-cap or an emerging-market name in size, far more. These are conventions for sizing an argument, not constants, get your own numbers, and then double them (see below).

Now the identity that makes all of this tractable. Over a period, with alpha the gross return per period in bps and turnover the fraction of NAV traded per period:

net = alpha − turnover × one_way_cost

and setting net to zero gives the number that ends most arguments about whether a strategy is real:

break-even cost  c* = alpha / turnover

Five configurations, all arithmetic:

Three readings, each worth more than the table:

• The same gross alpha is a good strategy, a marginal one, and a losing one, decided entirely by turnover. Rows 1–3 have identical signals.
• Compare c* with your cost estimate, and let the ratio decide how much you trust the strategy. Row 1's break-even is 36 bps against an assumed 6, a 6× margin, so the conclusion survives being wrong about costs. Row 2's is 9 against 6, a 1.5× margin, so the conclusion is the cost assumption. Row 3 was never alive.
• Now double the costs, because your cost model is a guess. Row 1 at 12 bps: net 6.00 bps/day, +15.12% a year: survives. Row 3 at 12 bps: net −15.00 bps/day, −37.80% a year: dead. This is the cost_sensitivity harness (exercise QD1.03-e09), and it is the second gate on the capstone. A strategy that dies at 2× costs is not a strategy; it is a bet on your cost model, and your cost model is the least researched object in your repository.

The last piece of arithmetic is the one people get wrong in code rather than in theory: a flip from +1 to −1 trades two units of NAV, not one. Turnover is Σ|Δposition|, and forgetting the flip halves every cost in a long/short backtest. Exercise QD1.03-e04 asserts it directly.

Slippage, and why it is not the same as cost

Cost is what you pay to trade at the expected price. Slippage is the difference between the price you assumed and the price you got. They compound, and they behave differently under stress: costs are roughly stable, slippage is worst exactly when you most want to trade. A defensible model has three terms:

• A fixed component: half-spread plus fees, in bps, per side.
• A size component: k·√participation, growing with your order relative to daily volume.
• A volatility component: slippage scales with volatility, so a fixed bps assumption under-charges precisely on the days that matter. At minimum, report the strategy's performance on the highest-volatility decile of days separately.

And a discipline: model slippage against you, always. If your model has a random slippage term with mean zero, you have modelled a world where half your trades get price improvement, which is not the world in which your order is the one demanding liquidity.

Capacity: two ceilings, and which one binds

"It returns 30% a year" is an incomplete sentence until it ends with "on how much money". Capacity is where fill assumptions become balance-sheet facts, and it has exactly two ceilings.

Ceiling 1, the participation cap. You refuse to be more than some fraction of a name's daily traded value (10% is a common self-imposed limit; 5% is conservative). That gives a hard AUM ceiling:

liquidity capacity = cap × ADV × n_names / turnover

Ceiling 2: impact erosion. Even below the cap, your impact grows with size until net alpha reaches zero. Setting alpha − turnover·(fixed + k·√p) = 0 and solving for participation gives a closed form, no search required, and exercise QD1.03-e08 grades the derivation rather than a bisection, because knowing when a root can be solved for beats searching for it:

√p* = (alpha/turnover − fixed) / k        p* = ((alpha/turnover − fixed)/k)²
impact capacity = p* × ADV × n_names / turnover

and if alpha/turnover ≤ fixed, the fixed costs alone have eaten the edge and capacity is zero at any size.

Worked on an Indian mid-cap basket: 25 names, median daily traded value ₹40 crore, fixed cost 6 bps one-way, k = 14 bps per unit √participation, participation cap 10%, gross alpha 9 bps/day.

The two books have identical gross alpha and their capacities differ by a factor of eleven. And the fast book is the more dangerous case, because it is dead from impact (₹46 crore) while still nominally "liquid" (₹100 crore), the constraint that binds is not the one you would have checked. Look at what happens as it grows: at ₹100 crore the fast book participates at 10% of volume, pays 4.43 bps of impact, and nets −1.43 bps/day = −3.60% a year. It was profitable on paper at every size below ₹46 crore and is a losing strategy above it, with no change to the signal at all.

Two practical consequences:

• Report capacity next to return, always. "18% a year at ₹500 crore, falling below zero above ₹4,600 crore" is a sentence an allocator can act on. "18% a year" is not.
• Capacity is the reason the best strategies are not published. A ₹46-crore-capacity strategy is a wonderful personal strategy and an irrelevant institutional one, and this asymmetry explains most of what you will read about "the anomaly stopped working."

Determinism, and the four harnesses

A backtest you cannot reproduce is an anecdote. Three requirements, all cheap:

• Seed everything and record the seed: every random draw, every shuffle, every stochastic optimiser, every train/test split.
• Freeze the data snapshot. Re-downloading between runs changes the answer for reasons that have nothing to do with your code. Hash the input file and record the hash next to the result.
• Version the configuration with the result. Parameters, cost assumptions, universe definition, date range, and the code commit, stored with the output, not in your memory of what you ran.

The test is trivial and non-negotiable: run it twice, byte-compare. If two runs differ, you do not have a result yet; you have a random number.

That is the fourth of the four harnesses this node exists to install. Together they are the gate on QD2.05, and they are the closest thing to a professional standard in this field:

Write them once, run them on everything, and never argue about any of the four again.

Common mistakes & how experts think differently

Mistake: treating a high Sharpe as good news. A daily-data Sharpe above about 3, with realistic costs, is not an achievement; it is a defect report. Experts have a prior on plausible Sharpes by strategy family and holding period, and a result above that prior triggers a bug hunt, not a celebration. The 7.77 in the look-ahead demonstration above is the archetype: it announced itself.

Mistake: debugging by reading. People try to catch look-ahead by reading their code carefully. It does not work, because the leak lives in a join's default, a library's center=True, or a helper written six months ago. Experts test the property (invariance to the future) and let the machine find it, which is why no_lookahead exists and why it is the highest-XP exercise in the pack.

Mistake: "I only tried a few parameters." You tried more than you remember, and so did whoever handed you the idea. Experts keep a research log with one line per configuration, count trials honestly, and deflate. The count is not a formality: it is the difference between a 2.23 Sharpe meaning something and meaning 0.746 probability of being real.

Mistake: perfecting the signal before pricing the trading. Weeks on feature engineering, an afternoon on costs. Experts compute c* = alpha / turnover in the first hour: if the break-even cost is below what they know trading costs, the idea is dead and no amount of feature work revives it. Kill it cheaply and early, the cost model is the fastest filter you own.

Mistake: confusing "no bug" with "no bias." A perfectly correct backtest of a survivorship-biased universe is a perfectly correct wrong answer. Experts audit the data layer separately from the code, and treat "where did this universe list come from and when was it built" as a first-class question.

Mistake: reporting one number. A point estimate with no error bar, no capacity, no cost sensitivity and no trial count is not a result. Experts report a small table: gross and net, at 1× and 2× costs, with turnover, break-even cost, capacity, trial count and deflated Sharpe. It takes four extra lines and it is the difference between research and marketing.

Mistake: the round-trip factor of two. Charging a one-way cost on a flip from long to short. It halves every cost in the backtest and is invisible in the output. Experts assert Σ|Δposition| in a test.

Mistake: iterating on the holdout. "It failed out-of-sample, so I adjusted it and re-ran." That data is now in-sample and you have no holdout. Experts treat the holdout as consumable (once used, gone) and either accept the verdict or go and get new data.

How experts think differently, in five sentences. They ask what would this look like if it were wrong before asking whether it is right, and they build the test that would show it. They price every assumption in currency rather than debating it in prose, because ₹1.50 crore ends an argument that "survivorship might matter a bit" does not. They separate the four assumption classes (data, timing, execution, selection) and audit each on its own, because a fix in one does nothing for the others. They keep the vectorised and event-driven implementations in agreement as a standing invariant, because that four-line test finds bugs neither implementation would confess to. And they treat their own enthusiasm as the leading indicator of a leak: the better a new backtest looks, the harder they look for the mistake, because in this field good news is usually a bug and the bug is usually theirs.

Worked examples

Every figure below is reproducible: stated seed, stated parameters, stdlib Python, no external data (R3). Currency is marked on each; India (₹) and US ($) are both represented, as are two market-neutral mechanics examples.

Worked example 1: The one-index difference, by hand (mechanics)

Setup. Closes [100, 110, 99, 108.9, 103.455]. A signal [1, 1, −1, 1, 0], each element decided at that bar's close. Compute terminal wealth with lag=1 and with lag=0.

Work. Bar returns are +10%, −10%, +10%, −5%.

With lag=1, the position over bar t is signal[t−1]: 1, 1, −1, 1 against +10%, −10%, +10%, −5% gives +10%, −10%, −10%, −5%, so terminal = 1.10 × 0.90 × 0.90 × 0.95 = 0.84645×.

With lag=0, the position over bar t is signal[t]: 1, −1, 1, 0 against the same returns gives +10%, +10%, +10%, 0%, so terminal = 1.10³ = 1.331×.

Read it. The leak converts a 15.4% loss into a 33.1% gain on four bars. It is not that the leaky version is better at trading; it is that on bar 2 it held a short because it had already seen the price fall. The whole of this node's first pitfall is that one substitution, and QD1.03-e01 grades exactly it.

Worked example 2: Pricing the missing shift over three years (US, $)

Setup. Seed 20240301, 756 bars (three years), start $184.00, daily drift 0.035%, daily volatility 1.35%. Two rules: FAST (long when the close is above its 10-bar mean) and SLOW (10-bar against 50-bar crossover). Run each with lag=0 and lag=1.

Work.

Buy-and-hold over the same path: 1.1406×, 4.49% CAGR, Sharpe 0.31.

Read it. Three things. (a) The phantom wealth is $96,704,280 on the fast rule: 130× the honest result. (b) The leak's value scales with trading frequency: 47 decisions a year buys 47 days of foresight a year; 3 decisions a year buys 3. (c) The honest fast rule loses money with a 60% drawdown, which is the ordinary fate of a hyperactive rule on a near-random series and is exactly what an honest backtest should show. The reassuring outcome is the disappointing one.

Worked example 3: The publication-lag join (US, $)

Setup. Seed 606, 1,008 bars (four years), 15 quarterly announcements, reporting lag 45 bars. Each announcement moves price by 3.0% × the standardised surprise on the day it is published, plus a 5.5 bp/day drift in the same direction for 40 days afterwards. Strategy: go long on a positive surprise, short on a negative one. Version A stamps each surprise with the period end; version B stamps it with the publication date.

Work.

Gap: 0.59 Sharpe, 16.37 percentage points a year, 1.65× terminal ratio.

Read it. Version A is not a different strategy. It is the same strategy with a different merge key, and the entire difference is 45 days of date arithmetic on 15 rows. Note that version B is still a perfectly good strategy, the post-announcement drift is real in this synthetic world and it captures it. The leak did not create the edge; it inflated it by 65%, which is far more dangerous than creating it, because an inflated real edge survives every sanity check you would apply to a fake one.

Worked example 4: Survivorship on a ten-year Indian panel (India, ₹)

Setup. Seed 515151. 60 names, 120 monthly periods (10 years), a common market factor, 12 names (20%) delisting at months spread from 31 to 99, each taking −80% in its final month. Equal-weight monthly rebalance. Version A uses point-in-time membership; version B uses only the 48 names still listed at the end.

Work.

Gap: 4.14 percentage points a year, terminal ratio 1.432×, phantom wealth ₹1,50,31,496.

Read it. Every statistic moved the flattering way at once (return up, drawdown down, Sharpe up) which is the fingerprint of a data problem rather than a modelling one. A single modelling error usually improves one statistic and worsens another; a biased universe improves all of them, because it deleted the same events from every calculation. When your whole results table improves together, audit the data, not the model.

Worked example 5: The best of 1,000 zero-edge strategies, and its deflated Sharpe (US, $)

Setup. Seed 90210, 1,260 bars, zero drift: the true edge is exactly zero by construction. 1,000 random ±1 strategies, each flipping with probability 0.20 per bar, independent of price. 755 in-sample bars, 504 out-of-sample.

Work.

• In-sample Sharpe distribution: mean 0.002, standard deviation 0.567, max 2.23, min −1.55.
• The winner's out-of-sample Sharpe: −0.12. The top ten in-sample averaged 1.65 in-sample and 0.01 out-of-sample.
• Theory: E[max] of 1,000 draws is z = 3.255 standard deviations, so the expected winner is 0.002 + 0.567 × 3.255 = 1.85 against the observed 2.23.
• Deflation: SR₀ = 1.847 annualised (0.1163/day), SR̂ = 2.231 annualised (0.1405/day), T = 755. DSR = Φ( (0.1405 − 0.1163)·√754 / √(1 + 0.1405²/2) ) = 0.746.
• 4.4% of the 1,000 zero-edge strategies posted an in-sample Sharpe above 1.0; 0.6% above 1.5.

Read it. A 2.23 Sharpe from a provably zero edge, and the extreme-value formula predicted it within 0.4. The deflated Sharpe of 0.746 says: given that this is the best of a thousand, the probability its true Sharpe is positive is about three in four, well below any bar you would act on. And the calibration figures are the ones to keep: at three years of daily data, one zero-edge strategy in twenty-three posts a Sharpe above 1.0, and one in 167 posts a Sharpe above 1.5. Choose among two dozen variants and you should expect to find one above 1.0 whether or not anything is there, which is why the number of variants, not the Sharpe of the winner, is the statistic that decides what the result means.

Worked example 6: The fill ladder on an Indian mid-cap (India, ₹)

Setup. Seed 1975, 504 bars (two years), start ₹1,450, with genuine positive return autocorrelation injected so that a gross edge really exists. Rule: long above the 5-bar mean, short below: 113 position changes in 503 bars. Costs: half-spread 3.0 bps; brokerage, exchange charges, STT, stamp duty and GST bundled at 12.0 bps round trip; impact 14·√0.06 = 3.43 bps at 6% participation.

Work.

On ₹1,00,00,000: ₹29.56 crore → ₹1.79 crore, phantom ₹27.77 crore, a 1,547% overstatement.

Read it. The free rungs destroy 92.0% of the claimed wealth; the entire cost stack destroys a further 24.2%. The fill assumption was roughly four times more expensive than every explicit cost combined, and it is the assumption that never appears in a results table. Note also what survives: rung 5 is still a 34% CAGR at a 1.36 Sharpe, because the injected autocorrelation is a genuine edge. Honest accounting did not destroy this strategy. It destroyed the claim, which was 13× too large.

Worked example 7: Break-even cost and the 2× stress (US, $)

Setup. Two candidate strategies with the same gross alpha of 9.0 bps/day and an assumed one-way cost of 6.0 bps. Strategy A turns over 25% of NAV a day; strategy B turns over 200%.

Work.

Read it. Identical signals, opposite verdicts, and you can reach both conclusions in under a minute with a calculator and no backtest at all. That is the point of the identity: c* = alpha / turnover is the cheapest triage in quantitative research. A break-even of 36 bps against a 6 bps assumption means the conclusion does not depend on the cost model; a break-even of 4.5 bps means the "strategy" is a statement about your cost model. Run this before you write the backtest, not after.

Worked example 8: Capacity on an Indian mid-cap basket (India, ₹)

Setup. 25 names, median daily traded value ₹40 crore each, fixed cost 6 bps one-way, impact k = 14 bps per unit √participation, self-imposed participation cap 10% of daily volume, gross alpha 9 bps/day. Two books: 20%/day and 100%/day turnover.

Work. Liquidity ceiling = cap × ADV × n / turnover; impact ceiling from √p* = (alpha/turnover − fixed)/k.

At ₹100 crore the fast book participates at 10.00% of volume, pays 4.43 bps of impact, and nets −1.43 bps/day = −3.60% a year.

Read it. Same alpha, capacities eleven times apart, and the fast book's binding constraint is the one you would not have checked, it is impact-dead at ₹46 crore while still inside its 10% liquidity limit up to ₹100 crore. The slow book's p* of 7.76 is a useful absurdity: it means impact would only kill it at 776% participation, which the liquidity cap forbids long before, so for that book capacity is a policy choice and for the fast book it is a physics one.

Worked example 9: Where the vector stops being enough (US, $)

Setup. Seed 20240301, the first 300 bars, a signal that alternates every 20 bars. Compare a vectorised backtest with an event loop, first frictionless, then with a 5% stop-loss filled at the trigger price.

Work.

Read it. Identical to machine precision (not "close", identical) which is exactly the assertion QD1.03-e05's hidden tests make at 1e-12. Then the stop opens a 1.34 percentage-point gap that the vector cannot reproduce at any effort, because the information required (did the low pierce the trigger before the close?) is not in a series of closes. Use this as a working rule: vectorise until a rule needs to know what happened inside the bar, then switch, and keep the frictionless equality as a permanent regression test on the switch. Note the remaining optimism even in the event-driven version, filling exactly at the trigger assumes price traded there, which a gap denies.

{
  "skill": "Using a copilot to build backtest scaffolding fast, without letting it write the parts that decide whether the answer is true",
  "use": "Once you can write the event loop by hand and your no_lookahead and cost_sensitivity harnesses pass, delegate the SCAFFOLDING — parameter sweeps, result tables, equity-curve plots, walk-forward fold bookkeeping, a results dataclass, argument parsing — and keep the four decision-bearing pieces for yourself: the alignment, the fill rule, the cost model and the trial count. Generate in your own tool, paste it back here, and let this node's hidden tests and your own harnesses grade it.",
  "tools": ["Chat assistants — Claude, ChatGPT, Gemini", "IDE copilots — GitHub Copilot, Cursor", "Your own four harnesses, which are the only participants in this list that can produce evidence"],
  "prompt": "Write a parameter-sweep harness around my existing backtest function. Requirements: (1) it must call MY backtest function unchanged — do not reimplement or 'improve' the alignment, the fill rule or the cost model; (2) it must record every configuration tried in a log, including the ones I discard, so I can count trials honestly; (3) it must report, per configuration, gross and net terminal wealth, Sharpe, turnover, and break-even cost alpha/turnover; (4) it must be deterministic given a seed and must not use wall-clock time or network access. Then, separately: tell me how many EFFECTIVE independent trials you think this grid represents, and why that number is smaller than the number of cells.",
  "verify": "Run the generated sweep twice and byte-compare the output — non-determinism here is a bug that will silently invalidate every later comparison. Re-read the cost line for the round-trip factor of two (a flip from +1 to -1 trades TWO units) and the alignment for a lost shift; both are common in generated code and neither raises an error. Check every pandas method the model used actually exists with those arguments, and re-derive one reported figure by hand before trusting the table (Primary-Source Guardrail: Trace, Recompute).",
  "diy": "Gated: you must have passed QD1.03-e01, e02, e04 and e09 by hand first. If you cannot state where the shift goes and why a flip costs two units, you cannot review the code that does it — and this is the exact code where a plausible-looking mistake is worth crores.",
  "market": "IN"
}

Practice set

Work each problem fully before reading its solution. Numeric answers are to about ±2%. Every figure traces to a seeded computation stated in this node or to the arithmetic in front of you.

P1 (guided). Price the missing shift by hand. Closes [200, 220, 198, 217.8], signal [1, −1, 1, 1] where signal[t] is decided at bar t's close. Give terminal wealth at lag=1 and lag=0, and say in one sentence why the leaky version wins.

Solution. Bar returns: 220/200 − 1 = +10%, 198/220 − 1 = −10%, 217.8/198 − 1 = +10%. lag=1 uses signal[0], signal[1], signal[2] = 1, −1, 1 → +10%, +10%, +10% → 1.1³ = 1.331×. lag=0 uses signal[1], signal[2], signal[3] = −1, 1, 1 → −10%, −10%, +10% → 0.9 × 0.9 × 1.1 = 0.891×. Here the honest version wins, because this signal happens to be right one bar late, which is the useful half of the lesson: the leak is not always favourable on a short sample. Its bias is favourable on average and grows with sample length and trade frequency, and the only reason to look at a four-bar case is to see the mechanism. Never conclude from a short sample that alignment "doesn't matter here".

P2 (guided). Turnover, and the factor of two. A long/short book holds the position path [0, 1, 1, −1, −1, 0] over 6 daily bars. Give total units traded, annualised turnover at 252 bars/year, and the daily cost drag at 8 bps one-way.

Solution. Σ|Δpos| = |1−0| + |1−1| + |−1−1| + |−1−(−1)| + |0−(−1)| = 1 + 0 + 2 + 0 + 1 = 4 units. The flip contributes 2, and treating it as 1 is the classic error. Elapsed time is 5 bars = 5/252 years, so annualised turnover = 4 × 252/5 = 201.6× NAV per year, i.e. about 80% of NAV per day. Daily cost drag = 0.80 × 8 = 6.4 bps/day ≈ 16.1%/year, enough to bury most signals, which is the point.

P3. Break-even triage. A strategy shows a gross alpha of 6.0 bps/day at 50% daily turnover. Your cost estimate is 8 bps one-way. Give the cost drag, the net in bps/day and annualised, the break-even cost, and the verdict at 2× costs.

Solution. Drag = 0.50 × 8 = 4.0 bps/day; net = 6.0 − 4.0 = 2.0 bps/day = 5.04%/year (2 × 252/100). Break-even c* = alpha/turnover = 6.0/0.50 = 12.0 bps, only a 1.5× margin over the 8 bps assumed, so the conclusion is the cost assumption. At 2× costs: 6.0 − 0.50 × 16 = −2.0 bps/day = −5.04%/year, it dies. survives_2x = False, and this is a strategy you do not fund without a real, measured cost study.

P4. Which architecture? For each, say whether a vectorised backtest is sufficient and why: (a) monthly rebalanced value factor on 500 names; (b) daily moving-average crossover with a 3% trailing stop; (c) pairs trade entering at 2σ and exiting at 0σ, close-to-close; (d) intraday VWAP execution of a daily target.

Solution. (a) Vectorised: no path dependence, no intrabar rules, state is a function of the signal alone. (b) Event-driven, the trailing stop needs the intrabar path and carries state (the running extreme since entry). (c) Vectorised is sufficient if entries and exits are evaluated on closes only, because the position is a deterministic function of the z-score series; it becomes event-driven the moment you add a stop, an intrabar trigger, or a capital constraint that couples the legs. (d) Event-driven, and finer than daily, you are simulating an intraday schedule, so bars must be intraday and the fill model is the whole exercise (QD2.01).

P5. The invariance property, stated precisely. In one sentence, state the property no_lookahead tests, and give one leak it cannot detect and one reason it might flag an honest strategy.

Solution. A strategy that uses only information at or before bar t produces the same decision for bar t under any modification of the data after bar t. It cannot detect leaks that do not flow through the perturbed array, a restated fundamental read from a file, a universe list built with hindsight, or a hard-coded constant that was itself fitted on the full sample. It can flag an honest strategy if the strategy is non-deterministic (an unseeded random draw makes two calls differ) or if it reads global mutable state that the harness's repeated calls disturb; both are real defects worth fixing anyway, so a false positive here is still a useful signal.

P6. Design the probes. Your no_lookahead uses only "multiply the future by 0.5". Give a leaking strategy it will miss, and the probe that catches it.

Solution. Any strategy that is scale-invariant in the future: for example 1.0 if mean(future returns) > 0 else −1.0, or 1.0 if prices[-1] / prices[t+1] > 1 else −1.0. Multiplying every future price by a constant leaves all future returns unchanged, so the decision does not move and the leak passes. Reversing the future flips the sign of the cumulative future return and catches it; flattening the future to prices[t] sets every future return to zero and catches it too. This is exactly why QD1.03-e02 specifies four probes rather than one, and why its hidden tests include a scale-invariant peeker.

P7. Read a flag pattern. no_lookahead returns [0, 5, 10, 15, 20, 25] on a 30-bar series. What does the pattern tell you, and where do you look first?

Solution. The leak occurs on every fifth bar and nowhere else, so it is not in the core signal logic (which would leak everywhere or nowhere) but in something that fires on a five-bar cycle: a weekly resample, a five-bar rebalance step, a .resample('W').last() that labels a bucket with its end rather than its start, or a group-wise transform applied to five-bar blocks. Localisation is the harness's most useful output, the flag list is a diagnosis, not just an alarm, and a periodic pattern points straight at the resampling or rebalancing code.

P8. Survivorship arithmetic. A universe of 200 names loses 3% of its members a year to delisting, and each delisted name takes a −70% return in its final period. Roughly how many names delist over 10 years, and in which direction does deleting them move (a) mean return, (b) volatility, (c) maximum drawdown, (d) Sharpe?

Solution. Survival is 0.97¹⁰ = 0.737, so about 26% of the universe delists, roughly 53 names. Deleting them removes the left tail of the cross-section, so: (a) mean return up; (b) volatility down (the −70% observations are the large ones); (c) maximum drawdown down; (d) Sharpe up, twice over: numerator up and denominator down. Everything improves together, which is the fingerprint of a data problem, and it is why the ten-year Indian panel above shows a lower drawdown for the biased version (24.2% against 25.7%).

P9. Expected maximum. You test 100 parameter sets and the dispersion of Sharpe estimates across them is 0.5. What Sharpe should the winner post if none of them has any edge? What if you had tested 1,000?

Solution. E[max] = σ · z(N) with z(100) = 2.5306 → 0.5 × 2.5306 = 1.27. At N = 1,000, z = 3.2551 → 1.63. Two readings: a 1.27 Sharpe from the best of 100 is exactly what nothing looks like; and the bar grows very slowly in N, a tenfold increase in search bought only 0.36 of extra Sharpe, because the normal tail is thin. That slowness is what makes the deflation practical: you do not need to count trials precisely, only to an order of magnitude.

P10. Deflate it. A strategy shows an annualised Sharpe of 1.60 over 1,008 daily observations. You tested 100 configurations; their Sharpe dispersion was 0.5 annualised. Compute the bar and the deflated Sharpe, and give the verdict at a 95% threshold.

Solution. Bar: SR₀ = 0.5 × 2.5306 = 1.265 annualised, i.e. 1.265/√252 = 0.0797 per day. Observed: 1.60/√252 = 0.1008 per day. Numerator = (0.1008 − 0.0797) × √1007 = 0.0211 × 31.73 = 0.670. Denominator = √(1 + 0.1008²/2) = 1.00254. DSR = Φ(0.668) = 0.748: fails a 95% bar. Note how close this is to the 0.746 of the worked-example winner: a 1.60 Sharpe from 100 trials and a 2.23 Sharpe from 1,000 carry almost identical evidential weight, which is the whole reason to compute the statistic rather than eyeball the Sharpe.

P11. Effective trials. A grid of 400 cells produces a best in-sample Sharpe 2.0 standard deviations above the grid mean. Estimate the effective number of independent trials, and say what it implies.

Solution. Invert the expected-maximum relation: find N such that z(N) ≈ 2.0. Since z(20) = 1.90 and z(50) = 2.28, interpolating gives N_eff ≈ 25. So 400 correlated cells were worth about 25 independent bets, a 16-fold discount, because neighbouring grid cells are near-duplicates of each other. Practical implication: deflating against N = 400 would be far too harsh and against N = 1 far too lenient; N_eff is a defensible middle, and reporting how you obtained it is more important than its exact value.

P12. Purge and embargo, by hand. With n = 20, a test block of bars 10–14 (inclusive), a label horizon of 3 bars and an embargo of 2, list the training indices and count them.

Solution. Test = [10, 11, 12, 13, 14]. Before the block, index i is purged when i + 3 ≥ 10, i.e. i ≥ 7, so 7, 8, 9 are purged. After the block, indices 15 and 16 fall in the embargo and are dropped. Training = [0, 1, 2, 3, 4, 5, 6, 17, 18, 19]: 10 indices out of 20. Note the asymmetry: indices after the block are not purged for label overlap, only embargoed, because a label reaches forward in time, not backward. Getting that asymmetry right is exactly what QD1.03-e10 grades.

P13. Capacity, both ceilings. A US small-cap book: 40 names, $5,000,000 average daily traded value each, 50% daily turnover, gross alpha 8 bps/day, fixed cost 2.5 bps one-way, impact k = 10, participation cap 5%. Give both ceilings and the binding one.

Solution. Liquidity = 0.05 × 5,000,000 × 40 / 0.50 = $20,000,000. Impact: alpha/turnover = 8/0.5 = 16; √p* = (16 − 2.5)/10 = 1.35; p* = 1.8225; capacity = 1.8225 × 5,000,000 × 40 / 0.50 = $729,000,000. Liquidity binds at $20 million. As with the slow Indian book, a p* far above 1 means impact is irrelevant at any size the participation cap permits, for this book, capacity is a policy number you could relax by raising the cap, not a physical limit.

P14. Diagnose a results table. A backtest reports: Sharpe 6.2, max drawdown 3%, 94% of days positive, turnover 300%/day, costs "assumed negligible". List the three most likely defects in priority order and the test for each.

Solution. (1) Look-ahead: a Sharpe of 6.2 with a 3% drawdown on daily data is physically implausible; run no_lookahead and inspect the alignment. (2) Costs, at 300% daily turnover, even 2 bps one-way is 6 bps/day ≈ 15%/year of drag, so "negligible" is false by construction; compute c* = alpha/turnover and run cost_sensitivity. (3) Fill assumption, a 94% hit rate at that turnover suggests filling at favourable intrabar prices; re-run filling at the next bar's open and compare. The ordering matters: fixing costs on a leaking backtest wastes the afternoon, because the leak is worth more than the costs.

P15. Why does the frictionless equality matter? Your event-driven engine returns 1.3319 where the vectorised version returns 1.3316 in the frictionless case. Is this acceptable? What are the three most likely causes?

Solution. No. With no costs and no path rules the two are algebraically the same product of the same factors and must agree to floating-point noise (< 1e-12); a difference in the fourth decimal is three orders of magnitude too large. Likely causes, in order: (1) an off-by-one in the position index, the loop applies target[t−1] where the vector applies target[t], or vice versa; (2) the loop starts at bar 0 and books a spurious first return, or stops one bar early; (3) the loop marks against the wrong reference price (previous open rather than previous close). This four-line assertion is the cheapest bug detector in the node precisely because the vectorised version is too short to hide anything.

P16. Design the experiment. You believe a signal works. Write the six-step protocol you will follow, in order, before you would put money behind it, and name what would stop you at each step.

Solution. (1) Triage on arithmetic: estimate gross alpha and turnover, compute c* = alpha/turnover, and stop if c* is not several times your real cost. (2) Build the data layer point-in-time: membership as (name, start, end), fundamentals stamped at publication, and stop if you cannot establish provenance. (3) Backtest with the honest alignment and a real cost stack, and stop if net is negative. (4) Run the harnesses: no_lookahead clean, cost_sensitivity surviving 2×, determinism byte-identical, and stop on any failure. (5) Validate the selection: walk-forward or purged CV, count trials honestly, deflate, and stop if the deflated Sharpe is below your threshold. (6) Size it: compute both capacity ceilings and stop if the binding capacity is below what the effort is worth. The order is deliberate, each step is cheaper than the one after it, so the protocol is designed to kill ideas as early as possible, which is the only way to test many of them without deceiving yourself about all of them.

Applied mini-project

Build a backtesting engine, then prove it lies to you, and by how much.

This project is the node. You will build one engine, run one strategy through it five times under progressively honest assumptions, and produce a report whose most valuable section is the one about what is wrong with it.

Data. Use the bundled datasets (us-eod, nse-eod for prices; nifty-index or us-index for a benchmark) or your own seeded synthetic panel. You must use one Indian and one US instrument or basket so that both cost regimes appear (R6). If you generate synthetic data, state the seed and the generating process; if you use the bundled sets, read their provenance note in MANIFEST.json and quote it, every one of them is a labelled synthetic look-alike, and saying so is part of the exercise.

Part 1, The engine. Build both, and keep them honest against each other.

1. A vectorised backtester: signal → shift → position → returns → equity, with a cost charge on Σ|Δposition|.
2. An event-driven backtester: a bar loop carrying position, entry price, cash and a fill model, supporting at least one intrabar rule (a stop-loss).
3. A test asserting they agree to 1e-12 in the frictionless, no-stop case. This test is a deliverable, not a courtesy.

Part 2, The ladder. Take one simple signal (a moving-average rule is ideal, precisely because it is not clever) and run it five times, each rung adding one honesty:

1. Signal applied to the same bar it was computed on, filled at the best intrabar price, no costs. (The fantasy.)
2. Signal shifted one bar, filled at the next bar's open, no costs.
3. Plus half-spread.
4. Plus the full fee and statutory stack, use real published rates for both markets and cite the schedule and the date you read it.
5. Plus impact k·√participation at a stated assumed size.

Report terminal wealth, CAGR, Sharpe and maximum drawdown at each rung, the step cost of each rung as a percentage of the previous rung's terminal wealth, and the total overstatement from rung 1 to rung 5 in both percentage and currency terms at a stated starting capital.

Part 3, The four harnesses. Implement and run all four, and paste their output verbatim.

• no_lookahead, on your own strategy function. Then deliberately break your own code (remove the shift, or normalise on the full sample) and show the harness catching it, with the flagged bar indices. A harness that has never caught anything is not evidence.
• cost_sensitivity, terminal wealth at 1×, 2× and 3× your assumed cost, plus the break-even cost in bps and how it compares with your assumption.
• determinism, two full runs, byte-compared, with the seed and the input-file hash recorded.
• oos_split, a walk-forward or a genuinely untouched holdout, with in-sample and out-of-sample results side by side.

Part 4: Selection honesty. Report how many configurations you tested (all of them, including the ones you discarded), the dispersion of their Sharpe estimates, the expected maximum under the null, your estimate of the effective number of independent trials, and the deflated Sharpe of your chosen configuration.

Part 5: Capacity. Both ceilings, which binds, the capacity in ₹ and $, and the net alpha at three AUM levels spanning it.

Part 6, The limitations memo (the section that is actually graded hardest). One page, in prose, naming: every assumption you made that could be wrong and in which direction it would move the result; what you could not test with the data you had; the single result you least believe and why; and what evidence would change your mind. "I found no problems" scores zero. There are always problems; the skill is knowing which ones matter.

Deliverable. One repository plus one report of at most four pages. The repository runs end to end from a single command, with the seed and configuration in a file rather than in the code. The report carries the ladder table, the four harness outputs, the selection-honesty numbers, the capacity table and the limitations memo.

Rubric (100 points; pass at ≥ 85 with no zero on any line, this is a crown node).

Self-check before you submit. Can you answer, in under two minutes and without opening the code: which single assumption, if wrong, would most change your conclusion; what your break-even cost is and how it compares with the cost you assumed; how many things you tried; and at what AUM the strategy stops working? If any of those four takes longer than thirty seconds, the report is not finished, because those are the four questions an allocator, a risk officer or a hiring manager will ask, in that order.

Reading & resources

Everything below is an anchor to learn from and verify against, never a source to quote a number from without re-deriving it (R3). Tags: ✓ free / $ paid, and ▲ beginner–intermediate / ▼ advanced.

The core spine, read one properly before adding a second:

• Marcos López de Prado, Advances in Financial Machine Learning, the standard modern treatment of exactly this node's second half: purged and embargoed cross-validation, the deflated Sharpe ratio, backtest overfitting, and the "backtesting is not a research tool" argument. Chapters 7, 11 and 12 are the ones to read first. [$] [▼]
• Ernest Chan, Quantitative Trading and Algorithmic Trading, the most practical introduction to building and pricing a backtest, with data-snooping, survivorship and transaction costs treated as first-class topics rather than appendices. [$] [▲]
• Robert Carver, Systematic Trading: the best available book on not overfitting, from a practitioner who argues for fewer parameters and simpler rules and shows why. [$] [▲]
• Bailey, Borwein, López de Prado & Zhu, "Pseudo-Mathematics and Financial Charlatanism" (Notices of the AMS, 2014), the minimum backtest length result and the clearest short statement of why the best of N is not an estimate of anything. Free PDF. [✓] [▼]
• Bailey & López de Prado, "The Deflated Sharpe Ratio" (Journal of Portfolio Management, 2014), the source of the DSR formula used above, including the skew and kurtosis correction. Working-paper version free. [✓/$] [▼]
• Harvey, Liu & Zhu, "…and the Cross-Section of Expected Returns" (Review of Financial Studies, 2016), the multiple-testing correction applied to the published factor zoo; the argument that a t-statistic of 2 is nowhere near enough. [✓ working paper] [▼]

Costs, impact and capacity:

• Almgren, Thum, Hauptmann & Li, "Direct Estimation of Equity Market Impact" (2005), the empirical basis for the square-root shape used in this node's impact term. [✓] [▼]
• Frazzini, Israel & Moskowitz, "Trading Costs of Asset Pricing Anomalies" (working paper), real institutional trading costs measured against real anomalies, and the capacity conclusions that follow. [✓] [▼]
• Your own broker's published schedule, and the exchange's, for India, the NSE/BSE transaction-charge pages plus the current STT, stamp-duty and GST rates; for the US, the SEC Section 31 fee rate and FINRA's TAF. Read the live pages and date your citation; these rates change, and a backtest carrying a stale rate is carrying a stale conclusion. [✓] [▲]

Engines to read rather than adopt:

• backtrader, vectorbt, zipline-reloaded, bt: read one engine's fill and cost model end to end and find where its defaults are optimistic. The exercise is not "which library should I use", it is "what does this library assume on my behalf, and would I have assumed it?" [✓] [▲–▼]
• pandas documentation on shift, rolling (especially the center argument), resample (especially label and closed), and merge_asof. merge_asof is the single most useful function in this node, it is the point-in-time join, and knowing it exists prevents mechanism 5 outright. [✓] [▲]

On the psychology, which is the actual failure mode:

• Campbell Harvey's talks and papers on backtesting standards and research protocol, the case for pre-registration in quantitative finance. [✓] [▼]
• Andrew Lo, Adaptive Markets: why edges decay, and why a strategy that worked for a decade stopping is the normal case rather than evidence of a bug. [$] [▲]

The finance bridge, and the only resource that answers back:

• Your own QD1.03 code pack: ten exercises whose hidden tests run a ₹ and a $ fixture, assert structural invariants as well as values, and fail the plausible wrong answer rather than only the lazy one. Exercise QD1.03-e02 in particular is a resource you will still be running years from now.

Do this, not just read: take any backtest you have already written, or the first one you write after this node, and run no_lookahead on it before you look at its returns. Then delete the shift on purpose and run it again. Watching your own harness light up on your own code is worth more than any chapter above, because it converts "I should be careful about look-ahead" into "I have a test for that."

{
  "skill": "Turning a copilot into a red team for your own backtest, instead of a co-author of it",
  "use": "After your report is written but BEFORE you show it to anyone, paste the assumptions section (not the code) and ask the model to argue that your result is spurious — to name the mechanism, predict the symptom, and specify the test. Then run the tests it names. This is the highest-value use of a model in this node, because the failure mode you cannot see is the one you have already rationalised, and a model has no stake in your idea.",
  "tools": ["Chat assistants — Claude, ChatGPT, Gemini", "Spaced-repetition tooling for this node's flashcard deck", "This node's code pack (QD1.03) as the grader — the app never calls a model at runtime"],
  "prompt": "Here are my backtest's assumptions: universe and how it was built, date range, signal, alignment rule, fill rule, cost model, number of configurations tested, and the headline result. Do NOT write code and do NOT tell me it looks reasonable. (1) Argue that this result is spurious: give the three most likely mechanisms, ranked, with the symptom each would produce in my reported statistics. (2) For each, name the single test that would settle it. (3) Tell me which number in my report an experienced allocator would attack first, and what they would ask next. (4) Write the paragraph of my limitations memo that I am most likely to have avoided writing.",
  "verify": "Treat every claim as a hypothesis to test, never a finding: run the tests it names and report what actually happened. Re-derive any statistic it quotes — models routinely confuse annualised and per-observation Sharpes in exactly the deflated-Sharpe formula this node uses, and produce a plausible number that is wrong by a factor of sixteen. If it proposes a library function to settle something, confirm the function exists with those arguments before believing the answer (Primary-Source Guardrail: Trace, Recompute).",
  "diy": "Gated: run your own four harnesses FIRST and write your own limitations paragraph before asking. A red team you cannot evaluate is just a second opinion, and on this material a confident second opinion is exactly what you do not need.",
  "market": "US"
}

Teach it back & journal

Feynman prompt. Explain to a capable colleague who builds models in Excel and has never written a backtest why the sentence "my strategy returned 340% a year in testing" is not yet a claim about anything. Use exactly three ideas and no jargon: that a backtest is a simulation whose every convenience points the same way; that the single most expensive mistake is using a number one day before it existed; and that the cost of trading is not a detail but frequently the entire result. Then make it concrete with one arithmetic example they can check, the four-bar toy where the same data and the same rule give 1.331× or 0.84645× depending on one index, and one sentence about why you now test your own code by changing the future and watching what moves. If they ask, unprompted, "so how many things did you try before you got that number?", you have transferred the most important idea in this node.

Journal prompt. Take one backtest you have actually run, or the first one you run after this node, and audit it against the four assumption classes, in writing. Data: where did the universe come from, when was that list built, and would it contain a company that delisted in year two? Timing: for every input, what is the difference between the timestamp you used and the moment a human could have acted on it, and can you name the join that resolves it? Execution: which bar did you fill on, what did you assume it cost, and what is your break-even cost alpha / turnover compared with that assumption? Selection: how many configurations did you try, including the ones you deleted, and what does the expected-maximum formula say the best of that many is worth under the null?

Then write three paragraphs. First, run no_lookahead on the strategy and record what it said, and, if it said nothing, deliberately break the alignment and record that it then said something, because a harness you have never seen fire is a harness you do not yet trust. Second, name the single assumption in the whole exercise that would most change your conclusion if it were wrong, state which direction it would move the result, and say what evidence would settle it. Third, answer honestly: when the numbers came out well, what was your first instinct, to check it, or to extend it? Over the next month, which habit will do more for how colleagues read your work: producing more backtests, or being the person in the room who can say "here is the number, here is what would have to be true for it to be real, and here is the test I ran"? Name the one harness from this node you will run on everything from now on, and the specific piece of work you will run it on first.

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.