Building a Backtesting Engine — Design & the Classic Pitfalls

Learning objectives

By the end you can:

1. State what a backtest is and what it is not, a simulation of a decision rule against a historical data-generating process, whose output is a distribution of possible histories and whose single realised path is one draw from it, and name the four assumption classes (data, timing, execution, selection) that every backtest silently makes.
2. Choose between vectorised and event-driven architectures on the merits, prove they are numerically identical in the frictionless case, and name precisely what breaks that equality: path dependence (stops, targets, margin), state (position, cash, inventory), and any rule that reads inside a bar.
3. Build an event loop with correct state discipline, the decide on t−1, act on t rule, the single ordering that makes look-ahead structurally impossible rather than merely avoided.
4. Detect look-ahead bias mechanically, by implementing and running the no_lookahead future-perturbation harness, and explain why invariance-to-the-future is a complete test for the class of leaks it covers and what it cannot see.
5. Name and quantify the six mechanisms of look-ahead, the missing shift, the full-sample normalisation, the centred window, the restated or revised data point, the publication-lag leak, and the survivorship-selected universe, and price each in terminal wealth on a worked case.
6. Build a point-in-time universe from listing and delisting records, explain why "the current index constituents" is the most expensive free dataset in finance, and measure the bias it induces.
7. Treat the best of N backtests as an order statistic. Compute the expected maximum Sharpe of N mean-zero trials, deflate an observed Sharpe against that bar, estimate the effective number of independent trials behind a correlated grid, and state why the deflated Sharpe is a floor on your scepticism rather than a certificate.
8. Model execution honestly: the fill-price ladder from fantasy to defensible, the half-spread you cross, the fee and statutory stack in both India and the US, and market impact as k·√participation, and derive the net-alpha identity net = alpha − turnover × cost and the break-even cost c* = alpha / turnover it implies.
9. Size capacity two ways (the participation ceiling and the impact-erosion ceiling) say which binds, and explain why an unrealistic fill assumption and an unstated capacity are the same error in different clothing.
10. Design a leak-free validation protocol: walk-forward selection, purged and embargoed cross-validation for overlapping labels, and the discipline of a genuinely untouched holdout.
11. Ship the four harnesses (no_lookahead, cost_sensitivity, determinism, oos_split) as executable gates on your own research, and read a backtest report the way a risk officer does: hunting for what is absent.
12. (Productivity objective: R10 duality.) Use a coding copilot on backtesting code the way a working quant developer does, after you can do it by hand, and review its output for the failure modes specific to this domain: silently vectorised look-ahead, cost models that charge one-way for a round trip, and confidently hallucinated pandas and vendor APIs.

The duality, stated once (R10). The gated skill is objectives 1–11. The gate is deliberately hostile to shortcuts: the pack's hidden tests check both a ₹ and a $ fixture, they assert structural properties (that the frictionless event loop equals the vectorised product to 1e-12; that a harness flags exactly the leaking bars and no others) as well as values, and several of them fail the most plausible wrong answer rather than only the lazy one. Objective 12 is the payoff you keep afterwards. The app never calls an AI at runtime, the callouts below ask you to generate something in your own tool, paste it back, and let the local hidden tests judge it. The machine drafts; the harness decides; you are the one who has to know why it failed.

Prerequisites & connections

Builds on. CS1.01 and CS1.02 for Python itself, CS1.04 for the cost of a loop, and CS1.05 for the structures an event-driven engine is made of, a backtester's event queue is a priority queue keyed on timestamp, and its order book is the tree-plus-queue from that node. DA1.02 and DA1.03 for pandas, because the vectorised backtest is a shift, a where and a cumprod, and because the single most dangerous pandas operation in this node, a join that silently reindexes: lives there. QD1.01 owns the data layer this node consumes: corporate actions, adjusted prices, and the construction of point-in-time, survivorship-controlled panels. QD1.02 owns signal and factor construction, the content of the signal that this node merely schedules and pays for.