Every VaR method produces a forecast: a claim about tomorrow, made today, before tomorrow's return is known. A forecast that is never checked against what actually happened is not risk management, it is a number on a slide. VaR backtesting is the discipline of checking, and this module treats it as a first-class graded skill, not an afterthought, for exactly the reason QD1.03 treated look-ahead defeat as first-class: the check is what makes the number trustworthy at all.
How do you count VaR exceptions?
The raw material every backtest consumes is a simple count: on how many days did the realized loss actually exceed the forecasted VaR? An exception (also called a breach) on day i occurs when realized_returns[i] < -var_forecasts[i], the day's actual loss was strictly worse than the VaR threshold predicted the day before. Get the boundary condition wrong here and every statistic built on top of the count inherits the error, however carefully those statistics themselves are coded: a loss landing exactly on the VaR threshold is, by the definition of VaR as a level not expected to be exceeded, not an exception, "exceeded" means strictly worse, not equal. Code pack exercise QD2.02-e04 grades exactly this counting rule, including a hidden test built specifically around a time-varying VaR forecast (not a constant one), which catches a solution that only ever compares against the first day's forecast.
Advertisement
The Kupiec proportion-of-failures test: turning a count into a verdict
A raw exception count on its own invites exactly the wrong kind of eyeballing: "4 breaches in 250 days, that's about 1.6%, close enough to our 1% target, ship it." Kupiec's proportion-of-failures (POF) test (Kupiec, 1995) replaces that eyeball with an actual hypothesis test, precisely the machinery QM1.02 built from first principles, applied here rather than re-derived.
Let p = 1 - confidence be the exception rate your VaR model claims (0.01 for a 99% VaR), T the number of observations, x the observed exception count, and p_hat = x / T the observed exception rate. The null hypothesis is "the model's claimed rate p is correct"; the test statistic is a likelihood-ratio statistic comparing how likely the observed data is under the claimed rate p versus under the observed rate p_hat:
LR = -2 * [ (T-x)ln(1-p) + x·ln(p) − (T-x)ln(1-p̂) − x·ln(p̂) ]Under the null, LR is asymptotically chi-square distributed with 1 degree of freedom (one estimated parameter, p_hat, compared against one hypothesised value, p, exactly the "degrees of freedom = parameters being tested" logic QM1.02 built up in general). Compare LR against the chi-square critical value at your chosen significance level (scipy.stats.chi2.ppf(sig_level, df=1); at the conventional 95% significance level, the critical value is ≈ 3.841) and reject the model's claimed calibration if LR exceeds it.
One implementation trap deserves its own paragraph because it is exactly the kind of "looks fine on the happy path, crashes or silently misbehaves on the edge case" defect CS2.06 trained you to hunt down, now appearing in a formula rather than a function you're reviewing. When x = 0 (the model was never breached) or x = T (it was breached every single day, a pathological but valid input), the formula above asks you to evaluate a count × log(probability) term whose probability can be exactly zero, a literal math.log(0) domain error, or worse, a silent 0 * -inf = nan that corrupts the whole statistic without ever raising. The fix is a small, explicit convention, standard in information theory and easy to get right once you know to look for it: a count * log(probability) term is defined as exactly 0.0 whenever the count is 0, regardless of what the paired probability is. Write that as a two-line helper and apply it to all four terms in the formula, and both edge cases resolve cleanly to a real number instead of a crash.
Worked example: when the Kupiec test rejects
Run the test on a hundred days of a dollar-denominated book's 95% VaR where exactly 10 exceptions occurred (5 more than the 5 the model's own 5% claimed rate would predict): LR = 4.131, against a critical value of 3.841: reject. Now compare that against a hypothetical run where exactly 5 of the 100 days breached, precisely matching the model's own claimed 5% rate: LR ≈ 0.000, as close to a perfect non-rejection as a real test can produce, because the observed rate and the claimed rate are identical. And here is the detail that should genuinely surprise you the first time you see it, and is worth sitting with rather than skimming past: run the same 250-day, 99%-VaR test at x = 0 exceptions (a model that was never once breached in a full year) and it also rejects (LR ≈ 5.025, comfortably above the 3.841 critical value). A model with zero breaches sounds, to ordinary intuition, like the safest possible outcome. To the Kupiec test it is equally strong evidence of miscalibration as too many breaches, because a properly-calibrated 99% VaR model is supposed to be breached about 1% of the time, a model that is never breached is most likely set far too conservatively (quietly costing the desk capital efficiency every single day) rather than genuinely, verifiably safe. "Too safe" is not free, and a backtest that only checks for too many exceptions and gives a free pass to zero has quietly stopped doing its job. This is exactly what code pack exercise QD2.02-e05 grades, the formula, the 0-count edge case handled without crashing, and the counterintuitive "too few is also rejected" behaviour, on both an India-flavoured and a US-flavoured backtest window.
Common mistake: numpy booleans in the test result
One more silent trap, purely about Python rather than statistics, and one this module takes as seriously as the formula itself: scipy.stats.chi2.ppf(...) returns a numpy scalar, and comparing it produces a numpy boolean, not a Python one. numpy.True_ is True evaluates to False (the two are not the same object) so a report field defined as "reject": lr > crit will silently fail an is True check anywhere downstream that relies on one (a dashboard, a JSON serializer with strict typing, another test). The fix is one explicit cast, bool(lr > crit), and the code pack's hidden tests check for exactly this (type(result["reject"]) is bool) because a risk report whose "did this model fail" field is secretly the wrong type of boolean is precisely the kind of defect that passes every casual glance and breaks the one time someone automates around it.