The Analyst's Path

Ledoit-Wolf shrinkage

Ledoit-Wolf shrinkage is a way of estimating a covariance matrix that pulls the noisy sample estimate toward a more structured target. It matters when the sample is short relative to the number of assets, because a raw estimate can make minimum-variance weights swing from one window to the next. Shrinkage trades that variance for some bias, so you report the stability gap and judge whether the trade is worth it.

6 min readPart of Portfolio Optimization in CodeOpen in course mode

Advertisement

The estimation-error maximizer section proved unconstrained MVO amplifies noise in μ. Here is a fact AA1.01 section 4 states but does not fully explore: GMV's formula never references μ at all, so is GMV actually immune to naive-optimizer blow-up? QD1.04-e08 answers this numerically, and the honest answer is no, not when the covariance matrix itself is poorly estimated. Pull five real tickers each from this program's bundled us-eod and nse-eod datasets (both seeded synthetic look-alikes, per this program's R3 discipline, see the data-honesty section) and take two different, non-overlapping 22-trading-day windows of daily returns for each market, a genuinely short sample relative to five assets. Estimate the covariance matrix the naive way, numpy.cov(returns, rowvar=False), and compute the GMV weights for each window: on the US menu, the two windows' GMV weights differ by as much as 10.3 percentage points on a single asset; on the India menu, by as much as 25.8 points, including a spurious short position appearing out of nowhere in one window purely from sampling noise. Now re-estimate the same two windows' covariance with Ledoit-Wolf shrinkage (sklearn.covariance.LedoitWolf, which shrinks the noisy sample estimate toward a more structured target) and recompute GMV: the US windows now differ by 0.0 points, the shrinkage is so strong at this sample size that GMV collapses to an exactly equal-weighted portfolio regardless of which window fed it, and the India windows differ by only 5.1 points, a large reduction from 25.8.

Read the honest nuance here, not just the win. Shrinkage traded variance (wild, sample-dependent weights) for bias (a covariance estimate pulled hard toward a generic structured target, potentially under-weighting real, persistent differences between assets), the classic bias-variance tradeoff, and precisely why "shrink harder" is not a free universal answer any more than "add more views" is free in Black-Litterman. The professional discipline is the same one AA1.01/AA1.02 teach throughout: run both estimators, report the stability gap, and let a human judge whether the reduction in noise is worth the increase in bias for this specific, real allocation decision, never accept either number, raw or shrunk, as self-evidently correct without having actually measured what changes when you re-estimate.

Advertisement

Michaud resampling, run rather than deferred

A third hardening technique attacks the same problem from a different angle. Michaud resampling (Michaud, 1989) takes the observation that your μ and Σ are estimates seriously enough to act on it. The procedure is four steps and each one is a few lines of numpy.

  1. Treat the point estimates μ and Σ as a data-generating process rather than as the truth.
  2. Draw a sample of T observations from that process, where T is the number of periods you actually have. This is the step that encodes how much evidence you own; a 60-month history and a 600-month history give very different answers.
  3. Re-estimate μ̂ and Σ̂ from that draw alone, re-run the same optimiser on them, and store the weight vector.
  4. Repeat many times and average the stored weight vectors. The average is the resampled portfolio.

Run it on this module's own Bharat Family Office menu, ten resamples of 60 monthly observations each, seeded so you can reproduce every line. The point estimate is Equity 24.51%, Debt 70.60%, Gold 12.26%, REIT −7.37%.

DrawEquityDebtGoldREIT
132.96%76.02%25.15%−34.14%
219.12%82.99%7.97%−10.07%
321.82%51.78%18.25%+8.15%
422.74%34.19%29.25%+13.82%
58.87%71.42%11.17%+8.54%
629.86%89.39%8.28%−27.53%
737.99%68.77%17.22%−23.97%
814.03%79.84%18.78%−12.64%
925.17%39.65%34.88%+0.31%
1024.82%65.85%12.87%−3.54%
Average23.74%65.99%18.38%−8.11%

Read the table twice, once down and once across. Down the REIT column, a single 60-month sample supports anything from a 34.14% short to a 13.82% long, and the standard deviation of that column across the ten draws is 16.48 percentage points. Debt ranges from 34.19% to 89.39%, a standard deviation of 18.48 points. Nothing here is a bug and nothing here is an unusual draw. This is what "the optimiser is reading an estimate, not a fact" looks like when you finally let yourself see more than one estimate.

Now read across the bottom row. The resampled average is Equity 23.74%, Debt 65.99%, Gold 18.38%, REIT −8.11%, against the point estimate's 24.51 / 70.60 / 12.26 / −7.37. The two are close. Gold rises by six points and Debt falls by five, and the REIT short survives essentially unchanged. That is the first honest finding, and it is not the one the technique's reputation promises: on this menu, averaging does not remove the short and does not produce a materially different portfolio.

What it does produce is stability. Any one of those ten rows is a portfolio some analyst would have handed a trustee with a straight face, and the ten rows disagree violently. The average is the portfolio you would have arrived at from any of them, which is exactly what you want when the input's sampling noise is the largest single thing moving the answer. Score them and the second finding follows: the resampled average earns a Sharpe ratio of 0.5712 under the true inputs, against the point estimate's 0.5780 and the individual draws' average of 0.5280. Resampling costs 0.7% of Sharpe against a point estimate that knew the answer, and buys 8.2% against the realistic alternative of optimising once on one sample.

State plainly what it does not fix. It does not correct a biased input. If your expected return for an asset class is wrong by two points for a structural reason rather than a sampling reason, every resample is wrong by two points and the average is wrong by two points with more confidence. It does not fix a Σ estimated from too few observations relative to the number of assets, because each resample inherits the same shortage. It has no closed form, so the resampled portfolio is not on any efficient frontier and cannot be defended by an optimality argument, only by a stability one. And with a long-only constraint it produces a well-known artefact: averaging many corner solutions yields a smooth interior portfolio that no single optimisation ever chose, which looks reassuringly diversified and is a statistical artefact of the averaging rather than a considered allocation.

The primitives are ones you already have. numpy.random.default_rng(seed).multivariate_normal(mu, Sigma, size=T) makes the draw, numpy.cov(X, rowvar=False) and X.mean(axis=0) re-estimate, and this module's own tangency_weights or gmv_weights closes the loop. Seed it, or the "stable" answer changes every time you run it, which would be a small joke at the technique's expense.

Advertisement

Part of the course

This page is part of Portfolio Optimization in Code (QD1.04), in Phase 12: Finance Plus, AI and the quant-code track. The module's flashcards and its quiz are in the app.