Start with the sample estimators and their properties. The sample mean is unbiased and has a standard error of σ/√n, which for financial returns is large enough to make the estimate nearly useless at any horizon a human will wait. The sample variance is estimated far more precisely, because variance can be estimated from higher-frequency data while the mean cannot: sampling daily instead of annually multiplies the observations by about 250 for volatility and does essentially nothing for the mean. That asymmetry is the single most useful statistical fact in this work. It says: build expected returns from economics, and estimate risk from data.
Even so, the covariance matrix is where an assumption set most often breaks, for a counting reason. An n-asset table needs n expected returns, n volatilities and n(n−1)/2 correlations, which is six at four assets, forty-five at ten and 190 at twenty. The parameter count grows as the square of the asset count while the data available to estimate them does not grow at all.
Three responses are standard.
Advertisement
How does shrinkage work?
Shrinkage pulls each sample correlation toward a structured target, trading a little bias for a large reduction in variance. Ledoit and Wolf's 2003 paper is the standard reference and the practical form is simple: pick a target, pick an intensity δ between zero and one, and set each shrunk value to (1 − δ) × sample + δ × target. The usual target is the average of all off-diagonal sample correlations, which imposes the structure "everything correlates about the same amount" and lets the data pull each pair away from it. On the four-asset Indian matrix of worked example 1 that average is 0.191667, and shrinking at δ = 0.30 moves equity-to-real-estate from 0.5500 to 0.4425 and equity-to-gold from minus 0.0500 to plus 0.0225.
What shrinkage buys is measurable stability. The smallest eigenvalue rises from 0.389563 to 0.517410 and the condition number, the ratio of largest eigenvalue to smallest, falls from 4.3814 to 3.1628. A lower condition number means the inverse the optimiser computes is less sensitive to small input changes, which is the whole objective.
How do factor-based and multi-regime models work?
Factor-based covariance models each asset's return as exposure to a few common factors plus an idiosyncratic residual, replacing forty-five correlations with n × k exposures, k(k+1)/2 factor covariances and n residual variances. The cost is a specific wrongness: whatever co-movement the factors do not span is assumed to be zero, and in a crisis that is the assumption that fails. Multi-regime estimation fits separate matrices for separate states and blends them by state probability, which is the honest answer to the equity-bond correlation problem and the one needing the most judgment, since the regime probabilities are themselves a forecast.
None of the three removes the underlying difficulty: a correlation you cannot estimate is not made estimable by a better estimator. What the tools do is stop a badly estimated correlation from producing an absurd portfolio, which is a real and sufficient benefit.