Ordinary least squares minimises the sum of squared residuals and nothing else. When predictors are nearly collinear that objective has a long flat valley in it: many very different coefficient vectors fit the data almost equally well, so the fitted surface is stable while the individual coefficients are not. The variance inflation factor measures the problem directly, and in the four-factor example in Worked Example 2 two of the four factors correlate at 0.9834 and carry variance inflation factors of 34.6 and 35.8.
Penalised regression changes the objective by adding a cost on coefficient size:
ridge (L2): minimise SSR + lambda * sum of b_j^2
LASSO (L1): minimise SSR + lambda * sum of |b_j|
elastic net: minimise SSR + lambda * ((1-a) * sum b_j^2 + a * sum |b_j|)Every one of them is biased on purpose. The estimator no longer aims at the coefficient that best fits this sample; it aims at a smaller one, and it accepts a known bias in exchange for a large reduction in sampling variance. Whether the trade pays is an empirical question with an empirical answer, which is the out-of-sample error.
Advertisement
How do ridge and LASSO differ, and what is elastic net?
Ridge, from Hoerl and Kennard's 1970 paper, has a closed form: b = (X′X + λI)⁻¹X′y. When the predictors are standardised and orthogonal, that collapses to a single shrinkage factor n/(n + λ) applied to every ordinary coefficient, which is the cleanest possible statement of what ridge does. It shrinks everything toward zero and sends nothing to zero exactly. Watch it work on the collinear pair: the ordinary coefficients are +4.2961 and −2.3476, which is a nonsense split with the wrong sign on one of them, while at λ = 10 they become +1.0304 and +0.9079. The sum barely moves, from 1.9486 to 1.9383, because the data always identified the sum precisely and never identified the split at all.
LASSO, from Tibshirani's 1996 paper, has no closed form in general, but on a standardised orthogonal design it reduces to a soft threshold: each coefficient loses λ from its absolute value, and anything that would go negative is set to exactly zero. That exact zero is the whole difference. Ridge produces a small coefficient you still have to think about; LASSO produces an absence. Fitting is done by coordinate descent, cycling one coefficient at a time until nothing moves.
Elastic net, from Zou and Hastie's 2005 paper, mixes the two penalties, which matters precisely when predictors come in correlated groups: LASSO tends to pick one member of a correlated group arbitrarily and drop the rest, while elastic net keeps or drops the group together.
How should you implement ridge and LASSO?
Two implementation rules are not optional. Standardise the predictors first, because a penalty on raw coefficient size is a penalty on the units you happened to measure in. Restate one predictor in basis points rather than percent and its coefficient falls by a factor of a hundred, so it contributes almost nothing to the penalty and is barely shrunk, while its small-scale neighbours take the whole of it. On a two-predictor demonstration at λ = 200, rescaling one column by 100 left it at 1.01 times its unpenalised value while the other was cut to 0.47. Choose λ by cross-validation, never by eye, because the in-sample error is monotone in λ and will always tell you to set λ = 0. The standard procedure fits on k − 1 folds, scores on the held-out fold, rotates, and averages; the λ with the lowest average error is the minimum-error choice, and the largest λ whose average error sits within one standard error of that minimum is the parsimonious choice known as the one-standard-error rule. On the factor example the minimum sits at λ = 0.20 with a cross-validated mean squared error of 7.1784, and the one-standard-error threshold of 7.8532 admits λ up to 0.50.
The subtlety that ruins more results than any other: the standardisation has to happen inside each training fold. Compute the means and standard deviations on the whole sample first and every fold's test rows have already contributed to the scaling of the features used to predict them. The leak is small, silent and always in the flattering direction.
What does it mean when LASSO sets a coefficient to zero?
What a LASSO's zero does and does not mean is worth a sentence of its own. On this sample the LASSO at λ = 0.20 keeps the first of the two twins at 1.8485 and sets the second to zero exactly. Resample thirty of the thirty-six months four hundred times and refit: it keeps the first twin alone 342 times, keeps the second alone 15 times, and keeps both 43 times. So the selection is a statement about this draw of the data, with roughly a one-in-seven chance of a different answer on a slightly different sample. Writing "LASSO selected the value factor and rejected the composite" as though it were a finding about markets is the error, and the honest sentence names the sample, the λ and the resampling stability.
Do ridge and LASSO beat ordinary least squares?
Does the penalty actually pay? Split the thirty-six months at random into twenty-four for fitting and twelve for testing, five hundred times, and average. Ordinary least squares gives a mean out-of-sample RMSE of 2.9615, LASSO at λ = 0.20 gives 2.8165 (a 4.90% improvement), ridge at λ = 3 gives 2.8956 (2.22%), and ridge at λ = 30 gives 3.8036, which is 28.44% worse. The dial has an optimum in the interior and both ends of it are bad, which is the same shape as every other complexity dial in this node.