Here is the arithmetic that ought to change how you read a broker note. Cascade Instruments' operating margin over twelve quarters fits a rising straight line well: slope +0.281636 points a quarter with a t-statistic of 5.73, R² of 0.7846, and an in-sample root mean squared error of 0.4666. An AR(1) on the same eleven target observations fits worse on every one of those measures, with R² of 0.5966 and an in-sample RMSE of 0.6386.
What happens when you score on data the models never saw?
Score both on the four quarters neither model saw and the ranking inverts. The trend's out-of-sample RMSE is 1.1502 and the AR(1)'s is 0.4664, a ratio of 2.47 to 1. The trend's errors are −0.6887, −1.1404 and −1.8620 across the last three quarters, growing monotonically, because a linear extrapolation cannot stop climbing and the series had already flattened at its mean-reverting level of 19.5314%.
The mechanism is visible in the rolling estimates before the forecast fails. Re-fit the trend slope on successive eight-quarter windows and it decays from +0.3876 to +0.3344 to +0.3217 to +0.1965 to +0.0345, with the last window's t-statistic collapsing to 0.48. A parameter that moves that much across overlapping windows of the same series is not a parameter, and the decay was legible two years before the forecast broke.
Advertisement
How do you test a model out of sample?
Three habits follow, and they cost about ten minutes each. Hold out the last few periods before you fit anything, and never look at them until both candidate models are frozen. Compare models on the same target observations, because a trend fitted to twelve quarters and an AR(1) fitted to eleven pairs are not comparable until you drop the trend's first quarter too. And re-estimate on rolling windows and print the parameter path, because the path is a diagnostic that no single-sample standard error can give you.
The mini-project at the end of this node is twelve series of this kind, and the point of it is the count. Out of twelve, the trend model wins in sample on six and out of sample on five, and on three series the in-sample winner is the out-of-sample loser, two of those three being series where the trend was the in-sample winner. That count is itself a statistic, and it belongs in your write-up, because the number of models you tried is an input to the deflation arithmetic the branch's multiple-testing discipline sets out.