Calibration is the journal's payoff and the app's core dashboard, so learn to compute it as well as admire it. There are two simple, durable tools. Neither needs anything beyond arithmetic.
The calibration table (the bucket method). Group all your resolved predictions by the confidence you assigned. For each bucket, compare stated confidence to the fraction that actually came true:
| You said (confidence) | # of such predictions | # that came true | Actual hit rate | Verdict |
|---|---|---|---|---|
| ~90% | 10 | 6 | 60% | badly overconfident |
| ~70% | 20 | 13 | 65% | roughly calibrated |
| ~50% | 12 | 6 | 50% | well calibrated |
For a perfectly calibrated forecaster, the actual hit rate matches the stated confidence in every bucket, so the numbers in the last two columns line up. Now look at the pattern above. High-confidence predictions hit far below their stated rate. That's the classic overconfidence signature, and it's what most people find the first time they run this on themselves. The fix is mechanical and immediate. Haircut your own confidence, especially at the top end, and let that flow into smaller positions.
Advertisement
How is the Brier score calculated?
The Brier score (one number for the whole record). When you want a single number to track over time, use the Brier score, the mean squared error of your probabilistic forecasts. For each resolved prediction, take your probability p as a decimal, and the outcome o, which is 1 if it happened and 0 if not. The Brier score is the average of (p − o)² across all predictions:
Brier = (1/N) × Σ (pᵢ − oᵢ)²
It runs from 0, which is perfect, to 1, which is perfectly, confidently wrong. Always predicting 50% gives you 0.25, the "know-nothing" benchmark you have to beat to claim any skill. Try a small example. Take three theses at p = 0.90, 0.70, 0.60, with outcomes 1, 0, 1. The errors are (0.90−1)² = 0.01, (0.70−0)² = 0.49 and (0.60−1)² = 0.16. Brier = (0.01 + 0.49 + 0.16)/3 = 0.66/3 = 0.22. That's just better than coin-flipping, and the confident miss did most of the damage.
What does a falling Brier score tell you?
Track this quarterly. A Brier score that falls over the years is objective evidence your judgment is improving, and it's immune to the luck that makes P&L a liar over any short horizon. Tetlock's Superforecasting, on the Phase 6 reading spine, is the deep treatment. The good news from that research is that calibration can be trained by keeping score and adjusting.