Calibration scores your forecasts. What follows scores your decisions, and it exists to solve the problem that has haunted the whole program and is stated bluntly in working agreement #8: a good decision can lose money, and a bad decision can make money. If you grade yourself on the money, you will systematically reward luck and punish skill, and because you learn from what you reward, you will train yourself, with total confidence, to become a worse investor. The defense is to build a scoreboard for decision quality that never looks at the outcome, and to run it alongside the P&L, knowing at all times which one is telling you the truth about your skill.
Why does luck dominate over short horizons?
Why the money lies over short horizons: the paradox of skill. Investing is a mix of skill and luck, and over any short horizon the luck dominates. Michael Mauboussin's The Success Equation gives the sharpest framework here. First, a test for how much luck is in an activity: can you lose on purpose? At chess you can throw a game deliberately (pure skill); at roulette you cannot lose on purpose any more than you can win on purpose (pure luck). Public-market investing sits closer to the luck end than almost anyone wants to believe. You cannot reliably lose on purpose over a quarter, because the market, not your skill, decides the quarter. Second, and more counter-intuitive, the paradox of skill: as the absolute skill of participants rises and becomes more uniform, and professional investing is a field of brilliant, superbly resourced, hyper-competitive people, the relative skill differences shrink, so luck explains more of the variation in who wins, not less. In a field of amateurs, skill shows through quickly. In a field of experts, everyone is so good that outcomes over short horizons are mostly noise.
This is why you cannot read your skill from your one-year return, why last year's top fund is a coin-flip to lead next year, and why the only honest short-horizon scoreboard for skill is one that grades the process directly. Over a long enough horizon skill does surface in results, but "long enough" in investing is many years, far longer than the interval at which you need feedback to improve, so you cannot wait for the P&L to teach you.
Advertisement
Can a good decision lose money?
The decision-quality × outcome-quality grid. Every resolved decision lands in one of four cells, and knowing which is the beginning of learning honestly:
| Good outcome (made money) | Bad outcome (lost money) | |
|---|---|---|
| Good decision (sound process) | Deserved success: keep the process, don't get cocky | Bad luck / variance: keep the process, log a well-decided miss |
| Bad decision (poor process) | Dumb luck: kill the process, the win is a warning | Deserved failure: the one everyone learns from |
The two diagonal cells are easy and misleading; the two off-diagonal cells are where all the real learning is, and where "resulting" (Annie Duke's term, M6.03, for judging the decision by the outcome) does its damage. The professional can say, and mean, "I lost money and it was the right decision" (top-right: a well-reasoned, well-sized bet that a 40%-likely risk happened to catch) and "I made money and it was a bad decision" (bottom-left: a reckless, oversized tip that happened to triple). They can keep the first process and kill the second, which is exactly backwards from what the P&L would tell them to do.
How does the process scorecard work?
The process scorecard: the instrument. To place a decision in the grid you need an outcome-blind measure of process quality. Score each resolved decision on a fixed rubric, 0/1/2 per dimension, without looking at what the stock did, ideally from the journal entry alone, which is why the journal had to be honest and complete:
| Process dimension | 0 | 1 | 2 |
|---|---|---|---|
| Inside written circle of competence | Outside it | At the edge, not flagged | Clearly inside, or edge honestly flagged and sized down |
| Full analysis run (the M8 system) | Story, no work | Partial / shortcut | Full teardown/deep dive; five questions answered |
| Checklist run as a do-confirm pause | Skipped | Run, flags unresolved | Run; every flag resolved in writing |
| Thesis falsifiable + own-hand bear case + base rate | None | Thesis only | All three, specific |
| Numeric confidence stated before outcome | Absent | Vague ("high") | A number, in the journal, pre-outcome |
| Margin of safety / reverse-DCF check | None | A multiple, no reverse-DCF | Priced-in story judged beatable; a range, not a point |
| Sized per policy, real downside floor | Oversized / unsized | Roughly | Fractional-Kelly/limit; downside floor verified real |
| Kill criteria + monitoring metrics pre-committed | None | Vague | Specific, observable, dated |
Sum to a score out of 16. This number is your decision quality, and it is available the moment you decide, years before the outcome. Now run two scoreboards: the process score (this) and the P&L. Over a handful of decisions they can point in opposite directions, and that divergence is information, not embarrassment: a high process score with a loss is variance (keep going), a low process score with a gain is a warning (stop). Over hundreds of decisions and many years, a rising average process score is what causes a good long-run P&L; you optimize the process score, and you let the P&L be the eventual, lagging, noisy confirmation. Optimize the scoreboard you control; the other one will follow, on its own schedule, with a lot of noise. Worked Example 3 runs the scorecard on two decisions, one that lost with a 15/16 process and one that won with a 2/16 process, and shows exactly which to reinforce.