Machine learning project steps

A supervised project on structured financial data runs through six steps, and skipping any of them shows up later as a result that will not replicate.

1. Conceptualise. Write down the decision the output will change and the metric that measures success, before touching data. A model with no named decision has no correct metric.
2. Collect. Assemble features and the target with an explicit as-of date on every field, so nothing enters a row that was unknown when the row's timestamp says it was.
3. Prepare and wrangle. Clean, handle missing values, encode categoricals, and standardise. Every transformation is fitted on the training data only.
4. Explore. Look at the distributions, the outliers and the pairwise correlations before you model, since collinearity and a broken join both show up here for free.
5. Train. Split chronologically, tune the complexity dial by cross-validation inside the training data, and touch the test set once.
6. Evaluate and monitor. Score on the metric from step 1, and re-score on live data as it arrives.

How do you score a classifier from its confusion matrix?

The evaluation set for a classifier starts from the confusion matrix: true positives, false positives, false negatives and true negatives. From those four numbers,

accuracy = (TP + TN) / N
precision = TP / (TP + FP)        of those I flagged, how many were real
recall    = TP / (TP + FN)        of the real ones, how many did I catch
F1        = 2 * precision * recall / (precision + recall)

Accuracy is the metric that fails first and fails silently. On a loan book with a 12% default rate, a model that never predicts a default is 88% accurate, and the classifier in Worked Example 7 at a 0.50 threshold scores 83%, which is worse than doing nothing while being genuinely informative. Precision and recall trade against each other as the threshold moves, and F1 is their harmonic mean, which is close to the smaller of the two and therefore punishes a model that wins one at the other's expense.

How do you read AUC and RMSE?

The ROC curve plots the true positive rate against the false positive rate across every threshold, and the area under it summarises the whole curve in one number. AUC has an exact interpretation worth carrying: it is the probability that a randomly chosen positive case scores higher than a randomly chosen negative one. It is 0.5 for a coin flip and 1.0 for a perfect ranker, and unlike accuracy it does not move when the base rate changes, because it is a property of the ranking rather than of any threshold. On the loan book in Worked Example 7 the AUC is 0.8804, computed three independent ways that agree to fifteen decimal places.

For regression targets, root mean squared error reports the error in the target's own units and penalises large misses quadratically, while mean absolute error is more forgiving of outliers. Use RMSE when a big miss is much worse than two small ones, which for most financial forecasts it is.

How do you pick the metric that fits the decision?

None of those metrics is the answer on its own, because the decision has a cost structure and the metric has to match it. Refusing a good borrower on the loan book costs ₹40,000 in forgone margin; funding one who defaults costs ₹8,50,000. Under that structure the cost-minimising threshold on the printed grid is 0.45, at a total expected cost of ₹40,30,000. The accuracy-maximising threshold is 0.70, at 89.50% accuracy and a cost of ₹1,54,20,000. Optimising the metric that was easiest to compute costs ₹1,13,90,000, or 3.8 times the minimum, and every number in that comparison came from the same model and the same scores.

{
  "skill": "Auditing a machine-learning result somebody else produced",
  "use": "Turn a model write-up or notebook you have been handed into a structured list of the choices it made and the checks it did not run, so you can interrogate the author rather than the output.",
  "tools": ["Chat assistants with code execution - Claude, ChatGPT, Gemini", "A notebook AI grounded on the actual code and data, never on the summary slide", "Python: scikit-learn for the recomputation, on the same split you were given"],
  "prompt": "Here is a model write-up and its code. Extract into a table: the target and its base rate; the exact train/test split rule and whether it is chronological; every preprocessing step and whether it was fitted inside the training folds or on the whole sample; the hyperparameters, their search grid and how the winner was chosen; the reported metric and every metric NOT reported; the number of model specifications tried in total. Then list every claim in the write-up that the code does not support, and every check that would change my read if it failed.",
  "verify": "Re-run the split yourself and confirm no test-period row precedes a training-period row in real time. Recompute the headline metric from the raw predictions, and recompute the always-predict-the-majority-class baseline beside it - a model that does not beat that baseline on the metric the decision cares about is not a model. Confirm the scaler and any imputer were fitted on training rows only. Ask for the count of specifications actually run, in writing, because that count is an input to the deflation arithmetic and a model tuned over 400 configurations and reported once is a selected maximum, not an estimate. Cite the notebook, the commit and the data snapshot date for each figure.",
  "diy": "You must be able to compute accuracy, precision, recall, F1 and the base-rate baseline from a confusion matrix unaided, and to state the expected cost at two thresholds given the two error costs - that is Worked Example 7. An assistant can list the choices; only you can say which of them would change the decision your firm is about to take.",
  "market": "US"
}