CreditAI

Population Stability Index

PSI measures how far one distribution has moved from another. It is normally used to detect drift over time; here it is pointed at the problem this model actually has. The applicants this lender declined were never observed repaying, so the training data is the population the incumbent policy already selected. PSI turns "the rejects are a different population" from an assertion into a number.

What the two panels say together

Read alongside the KS separation from the reject-inference comparison, the ordering is the interesting part.

Hard cutoff scores best on KS while moving the model least. It labels almost every declined applicant bad, which largely restates what the incumbent policy already believed — so KS rewards it for agreeing with the assumption being tested, and the coefficients barely move. Parceling labels fewer declines as bad and shifts the model the most.

Two independent diagnostics therefore point the same way: the method that wins on separation is the one that confirms, and the method that changes the model is the one that scored lower. Selecting a reject-inference method by a metric computed on labels that method invented is circular, and the score shift is what exposes it.

One finding came out of recalibrating the scorecard. Parceling used to bin applicants by their 300–850 score, using equal-width bins. That score is a cosmetic transform of the model’s log-odds, so rescaling the card — a relabelling that changes no decision — moved the bin edges and reassigned people: the same data and the same seed produced 261, 281 and 278 declines labelled good under three different display scales. The labels a model trains on should not be a function of how its output is drawn. Parceling now bins on quantiles of predicted probability, which has no scale in it, and the assignment is identical under all three. That also fixed a second problem: the old equal-width bins held as few as 3 approved applicants, so some declines inherited a bad rate estimated from three observations.

Cross-check, computed in your browser

The figures above come from src/psi_analysis.py, which bins on deciles of the approved distribution. The panel below recomputes PSI live from the histograms already on this page, which use equal-width bins. Same data, same formula, different binning.

The numbers do not match, and that is the point. PSI is not a property of two distributions alone — it is a property of two distributions and a choice of bins. A dashboard that reports a single PSI without saying how it binned is hiding a parameter that moves the answer. The direction and the ordering survive the change; the magnitudes do not.