AI-Powered Credit Scoring & Risk Analysis
Real-time execution logs of the Multi-Agent System
Run the Credit Scoring analysis to see the pipeline execution.
PSI measures how far one distribution has moved from another. It is normally used to detect drift over time; here it is pointed at the problem this model actually has. The applicants this lender declined were never observed repaying, so the training data is the population the incumbent policy already selected. PSI turns "the rejects are a different population" from an assertion into a number.
How different the declined applicants are from the approved ones, feature by feature. Late payments and debt ratio are the features the incumbent policy is most strongly selecting on — which is also where a model trained only on approvals is least able to extrapolate.
Each method assigns outcomes to the declined applicants, a model is retrained on the result, and the whole population is rescored. This compares those scores against the accepts-only baseline — it measures how much each method actually changes the model's behaviour.
Read alongside the KS separation from the reject-inference comparison, the ordering is the interesting part.
| Method | Declined labelled bad | KS | Score shift |
|---|---|---|---|
| Hard cutoff | 98.5% | 51.6% (best) | 0.13 |
| Fuzzy augmentation | 91.7% | – | 0.10 |
| Parceling | 86.8% | – | 0.37 |
Hard cutoff scores best on KS while moving the model least. It labels almost every declined applicant bad, which largely restates what the incumbent policy already believed — so KS rewards it for agreeing with the assumption being tested, and the coefficients barely move. Parceling labels fewer declines as bad and shifts the model the most.
Two independent diagnostics therefore point the same way: the method that wins on separation is the one that confirms, and the method that changes the model is the one that scored lower. Selecting a reject-inference method by a metric computed on labels that method invented is circular, and the score shift is what exposes it.
One finding came out of recalibrating the scorecard. Parceling used to bin applicants by their 300–850 score, using equal-width bins. That score is a cosmetic transform of the model’s log-odds, so rescaling the card — a relabelling that changes no decision — moved the bin edges and reassigned people: the same data and the same seed produced 261, 281 and 278 declines labelled good under three different display scales. The labels a model trains on should not be a function of how its output is drawn. Parceling now bins on quantiles of predicted probability, which has no scale in it, and the assignment is identical under all three. That also fixed a second problem: the old equal-width bins held as few as 3 approved applicants, so some declines inherited a bad rate estimated from three observations.
The figures above come from src/psi_analysis.py, which bins on
deciles of the approved distribution. The panel below recomputes PSI
live from the histograms already on this page, which use
equal-width bins. Same data, same formula, different binning.
The numbers do not match, and that is the point. PSI is not a property of two distributions alone — it is a property of two distributions and a choice of bins. A dashboard that reports a single PSI without saying how it binned is hiding a parameter that moves the answer. The direction and the ordering survive the change; the magnitudes do not.
src/psi_analysis.py on the 7,000-row synthetic
dataset in data/raw/telecom_data.csv (5,000 approved, 2,000 declined).
Parceling is seeded, so the pipeline is deterministic — re-running reproduces
reports/psi_analysis_report.txt exactly. These are properties of a
simulation, not business outcomes.
CreditAI demonstrates an End-to-End Credit Scoring Pipeline typically used in the telecommunications and banking sectors. It relies on a Logistic Regression model trained on synthetic data to predict the likelihood of a customer paying their bills.
To enable AI features, click the OpenAI button in the header and enter your API key. The key is stored in browser memory only and never persisted.