AI-Powered Credit Scoring & Risk Analysis
Real-time execution logs of the Multi-Agent System
Run the Credit Scoring analysis to see the pipeline execution.
PSI measures how far one distribution has moved from another. It is normally used to detect drift over time; here it is pointed at the problem this model actually has. The applicants this lender declined were never observed repaying, so the training data is the population the incumbent policy already selected. PSI turns "the rejects are a different population" from an assertion into a number.
How different the declined applicants are from the approved ones, feature by feature. Credit history length dominates — PSI 2.07, far beyond the 0.25 severe threshold — so the incumbent policy was selecting mainly on how long someone had been borrowing. Age and number of accounts are stable, meaning the policy barely used them.
Each method assigns outcomes to the declined applicants, a model is retrained on the result, and the whole population is rescored. This compares those scores against the accepts-only baseline — it measures how much each method actually changes the model's behaviour.
Reject inference is normally untestable. You assign outcomes to the applicants you declined, train on them, and never find out whether the assignment was right, because nobody observes what those applicants would have done. Choosing a method by KS is circular — the metric is computed on labels the method itself invented.
This dataset carries the answer. It records the true outcome for every applicant including the declined ones, which a real lender never has. So the guess can be marked.
| Method | Declines it called bad | Actually bad | Error | Score shift |
|---|---|---|---|---|
| Hard cutoff | 76.8% | 60.3% | +16.5 | 0.34 |
| Fuzzy augmentation | 67.6% | 60.3% | +7.3 | 0.03 |
| Parceling | 53.8% | 60.3% | −6.5 | 0.08 |
Hard cutoff is the most confident and the most wrong. Labelling almost every decline bad mostly restates what the incumbent policy already believed, and it overshoots the real rate by seventeen points — more than twice the error of either of the other two. It also moves the model the most, which is the combination to be suspicious of: a large shift toward a conclusion that turns out to be wrong.
But getting the rate right is not the same as ranking better. Scored against the declined population’s real outcomes, all three methods land within 0.001 AUC of doing no reject inference at all. They recover roughly the right level of risk and add nothing to the ordering.
That is not surprising once you look at where the labels come from. All three derive them from the baseline model’s own predicted probabilities, so none of them can carry information the baseline does not already encode — they only redistribute the fit. Real lift on the declined population needs information from outside the model: bureau data on those applicants, their performance at another lender, or a deliberately approved randomised slice of declines.
The honest use of reject inference here is calibration, not discrimination. It puts the expected bad rate of the declined population roughly in the right place, which matters for pricing and for reserving. It does not tell you which declines to reconsider.
The figures above come from src/psi_analysis.py, which bins on
deciles of the approved distribution. The panel below recomputes PSI
live from the histograms already on this page, which use
equal-width bins. Same data, same formula, different binning.
The numbers do not match, and that is the point. PSI is not a property of two distributions alone — it is a property of two distributions and a choice of bins. Here the gap is five to seventeen percent depending on the feature, and on an earlier version of this data one feature moved threefold on the binning alone. Bands and ordering survive; magnitudes do not. A dashboard reporting a single PSI without saying how it binned is hiding a parameter that moves the answer.
src/psi_analysis.py on the 7,000-row synthetic
dataset in data/raw/telecom_data.csv (5,000 approved, 2,000 declined).
Parceling is seeded, so the pipeline is deterministic — re-running reproduces
reports/psi_analysis_report.txt exactly. These are properties of a
simulation, not business outcomes.
CreditAI demonstrates an End-to-End Credit Scoring Pipeline typically used in the telecommunications and banking sectors. It relies on a Logistic Regression model trained on synthetic data to predict the likelihood of a customer paying their bills.
To enable AI features, click the OpenAI button in the header and enter your API key. The key is stored in browser memory only and never persisted.