When Models Must
Explain Themselves
to Regulators

The conversation used to be about statistical validity - Gini coefficients, population stability indices, that kind of thing. Now the first question is often: "For this specific applicant who was declined, which inputs drove that outcome and by how much?" That is a fundamentally different question. It requires local explainability, not just aggregate model performance.

South Africa's National Credit Act and the broader Basel model risk guidelines have pushed banks toward individual-level accountability. When an applicant disputes a decision, the bank needs to reconstruct the reasoning in plain language. A gradient boosting model with 200 trees cannot do that on its own - you need an explainability layer sitting alongside it.

SHAP earns its place for tree-based models. TreeSHAP runs fast enough for production scoring, the values are consistent with game-theoretic fairness properties, and they aggregate cleanly across a portfolio. When a risk manager asks "which feature group contributed most to this month's score distribution shift," SHAP gives a coherent answer.

LIME is more flexible across model types but introduces instability that makes auditors uncomfortable. Run LIME twice on the same input with slightly different neighbourhood samples and you can get meaningfully different explanations. That is a problem when you are trying to document a decision made six months ago. I have seen teams abandon LIME specifically because they could not reproduce their own explanations during a regulatory review.

The honest answer is that most banks have standardised on SHAP not because it is theoretically superior in every case, but because it is reproducible and the outputs are interpretable enough for non-technical stakeholders. Consistency under audit pressure matters as much as mathematical elegance.

The gap between what teams produce and what auditors need is usually about traceability, not volume. A 120-page model development report full of charts does not help if you cannot answer: "On 14 March, applicant reference X received score Y. Show me the feature values, the SHAP contributions, and the decision threshold in effect at that moment."

Useful documentation captures three things at scoring time: the raw input values, the transformed feature values the model actually saw, and the explanation outputs. All three need to be stored together with a timestamp and the model version identifier. Teams that log only the final score create enormous problems for themselves later.

There is also a communication layer that gets overlooked. The technical documentation satisfies the model validation team. But credit officers need a separate document that translates the same findings into language a branch manager can read. Writing both from the same underlying data is not optional if the bank wants to use the model across business lines.

Drift in feature importance is different from drift in model performance, and teams often conflate them. A model can maintain its Gini coefficient while the features driving predictions shift substantially - income variables might lose weight while bureau tradeline features gain it during a credit cycle downturn. The model looks stable by traditional metrics but is making decisions for different reasons than when it was validated.

The practical approach is to run a monthly SHAP summary across a stratified sample of recent applicants and compare the rank ordering of mean absolute SHAP values against the baseline established at deployment. Set a threshold - say, if any feature moves more than three positions in the top-ten ranking - and flag it for review rather than automatic retraining.

Retraining is expensive and introduces new validation cycles. Monitoring buys time and creates a documented record that the bank was aware of the change and assessed it. That documented awareness is itself valuable during an audit, even if the conclusion was "no action needed yet."

Interaction effects. SHAP handles them, but the outputs become difficult to communicate once you move beyond simple additive contributions. If a model has learned that high income combined with short employment tenure is riskier than either factor alone, the SHAP values for those features in isolation do not capture that relationship cleanly. Interaction SHAP values exist but add another layer of complexity that most credit officers cannot action.

Scale is the other genuine problem. Generating SHAP values for every application in a high-volume retail bank is computationally feasible with TreeSHAP, but storing, indexing, and querying those explanations across millions of records requires infrastructure investment that is often underestimated. Teams that prototype explainability in a notebook environment and then try to deploy it at production scale frequently hit this wall.