Post-hoc Explainability#
Interpreting an already-trained model rather than building it interpretable.
Important
✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.
What it is#
Post-hoc explainability means explaining a model’s behaviour after training, without changing its internal structure — making black-box models (deep nets, ensembles, gradient boosting) interpretable. Post-hoc means after the fact: you do not train the model to be interpretable, you analyse its outputs afterward. It contrasts with intrinsically interpretable models (linear regression, small decision trees) that are transparent by design.
Why it matters#
Many high-performing models are opaque, yet users, regulators and businesses need to know why a prediction was made — for debugging, trust and transparency, and compliance (finance, healthcare, GDPR / AI Act).
The techniques#
The toolkit spans feature importance (global permutation or gain-based, and local), surrogate models (a simple tree approximating the black box), the local methods LIME and SHAP, visualisations — partial-dependence plots, ICE plots, and saliency maps / Grad-CAM for images — and counterfactual explanations. For a black-box loan model, SHAP might flag low income and short employment, while a counterfactual says “two more years of employment would flip the decision.”
Limitations#
Post-hoc explanations are approximations of the true model logic, risk being misleading (the faithfulness problem), can be computationally expensive, and are diagnostic only — not a substitute for fair training practices.
Theme: Explainability & Governance · All terminology
Hint
Mind map — connected ideas
SHAP (SHapley Additive exPlanations) · LIME (Local Interpretable Model-agnostic Explanations) · Counterfactual Explanations · Decision Trees · Feature Values · Discriminatory Power
Hint
More in Explainability & Governance
Basel III · Counterfactual Explanations · Fair Lending laws · High-Stakes Domains · LIME (Local Interpretable Model-agnostic Explanations) · SHAP (SHapley Additive exPlanations)
See also
Source article Adapted (context, re-expressed) in our own words from: Post-hoc Explainability (insightful-data-lab.com).