Explainable AI

Explainable AI (XAI) methods help people understand why a model produced a given output, within technical limits of fidelity.

Tools and techniques that try to make a model’s decisions more understandable to humans.

Methods include intrinsically interpretable models and post-hoc explanations (feature attributions, prototypes, counterfactuals). Explanations can be faithful to a local approximation without fully describing the model.

Regulated industries often require understandable decision factors for high-impact outcomes.

• Credit and insurance decision support
• Clinical decision support review
• Model debugging

• Supports human oversight
• Aids debugging and trust calibration

• Explanations can mislead if oversold
• Tradeoffs with accuracy for some interpretable models

• A heatmap is not proof of causal understanding

Is XAI required by law?

Requirements vary by jurisdiction and use case. Brel cites frameworks; it does not provide legal advice.

• NIST AI RMF

• Guidotti et al. — survey of explanation methods