What does Explainable AI mean?

Explainable AI refers to methods that make it traceable how a model arrived at a particular output. They answer which inputs tipped the outcome and how the output would have changed under different values. This differs from a justification in the legal sense in that it describes the calculation, not the legal basis.

Two approaches are common. Inherently interpretable models such as decision trees or linear regression expose their rule directly. Post-hoc methods, by contrast, explain a finished model from the outside. SHAP distributes the contribution to a result across the individual input features, while LIME approximates a decision in the neighborhood of a case with a simple model. Counterfactual explanations state the smallest change that would have led to a different result.

Explainable AI pays off wherever a model produces recommendations that people need to review and ultimately take responsibility for. Typical cases include assessing applications and credit requests, pre-selecting job applications, and flagging anomalies in audits. For technical predictions without any connection to individuals, the need recedes.

The advantage over an output with no explanation is that a specialist can actually review the recommendation instead of simply adopting it. On request, affected individuals receive information that goes beyond a reference to some calculation procedure. Conspicuous weights on individual features also reveal biases before they surface as contradictions.

An explanation describes how the model calculated its result; it does not prove that the result is correct. The EU AI Act requires that outputs of high-risk systems be interpretable for the body applying them. The justification for a formal decision must still follow from the applicable legal rule.