Towards agent explainability

1. The story of a research question

The seminar opens with the evolution of its own title across the years of a PhD: Humility → Epiphany → Frustration → Intervention → Incompleteness. The arc is the research process itself: first humility before a huge field, then the epiphany of a promising idea, then the frustration of its limits, then an intervention — a concrete method — and finally the honest acknowledgment of incompleteness. The presentation is guided by four pillars as a guide to XAI: getting started (context, current XAI, why we should care), interventions & agents, intentions, and reliability.

Read the title trajectory as the seminar’s methodology statement: the speaker is not claiming to have solved explainability, but to have made a sound, bounded contribution — and the chapter will end, accordingly, with future research lines rather than conclusions.

2. Artificial intelligence and the paradox of understanding

The seminar sets the stage with a deflationary definition of the field’s name:

• Artificial = things are synthesised (though not with full forethought) by humans, and they can be characterised in terms of functions and goals;
• Intelligence = ?? — hard to find a good definition. Things that are ‘hard to understand’ appear more intelligent; ‘understanding’ how something works makes its intelligence appear to vanish.

Hence the provocative formulation: the goal of Artificial Intelligence is to be incomprehensible. The paradox is exactly the tension XAI lives in: as systems get better at their tasks, they are perceived as more intelligent — and the less intelligible they become. Chapter 7’s intelligence discussion comes back with a twist: here the perceived intelligence is inversely correlated with understandability, which is why explainability must be engineered, not assumed.

3. What is explainability? Two families of enquiry

Explainability sits at the intersection of many notions: causality, decisions, autonomy, inquiry, communication, human-computer interaction, justification. The seminar’s key move is to locate explainability in the interstice between two families of enquiry:

• scientific inquiry — how did this decision come about? (causal, mechanistic);
• social enquiry — why should I accept this decision? (justificatory, communicative).

The two families have deep historical roots: the Aristotelian tradition explains by function, intention, teleology (why something exists, what it is for); the Galilean tradition explains by cause, strict origin, mechanism (how something came to be). The Copernican revolution — roughly, the transition from teleological to mechanistic explanation — is what modern science is built on. XAI inherits both: a user may want a mechanistic answer (“which features drove this output?”) or a teleological one (“what was the agent trying to do?”). Both are legitimate; they answer different questions.

4. What do we want XAI for?

The seminar surveys the reasons for XAI, following [Adadi et al.]. Three clusters of goals:

• For humans & law — the most prevalent: accountability. Systems that decide about people must be able to justify themselves before regulation (think the European AI Act, GDPR’s right to explanation).
• For machines & humans — bias control: explanations let developers detect and correct systematic errors; and the model itself can learn from them.
• For machines & humans — others learn from the model: explanations as teaching devices, transferring competence.

Each cluster demands a different kind of explanation: a legal justification, a debugging report, or a lesson. One size does not fit all — and the choice of audience determines what “good” means.

5. How XAI should be: the state of the art and the “Focus!” truth

In the mainstream literature, “interpretability über alles” dominates: feature relevance methods and user studies are the currency of the field [Lipton; Montavon et al.; Winnikof et al.]. The state of the art in XAI for non-agent systems is well stocked:

• SHAP — Shapley-value-based feature attribution;
• coefficients (for linear models) — direct reading of weights;
• “counterfactuals” — the closest similar instance of another class;
• attention/saliency maps [Selvaraju et al.; Montavon et al.], and so on.

But how should XAI actually be? The seminar’s answer, attributed to [Arias-Duart, 2022]: “Focus!” Explanations should be:

• short — not exhaustive;
• true — faithful to the system;
• given in context — for the specific user, situation and question;
• understandable — by that user.

The four adjectives are a quiet critique: much of the field optimises elaborate techniques and user-study scores while forgetting that an explanation nobody can act on is not an explanation. The “less spoken truth” is that feature-relevance maps, however technically sound, often fail the short/context/understandable test.

6. Agents, interventions and causality

The seminar then asks what is special about explaining agents. The bridge is the classical trio of problems:

1. correlation does not imply causation;
2. then what is causation?
3. what can causation be used for?

The answer to the second question is J. Pearl’s Ladder of Causation: association → intervention → counterfactuals. The key formal distinction is between conditioning and intervening:

P(X|A) ≠ P(X|do(A))

Observing that A happened is not the same as making A happen. Relevance of agency: agents are precisely the entities that perform autonomous interventions — they have freedom of exploration, and the quality of the knowledge they gather depends on how they intervene. This is what makes agent XAI different from static-model XAI: an agent’s behaviour is the product of its interventions in a world it changes, so explaining it requires causal (not merely correlational) vocabulary — Chapter 4’s autonomy, now from the epistemic side.

7. Agent XAI today: theory, practice, and the cynical triangle

How does “theory” look today? Two families of approaches coexist:

• ‘XAI-by-Design’ — architectures conceived to be explainable from the start;
• “General” methods — model-agnostic techniques applicable to any agent [Winnikof et al.; Puiutta & Veith; Dazeley et al.; Milani et al.].

And how does practice look today? At most, rule-based explanations of the form: “I do ‘up’ when I am between x=[1,3] and y=[-2,3] and there is no screw at x,y=(1,4)”. Functional, but far from the intentional vocabulary the course has been building.

Why is it like it is today? The seminar offers a (cynical) triangle of incentives:

• European commission — must make XAI laws: wants general definitions, system-agnostic, doable by industry; focus on what ‘matters’, not on what ‘implementing’ means;
• Academy, scientists & techies — must research more XAI techniques: hard to implement (which justifies writing a paper), model-dependent, should be ‘good’ and fulfil advanced definitions of explanation;
• Industry — must justify projects before capitalistic stakeholders in a law context: anything that fulfils the commission’s demands — SHAP/LIME works. Who cares about correlation/causation?

The result: regulation asks for things that are doable, academia produces things that are impressive but often unusable, and industry uses whatever passes the bar. The seminar’s intervention — the rest of the chapter — tries to break this triangle by grounding explanations in the agent’s actual behaviour.

8. How to check if an explanation is good

How to check whether agent XAI is good? The standard answer is user studies — but the seminar interrogates them:

• who makes the user study? (with whom? which users? what background?)
• which model is being explained, which method explains it, in what context?
• which metrics are evaluated? how much noise?

It is unending work: middle-guys made tons of papers with cherrypicking; users can be deceived or wrong; formal guarantees are not always possible. Two additional currents complicate the picture: LLMs are great! (a user, probably) — large language models as explanation providers; and the question of metrics for reliability — can we measure whether an explanation is truthful, not just liked? This is the gap the seminar’s own work attacks: truthfulness as a measurable property of the explanation relative to the explained policy.

9. Explaining an agent: Q-learning

To make the problem concrete, the seminar picks a minimal architecture: Q-learning.

• the agent acts according to the ‘best known action value in a state’: argmaxa Q(s,a);
• Q is learnt from experiencing states (S), actions (A), and rewards from the environment (R);
• but you cannot learn Q if you are only exploiting what you know: you have to sometimes explore alternatives, be curious, or you will converge to local minima;
• hence multiple modes (M): exploring / exploiting. When exploring, pick A randomly!

Then comes the gutpunch: now you know Q-learning. You can ‘ask’ XAI about this model’s particular decision, study how Q is created, or why this action was taken then. You can measure XAI across 20 metrics, and run 50 user studies in 30 use-cases. What about the 300 other, equally valid architectures? And… what about people that don’t work on AI? Explaining one architecture in depth does not scale: the field needs methods that work for any architecture — including humans — and that produce explanations non-experts can use.

10. Intention Policy Graphs

The seminar’s intervention builds on policy graphs:

• Hayes et al. (original): a graph over discrete states and actions, with edges carrying P(s′|a,s) and P(a|s) — purely model agnostic;
• refinements by Climent et al., Domenech i Vila et al., Tormos et al., … (quality);
• Gimenez-Abalos et al. — the IPG (Intention Policy Graph), adding intentions on top of the policy graph.

The original Hayes explanations are three algorithms, each answering one question:

1. What will you do when you are in state X? → “If I am in state X, I will perform action A” (or with probabilities: “… action A with probability p_A, … or action B with probability p_B”);
2. When do you perform action X? → “I will perform action A in states S1, S2, … or Sn”;
3. Why did you not perform action X in state Y? → “In state S, I would choose action A1. However, I would rather choose A2 if, instead, predicate p1 had value v1, … and pn had value vn”.

These are the basic explainable units: single-action statements, action-location statements, and counterfactual conditions. The IPG extends the graph’s edges so that intentions can be read out of them — the topic of the next section.

11. Intentions and telic XAI

Before adding intentions to graphs, the seminar asks whether it is safe: ALARM: INTENTIONS! Is it ‘safe’ to say a code has an intention? When is it ‘safe’? Will people misunderstand and think machines are conscious?

The philosophical grounding is telic XAI: artificial things can be characterised in terms of functions, goals, adaptation [Malle et al.; Saxe et al.; Simon]. The classic example:

Why did you fill the pot with water?
• I filled the pot with water because it was empty, and because the hob was free, and because there is no fire alarm going on.
• I filled the pot with water because I need boiling water, I intend to boil pasta, I am cooking pasta carbonara.

The first answer is mechanistic (Galilean): a conjunction of enabling conditions. The second is teleological (Aristotelian): an intention chain — boiling water serves cooking pasta, which serves eating carbonara. Both are true; they answer different questions. The seminar asks: are we OK to say a DL model has intentions? ‘Why that policy?’

How it looks for IPGs: teleology. Given a policy graph, the IPG supports teleological readings:

• What does it intend to do? → Desire to SERVE SOUP: 0.625;
• Why would it INTERACT? → Intentions attributed for SERVING SOUP expected to increase by 0.05;
• How would it fulfil its desire to SERVE SOUP? → INTERACT → DOWN → RIGHT → DOWN → INTERACT.

Intentions formalised. The procedure is explicit:

1. attributing a desire d;
2. suppose the agent ‘believes’ PPG(s′|a,s);
3. suppose the agent acts PPG(a|s) — if the agent ‘desires’ d, then it will ‘act’ with the intention of achieving it: (existing) desires condition action choice;
4. compute Id(s) = P(d|s);
5. cut off ‘low’ (below threshold) Id to avoid attributing unfeasible or unlikely intentions.

The result is a probabilistic, testable notion of intention: an intention is not asserted but attributed with a degree, and the attribution is anchored to the graph’s empirical probabilities — not to any claim about consciousness. Safety is engineered by the threshold: only intentions that the behaviour actually supports are attributed.

12. IPGs in practice: Overcooked, rationality and debugging

The evaluation domain is Overcooked (the cooperative cooking game), where the agent’s desire is delivering soup. The questions the IPG answers empirically:

• How much of the time can intention d be attributed? — coverage of the intention;
• How much behaviour can be explained (with intentions)? — explanatory power;
• How probable is it that an intention is fulfilled? — success probability;
• How much behaviour explained is truthful? — truthfulness of the attribution.

Rationality & XAI. The IPG also supports normative readings: between best performance and random performance (0 reward), behaviour that neither pursues a desire nor matches the environment’s affordances is irrational — and irrationality is where debugging begins.

Debugging & XAI. Locating irrational behaviour means:

• agents not doing any desire — aimless behaviour;
• desires not matching agent behaviour — the graph says one thing, the agent does another;
• affordances & rationality — the environment’s possibilities vs. what the agent attempts;
• locating irrational contexts — the specific states where behaviour breaks down (e.g. an unseen plate, no plate at all).

Explainability and debugging merge: a faithful explanation of an agent’s intentions is also a diagnosis of its failures — Chapter 13’s “explainability = rationality” idea, applied.

13. Reliability: advantages and limitations

The seminar closes the IPG part with an honest balance sheet. Advantages:

• simple & intuitive;
• multiple applications;
• public repository implementation;
• applicable to any architecture — including humans;
• frequentism ↔ epistemically sound (grounded in observed frequencies).

Limitations:

• finding the desires is non-trivial;
• continuous, complex domains with sparsity ↔ efficiency;
• desire prioritisation & deliberation, values, virtues… not modelled;
• interventionism: “why not do this thing you never do? I don’t know!” — the graph cannot answer about actions outside its support;
• higher-order ‘causes’ (causal history of reasons) out of scope;
• ignores transparent information from the model;
• ignores questions like ‘why does the agent believe P(s′|a,s)?’

The limitations are the honest boundaries of the intervention — and they motivate the final, more ambitious framework: the Ladder of Intentions.

14. A Ladder of Intentions

The final framework is a unifying model of behaviour explanation. Why is a unifying model important? Because otherwise every question multiplies:

• how to do good XAI?
• how do we evaluate manner? (cohorts with explainees);
• how do we evaluate truthfulness for a particular case?
• do the user studies extrapolate?
• must we repeat every user study with every architecture, kind of possible explainee, situation and context?

The Ladder of Intentions is a holistic, universal taxonomy of behaviour, intention-centric, spanning praxia vs gnosia (acting vs knowing) and declarative vs imperative knowledge. It applies to BDI, RL, LLM-agents, custom code… and it decouples how to make interpretable XAI from truthful & feasible XAI.

Key idea. The ladder answers recursively: Why π? Reasons(π). Then… why Reason(π)? Reasons(Reason(π)). And so on… The agent can be ‘summarised’ into π; splitting it to see its causes makes it more understandable.

The levels, reification, robustness. Splitting into levels separates statements (S) from code (I) using them:

• the lowest rung is the policy/IPG level: actions and percepts;
• above it, utility and consequences: statements about the policy’s outcomes;
• statement reification: s2 = Consequence of [i0] in [s1] is [s1] — elaborations of lower levels become statements themselves: “desirable s1”, “value of a in s1”, “confidence in transition s1 to s1′”, “preferences between desires”…

Folk-psychology grounding. The ladder’s rungs map onto ordinary explanation:

Why go right? Because [there is a wall on top] and [it’s the best action] — Belief.
Why is it the best? Because [] [I am exploring random actions] — Desire.
Why are you exploring random actions? Because [I have an estimator of when it’s better to exploit my knowledge vs explore] [and I use it to decide to explore now] — IPG is here.
Why do you use a value to decide to explore or exploit? Because [that’s the intent of my creator] — the top of the ladder.

Ladder learning. Learning a statement si implies learning something about si-1, ii-2. Statements can be given a priori, or learnt by experiencing the lower level; an agent could be designed to climb the ladder.

The 3 kinds of questions. The ladder organises XAI questions by their target rung:

• Intentional (I to I): goal oriented, climbs up. “Why up? — Because I am going to the pot.” Concise but not comprehensive; may need clarification; high predictability.
• Weakly epistemic (I_i to S_i+1 in the context of I_i+1): “Why go up when going to the pot? — In this state, the pot is above me.” Declarative relevance, supports a way of acting, bounded rationality; how S came to be remains unclear; cognitively harder on the explainee.
• Strongly epistemic (S to S, I of a lower level): “Why did you think that going up results in you being closer to the pot? — I have experienced that going up decreases my distance to the pot, increasing reward.” Learning explanations; understanding how E works is non-trivial; aggregates tons of data, currently unfeasible in many domains.

Level 2 formalisation (general S2). The general idea is to establish an action preference / deliberation mechanism over action preference; the bottom can reduce to the top. From easiest (least informative) to hardest (most expressive):

• easiest: action preference per state — “why a in s? because it’s the best” (a strongly epistemic question follows…);
• medium: probability about characteristics of future states — “why a in s? it results in a future state where x happens” (tied to I2: x is part of its intention);
• hardest: causal model of state transitions — “why a in s? it likely changes the state such that… and then I will do a′… and I get x”.

15. Two processes, and future research

The seminar concludes by separating two roles in the XAI supply chain:

• Process 1 (Agent Expert, technical XAI): deeply understands an agent architecture; conceives mechanisms for transforming and modelling the agent in the formalisations (S) established by Process 2’s needs; builds from agency to an S thought useful in some context; remains unconcerned with use-cases, but provides benchmarks of reliability; is concerned with reliable tools, not how they are used; does AI/agents, and is an expert on it.
• Process 2 (Sociologist, others): studies the required desiderata from explanations; demands intermediate formalisations (S) and the 3 queries from which desiderata can be fulfilled (think requirements or imports); builds from S to a natural-language (or other) explanation, even before an architecture implements S; remains unconcerned with the substrate of agency, only with reported reliability; evaluates which kinds of explanations are most useful (interpretable, accountability-driven, …) for a context; does user studies, and is an expert on them.

The two processes are the division of labour behind the ladder: Process 2 specifies what explanations should satisfy; Process 1 builds formalisations that can satisfy them, with reliability benchmarks. The deck’s closing slide lists future research lines: IIPGs (adding interventions); Deep IPGs (from Overcooked to Chess); Pgeon 2.0 (bringing XAI to industry via a good repository); truthfulness & benchmarking reliability; mind the gap (from technical to law and back); Red XAI (explainability = rationality); empowerment and higher-order empowerment; level formalisations; ladder extensions; the climbable ladder; the XAI inversion (AI for X, not X for AI); the XAg inversion (explain AI as agents).

And, in the speaker’s group: virtuous agents for modelling waste-water management and policy impact; curiosity-driven agents and transfer learning; LLM-based explanations (are they good? not generally — are there good use cases?); abduction-driven agents; IPG applications to LLMs, humans, and complex use-cases; empowerment research; evaluating LLMs critically (do they really reason or plan?); benchmarking XAI; and two Horizon projects on Agentic AI with focus on XAI.

Check your understanding