Simple reflex agents select actions on the basis of the current percept, ignoring the rest of the percept history. They have no state: the same percept always produces the same action, through condition–action rules. This is the architecture embodied by the classic subsumption architecture line of work and by rule-based reactive systems.
Plate 13.1 — A simple reflex agent maps the current percept directly to an action through condition–action rules; the percept history is ignored and no internal state is kept.
2. Subsumption and layered architectures
The subsumption architecture is the best-known architecture for reactive agents, developed by Rodney Brooks. The agent’s decision-making is realized through a set of task-accomplishing behaviours arranged into hierarchical layers:
each behaviour may be thought of as an individual action selection function, which continually takes perceptual input and maps it to an action to perform;
higher layers (lower priority) represent more abstract behaviours; lower layers have higher priority;
e.g. in a mobile robot, the behaviour ‘avoid obstacles’ is given high priority and encoded in a low-level layer, while ‘explore’ sits in a higher, lower-priority layer.
When the layers are combined, higher layers subsume the lower ones when appropriate, giving reactivity with a hierarchical organization of behaviours.
Plate 13.2 — The subsumption architecture: task-accomplishing behaviours in hierarchical layers. Lower layers carry higher priority (e.g. avoid obstacles), higher layers are more abstract and subsume them when appropriate.
The general family of layered agent architectures organizes behaviours, or more generally reasoning capabilities, into layers that interact through defined interfaces — the subsumption architecture being the reactive member of the family.
3. Model-based reflex agents
Model-based reflex agents overcome the statelessness of simple reflex agents: the agent has an internal state that depends on the percept history, used to reflect at least some of the unobserved aspects of the current state. The internal state is updated as time goes by, using two kinds of knowledge:
Model of the world — “how the world works”: how the world changes over time, the effects of the agent’s actions, how the world evolves independently of the agent.
Sensor model — how the state of the world is reflected in the agent’s percepts.
Together, the transition model and the sensor model allow an agent to keep track of the state of the world, to the extent possible given the limitations of its sensors.
Plate 13.3 — A model-based reflex agent maintains an internal state, updated from percepts using a model of the world and a sensor model; the condition–action rules then fire on the internal state rather than on the raw percept.
4. Goal-based agents and the sense-plan-act cycle
A model-based goal-based agent has an explicit knowledge and representation of the goal/task to accomplish. The agent program combines this with the model (the same information used by the model-based reflex agent) to choose actions that achieve the goal. Search and planning are the subfields of AI devoted to finding action sequences that achieve the agent’s goals.
Decision making of this kind is fundamentally different from condition–action rules: it involves consideration of the future. The classic realization is the Sense-Plan-Act architecture: sense the environment, plan a course of action that achieves the goal, act on it; then repeat.
Plate 13.4 — The goal-based agent decides with an eye on the future: search and planning find action sequences that achieve the explicit goal, realized through the sense-plan-act cycle.
5. Hybrid architectures: TouringMachines and InterRRaP
Reactive and deliberative layers each have weaknesses: pure reactivity cannot pursue long-term goals; pure deliberation is too slow for dynamic environments. Hybrid architectures combine both in layers, with a control mechanism deciding which layer drives behaviour at each moment. Two classic examples:
TouringMachines — layers for reactive, planning and modelling behaviour, coordinated by a control subsystem that can suppress and censor layer outputs.
InterRRaP — a layered architecture with a social layer (cooperation), a local planning layer and a reactive behaviour layer, with hierarchical control and knowledge bases at each level.
Why this matters for Chapter 14
The layering tension — react while deliberating, deliberate while reacting — is resolved in the BDI architecture by interleaving plan selection and action execution. The BDI reasoning cycle of Chapter 14 is, in a precise sense, the mature answer to the hybrid-architecture problem.
6. Utility-based agents
In many real-world scenarios, goals alone are not enough to generate high-quality behaviour: there may be different action sequences achieving the goal, some evaluated better than others with respect to the performance measure — goals provide only a crude binary distinction between “happy” and “unhappy” states. A more general performance measure allows a comparison of different world states according to exactly how happy they would make the agent. Because “happy” does not sound scientific, economists and computer scientists use the term utility.
An agent’s utility function is an internalization of the performance measure: an agent that chooses actions to maximize its utility will be rational according to the external performance measure.
The notion of utility is essential for decision making under uncertainty — e.g. partially observable environments or conflicting goals.
A rational utility-based agent chooses the action that maximizes the expected utility of the action outcomes: the utility the agent expects to derive, on average, given the probabilities and utilities of each outcome.
Plate 13.5 — Utility-based agents internalize the performance measure as a utility function and choose the action maximizing expected utility, handling uncertainty from partial observability or conflicting goals.
7. Learning agents: the four components
The central question: how do the agent programs come into being? In his famous early paper, Turing (1950) considers programming his intelligent machines by hand, estimates the work, and concludes that “some more expeditious method seems desirable” — the method he proposes is to build learning machines and then to teach them. In many areas of AI this is now the preferred method for creating state-of-the-art systems; any type of agent (model-based, goal-based, utility-based, etc.) can be built as a learning agent or not.
A learning agent can be divided into four conceptual components:
Performance element — responsible for selecting external actions; what we have previously considered to be the entire agent (it takes percepts in and decides on actions).
Critic component — tells the learning element how well the agent is doing with respect to a fixed performance standard; necessary because percepts themselves provide no indication of the agent’s success (a chess program receives a checkmate percept, but needs a performance standard to know that this is good). The performance standard must be fixed — conceptually outside the agent, since the agent must not modify it to fit its own behaviour.
Learning element — responsible for making improvements: it uses feedback from the critic and determines how the performance element should be modified to do better in the future. Its design depends very much on the design of the performance element: the first question is not “How am I going to get it to learn this?” but “What kind of performance element will my agent use once it has learned how?”
Problem generator — suggests actions that will lead to new and informative experiences: if the performance element had its way it would keep doing the best actions it knows, but exploring a little — doing possibly suboptimal actions in the short run — may discover much better actions for the long run. This is what scientists do when they carry out experiments.
Plate 13.6 — The learning agent in four components: performance element (acts), critic (evaluates against a fixed standard), learning element (improves), and problem generator (explores). After Russell & Norvig.
8. Reinforcement learning agents as an example
Reinforcement Learning (RL) agents instantiate the four components concretely. The policy π(s) selects the action; a state estimator may be needed if percepts ≠ states. Typical implementation:
state s(t) = f(percept(t), internal_state);
action selection — exploit: a(t) = arg maxa Q(s(t), a), optionally combined with exploration from the problem generator (e.g. ε-greedy).
The four components map onto RL machinery:
Critic element — evaluates how well the agent is doing: in RL it receives (state, action, reward, next state) and produces an evaluation signal (e.g. the TD error δ) sent to the learning element; positive δ means “better than expected”, negative “worse than expected”.
Learning element — updates the policy/value function so future action choices improve: it uses the critic’s signal to update internal parameters (Q-table, neural network weights); e.g. tabular Q-learning updates Q(s,a) from δt.
Problem generator — injects exploratory behaviour into the policy to discover better actions/states: it decides when to explore (choose non-greedy actions) balancing exploration vs exploitation; typical implementation — ε-greedy (with probability ε choose a random action, with probability 1−ε choose the greedy one), or more advanced: softmax, UCB, curiosity bonuses.
Plate 13.7 — An RL agent instantiates the learning-agent components: the policy exploits Q-values, the critic produces the TD error δ, the learning element updates Q, and the problem generator injects exploration (e.g. ε-greedy).
9. AI agents and language agents
AI agents / “language agents” — also called LLM-based autonomous agents or “generative agents” — are the “AI Agents” of Agentic AI: agent systems whose reasoning core is a large language model. The reference model used by the module is Sumers, Yao, Narasimhan and Griffiths, Cognitive Architectures for Language Agents (TMLR, 02/2024), which explicitly frames language agents as a species of cognitive architecture — the connection to Chapter 14.
Their architecture is based on prompt-engineering techniques and runs a continuous cycle of perception, reasoning, planning, action and reflection (see Plate 13.8).
Plate 13.8 — The AI-agent components from the Google Agent SDK: a continuous cycle of perception, reasoning, planning, action and reflection, with learning and improvement over time.
10. AI agent components and agent SDKs
Several in-progress technologies (libraries, frameworks, platforms) support building such agents. Main examples:
Google Agent Development Kit (google.github.io/adk-docs).
Microsoft 365 Agents SDK (github.com/microsoft/Agents).
The programming style is declarative configuration around an LLM core, with tools, instructions and input schemas. The course’s example comes from Google ADK (2025):
Note the architectural echoes: instruction is a goal in natural language, tools are the action repertoire, and inputSchema defines the agent’s interface with the world — the vocabulary of Chapter 12 reappears in the agentic wave.
11. Emerging standards: MCP and A2A
Two emerging “standards” frame the agentic wave:
Model Context Protocol (MCP) — an open standard and open-source framework introduced by Anthropic in November 2024 to standardize the way AI systems like LLMs integrate and share data with external tools, systems, and data sources; adopted by major AI providers, including OpenAI and Google DeepMind (modelcontextprotocol.io).
Agent to Agent (A2A) protocol — an open standard proposed by Google to enable seamless communication and collaboration between AI agents (a2a-protocol.org).
MCP and A2A are complementary: MCP connects an agent to its tools and data sources (the agent–environment interface), while A2A connects agents to each other (the agent–agent interaction). Chapter 15 will revisit this complementarity from the MAS side, contrasting “tool”/MCP with artifacts (A&A) and A2A with agent communication languages.
Plate 13.9 — MCP standardizes how agents integrate tools and data sources; A2A standardizes communication between agents. The two protocols are complementary rather than competing.
Check your understanding
Describe simple reflex agents and the subsumption architecture.
Simple reflex agents select actions on the basis of the current percept, ignoring percept history, with no state — condition–action rules. The subsumption architecture (Brooks) realizes reactive decision-making through task-accomplishing behaviours arranged in hierarchical layers; each behaviour maps perceptual input to an action, higher layers (lower priority) are more abstract, and low-level layers carry high priority (e.g. avoid obstacles).
How do model-based reflex agents keep track of the world?
They maintain an internal state that depends on the percept history, reflecting unobserved aspects of the current state. The state is updated over time using two kinds of knowledge: a model of the world (how the world changes, effects of the agent’s actions, how it evolves independently) and a sensor model (how the world state is reflected in percepts). Together they let the agent track the world to the extent its sensors allow.
What distinguishes goal-based agents, and what is the sense-plan-act cycle?
Goal-based agents have an explicit representation of the goal/task and combine it with the world model to choose actions that achieve it; decision making involves consideration of the future. Search and planning are the AI subfields that find action sequences achieving goals. The sense-plan-act cycle realizes this: sense the environment, plan a course of action, act, and repeat.
Why do utility-based agents need a utility function rather than just goals?
Goals give only a crude binary distinction between “happy” and “unhappy” states, while different action sequences achieving a goal may be evaluated differently by the performance measure. The utility function internalizes the performance measure, ranking states by how good they are, and is essential for decision making under uncertainty (partial observability, conflicting goals). A rational utility-based agent chooses the action maximizing expected utility of outcomes.
Name and describe the four components of a learning agent.
Performance element — selects external actions (what we previously called the whole agent). Critic — tells the learning element how well the agent is doing against a fixed performance standard (percepts alone do not indicate success). Learning element — uses critic feedback to modify the performance element; its design depends on the performance element’s design. Problem generator — suggests exploratory actions leading to new, informative experiences, balancing short-run suboptimality against long-run discovery.
How do the four learning-agent components map onto reinforcement learning?
The policy π(s) is the performance element, selecting actions (exploit: argmaxa Q(s,a)). The critic receives (state, action, reward, next state) and produces an evaluation signal — the TD error δ, positive meaning “better than expected”, negative “worse”. The learning element uses δ to update internal parameters (Q-table or network weights), e.g. tabular Q-learning. The problem generator injects exploration into the policy (ε-greedy, softmax, UCB, curiosity bonuses), balancing exploration vs exploitation.
What are AI agents / language agents, and what cycle do they run?
AI agents (LLM-based autonomous agents, generative agents) are agent systems whose reasoning core is a large language model, framed as cognitive architectures for language agents (Sumers et al., TMLR 2024). Their architecture is based on prompt-engineering techniques and runs a continuous cycle: perception (gather information from sensors, databases, UIs), reasoning (LLM analyzes the data and formulates solutions), planning (set goals, break them into steps), action (perform tasks, interact with systems), and reflection (evaluate results and adjust future plans).
What are MCP and A2A, and why are they complementary?
MCP (Model Context Protocol, Anthropic, Nov 2024) is an open standard for how AI systems integrate and share data with external tools, systems and data sources — the agent–environment interface. A2A (Agent to Agent, Google) is an open standard for communication and collaboration between AI agents — the agent–agent interface. They are complementary because they standardize two orthogonal axes of the same system: tools/data access and inter-agent communication.
Why does the module call LLM-based agent programming “the newest kind of agent program”?
Because the same abstraction holds: an explicit goal (the instruction), an environment (tools, data sources, input schema), autonomy (the LLM decides the steps), perception (tools/sensors), and action (tool calls). The declarative LlmAgent example from Google ADK shows goals, action repertoires and interfaces reappearing in the agentic wave — the vocabulary of Chapter 12 remains the vocabulary of AI agents.