Ch. 12 — Reinforcement learning

1. What is reinforcement learning?

The chapter opens with a definition: “Reinforcement learning is the science of decision making. It is about learning the optimal behavior in an environment to obtain maximum reward.”

Reinforcement Learning (RL) is a general framework in which an agent learns to behave in an environment by performing actions and seeing the results of actions. For each good action, the agent gets positive feedback; for each bad action, it gets negative feedback or a penalty. The agent learns with the process of hit and trial, and based on the experience it learns to perform the task in a better way using feedback, without any labeled data. RL is an important model of how we (and all animals in general) learn: praise from our parents, grades in school, salary at work — these are all examples of rewards.

What characterizes reinforcement learning, according to the slides:

Objective: select actions to maximize total future reward.
No supervisor: a reward is given for each action instead.
Importance of time: RL pays much attention to sequential data, unlike other paradigms where you receive random inputs; the input at the next step will always be dependent on the input at the previous state.
The agent action affects its next input: after each action, the input at the next time step will be different depending on the particular action chosen.
Actions may have long term consequences: it may be better to sacrifice immediate reward to gain more long-term reward.
Delayed asynchronous rewards: each step does not necessarily have a reward; a reward may be delayed or given only after the completion of the entire task — a financial investment (may take months to mature), or refueling a helicopter (might prevent a crash in several hours).

2. Applications

The applications span fields far beyond games: game playing, robotics, adaptive control, business strategy, chemistry, manufacturing, finance sector, healthcare and education. The common thread: any setting where an agent must make a sequence of decisions under delayed feedback.

3. The vocabulary: agent, environment, state, reward, policy

Term	Meaning
Agent	an entity that can perceive/explore the environment and act upon it.
Environment	where the agent learns and decides what actions to perform; anything that the agent cannot change arbitrarily is part of the environment. Usually the environment is stochastic: the next state may be somewhat random.
Action	the moves taken by an agent within the environment.
State	a situation returned by the environment after each action — how the environment changes in response to the agent’s action. If only a partial description of the state is available to the agent, it is called an observation.
Reward	a scalar feedback returned to the agent from the environment when it performs specific actions.
Policy	a strategy applied by the agent for the next action based on the current state; it defines the agent’s behavior at a given time by mapping state to action.

4. The reinforcement learning process

At each time step t, the loop is:

the agent analyzes the current environment state s_t;
out of the possible actions, it chooses and executes an action a_t;
the environment receives the action a_t;
the environment emits a new state s_t+1;
the environment returns a scalar reward r_t+1;
the agent updates its knowledge with the reward r_t+1.

The goal of the agent is to maximize the reward in the long run.

_t reward r_t+1 state s_t+1 the agent acts, the environment reacts, the reward is the only teaching signal

Plate 12.1 — The RL loop. The agent picks an action from the current state; the environment answers with a new state and a scalar reward. There is no supervisor: the reward stream is the only feedback.

The PacMan example

RL is easily explained with PacMan: the goal of PacMan (the agent) is to eat the food in the grid while avoiding the ghosts. The grid is the interactive environment; PacMan can make four actions (up, down, left, right); it receives rewards for eating food and punishments if it gets killed by a ghost. The state is represented by the locations of PacMan, ghosts and food in the grid world; the total cumulative reward is PacMan winning the game.

5. Markov decision processes

The most common method to formalize an RL problem is to represent it as a Markov Decision Process (MDP): a mathematical framework to formalize RL in sequential decision making where outcomes are partly random and partly controlled by the decision maker.

A MDP is a tuple ⟨S, A, P_a, R_a⟩ where:

S is the set of states (state space);
A is the set of actions the agent can choose to take (action space);
P_a(s, s′) = p(s_t+1 = s′ | s_t = s, a_t = a) is the probability that action a in state s at time t will lead to state s′ at time t + 1;
R_a(s, s′) is the immediate reward received after transitioning from state s to state s′, due to action a.

The Markov property

A system has the Markov property if the current time step contains all the pertinent information about the state of the environment from previous time steps. In other words, the state transition depends upon the current state and current action, and not on the entire trajectory and actions that happened in the past:

p(s_t+1 | s_t, a_t, s_t−1, a_t−1, … s₀, a₀) = p(s_t+1 | s_t, a_t)

With respect to the PacMan game, this means PacMan would elect the action to take at a given time step t by considering only the state (e.g., its location) at that particular time step t.

_a(s₀,s₁) = 0.7, r = +1 a′: P_a′(s₁,s₃) = 0.9, r = +2 a″: P_a″(s₂,s₃) = 0.8, r = 0 self-loop: P = 0.3, r = 0 MDP = (S, A, P_a, R_a): the next state depends only on the current state and action (Markov property)

Plate 12.2 — A Markov decision process. Transitions carry probabilities P_a (partly random) and immediate rewards R_a (partly controlled). The Markov property says the whole past is summarized by the current state.

6. The reward hypothesis and discounting

A policy π is a function that specifies the action π(s) the agent will choose when in state s. In RL, the objective is to find the optimal policy π*, the policy that maximizes the long-term reward — called expected cumulative reward. The total future reward from time step t onward can be expressed as:

ℝ_t = ∑_i=t..∞ R_{π(s_i)}(s_i, s_i+1)

Because the environment is stochastic, the more into the future we go, the more the reward may diverge. For this reason, it is common to use the discounted future reward instead:

ℝ_t = ∑_i=t..∞ γ^i−t · R_{π(s_i)}(s_i, s_i+1)

where γ ∈ (0; 1) is a hyper-parameter called the discount factor (typically γ = 0.9). The more into the future the reward is, the less we take it into consideration:

if γ = 0, the strategy relies only on immediate rewards;
if 0 < γ < 1, the strategy balances between immediate and future rewards;
if γ = 1, the strategy supposes the environment is deterministic, with the same actions always resulting in the same rewards.

The discounted future reward at time step t can be expressed recursively as the sum of the immediate reward and the discounted future reward at time step t + 1:

ℝ_t = R_{π(s_t)}(s_t, s_t+1) + γ · ℝ_t+1

This recursion is the backbone of every value-based algorithm in this chapter.

7. Exploration vs exploitation

An agent needs to explore the environment to assess its reward structure. After some exploration, the agent might have found a set of apparently rewarding actions — but how can it be sure they are the best? When should it continue to explore, and when should it just exploit its existing knowledge?

To build an optimal policy, the agent faces the dilemma of exploring new states while maximizing its reward at the same time. This is the exploration vs exploitation dilemma:

exploration means exploring and capturing more information about the environment in the hope of finding better actions;
exploitation involves using the already known information to maximize the rewards.

The ε-greedy strategy

A simple choice proposed in the literature is ε-greedy: the agent randomly explores with probability ε while it takes the optimal action with probability 1 − ε (with 0 < ε < 1):

π(s) = π*(s)          with probability 1 − ε
         a random action  with probability ε

With partial or no knowledge about future rewards, the ε-greedy policy gives the best results, as it balances exploitation of current knowledge and exploration of unknown actions. Usually, ε starts close to 1 (more exploration) and decreases to 0 in steps as the agent learns more and more about the environment.

Plate 12.3 — The ε-greedy policy. Most of the time the agent follows the greedy action; with probability ε it picks a random action instead. As learning proceeds, ε decays from near 1 toward 0.

8. Two approaches: policy-based and value-based

How do we build an agent able to select the actions that maximize its expected cumulative reward — in other words, how do we train the agent to find the optimal policy π*? Two main approaches exist:

Policy-based methods: the agent directly learns a policy function mapping from each state to the best corresponding action (or a probability distribution over the set of possible actions). The agent learns which action to take, given a state.
Value-based methods: instead of a policy function, the agent learns a value function that maps a state or state-action pair to the expected value of being at that state. The agent learns which state is more valuable and takes the action that leads to it.

The rest of the chapter follows the value-based route, ending with its deep-learning incarnation.

9. Q-learning

Q-learning is the most commonly used RL value-based method. It defines a Q-function Q(s, a) representing the discounted future reward when the agent performs action a in state s and continues optimally from that point on:

Q(s_t, a_t) = max_{π(s_t)=a_t} ℝ_t+1

Q(s, a) represents the best possible reward at the end of the process after performing action a in state s. Given Q(s, a), the optimal policy π* at the current state s is simply:

π*(s) = argmax_a Q(s, a)

How can we estimate the reward at the end of the process if we know just the current state and action, and not the actions and rewards coming after? Instead of calculating for each state-action pair the expected cumulative reward (a long process), we can use the Bellman equation. Given a transition ⟨s_t, a_t, r_t+1, s_t+1⟩, Q(s_t, a_t) can be expressed in terms of the Q-value of the next state s_t+1:

Q(s_t, a_t) = r_t+1 + γ · max_a Q(s_t+1, a)

The maximum future reward for the current state and action is the immediate reward r_t+1 plus the maximum future reward for the next state s_t+1.

The main idea in Q-learning is to iteratively approximate the Q-values using the Bellman equation:

Q_new(s_t, a_t) ← (1 − α) · Q(s_t, a_t) + α · ( r_t+1 + γ · max_a Q(s_t+1, a) )

where α is the learning rate that controls how much of the current Q-value and the newly proposed Q-value is considered. When α = 1 the update is the same as the Bellman equation. α is generally initialized to 0.5 and progressively reduced during learning.

The Q-learning algorithm:

initialize Q[num_states, num_actions] arbitrarily
for m episodes
    t = 0
    repeat
        with probability ε select a random action a_t
        otherwise select a_t = argmax_a Q(s_t, a)
        execute action a_t and observe reward r_t+1 and new state s_t+1
        Q[s_t, a_t] = (1−α) · Q[s_t, a_t] + α · ( r_t+1 + γ · max_a Q[s_t+1, a] )
        t = t + 1
    until terminated
end for

Note that max_a Q[s_t+1, a] used to update Q[s_t, a_t] is only an estimation, and in the early stages of learning it may be completely wrong. However, the estimations get more and more accurate with every iteration, and the Q-function converges to the true Q-value.

_U Q(s,a_U) = 0 a_L Q = 0.5 a_D Q = 1 a_R Q = 0.5 π*(s) = argmax_a Q(s, a): here the down action, with Q = 1

Plate 12.4 — Acting greedily from a Q-function. From state s each action has an estimated Q(s, a); the optimal policy picks the action with the highest value (here a_D). During training, ε-greedy exploration still sometimes deviates from this choice.

10. The Q-learning worked example

The slides build a complete worked example — a building with 5 rooms numbered 0 to 4, where an agent is placed in any one of the rooms and the goal is to reach outside the building (number 5). The outside can be reached from rooms 1 and 4; a reward value is associated with each door. The hyper-parameter values are fixed: ε = 0, α = 0.5, γ = 0.9.

The problem can be represented by a graph where each room is a node (state) and each door is a link (action). Doors that directly connect to the outside (room 5) are given a positive reward (r₁₅ = 1, r₄₅ = 1, r₅₅ = 1). A matrix Q is constructed to represent the memory of what the agent has learned so far: the rows of Q represent the current state (room), the columns represent the possible actions (doors) leading to the next state. In the beginning all possible Q-values are initialized to 0; Q(s, a) = −1 means that action a cannot be executed from state s.

Plate 12.5 — The rooms problem. Six nodes, doors as actions, and a reward of 1 on the doors that reach the outside (5) directly. The agent learns a Q-table whose rows are rooms and whose columns are doors.

11. Deep reinforcement learning and DQN

In many practical decision-making problems, the states s are high-dimensional (e.g., images from a camera or raw sensor streams from a robot) and cannot be solved by traditional RL algorithms. Moreover, the amount of time required to explore each state to create the required Q-table would be unrealistic. Deep RL combines deep neural networks and RL to solve such problems, representing the policy π or other learned functions as a deep neural network. Deep RL algorithms can take in very large inputs (e.g., every pixel rendered to the screen in a video game) and decide what actions to perform to optimize an objective (e.g., maximizing the game score), without manual engineering of the state space.

One of the fundamental problems of Q-learning is that the amount of memory required to store data rapidly expands as the number of states increases. With deep Q-learning, the Q-values are estimated with neural networks: the neural network takes the state as input and outputs Q-values for all the different actions the agent might take.

The Deep Q-Network

In 2013, a small company called DeepMind (immediately bought by Google) developed the Deep Q-Network (DQN). DQN learned to play Atari video games by observing just the screen pixels and receiving a reward when the game score increased. It was trained on 49 different Atari games using the same algorithm, architecture and hyper-parameters, and reached human-level performance on 29 of them.

The DQN consists of three convolutional layers and two fully-connected layers. Note that there are no pooling layers, because they introduce translation invariance, and the network would become insensitive to the location of an object in the image. The input is the screen image (84 × 84 × 4: four grayscale, cropped and resized frames stacked to give the network a sense of motion). The first layer convolves 32 kernels of 8×8 with stride 4; the second 64 kernels of 4×4 with stride 2; the third 64 kernels of 3×3 with stride 1, all with ReLU, padding 0; then a fully-connected layer of 512 units with ReLU, and a linear output layer with one unit per action.

Input preprocessing: each frame is (1) transformed to grayscale, (2) cropped to select the region of interest, (3) resized to 84 × 84; and to solve the problem of temporal limitation and give the network the sense of motion, DQN takes a stack of four frames as input.

Plate 12.6 — The DQN. Three strided convolutional layers and two fully-connected layers map the four-frame state directly to one Q-value per action. No pooling anywhere: translation invariance would blur where objects are, which matters for Atari.

12. DQN loss, target and training

Q-values can be any real values, which makes this a regression task, optimized with a simple square error loss:

L = ( y_t − ŷ_t )²

where y_t and ŷ_t are the true (or target) and predicted values at time step t. Since the true value y_t is unknown (RL is unsupervised, no labels are available), it is estimated using the Bellman equation:

y_t = r_t+1 + γ · max_a Q(s_t+1, a)

Given a transition ⟨s_t, a_t, r_t+1, s_t+1⟩, the loss is computed as:

L = ( r_t+1 + γ · max_a Q(s_t+1, a) − Q(s_t, a_t) )²

where the first part is the true (or target) value and Q(s_t, a_t) is the predicted value. The Q-table update rule of Q-learning is replaced with the following procedure:

do a forward pass for the current state s_t to get predicted Q-values for all actions;
do a forward pass for the next state s_t+1 and calculate the maximum over all outputs (max_a Q(s_t+1, a));
set the target Q-value for action a_t to r_t+1 + γ · max_a Q(s_t+1, a); for all other actions set the Q-value target to the same value returned in step 1, making the error 0 for those outputs;
do a backward pass to update the weights.

The basic training algorithm:

initialize network Q with random weights
for m episodes
    t = 0
    repeat
        with probability ε select a random action a_t
        otherwise select a_t = argmax_a Q(s_t, a)
        execute action a_t and observe reward r_t+1 and new state s_t+1
        estimate the target value y_t = r_t+1 + γ · max_a Q(s_t+1, a)
        gradient descent step updating Q’s weights by minimizing l = ( y_t − Q(s_t, a_t) )²
        t = t + 1
    until terminated
end for

13. Experience replay

There is an issue when using a neural network as a Q approximator: the transitions are very correlated since they are all extracted from the same episode, reducing the overall variance of the estimates. As a result, the network tends to forget the previous transitions as it overwrites them with new ones, resulting in a network overfitted on the current episode. For instance, if we are in the first level and then in the second (which is totally different), the Mario agent can forget how to behave in the first level.

To remove correlations and make DQN training more stable, the experience replay technique can be used:

during training, all transitions ⟨s_t, a_t, r_t+1, s_t+1⟩ are stored in a replay memory;
when updating the network, mini-batches are randomly sampled from the replay memory and used instead of the most recent transition.

The three benefits highlighted by the slides:

experience replay breaks the correlation of subsequent training samples, which otherwise might drive the network into a local minimum;
each transition is potentially used in many weight updates, which allows for greater data efficiency;
by using experience replay, the behavior distribution is averaged over many of its previous states, smoothing out learning and avoiding oscillations or divergence in the parameters.

14. Fixed Q-value targets

As discussed before, both the predicted and target values used to calculate the loss are estimated using the network itself. Therefore there is a big correlation between the target value and the network weights to update: at every step of training the Q-values shift, but the target value also shifts. While Q-values get closer to the target, the target is also moving — leading to a big oscillation during training (the slides draw it as a dog chasing its own tail).

To solve the moving-target problem, two different DQNs can be used:

the action network Q, used to move the agent, is updated every u steps;
the target network Q̂ (a clone of Q) is used only to define the targets, and is updated every c steps (c ≫ u) by replacing its weights with those of the action network.

In practice, a snapshot of the network weights from a few iterations before is used instead of the last iteration. Generating the targets using an older set of weights adds a delay between the time an update to Q is made and the time the update affects the targets, making divergence or oscillations much more unlikely. Moreover, since the target network is updated much less often than the action network, the Q-value targets are more stable. The loss becomes:

L = ( r_t+1 + γ · max_a Q̂(s_t+1, a) − Q(s_t, a_t) )²

The improved training algorithm:

initialize replay memory D to capacity N
initialize action and target networks Q and Q̂ with the same random weights
T = 0
for m episodes
    t = 0
    repeat
        with probability ε select a random action a_t
        otherwise select a_t = argmax_a Q(s_t, a)
        execute action a_t and observe reward r_t+1 and new state s_t+1
        store transition ⟨s_t, a_t, r_t+1, s_t+1⟩ in D
        if T mod u = 0 AND D ≥ b_s then
            sample a random mini-batch m_b of b_s transitions from D
            for each transition ⟨s_i, a_i, r_i+1, s_i+1⟩ in m_b
                estimate the target value y_i = r_i+1 + γ · max_a Q̂(s_i+1, a)
                calculate the loss L_i = ( y_i − Q(s_i, a_i) )²
            end for
            gradient descent step updating Q by minimizing C = (1/b_s) · ∑_{i=1..b_s} L_i
        end if
        if T mod c = 0 then copy weights from Q to Q̂
        t = t + 1
        T = T + 1
    until terminated
end for

_t,a_t,r_t+1,s_t+1⟩ action net Q target net Q̂ mini-batch sampled at random (breaks correlation, reuses transitions) weights cloned every c steps targets are computed with the frozen clone Q̂, so the target stops moving while Q learns

Plate 12.7 — Stabilizing DQN. Transitions accumulate in a replay memory from which mini-batches are drawn at random; the target values come from a clone Q̂ whose weights are refreshed from Q only every c steps. Correlated updates are broken and the moving-target oscillation is removed.

15. Chapter summary

RL is a general-purpose framework to discover which action yields the highest reward over the longer period; there is no supervisor, only an agent that learns from the environment through trial and error, receiving rewards as feedback.
The fundamental concepts are agent, state, reward, environment, action and policy.
An RL problem can be formulated as a Markov decision process satisfying the Markov property (“future is independent of the past given the present”); the goal is to maximize the expected cumulative reward while still exploring.
Rewards are usually discounted by γ: rewards that come sooner are more probable and more predictable than long-term ones.
The optimal policy can be found directly (policy-based) or through a value function (value-based).
Q-learning is a value-based algorithm that iteratively estimates Q(s, a) with the Bellman equation; the optimal policy is argmax_a Q(s, a).
Q-learning’s fundamental problem is the memory needed for the Q-table as states and actions grow; deep Q-learning approximates Q(s, a) with a deep neural network (DQN), taking raw high-dimensional states as input.
As in supervised learning, DQN uses a square error loss, but the true value is unknown: it is estimated by the DQN itself through the Bellman equation.
Experience replay stores transitions and samples random mini-batches, reducing correlation and stabilizing training.
Using the same model for predictions and targets makes the target move while the network learns; two networks (action Q and target Q̂) fix this: the target network is updated much less often and is used only to define targets.

Check your understanding

What characterizes reinforcement learning with respect to supervised learning?

There is no supervisor and no labeled data: the agent learns by trial and error through feedback (rewards and penalties). RL pays much attention to sequential data (the input at the next step depends on the previous state), the agent’s action affects its next input, actions may have long-term consequences, and rewards may be delayed or asynchronous. The objective is to select actions that maximize total future reward.

Define agent, environment, action, state, observation, reward and policy.

Agent: the entity that perceives/explores the environment and acts upon it. Environment: where the agent learns and decides; anything the agent cannot change arbitrarily, usually stochastic. Action: the moves taken by the agent. State: the situation returned by the environment after each action (a partial description is an observation). Reward: the scalar feedback returned for specific actions. Policy: the strategy mapping state to action.

Describe the RL loop at a single time step.

The agent analyzes the current state s_t, chooses and executes an action a_t; the environment receives the action, emits a new state s_t+1 and returns a scalar reward r_t+1; the agent updates its knowledge with the reward. The goal is to maximize the reward in the long run.

What is an MDP and what is the Markov property?

An MDP is the tuple ⟨S, A, P_a, R_a⟩: state space, action space, transition probabilities P_a(s, s′) = p(s_t+1=s′ | s_t=s, a_t=a), and immediate rewards R_a(s, s′). The Markov property holds when the current time step contains all pertinent information: p(s_t+1 | s_t, a_t, s_t−1, a_t−1, …) = p(s_t+1 | s_t, a_t) — the future is independent of the past given the present.

What is the reward hypothesis, and what does the discount factor do?

The objective of RL is to find the optimal policy π* that maximizes the expected cumulative (discounted) reward. Because future rewards are uncertain, they are discounted: ℝ_t = ∑_i≥t γ^i−t R(·) with typically γ = 0.9. If γ = 0 only immediate rewards count; if γ = 1 the environment is treated as deterministic; between 0 and 1 there is a balance. The discounted return is recursive: ℝ_t = R_t + γℝ_t+1.

Explain the exploration vs exploitation dilemma and the ε-greedy strategy.

The agent must explore to discover the reward structure but exploit known-good actions to maximize reward — the two goals conflict. ε-greedy resolves it: with probability ε pick a random action (exploration), otherwise pick argmax_a Q(s, a) (exploitation). Usually ε starts near 1 and decays toward 0 as the agent learns.

Compare policy-based and value-based methods.

Policy-based methods directly learn a policy function mapping each state to the best action (or a distribution over actions): the agent learns which action to take. Value-based methods learn a value function mapping states (or state-action pairs) to expected values: the agent learns which state is more valuable and takes the action that leads to it. Q-learning is the most common value-based method.

Write the Bellman equation and the iterative Q-learning update, explaining each symbol.

Bellman: Q(s_t, a_t) = r_t+1 + γ · max_a Q(s_t+1, a). Iterative update: Q_new(s_t, a_t) ← (1−α)Q(s_t, a_t) + α(r_t+1 + γ max_a Q(s_t+1, a)), where α is the learning rate (how much of the new estimate is taken; α = 1 reproduces the Bellman equation, usually initialized at 0.5 and reduced), γ the discount factor, and the max is over actions of the next state. The optimal policy is π*(s) = argmax_a Q(s, a).

Walk through one update of the rooms example (episode 1).

Starting state s₀ = 1, two actions (a₁₃, a₁₅) with Q-values 0, so the action is randomly selected: a₁₅. Reward r₁ = 1, new state s₁ = 5 (goal). Update: Q(1, a₁₅) = 0.5·0 + 0.5·(1 + 0.9·0) = 0.5 (maxQ(5, a) = 0 since row 5 is all zeros). The episode ends because state 5 is the goal. Later episodes propagate value backward: Q(3, a₃₁) = 0.22, then Q(1, a₁₅) = 0.75, then Q(2, a₂₃) = 0.1, Q(3, a₃₁) = 0.45, Q(1, a₁₅) = 0.88, converging to the optimal Q-table (e.g., row 1: a₁₃ = 8.1, a₁₅ = 10).

Why does DQN use a neural network, and what architecture does it use?

Because high-dimensional states (e.g., raw screen pixels) cannot be explored state by state to build a Q-table: memory grows with the number of states and exploration would be unrealistic. DQN takes the state as input and outputs Q-values for all actions. The architecture: three convolutional layers (32×8×8 stride 4, 64×4×4 stride 2, 64×3×3 stride 1, all ReLU, no padding) and two fully-connected layers (512 units with ReLU, then a linear output). There are no pooling layers because pooling’s translation invariance would make the network insensitive to object location. Input: four 84×84 grayscale, cropped frames stacked for a sense of motion.

What is the DQN loss, and how is the target obtained without labels?

The loss is the square error L = (y_t − Q(s_t, a_t))². The true value is unknown, so the target is estimated with the Bellman equation: y_t = r_t+1 + γ max_a Q(s_t+1, a). In the 4-step procedure: forward pass on s_t for all Q-values; forward pass on s_t+1 for the max; set the target for a_t to the Bellman estimate and for all other actions to the current predictions (zero error); backward pass.

Why experience replay, and why fixed Q-value targets?

Experience replay: consecutive transitions from one episode are highly correlated and the network overfits to the current episode (Mario forgets level 1 while playing level 2). Storing all transitions in a replay memory and sampling random mini-batches breaks the correlation, reuses each transition in many updates (data efficiency), and averages the behavior distribution over past states (stability). Fixed Q-value targets: when the same network computes predictions and targets, the target moves as the weights update (a dog chasing its own tail), causing oscillation. Using a clone Q̂ updated every c steps (c ≫ u) for targets only adds a delay between an update to Q and its effect on the targets, making training stable.

In this chapter