The chapter opens with the definition that started the revolution, from the paper “Attention is All You Need” (A. Vaswani et al., NIPS, December 2017):
“Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.”
Today, transformers are used in many modelling language understanding and time series tasks, including: machine translation, document summarization, document generation, named entity recognition, biological sequence analysis, image captioning and video understanding.
Human cognitive attention allows us to focus on a certain region with “high resolution” while perceiving the surrounding image in “low resolution”, and then adjust the focal point. Similarly, we can explain the relationship between words in a sentence: some words are important for the meaning of the sentence, others are almost decorative.
In artificial neural networks, attention mechanisms simulate human cognitive attention by focusing on some parts of the input data while ignoring (or blurring) other parts. Attention is one component of an artificial neural network in charge of managing and quantifying the interdependence between the input and output elements. In other words, attention can be interpreted as a mechanism by which a network can weigh features by level of importance to a task and use this weighting to help achieve the task.
For instance, in a translation task, for each word in the output sentence the most relevant words from the input sentence are identified and higher weights are assigned to these words, enhancing the accuracy of the output prediction.
A second quote from the same paper fixes the definition: “Self-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.”
The slides build a sequence classifier trained to predict whether a restaurant review is positive or negative. If each word can only contribute to the output score independently of the others, then the word terrible would probably cause the classifier to predict a negative review. But consider the sentence:
This restaurant was not too terrible
To obtain a positive review, the model needs to recognize that the meaning of the word terrible is moderated by the words not and too. The problem: how to assign “influence scores” systematically?
Self-Attention, or intra-attention, is a sequence-to-sequence operation that takes in n inputs xi ∈ ℝd (with 1 ≤ i ≤ n) and returns n outputs yi ∈ ℝd. The mechanism allows the inputs to interact with each other (“self”) and find out who they should pay more attention to (“attention”). The outputs are aggregates of these interactions and attention scores.
Self-attention enables us to find correlations between different input elements indicating the syntactic and contextual structure of the sequence. The idea is to let each input build its hidden representation by focusing on some of the other inputs.
The computation is fully specified by three equations:
yi = ∑j ωij · xj (the output is a weighted sum of all inputs)
ω′ij = xiT · xj / √d (raw score: how aligned are i and j)
ωij = eω′ij / ∑j eω′ij (softmax normalization over j)The raw scores are the dot products between pairs of inputs, scaled by √d; the softmax turns each row of scores into a distribution over the sequence. If the weight ωterrible, not is high, the word not influences the meaning of terrible in the output sequence yterrible.
Self-attention as defined so far has two serious limitations.
Self-attention is permutation equivariant: it sees the inputs as a set, not a sequence. If we permute the input sequence, the output sequence is exactly the same, but permuted. This poses challenges in applications (e.g., NLP) where permutations may result in completely different meanings. The slides make it concrete: the architecture will necessarily return the same output label for the two sentences
although their meanings are opposite. A global-pooling classifier over the self-attention output cannot tell them apart because the multiset of representations is identical.
With only dot products of the inputs themselves, the model is not able to learn deeper connections between inputs. And in many sequences there are multiple relations to model at the same time: in “This restaurant was not too terrible”, not inverts, too moderates, and the whole phrase is a property of the restaurant. One fixed similarity function cannot represent all of these simultaneously.
To overcome permutation equivariance and exploit the sequential information, a model needs to be able to access the position of the inputs in the sequence. Positional encoding maps the location of an input in a sequence to a vector so that each position is assigned a unique representation.
Given a sequence of n inputs of size d, the positional vector of position k (pk ∈ ℝd) is obtained using sine and cosine functions of varying frequencies:
pk(2i+1) = sin( (k−1) / δ2i/d )
pk(2i+2) = cos( (k−1) / δ2i/d )
with 1 ≤ k ≤ n, 0 ≤ i < d/2, δ = 10000The output of the positional encoding layer is a sequence of n positional encodings obtained as the sum of each input of the sequence with the corresponding positional vector: ek = xk + pk. The scheme has three advantages:
[−1, 1], which keeps the values of the positional encoding in a normalized range;In self-attention, each input xi plays three distinct roles:
qi) — it is compared with all other inputs to construct weights (ωi1, …, ωin) for its own output yi;ki) — it is compared with every other input xj to construct weights (ω1i, …, ωni) for their output yj;vi) — it is used to finally synthesize each actual output y1, …, yn.Self-attention is similar to a soft dictionary: every key (kj) matches the query (qi) to some extent, as determined by their dot product (ω′ij); a mixture of all values (vj) is returned, with softmax-normalized dot products (ωij) as mixture weights.
To learn the deepest connections between inputs, trainable weights are introduced at three locations, corresponding to the three roles of the inputs:
qi = Wq xi , qi ∈ ℝdₖ
ki = Wk xi , ki ∈ ℝdₖ
vi = Wv xi , vi ∈ ℝdᵥwhere Wq, Wk ∈ ℝdₖ×d and Wv ∈ ℝdᵥ×d are learnable projection matrices defining the roles of the inputs. Therefore we get:
yi = ∑j ωij · vj , yi ∈ ℝdᵥ
ω′ij = qiT · kj / √dₖThe slides illustrate the roles with the sentence “the ball sinks into the basket”. For yball, the query qTball is compared with every key (kthe, kball, …, kbasket), and the softmax weights blend the corresponding values: ball collects what it needs from sinks and basket to know its role. The same procedure runs for ybasket with its own query.
Recall the limitation: in many sequences there are multiple relations to model at the same time. In “This restaurant was not too terrible”, one relation inverts, another moderates, a third captures property-of. The idea behind multi-head self-attention is that multiple relations are best captured by different self-attention operations.
A multi-head self-attention contains h self-attention operations that work in parallel. This is the same analogy as using multiple filters in a convolutional layer (Chapters 6–7): each head gets its own projections Wq, Wk, Wv and therefore its own notion of what “related” means.
Given a multi-head self-attention with h self-attention layers, they return h outputs yji ∈ ℝdᵥ (with 1 ≤ j ≤ h) for each input xi. Since the number and size of the outputs must equal the number and size of the inputs:
xi, the corresponding h outputs (yji) are concatenated into a single vector ci ∈ ℝdᵥ·h;ci is projected into a d-dimensional space through a linear layer: oi = Wo ci, with Wo ∈ ℝd×(dᵥ·h), oi ∈ ℝd.The original transformer architecture follows an encoder-decoder structure. The encoder takes the input sequence and maps it to a latent representation of the whole sequence; the output of the encoder is then passed to a decoder which unpacks it to the desired target sequence (for instance, the same sentence in another language).
The slides translate “a dog on a skateboard” into Italian: the encoder receives the input sequence a dog on a skateboard; the decoder receives the shifted output sequence <start> un cane su uno skateboard and produces un cane su uno skateboard <end>, using the encoded sequence from the encoder.
The encoder consists of a stack of six identical layers composed of two sublayers:
The decoder consists of a stack of six identical layers composed of three sublayers:
In the training phase, it is important that the self-attention output at position k is computed by paying attention only to the elements in the input sequence before that position, working as an autoregressive model. Otherwise the decoder could “cheat” by looking at future elements of the target sentence it is supposed to predict.
Therefore, all weights ωkj with j > k are set to 0 by imposing ω′kj = −∞ before the softmax. Since e−∞ = 0, the softmax gives those positions exactly zero weight, and only input elements that precede the output position are considered in the computation of the k-th output element.
Training is carried out through self-supervised learning: the model is trained using the data itself to generate supervisory signals (e.g., labels); an unsupervised problem is transformed into a supervised problem by auto-generating the labels. For a translation model, the parallel corpus provides the target sentence; the decoder is trained to predict each next token from the shifted target, with masking hiding the future.
To regularize the training and avoid overfitting, dropout (p = 0.1) is applied before each addition with the residual connections. At the end of the training, the transformer can be used to creatively complete or generate texts.
Given an input sequence, the model iteratively:
From their introduction, transformers drove a revolution in artificial intelligence and especially in Natural Language Processing (NLP), and enabled modern Large Language Models (LLMs) such as Generative Pre-trained Transformer (GPT, OpenAI). Other leading ICT companies released their own LLMs: Gemini (Google), Claude (Anthropic), Grok (xAI), DeepSeek (DeepSeek-AI), and LLaMA (Meta). Recently, transformers also emerged as a competitive alternative to CNNs in different image recognition tasks (the Vision Transformer, ViT).
Modern LLMs adopt a decoder-only transformer because it directly matches the autoregressive objective of predicting the next token. The arguments from the slides:
| Model | Year | Layers × heads | Parameters | Training corpus |
|---|---|---|---|---|
| GPT-1 | 2018 | 12 decoder layers · 12 heads | ~120 million | — |
| GPT-2 (XL) | 2019 | 48 decoder layers · 25 heads | ~1.5 billion | — |
| GPT-3 | 2020 | 96 decoder layers · 96 heads | ~175 billion | ~500 billion tokens, ~1.3 GWh |
| GPT-4 | 2023 | details not available | assumed > 1 trillion | ~16 trillion tokens, ~50 GWh |
To put the energy numbers in perspective: GPT-3 was trained on about 500 billion tokens requiring about 1.3 GWh (the electricity consumed by 120 US homes in a year); GPT-4 on about 16 trillion tokens requiring about 50 GWh. The International Energy Agency (IEA) estimated that the average electricity demand of a typical web search would increase tenfold (10×) using GPT technology (2.9 Wh) instead of a traditional search engine like Google (0.3 Wh).
In 2022 OpenAI developed a chatbot called ChatGPT built on top of GPT-3, fine-tuned using both supervised learning and reinforcement learning techniques (a preview of Chapter 12). It is capable of generating human-like text with a wide range of applications: debugging code, writing code for a particular problem, playing simple games, explaining things, getting ideas for art, decoration and themes, writing music, translations, and solving math questions.
GPT-4 exhibits unexpected reasoning and problem-solving skills although the model has been trained only to predict the next word: summarization, mathematical and problem-solving abilities, common sense, and coding. The slides show a worked template solution for the tomato-harvest problem (“Andy harvests all the tomatoes from 18 plants that have 7 tomatoes each…”) expressed symbolically (P × T, D = P×T/2, M = (P×T − D)/3, L = P×T − D − M), a stable stacking plan for a book, nine eggs, a laptop, a bottle and a nail, and a complete 3D game in HTML/Javascript.
On the other hand, it often makes mistakes:
In 2021, the Google Research Brain Team proposed ViT, the first transformer for image recognition trained on ImageNet able to obtain results comparable to CNNs. The obstacle to applying transformers to images is scale: an image can contain thousands to millions of pixels, and in a transformer each pixel does a pairwise operation with every other pixel — a huge task even with multiple GPUs.
The authors solve this problem by splitting the input image into fixed-size non-overlapping patches. Then:
The model learns to encode the relative location of the image patches to reconstruct the structure of the image — the patch order plays the role that word order plays in text.
Attention mechanisms simulate human cognitive attention by focusing on some parts of the input data while ignoring (or blurring) other parts. Attention is the component in charge of managing and quantifying the interdependence between input and output elements: a network can weigh features by their level of importance to a task and use this weighting to help achieve the task.
yi = ∑j ωij · xj (each output is a weighted sum of all inputs); ω′ij = xiTxj / √d (scaled dot product scores); ωij = eω′ij / ∑j eω′ij (softmax normalization over j).
Self-attention sees its inputs as a set: permuting the input sequence yields exactly the same output sequence, permuted. Therefore a global-pooling classifier returns the same label for “This is not a real restaurant, it is a filthy burger joint” and “This is not a filthy burger joint, it is a real restaurant”, even though the meanings are opposite. The fix is positional encoding, which injects order information.
Because the model is not able to learn deeper connections between inputs: the scores are fixed dot products of the inputs themselves. Many sequences contain multiple simultaneous relations (in “This restaurant was not too terrible”: not inverts, too moderates, and the phrase is a property of the restaurant) that one fixed similarity function cannot represent. Trainable projections (queries, keys, values) and multiple heads address this.
Each position k gets a vector pk of size d built from sine and cosine functions of varying frequencies (p(2i+1) = sin((k−1)/δ2i/d), p(2i+2) = cos((k−1)/δ2i/d), δ = 10000), and is added to the corresponding input: ek = xk + pk. Advantages: values stay in [−1, 1] (normalized range), each position is encoded uniquely, and similarity between positions quantifies relative order.
Query qi = Wqxi: compared with all other inputs to build the weights for its own output. Key ki = Wkxi: compared with every other query to build weights for their outputs. Value vi = Wvxi: the content actually mixed by the weights. Self-attention is a soft dictionary: keys match the query by dot product, and a softmax-weighted mixture of values is returned.
Because multiple relations (invert, moderate, property-of) are best captured by different self-attention operations — the same analogy as multiple filters in a convolutional layer. Each head runs in parallel with its own projections; the h outputs for each input are concatenated into ci ∈ ℝdᵥ·h and projected by a linear layer oi = Woci back to the input dimension.
Six identical layers, each with two sublayers: multi-head self-attention, then a feed-forward network (two linear transformations with ReLU between, output size equal to input size). Both sublayers have a residual connection followed by a normalization layer.
Six identical layers with three sublayers: masked multi-head self-attention (attends only to preceding inputs), multi-head attention that receives the queries from the previous sublayer and the keys and values from the encoder output (cross-attention over the whole input sequence), and a feed-forward network. All sublayers have residual connections and normalization.
During training the decoder must be autoregressive: the output at position k may only attend to positions before it. All weights ωkj with j > k are zeroed by setting the raw scores ω′kj = −∞, so e−∞ = 0 and the softmax assigns exactly zero weight to future elements.
Training is self-supervised: the data itself generates the supervisory signals (the next token is the label for the previous ones), turning an unsupervised problem into a supervised one; dropout p = 0.1 is applied before each residual addition. Prediction is autoregressive: the model predicts the next output from the input sequence, appends it to the input, and repeats.
Decoder-only directly matches the autoregressive next-token objective, scales well with a simple uniform training setup, adapts to many tasks through prompting without architectural changes, and allows caching keys/values during generation. ViT splits the image into fixed non-overlapping patches, linearly embeds each patch, adds learnable position embeddings, prepends a [Class] token, and feeds the sequence to a standard transformer encoder, read out by an MLP head.