The module is taught from a practical, software engineering perspective, and it says so explicitly: the goal is to understand the fundamentals of natural language processing and language models, first the basic concepts, architectures and common tasks, then how to use them in practice through APIs and libraries and how to tune them for specific tasks. It deliberately does not dive into the algorithms and the mathematics behind them, for which it defers to the NLP course.
The empirical case for caring is stated with a figure: AI is becoming a standard in developers lives, with 85 percent of developers regularly using AI tools for coding and development, and 62 percent relying on at least one AI coding assistant, agent or code editor, according to the JetBrains State of Developer Ecosystem 2025.
The framing device is a three-way comparison of how the same task, sentiment classification, is expressed in three eras.
def sentiment(text):
good = ["great", "good"]
bad = ["bad", "awful"]
score = 0
for w in text.split():
if w in good: score += 1
if w in bad: score -= 1
return score > 0The behaviour is written out as code. Everything it does is inspectable, and everything it fails to do has to be added by hand.
model = nn.Sequential(
nn.Embedding(1000, 16),
nn.LSTM(16, 32),
nn.Linear(32, 1),
nn.Sigmoid()
)
# Requires dataset
# and training loop
model.fit(X_train, y_train)The behaviour is learned from a labelled dataset. The programmer writes the architecture and the training loop; the weights, which are the actual program, are found by optimisation.
prompt = f"""
Classify sentiment:
'{text}'
Output Positive or Negative.
"""
response = llm.generate(
prompt=prompt
)The behaviour is requested in natural language from a model that was never trained on this task in particular. There is no dataset and no training loop, only an instruction, and the engineering problem shifts to making that instruction unambiguous.
The lecture then lists what LLMs can now handle that previously required human expertise: smart code completion, automated documentation generation, assisted refactoring and optimization, test case generation. And it poses the key questions honestly, adding that the lecturer does not have the answers: what will our role be in this AI-augmented future, how can we best leverage NLP to enhance productivity, and which skills remain uniquely human in software development. The conclusion is only that understanding this technology is not optional, it is essential for staying relevant.
Natural language processing is a subfield of artificial intelligence focused on understanding, interpreting and generating human language. Its goal is to identify the structure and meaning of words, phrases and sentences in order to enable computers to understand and generate human language; the why is improving human-computer interaction, closing the gap between human communication and computer understanding. Its applications are all around us: chatbots, machine translation, speech recognition, sentiment analysis, question answering, code generation, image captioning, summarization and text classification.
| Challenges | Approaches |
|---|---|
| Ambiguity: multiple meanings for words and phrases. Context: meaning shifts with linguistic and cultural context. Syntax: sentence structure affects meaning. Sarcasm and idioms: non-literal language interpretation. | Rule-based: hand-crafted linguistic rules, for example Georgetown-IBM. Statistical: probabilistic language modelling, for example hidden Markov models. ML and deep learning: algorithms learn from data, and neural networks model complex patterns (RNN, LSTM, GRU). |
The statistical row is not a historical curiosity for this course. Hidden Markov models are the same family of object as the discrete-time Markov chains of chapter 10: a state space plus a transition probability distribution. Where chapter 10 uses them to model a communication channel, early NLP used them to model a sentence.
A machine learning model that aims to predict and generate plausible text. When a language model is used to generate new content, whether text, code or images, it is called a generative AI model. The deck contrasts this with discriminative AI: discriminative models classify or predict data, focusing on the boundary between classes, as in "is this a cat?"; generative models are designed to create new content, focusing on the structure of the data itself in order to generate new instances, as in "generate a cat".
The fundamental idea is one sentence: text is a sequence of words, namely a prompt, and language models learn the probability distribution of a word given the previous words in context. The example given, with its probabilities, is worth keeping because the temperature widget below reuses it:
The software engineer was very happy with the <*>
↓
The software engineer was very happy with the coffee. (80%)
The software engineer was very happy with the unit-tests. (15%)
The software engineer was very happy with the codebase. (0.0001%)A language model is decomposed into four conceptual phases. The deck immediately adds two caveats: tokens and words are used interchangeably for illustration, while in practice tokens are the actual units processed by the model; and modern LLMs integrate these phases into a massive end-to-end pipeline, for example transformers, that learns all components jointly during training.
["Un", "believ", "able", "!"].["Un"] becomes [0.25, -0.75, 0.5, ..., 1.0].Tokenization splits text into discrete subword units for the model to process and generate. The byte-pair-encoding example given is "Tokens are subwords!", split as ["Tok", "ens", " are", " sub", "words", "!"] and mapped to ids, with the GPT-5 ids listed as Tokens 30325, are 553, sub 1543, words 10020, ! 0. In practice, modern vocabulary sizes reach up to 250k (Gemini 3.1); the vocabulary is used bidirectionally, encoding input prompts and decoding text during generation; unseen text is dynamically split into known subword fragments; and special tokens manage flow, such as <|begin_of_text|> and <|eot_id|>.
Embedding translates token ids into dense numerical arrays that capture semantic meaning in context. The illustration is Dog, Cat and Car: if the model understands that Dog and Cat are more similar to each other than to Car, the vectors will reflect that, with a smaller distance between Dog and Cat. In practice, modern embeddings are typically computed inside the same model architecture, for example inside the transformer layers, whereas in the past they were often pre-trained separately with Word2Vec or GloVe. Dimensionality varies widely: GPT-2 uses 768 dimensions, DeepSeek V3 uses 7168.
Three families of approach to the modelling phase are compared.
| CNN | RNN / LSTM / GRU | Self-attention |
|---|---|---|
| Fixed-size sliding windows over text; good at capturing local patterns; limited by a fixed receptive field. | Process tokens sequentially; can capture order and context; struggle with long-range dependencies. | Each token attends to all others; captures arbitrary-distance relationships; fully parallelizable. |
Self-attention answers one core question: for each token, how much does each other token affect its interpretation? Attention weights determine token relationships, and they capture arbitrary-distance dependencies. The canonical example is "The animal didn't cross the street because it was too tired.": the pronoun "it" is ambiguous, and self-attention resolves it to "animal". Multi-head attention captures different relationship types simultaneously.
Mechanically, the input is a sequence of token embeddings and the output is a sequence of context-aware vectors, each a mixture of information from other tokens weighted by their relevance. Conceptually, each token is projected into three views: a Query, what information the token is looking for; a Key, what information the token contains; and a Value, the actual content to be passed along. Attention computes the similarity between queries and keys to find relevance, and aggregation computes the weighted sum of values based on the attention scores.
Transformers are the dominant architecture for LLMs. They merge word embedding and modelling into a single end-to-end system; the architecture relies on layers of multi-head self-attention and feedforward networks, where multi-head attention captures various relationships such as syntax and semantics simultaneously and feedforward layers introduce non-linearity for complex pattern learning. It is highly parallelizable, which allows training on massive datasets, and its output is a vector for each token, which results in a probability distribution used to generate the next token. The deck points at the Transformer Explainer visualiser at poloclub.github.io/transformer-explainer/.
Generation proceeds in five steps: the model receives a prompt or seed text, for example "The cat sat on the…"; it predicts probabilities for the next token based on the self-attention encoding; a token is selected from that distribution; the selected token is added to the sequence; and the process repeats until a stopping criterion is met. The key idea is building a sequence one token at a time.
| Decoding strategy | What it does |
|---|---|
| Greedy | Always choose the highest probability token. |
| Random | Sample from the probability distribution. |
| Top-k | Sample from the k most likely tokens. |
| Top-p / nucleus | Sample from the smallest set with probability greater than p. |
| Beam search | Track multiple candidate sequences. |
Temperature modifies the probability distribution before sampling. It is applied by dividing the logits by the temperature value, after which the softmax function is applied to get new probabilities, so that probabilities = softmax(logits / temperature). A high temperature, at or above 1.0, gives a flatter distribution and more random, creative output; a low temperature, around 0.2, gives a sharper distribution and more coherent but repetitive output; a temperature of zero is equivalent to greedy decoding. The analogy offered is that temperature controls the spice level of the text.
Temperature and the decoding strategy are orthogonal knobs, and confusing them is a classic mistake. Temperature reshapes the distribution; the strategy decides how to draw from whatever distribution it is given. Temperature zero collapses any sampling strategy into greedy decoding, which is why the labs recommend low temperature when you want reproducible tests, and why chapter 15 recommends high temperature when you want sample diversity for a pass@k estimate.
If you remove the last layer of a transformer, you get a contextual embedding for each token. The example is the word "bank", which gets a different vector depending on the surrounding text: "River bank" maps to something like [0.12, −0.85, 0.33, …] while "Secure bank" maps to [−0.45, 0.22, 0.91, …]. Two properties follow: the representation is deeply contextual, since the vector shifts with the sentence, and geometric distance equals semantic similarity.
In practice these are highly dimensional, typically 1024 to 12288 or more dimensions per token, and standard models include OpenAI text-embedding-3 and the open-source BGE or Nomic families. The deck points at the Cohere embedding playground for hands-on exploration.
The line "geometric distance equals semantic similarity" is the entire technical basis of retrieval-augmented generation, which appears twice more in this course: as the mechanism behind a coding assistant that knows your repository (chapter 5) and as the long-term memory of an agent (chapter 15). Both are built on the ability to turn a query and a document chunk into vectors and compare them.
A large language model is a language model with a large number of parameters, trained on a large corpus of text. Its implementation strategy has three ingredients: transformers as the foundational architecture, characterised by long-range context thanks to attention, efficient large-scale training thanks to parallelization, and model growth thanks to scalability; pretraining, which trains the model on a vast corpus of text to learn a wide range of language patterns and structures; and fine-tuning, which refines the pretrained model for specific tasks.
Pretraining is self-supervised: the data creates its own supervision signal, no human annotations or labels are needed, and the model learns to predict parts of its own input, as in "The people of sleepy town weren't __" → "happy". It leverages the natural structure in language itself. The advantages are that it uses unlimited text data from the internet, scales efficiently with more data and compute, creates rich representations of language, learns grammar, facts, reasoning and more, and forms the foundation for downstream adaptation.
Modern LLM training is then organised as a pipeline of three phases, and the deck notes that steps 2 and 3 are often iterated to fix regressions and target specific domains.
A foundation model is a large model that serves as the basis for a wide range of downstream applications, and it marks a paradigm shift. The traditional ML pipeline used task-specific datasets, built models for single purposes, followed a linear development pipeline, required retraining for new tasks and transferred little knowledge. The foundation model approach acquires general knowledge first, adapts to downstream tasks, transfers knowledge efficiently, and has zero-shot and few-shot capabilities.
The deck is careful about what "adaptation" now means. Adaptation is a kind of transfer learning to other tasks, but with foundational LLMs it may not require additional learning at all: the parameters are frozen and the model is used as-is with just the right instructions. LLMs function as zero-shot or few-shot learners, and with just the right prompts they can perform a wide range of tasks. That is the fundamental paradigm shift in AI and NLP development, and it is also why "prompt engineering" becomes an engineering discipline rather than a knack.
Scalability brings emergent properties: capabilities that appear at scale rather than being designed in. Applications span chatbots and assistants (general purpose interaction, content generation, coding assistants), specialised domains (medical diagnosis with Med-PaLM, legal analysis, scientific research) and embodied AI (robotics control, generalist agents such as Gato, multi-modal interaction).
Training costs: massive compute requirements, petabytes of data and exaflops of compute, environmental impact, centralization of power. Privacy and legal: data leakage risks, copyright infringement, GDPR compliance. Reliability: hallucinations, meaning confidently stating false facts, lack of grounding, and bias and fairness issues.
| Kind | What is released |
|---|---|
| Open weights | The model parameters are released; you can run and fine-tune locally, but training code and training data may remain undisclosed. |
| Open source | The model code, and often the full training pipeline, is released; you can inspect, modify, retrain and redistribute under the licence, though weights and data may still differ in openness. |
| Open data | The training dataset, or a meaningful reusable disclosure of it, is released; this enables stronger reproducibility and auditing, but is comparatively rare in practice. |
| Closed | No public code, weights or data; typically API access only, vendor-controlled. |
The key distinction is stated compactly: open weights implies mainly usage; open source implies modifiability; open data implies data transparency and reproducibility.
Examples of each type: GPT-* is closed, OpenAI flagship series, known for strong language generation and multi-modal capabilities, although OpenAI also offers open-weight variants such as GPT-4.1 nano; Llama is open weights, Meta family of LLMs, notable for being among the first large-parameter models available for public use; Gemini is closed, Google flagship series comparable to GPT, with Gemma as its open-weights counterpart. Several others exist: Claude (Anthropic, strong at code), Mistral (one of the few European models), DeepSeek, Qwen. They all share the same core idea, sequence in and next token out, and differ in architecture details, training data and fine-tuning techniques.
LLMs may have billions of parameters and typically require specialised hardware. The rule of thumb given is that 1B parameters requires roughly double that amount of GPU memory for inference; the worked case is DeepSeek-V3, with 671B parameters, requiring at least 800 GB of GPU memory for inference, for example eight A100 80 GB cards. Hence it is common to use LLMs via a provider that hosts the model, manages the infrastructure and exposes an interface, typically a RESTful API with a standardised format such as JSON.
The cost model is pay-as-you-go: users are charged based on API usage, priced on tokens processed, input plus output, per million tokens; some providers offer subscription plans or enterprise agreements; and access typically works with an API key that tracks usage and billing. The pricing examples given, as of March 2026, are:
Cost optimization strategies: batch processing, a 50 percent discount for non-real-time workloads with a 24-hour turnaround; prompt caching, up to 90 percent savings on repeated content, for example with Anthropic; and combined, batch API plus caching can achieve a 95 percent cost reduction. Access tiers matter too: free tiers with limited usage exist for experimentation, rate limits are tiered (Google is quoted at 15 requests per minute on the free tier rising to 1000+ on Tier 2), and higher tiers are unlocked by spending thresholds of $250 or more, or by time.
Models also vary along other axes: size, small (under 3B, local usage, edge AI) versus large (more than 10B, requiring specialised hardware); modality, text-only versus multi-modal with images and audio; and specialisation, general-purpose versus domain-specific such as medical, legal or code. Choosing a model means weighing the task requirements, the computational resources available, privacy and security needs, and the cost of API use versus local inference.
The core mechanism is HTTP POST requests with a JSON payload carrying inputs and configuration such as temperature. Chat completions are the modern standard, using an array of structured messages with roles system, user and assistant to maintain dialogue context; examples are v1/chat/completions at OpenAI and v1/messages at Anthropic. Embeddings endpoints convert text into high-dimensional vectors, taking a list of strings and returning a list of vectors, one per input string.
Streaming matters for user experience: output is streamed incrementally via server-sent events, triggered by "stream": true, which drastically reduces the time to first token, giving immediate visual feedback and preventing HTTP timeouts.
Cloud APIs are convenient, but running LLMs locally or on self-hosted infrastructure gives full control, strict data privacy since no data leaves the machine, and zero recurring API costs. The focus of the module is turnkey platforms that provide an OpenAI-compatible API out of the box, encapsulating the complexity of model execution so that local LLMs can be treated like standard microservices; explicitly out of scope are low-level manual loading of models, for example raw PyTorch or HuggingFace Transformers pipelines, and the orchestration of complex distributed serving frameworks for enterprise data centres.
A good local engine should offer API compatibility (a standardised REST API, often drop-in compatible with the OpenAI format), hardware abstraction (automatic GPU offloading, memory management, execution optimization) and model management (discovering, downloading and updating weights). The common solutions named are Ollama, which wraps LLMs in a background service optimized for native local performance with easy CLI management; LM Studio, focused on prototyping with a friendly GUI to discover models and test prompts; and vLLM / TGI, enterprise-grade engines focused on high throughput and scalability.
Quantization is the key to running large models locally: it reduces memory and computational requirements by representing model weights with lower precision, for example moving from 16-bit floats to 8-bit or 4-bit integers.
The benefits are a lower memory footprint, making otherwise inaccessible models fit into available RAM or VRAM, and faster inference, since memory bandwidth is often the primary constraint for LLMs. The trade-offs are that it may introduce slight losses in model accuracy, scaling with aggressive shrinking below about 4 bits, and that it requires specific file formats such as GGUF or AWQ, optimized for the targeted inference engine.
ollama serve # starts a local server at http://localhost:11434
# with endpoints for generation and embeddings
ollama pull qwen3:0.6b # downloads the Qwen3 model for local use
ollama run qwen3:0.6b # starts an interactive REPL for testing prompts
>>> Hello, World!
Thinking…
Okay, the user said "Hello, World!" and I need to respond. Let me
start by acknowledging their message. Since it's a simple greeting, a
friendly response is appropriate...
…done thinking.
Assistant: Hello, World! How can I assist you today?That transcript also illustrates reasoning or thinking models: LLMs perform significantly better on complex reasoning tasks when allowed to think step by step, and newer models are natively trained to generate intermediate reasoning steps before outputting the final answer, giving drastic improvements in accuracy and logical coherence with a significant reduction in reasoning errors.
LLMs are powerful but need the right instructions, because human language is inherently ambiguous: the same prompt can lead to wildly different outputs. The deck demonstrates this with one prompt, "Write a function to filter a list.", and three results: Python, a list comprehension filtering integers greater than 10; JavaScript, .filter() removing null values from objects; and C++, std::copy_if extracting even numbers. Same prompt, three completely different interpretations, with language, data type and filter logic all varying.
The diagnosis is that between humans we naturally disambiguate through shared context, while models lack implicit context and must be explicitly guided.
| Dimension | Between humans | With LLMs |
|---|---|---|
| Role | We know who we are talking to, for example a senior developer helping a junior. | Must be stated in the prompt, for example "You are an expert Python developer." |
| Context | We know what we are working on, for example the data-processing module. | Must be provided as input, for example "We are building a REST API in FastAPI." |
| Outcome | We know what we need, for example remove duplicates from a dataset. | Must be clearly specified, for example "Return a function that removes duplicates." |
The discipline of crafting these explicit instructions is called prompt engineering: crafting inputs to guide LLMs toward desired outputs, with the goal of reducing ambiguity and constraining the model probability space, and the key being iterative refinement of clear, specific and structured instructions. A prompt typically has two components: system instructions, high-level guidelines that set the model overall behaviour and persona, and user input, the specific request or question. Prompts range from a few words to structured multi-paragraph instructions; they can steer the model toward a specific task, style, format or constraint; and small changes in phrasing can significantly influence output quality and relevance.
Four groups of best practice are listed: set clear goals (use action verbs, define the length and format of the output, specify the target audience); provide context (include relevant facts and data, reference specific sources or documents, define key terms and constraints); be specific (use precise language and avoid ambiguity, quantify requests whenever possible, break down complex tasks into steps); and iterate and experiment (try different phrasings and keywords, adjust the level of detail, test different prompt lengths).
Zero-shot prompting means direct instructions with no examples: the model relies entirely on its pretraining knowledge and on the instructions in the prompt, and this is the most common interaction mode, as in everyday chat. Its general structure has four parts: role, who the model should act as; context, background information relevant to the task; task instructions, what the model should do; and output format, how the result should be structured. Use it for tasks where the model already has strong knowledge such as translation, summarization or classification; for quick prototyping, since no examples need preparing; and for simple, well-defined requests where ambiguity is low.
Few-shot prompting means providing examples to demonstrate the desired pattern: instead of only describing the task, you show the model what you expect through a small number of input-to-output pairs. Examples act as implicit instructions, since the model learns the pattern and mimics it; the technique significantly improves performance over zero-shot for ambiguous or domain-specific tasks; the number of examples matters, with typically two to five shots being enough, though more can help at the cost of context; and examples should be representative and diverse to avoid bias. Use it when the desired output format is unusual or strict, such as a custom schema; when the task is ambiguous and a verbal description alone is insufficient; when the model underperforms in zero-shot mode; or when you want to enforce a consistent style or tone across outputs.
System: You are a sentiment analysis expert.
User: Classify the sentiment of this review as Positive, Negative, or Mixed:
"This restaurant had amazing food but terrible service."
Assistant: MixedRole, task and the admissible output values are all stated. Nothing is shown, only described, which is exactly what makes it zero-shot.
System: You are a senior Python developer. Always include type hints and docstrings.
User: Write a function that takes a list of integers and returns only the even
numbers using a list comprehension.
Assistant:
def filter_even(nums: list[int]) -> list[int]:
"""Return only even numbers from the input list."""
return [n for n in nums if n % 2 == 0]System: You are a data extraction assistant. Always respond with valid JSON.
User: Extract name, email, and company from:
"Hi, I'm Alice from Acme Corp. Reach me at [email protected]."
Assistant:
{"name": "Alice", "email": "[email protected]", "company": "Acme Corp"}Compare the first snippet with the ambiguity example of section 11: the language, the data type and the filter logic are now all pinned down, so the three divergent answers collapse into one.
User: Extract the programming language and framework from each sentence:
"We built the API in Go with Gin." → Language: Go, Framework: Gin
"The frontend uses TypeScript and React." → Language: TypeScript, Framework: React
"Our pipeline is written in Rust using Actix." →
Assistant: Language: Rust, Framework: ActixThe output schema is never described in words: it is demonstrated twice and then imitated. That is the defining move of few-shot prompting.
User: Generate a conventional commit message from the diff description:
"Added null check in UserService.getById"
→ fix(user): add null check in getById to prevent NPE
"Renamed Config to AppConfig and moved to config pkg"
→ refactor(config): rename Config to AppConfig and relocate to config package
"Added retry logic with exponential backoff to HttpClient" →
Assistant: feat(http): add retry logic with exponential backoff to HttpClientTwo shots are enough to teach a strict convention (type, scope, imperative summary) that would take a paragraph to describe and would still be applied inconsistently.
Prompt engineering gets you useful outputs; building real applications on top of LLMs requires more. The challenges listed are switching between providers such as OpenAI, Ollama and Gemini without rewriting code; chaining multiple LLM calls and processing steps together; integrating external data sources such as databases, APIs and documents; and managing conversation history and context windows. What is needed, correspondingly, is a unified abstraction layer over different providers, composable pipelines for multi-step tasks, built-in support for tool use and external integrations, and a principled way to build agentic workflows. In short, a framework that bridges the gap between prompt engineering and software engineering.
LangChain4J is a Java framework for building LLM-powered applications, inspired by the original Python LangChain library and redesigned to be idiomatic for the Java ecosystem, also drawing ideas from LlamaIndex; it lives at github.com/langchain4j/langchain4j. Its core features are a unified provider API, allowing you to swap between OpenAI, Ollama, Gemini and Anthropic with one-line config changes; AI Services, composing multi-step LLM workflows in the shape prompt, call, parse, act; and tool integration, letting the model call external functions such as search, database queries and REST APIs, which is the basis for agentic AI. It offers first-class Java and Kotlin support with type-safe APIs, built-in memory management for multi-turn conversations, native embedding stores for retrieval-augmented generation, and an active community with Spring Boot integration.
Its main abstractions are five:
Getting a model running takes two dependencies and a builder:
libraryDependencies += "dev.langchain4j" % "langchain4j" % "1.11.0"
libraryDependencies += "dev.langchain4j" % "langchain4j-ollama" % "1.11.0"Integrating LLMs into software requires bridging the gap between structured domain models and unstructured natural language, and the deck splits that bridge into three concerns.
| Concern | What it means |
|---|---|
| Input: prompt construction | Translating the application state, for example a game Board, into text. Often done with prompt builders or templates that inject dynamic values at runtime to create context-aware prompts. |
| Output: response parsing | The LLM returns free-form text, so structured data must be extracted from it, for example row,col coordinates, using regular expressions, JSON parsing or dedicated output parsers. Modern LLMs can be trained to produce strictly formatted outputs, which is called structured output and also accepts a schema definition to validate the format. |
| Resilience: validation and fallbacks | LLMs are non-deterministic and can produce invalid output, hallucinating bad moves or breaking format. Production code must include retry mechanisms, data validation and safe fallbacks to guarantee stability. |
The module closes on questions rather than answers, and they set up the two chapters that follow. On testing LLM integrations: we tested our code by mocking the LLM API calls (the technique of chapter 3), but how do we verify that the actual model behaves as expected in reality, and how do we test for edge cases, non-determinism and hallucinated failure modes? On evaluating LLM applications: how can we systematically compare the performance of different models or prompting strategies, and do we need new metrics to score subjective natural language outputs instead of exact matches? On AI-assisted quality assurance: can we use LLMs themselves to generate validation pipelines or bootstrap test cases, and how do we integrate these checks securely into continuous integration pipelines? Chapter 15 answers all three.
LLMs are the state of the art in generative AI for text processing and generation; a paradigm shift toward foundational models that can be adapted to many tasks and domains; adaptation happens through fine-tuning, not covered here, and prompt engineering, meaning few-shot and zero-shot learning; many models are available with different characteristics. Why we care: they may transform the software engineering landscape toward Software Engineering 3.0, and as software engineers we should be able to use them, understand them and possibly adapt them. The two questions are how to create applications with LLMs, which is today, and how to test them, of which we get a glimpse.
https://ollama.com/, and its model search for small models under 1B, at most 3B parameters.https://github.com/langchain4j/langchain4jhttps://github.com/cric96/asmd-public-03-llm-intro-labhttps://github.com/cric96/asmd-llm-codeThe general goals are to be operative with Ollama and LangChain4J, to experiment with prompt engineering, and to pre-check how LLMs can be used to create software applications.
it.unibo.basics runs without errors.it.unibo.basics, which contains the basic utilities for interacting with Ollama, and try modifying models and parameters to observe different results; look at the prompt package with its prompt engineering examples and write your own prompt, for instance asking for code generation or explanations of code, experimenting also with few-shot prompting by providing examples; finally look at the tictactoe package, a simple application that uses an LLM to play tic-tac-toe, focusing in particular on the controller package and on the formatter, parser and prompt packages.e1 package, an application that uses LLMs to create a role-play text-based game, and follow e1/EXERCISES.md: first implement the Advance and Begin prompts, then unit test the logic of LLMStoryEngine by mocking the LLM interactions.| Task | What it asks |
|---|---|
| MEMORY MANAGEMENT | Add memory management to the story engine for long-term narrative coherence across LLM calls. Primary: design and implement a prompt decorator that maintains a summary of previous story beats and injects it into each new request, compensating for the stateless nature of LLM interactions. Advanced: investigate different context management strategies, for example fixed-window history versus LLM-based summarisation, and analyse the trade-offs between narrative coherence, context length and token cost. |
| LLMs VERIFICATION | LLM outputs are non-deterministic: the same prompt can yield different results depending on temperature, sampling and model state. Primary: design a testing strategy for the story engine that accounts for non-determinism, experiment with temperature settings to observe their effect on output stability, and implement checks that tolerate variability while still asserting correctness, for example a second LLM acting as a judge. Advanced: reflect on what correctness means for generative outputs and how confidence and reproducibility relate to temperature; investigate whether lowering temperature is sufficient for reliable testing. |
| AI-APP DESIGN | Integrate an LLM into one of your existing applications or projects, replacing static rules or simple AI logic with a language-model-driven component. Primary: identify a decision point in a past project where an LLM could add value, implement the integration, and reflect on the design choices and limitations introduced by the non-deterministic nature of the model. |
The LLMs VERIFICATION task is the one with the strongest links to the rest of the course, which makes it excellent exam material. Lowering temperature reduces variance but does not make the component deterministic, so a test suite over it is closer to the approximate model checking of chapter 11 than to a JUnit assertion: you run N samples, you accept a result within an approximation ε with a confidence δ. Being able to make that connection explicitly is exactly the kind of cross-module link the oral discussion looks for.
Tokenization splits raw text into discrete subword units; word embedding maps tokens into dense numerical vectors; modelling learns contextual relationships and probabilities, for example P(able | Un, believ) = 0.95; generation samples from those probabilities to produce output. The caveat is that this is a conceptual decomposition: modern LLMs integrate the four phases into a massive end-to-end pipeline, typically a transformer, that learns all components jointly during training. A second, smaller caveat is that tokens and words are used interchangeably only for illustration; the model processes tokens.
Each token is projected into three views: the Query is what information the token is looking for, the Key is what information the token contains, and the Value is the actual content to be passed along. Attention computes the similarity between queries and keys to find relevance, then aggregation computes the weighted sum of values according to those scores. In "The animal didn't cross the street because it was too tired", the query of "it" matches most strongly the key of "animal", so the value of "animal" dominates the new context-aware vector for "it". Multi-head attention runs several such comparisons in parallel, capturing different relationship types simultaneously.
Temperature modifies the probability distribution before sampling: the logits are divided by the temperature value and softmax is then applied, so probabilities = softmax(logits / temperature). High temperature, at or above 1.0, flattens the distribution, giving more random and creative output; low temperature, around 0.2, sharpens it, giving more coherent but repetitive output; temperature zero is equivalent to greedy decoding. It is orthogonal to the decoding strategy: temperature reshapes the distribution, while greedy, random, top-k, nucleus and beam search decide how to draw from whatever distribution they are handed.
Open weights means the model parameters are released, so you can run and fine-tune locally, though training code and data may remain undisclosed. Open source means the model code, and often the full training pipeline, is released, so you can inspect, modify, retrain and redistribute under the licence, with weights and data possibly differing in openness. Open data means the training dataset, or a meaningful reusable disclosure of it, is released, enabling stronger reproducibility and auditing, which is comparatively rare. Closed means no public code, weights or data, typically API access only. The key distinction: open weights implies usage, open source implies modifiability, open data implies data transparency.
Pretraining: the goal is general language patterns and world knowledge, the signal is next-token prediction on massive unlabelled corpora, and the result is a fluent but not instruction-following model; the guiding question is "what continuations are likely in text?". Instruction tuning: the goal is following instructions and specific tasks, the signal is labelled prompt-to-ideal-response pairs, and the result is an instruction-tuned model; the question is "what should I do when asked?". Alignment: the goal is matching human preferences for safety and helpfulness, the signal is human preference comparisons of A versus B, and the result is a safe and helpful assistant; the question is "what response is preferred or safe?". Phases two and three are often iterated.
Because the data creates its own supervision signal: no human annotations or labels are needed, and the model learns to predict parts of its own input, as in "The people of sleepy town weren't __" giving "happy". It leverages the natural structure in language itself. What it buys is the ability to use unlimited text data from the internet, efficient scaling with more data and compute, rich representations of language, learned grammar, facts and reasoning, and a foundation for downstream adaptation.
Quantization reduces memory and computational requirements by representing model weights with lower precision, for example from 16-bit floats to 8-bit or 4-bit integers. Its impact on VRAM, approximately: 7B or 8B parameters go from 14 GB at 16-bit to 4.5 GB at 4-bit, running on a standard laptop; 32B goes from 64 GB to 18 GB, running on a Mac M-series or a consumer GPU such as an RTX 4090; 70B goes from 140 GB to 40 GB, running on high-end workstations such as two 24 GB GPUs or a Mac Studio. The benefits are a lower memory footprint and faster inference, since memory bandwidth is often the primary constraint. The trade-offs are slight accuracy losses that grow with aggressive shrinking below about 4 bits, and the need for specific formats such as GGUF or AWQ.
Roughly, 1B parameters requires double that amount of GPU memory for inference. DeepSeek-V3 has 671B parameters and therefore requires at least 800 GB of GPU memory for inference, for example eight A100 cards of 80 GB each. This is the concrete reason why it is common to use LLMs through a provider that hosts the model, manages the infrastructure and exposes a RESTful API, rather than running them locally.
When the desired output format is unusual or strict, for example a custom schema; when the task is ambiguous and a verbal description alone is insufficient; when the model underperforms in zero-shot mode on that specific task; or when you want to enforce a consistent style or tone across outputs. Typically two to five shots are enough; more can help but consume context. Examples should be representative and diverse to avoid bias, since they act as implicit instructions that the model learns from and mimics.
ChatModel is the model used to generate responses, for example OpenAI, Ollama or Gemini. ChatMessage is the input message, with the system, user and assistant roles, defining the conversation context. Response is the output generated by the model, which can be further processed or parsed and which also carries metadata about the generation such as tokens used and latency. ModelBuilder is a fluent API for configuring and instantiating ChatModel instances with parameters such as temperature, max tokens and provider. A similar architecture persists for EmbeddingModel, adjusted for its different input and output formats.
Input, that is prompt construction, translating the application state into text with prompt builders or templates injecting dynamic values at runtime. Output, that is response parsing, extracting structured data from free-form text with regular expressions, JSON parsing or dedicated parsers, possibly helped by structured output with a schema. And resilience, that is validation and fallbacks. The third is unavoidable because LLMs are non-deterministic and can produce invalid output, hallucinating bad moves or breaking the format, so production code must include retry mechanisms, data validation and safe fallbacks to guarantee stability.
Because output arrives incrementally via server-sent events rather than as one final blob, which drastically reduces the time to first token, gives immediate visual feedback to the user and prevents HTTP timeouts on long generations. It is triggered by setting "stream": true in the JSON payload of the POST request. This is a purely engineering concern, invisible to the model, and a good illustration of why the module treats an LLM endpoint as an ordinary networked service to be engineered around.