Part A — Robust software engineering · Chapter 1

Robust software engineering and pervasive validation

~25 min read4 interactive widgets3 plates

In this chapter

  1. ASMD inside the software architect thread
  2. Learning outcomes and the philosophy behind them
  3. The map of the course
  4. Labs, exam and the software stack
  5. Software engineering is a young field
  6. The industry today, and the generative AI turn
  7. Robust software engineering: an internal definition
  8. A digression on complexity
  9. Pervasive computing, and four faces of complexity
  10. Testing in a nutshell
  11. Pervasive testing and the shoring metaphor
  12. Verification, validation and the V-model
  13. Testing and models in model-driven engineering
  14. Test your knowledge

1. ASMD inside the software architect thread

Advanced Software Modelling and Design is one of three courses that together form the software architect thread of the master degree in Ingegneria e Scienze Informatiche at the Cesena campus. The three are meant to be read as a set, and it is worth knowing what the other two do, because ASMD deliberately does not repeat them.

CourseWhat it introduces
SAAP
Software Architecture and Platforms
The essential concepts of software architecture and the related abilities, including the application of modern architectures to concrete application contexts, cyber-physical systems and the IoT among them.
SPE
Software Process Engineering
Advanced techniques for organising the software development process: build automation, continuous integration and delivery, domain-first analysis in a technology-independent fashion, and the creation of domain-specific languages.
ASMD
Advanced Software Modelling and Design
Enhances the ability to construct models of software systems in a variety of contexts, again including cyber-physical systems and the IoT, and to turn those models into concrete designs of reliable and effective systems and applications.

So SAAP is about the shape of the system, SPE is about the machinery that builds it, and ASMD sits in between: it is about the models you write before there is a system, and about the disciplined route from such a model to running code that can be trusted. The course is taught by Prof. Mirko Viroli and Prof. Gianluca Aguzzi, with seminar contributions from Nicolas Farabegoli on effect systems and Danilo Pianini on simulation.

Editor's note

Throughout this site the word model is used in the model-driven engineering sense, not in the machine learning sense: a model is a specification of a software system, useful both as a template for an implementation and as an object that properties can be proved about. When the machine learning meaning is intended, from chapter 4 onwards, it is always called a language model, a neural approximator or similar.

2. Learning outcomes and the philosophy behind them

The stated goal of the course is to enhance the abilities of prospective software architects to construct models of software systems in a variety of contexts, including cyber-physical systems and the IoT, and turn them into concrete designs of reliable and effective systems and applications. Concretely, the student will learn to:

Behind these three lines there is a diagnosis of the present moment, which the opening lecture states bluntly. Software virtually runs the world, but software development and engineering are at a turning point. New powerful tools, the large language models, are here to stay and are changing the game radically. The complexity of software and of software engineering is rising, partly because systems now embed AI components of their own. And yet quality and competence are shrinking, under the combined pressure of technical debt, lowering skills and AI-generated pollution.

From that diagnosis follows what the course claims to provide to a software architect: training in software modelling and design; training in the correct transfer of a specification into validation and testing; a span from industry-ready approaches all the way to research; and a span from standard, generic contexts to emerging scenarios. The two arrows of that last point are worth memorising, because they explain the shape of the whole syllabus: from general-purpose software to large-scale distributed systems on the IoT and the cloud-edge continuum, and from software that is merely built by people to software that uses AI both in its development methods and as a part of the system being developed.

Key idea

The course is organised around a single tension: the systems we are asked to build are getting less predictable (they are distributed, open, stochastic, and now partly synthesised by a model), while the guarantees we are asked to give about them are getting stronger. Every technique in the next fourteen chapters is an attempt to buy back some predictability, either by writing a more rigorous model, or by validating more pervasively, or by explicitly capturing the uncertainty instead of ignoring it.

3. The map of the course

The syllabus is declared as four parts of twelve modules. Because this site reorganises the twenty-eight decks by topic rather than by date, it is useful to see immediately how the official schedule maps onto the fifteen chapters you are about to read.

PartModules in the scheduleChapters here
A — Robust software engineering1. Pervasive validation for robust software engineering
2. Advanced testing: principles and techniques
3. Code synthesis with LLMs
this chapter, plus 2, 3, 4 and 5
B — High-level specification and programming4. High-level programming patterns
5. Advanced functional programming
6, 7 and 8
C — Modelling with complexity6. Modelling nondeterministic systems
7. Modelling stochastic systems
8. Stochastic analysis
9. Simulation
9, 10, 11 and 12
D — Learning and distribution10. Reinforcement learning
11. Aggregate computing
12. Aggregate and multiagent reinforcement learning
13, 14 and 15

The ingredients listed by the intro deck cut across those parts. Under advanced testing, programming and design (in Java and Scala) sit acceptance testing and integration testing, and the idea of addressing correctness by design through declarativity, laws and effects. Under design in unpredictable settings sit non-determinism, large networks of devices, stochastic modelling, model checking, probabilistic and approximate model checking with discrete-time and continuous-time Markov chains, and simulation. Under AI in software development and design sit large language models and their role in testing and program completion, design with self-organisation through patterns, macro-programming and aggregate computing, and design with learning through Markov decision processes, reinforcement learning and multi-agent reinforcement learning.

4. Labs, exam and the software stack

Every module has a lab activity, and every lab proposes several tasks that exercise different skills. The first task of each lab is introductory, meant to let you grasp the technology and the main ideas; the others are there for you to pick from for possible additional explorations. Students work alone or in small groups, typically in pairs, and the teachers provide assistance at need. In the later labs the two classes of exercise are named explicitly: operational steps, designed to be quick and completed during the lab session, and R&D tasks, which involve more in-depth research and development work, resemble exam projects and typically require additional work outside the lab.

The R&D tasks are not meant to be all completed. You pick one, or two, or more, depending on your interest and the time you have; you can start in the lab and continue at home. When you have an interesting result or a completed task, you share it with the teacher: the labs ask for an email with subject [ASMD24-LABXX-TASKYY] carrying authors and repository in the body.

The exam assumes roughly 90 to 100 hours of work in addition to the labs, and consists of a discussion of the tasks you completed. That can be one big task producing a software artifact, which can later become a scientific paper, or a set of smaller tasks, explicitly not all of them. In either case the discussion also covers the links to other parts of the course, which is the single most useful thing to keep in mind while studying: the examiner is interested in whether you can connect, say, a Petri Net safety property to a ScalaCheck property, or a Gillespie simulation to an approximate model checking result.

The tooling is fixed and worth installing early: Java (for example Java 24, from Adoptium), Scala (for example Scala 3.6), the IntelliJ IDE with the Scala plugin, Git, the SBT build tool, plus various LLMs and various testing frameworks as the modules require them.

For the exam

The exam is not a set of questions on the slides: it is a discussion built on the artifacts you produced, extended to the connections between modules. Practically, that means every lab task you complete should end with a written paragraph on what it has in common with, or what distinguishes it from, the corresponding task in another part of the course. That paragraph is the exam.

5. Software engineering is a young field

The first substantive lecture opens with a piece of perspective. Compared with other engineering disciplines, software engineering is extremely young: civil and mechanical engineering go back to antiquity, chemical and electrical engineering to the eighteenth century, nuclear engineering to the first part of the twentieth century, and software engineering only to the second half of the twentieth century. That youth is not an excuse, but it does explain why the field keeps reinventing its own methods roughly once a decade.

Read as a sequence, each stage is an answer to the dominant scarcity of its decade. In the sixties the scarce resource was the machine; in the nineties it was reusable structure; in the 2000s it was time to market. In the 2020s the scarce resource, the course argues, is trust: we can now produce code faster than we can convince ourselves that it is correct.

6. The industry today, and the generative AI turn

The deck states the current situation as a list of facts rather than opinions. The industry is under stress: software de facto runs modern life, it accounts for a big share of the world economy (around 20 percent in the United States), several science and engineering fields are basically just applications of software engineering, practical abilities are increasingly requested more than foundational ones, the reliability of software as both product and process is increasingly crucial, and generative AI is being exploited as a brute-force tool to speed up development and reduce labour.

Meanwhile software products are large and complex, and grow exponentially, with continuous evolution and deployment; because of that complexity, failure is becoming the norm. And they remain costly: productivity is well under 100 lines of code per day per person over the whole life cycle, and both production cost and unitary cost increase with production size, which is the opposite of what happens in other engineering fields, where scale usually drives unit cost down.

Onto that background arrives generative AI, and the lecture is careful to separate the expectations from the likely outcomes.

All development problems will be quickly solved by LLMs. More productivity with fewer engineers. An increasing ability to understand code. An increasing ability to avoid and correct faults.

The complexity and size of software will grow even faster. Technical debts, defects and bugs will dramatically increase. Tracking both product and process will become much more difficult.

Highly technical skills will be needed to really exploit all the benefits of generative AI. The tool amplifies whoever is holding it, which is exactly the argument chapter 5 develops as the knowledge paradox.

The recap of the problem is stated in four lines that are worth learning verbatim: the complexity of problems is rising; the complexity of solutions is rising further; quality and competence are not rising, if not actually decreasing; and the causes of that last point are pressure, technical debt, lowering skills, software pollution and AI-generated code.

7. Robust software engineering: an internal definition

The course then gives its own working definition, described honestly as an internal one, that is, the definition adopted by these lectures rather than a standard term of the literature.

Key idea — robust software engineering

Software engineering driven by the need to achieve robustness of production, to be managed by highly skilled software architects, and tightly coupled with the emerging AI-native software engineering.

That definition is then unfolded into three pillars, each with a technique, an emphasis and a consequence. The consequences are the interesting part, because each of them is about making the collaboration with an LLM safer.

PillarHowEspeciallyConsequence
Address complexity by careful high-level abstractionRigorous, coherent, fitting specification, then design, then programmingComputer-based formal definition of models and designsBetter training of LLMs
Achieve better process and product controlPervasive validation, at all stages, to track, enforce and guaranteeTesting and verification of early models and designsBetter trust in LLM-generated code
Tackle unpredictability, limiting negative effectsCapturing and harnessing uncertaintyCarefully designing and analysing non-determinism; relying on controlled program synthesis with LLMs and reinforcement learningUncertainty becomes a modelled quantity rather than an unpleasant surprise

Notice how neatly the three pillars partition the rest of the course. Pillar one is Part B: writing the specification in a language rigorous enough that it can be typechecked, tested and reused. Pillar two is the rest of Part A: pushing validation as early and as often as possible. Pillar three is Parts C and D: modelling the non-determinism explicitly, quantifying it with probabilities and rates, and then synthesising behaviour by learning inside the space that remains.

8. A digression on complexity

Since robustness is defined against complexity, the lecture pauses to say what complexity means here. The definition given is a systems-theoretic one rather than a code metric.

Editor's note — a possible definition of complex system

A system in which sub-components are intertwined, by interaction, in a way that makes overall system behaviour not directly reducible to the behaviour of the single parts, and highly sensitive to small changes in environmental conditions.

Both clauses matter. The first kills the hope of understanding the whole by understanding each part in isolation, which is exactly the hope that unit testing rests on, and is why chapter 3 spends so long on integration testing. The second kills the hope of a reproducible experiment, which is why chapter 12 spends so long on making simulations strictly reproducible.

Four sources of complexity are listed, at various levels:

The last one is the sharpest claim: beyond a certain size you do not get to choose whether your system is complex. That is the premise of aggregate computing in chapter 14, which stops trying to control each device and programs the ensemble instead.

9. Pervasive computing, and four faces of complexity

The concrete setting for all of this is pervasive computing systems, also called socio-technical or cyber-physical systems. The deck decomposes them into four elements:

And it summarises the engineering problem in one line: engineering collective pervasive computing systems is all about creating a coherent and working ensemble out of those entities and interactions.

Four scenarios are then used as running examples of what makes such ensembles hard.

ScenarioComplexity elements
Cooperative team controlSending commands to a team in an explicit or implicit way; dealing with human factors; handling the heterogeneity of the infrastructure.
Sensing global patterns of dataDiscovering emergent patterns over space and time; creating communities out of a collective of wearables.
Crowd-aware steeringDynamically performing complex situation recognition; adapting decisions to ongoing context changes.
Controlling fleets of dronesCapturing swarm behaviour with the right abstraction; defining complex behaviour through libraries of composable units; seamlessly dealing with synthesis by learning.

The last row is a small manifesto for Part D of the course: the right abstraction is the computational field, the composable units are the building blocks of aggregate computing, and the synthesis by learning is reinforcement learning, eventually multi-agent.

10. Testing in a nutshell

Testing gets a definition broad enough to cover the whole of Part A.

Key idea — a definition of software testing

The overall process of planning, preparing and carrying out a suite of different types of tests designed to validate a system under development, in order to achieve an acceptable level of quality and to avoid unacceptable risks.

Two details of that definition do a lot of work. The first is a suite of different types of tests: the plural is deliberate, and chapter 3 will enumerate the levels. The second is what counts as the system under development: not only your code, but also the companion data, specifications, models and designs. Once specifications and models are part of the system under development, it becomes meaningful to test them, which is precisely the move that pervasive validation makes.

Six reasons to test are listed, and they are usefully different from each other:

Only the first is about finding bugs. The second is why acceptance tests are written in Gherkin rather than in JUnit (chapter 2); the third is the Factory school of testing; the fourth is the Context-Driven school; the fifth and sixth are the Agile school (chapter 3).

11. Pervasive testing and the shoring metaphor

Here is the central idea of the whole first module. Software development is not a pipeline but an intricate workflow of heterogeneous micro-activities, each sitting in one of requirements analysis, design, implementation, integration, deployment or maintenance. Those micro-activities are carried on in an interleaved way, by a group of people, in dynamic contexts, with feedback loops. From that observation two consequences follow.

First: because of all its motivations and benefits, testing has to be used at each micro-stage, as a means to secure the internal coherence of the micro-system under test, as well as its coherence with respect to other parts, implemented in the past or in the future. Second, and less comfortable: software tests form a challenging system to engineer, requiring ad-hoc considerations. Theory, books and courses only give you templates.

The metaphor explains why the props are drawn in vermilion and never removed. A test written during design is not thrown away when the code exists: it keeps the design stage from collapsing while the implementation leans on it. That is the difference between testing as an activity that happens after coding and pervasive testing as a property of the whole workflow.

12. Verification, validation and the V-model

The two words are constantly confused in practice, so the deck pins them down as two questions.

TermThe question it answers
ValidationIs my specification right with respect to stakeholder expectations?
VerificationIs my system right with respect to the specification?

The consequence is stated explicitly, and it is the hinge of the whole course: note the key role of the specification, which is essentially a model. Validation compares a model with the world; verification compares a system with the model. If you have no model, you cannot do either, you can only test against your own intuition, which is another way of saying you can only test against yesterday's intuition.

13. Testing and models in model-driven engineering

The closing part of the deck lines up model-driven engineering with testing, splitting the life cycle in two halves that meet at the code.

Requirement-to-code

You start from abstract models and iteratively concretise until coding. Model-driven engineering advocates concretisation via model-to-model transformation. In high-level languages, coding is rather abstract too, which is exactly the observation that Part B of this course exploits: if the programming language is expressive enough, the last model and the first program are the same artifact. Verification and validation are based on models, and are about three things: correctness with respect to requirements, corresponding to stakeholder expectations; self-coherence; and, for so-called executable models, some behavioural correctness that can actually be checked.

Code-to-operation

You start from code and iteratively put everything together until operation. Testing here is executed in code: it checks the actual execution of code, and verifies it in different contexts, namely in isolation, in integration and deployed.

For the exam

Be able to explain why the same diagram carries both validation and verification arrows, and where each sits: validation is the horizontal link between the requirements and the executable specification, because it asks whether the specification captures what stakeholders wanted; every arrow below it is verification, because it asks whether an artefact conforms to the model above it. The role of MDE is precisely to give verification something to be checked against at each level.

Watch out

Do not read the diagram as a waterfall. The lecture is explicit that development is an interleaved workflow with feedback loops, and that pervasive testing applies at each micro-stage. The diagram says which artefact is the oracle for which test, not in which order the work happens.

Test your knowledge

What distinguishes ASMD from the other two courses of the software architect thread?

SAAP introduces the essential concepts of software architecture and their application to concrete contexts including cyber-physical systems and the IoT. SPE introduces advanced techniques for organising the development process: build automation, continuous integration and delivery, domain-first analysis and the creation of domain-specific languages. ASMD sits between them: it enhances the ability to construct models of software systems in a variety of contexts and to turn them into concrete designs of reliable and effective systems. In one line: SAAP is the shape of the system, SPE is the machinery that builds it, ASMD is the model that precedes both and the route from it to trusted code.

State the internal definition of robust software engineering used by this course, and its three pillars.

Software engineering driven by the need to achieve robustness of production, to be managed by highly skilled software architects and tightly coupled with the emerging AI-native software engineering. The three pillars are: address complexity by careful high-level abstraction (rigorous specification, then design, then programming, especially via computer-based formal definitions of models and designs); achieve better process and product control through pervasive validation at all stages, especially testing and verification of early models and designs; and tackle unpredictability by capturing and harnessing uncertainty, especially by carefully designing and analysing non-determinism and by relying on controlled program synthesis. The stated consequences of the first two are better training of LLMs and better trust in LLM-generated code.

What is a complex system, and what are the four sources of complexity listed?

A system in which sub-components are intertwined by interaction in a way that makes the overall behaviour not directly reducible to the behaviour of the single parts, and highly sensitive to small changes in environmental conditions. The four sources are defects in design or code, which make behaviour unpredictable on changes; distribution and heterogeneity, which make global behaviour hard to control; openness, which leaves little knowledge of the actual environmental conditions; and scale, because as the system grows elements of complexity become unavoidable.

Why does the course claim that generative AI will make things harder before it makes them easier?

Because the expectations and the likely outcomes diverge. The expectation is that LLMs will quickly solve development problems, bringing more productivity with fewer engineers, better understanding of code and better fault avoidance. The likely outcomes are that the complexity and size of software will grow even faster, that technical debts, defects and bugs will dramatically increase, and that tracking product and process will become much more difficult. The consequence drawn is that highly technical skills are needed to actually exploit the benefits, since the tool amplifies the engineer holding it.

Define validation and verification, and explain why the specification is the pivot between them.

Validation asks whether my specification is right with respect to stakeholder expectations; verification asks whether my system is right with respect to the specification. The specification is the pivot because it is the only artefact both questions refer to, and because it is essentially a model. Without an explicit specification there is nothing for verification to compare the system against, and nothing for validation to show to stakeholders, so both activities degenerate into ad-hoc checking against the developer's own current intuition.

What is pervasive testing, and what two consequences does the lecture draw from it?

Pervasive testing starts from the observation that development is an intricate workflow of heterogeneous micro-activities, each sitting in requirements analysis, design, implementation, integration, deployment or maintenance, carried on in an interleaved way by a group of people, in dynamic contexts, with feedback loops. The first consequence is that testing has to be used at each micro-stage, to secure both the internal coherence of the micro-system under test and its coherence with respect to other parts implemented in the past or in the future. The second is that software tests form a challenging system to engineer in their own right, requiring ad-hoc considerations, with theory and books supplying only templates.

List the six reasons to test given in the lecture, and say which of them are not about finding bugs.

To enforce internal and external quality properties; as specification and documentation of system behaviour; to track progress in system development; as a basis for a contract with stakeholders; to mitigate risks in case of changes; and as a safety guide to maintenance and evolution. Only the first is directly about defects. The second turns tests into readable specifications, which motivates Gherkin; the third turns them into a project-management instrument; the fourth turns them into a contract; the fifth and sixth turn them into the safety net that makes refactoring and evolution possible.

In the V-model, at what point are the tests for a given level designed?

At the same time as the level itself, not after the implementation. Each rung of the V pairs a specification or design artefact on the descending arm with the test level that will check it on the ascending arm: requirements with acceptance tests, the architectural model with system tests, the design model with integration tests, and the code with unit tests. Designing the tests together with the artefact is what makes the artefact testable, and is the structural reason why test-driven approaches work at every level, not only at the unit level.

What does model-driven engineering contribute to testing, according to the framing diagram?

It supplies the oracles. Requirement-to-code produces, by model-to-model transformation, an executable specification, an architectural model and a design model; each of them is the reference against which acceptance, system and integration tests respectively are checked. For executable models, some behavioural correctness can be checked directly on the model, before any code exists. Code-to-operation then executes tests in code, verifying it in isolation, in integration and deployed.

Why is "the system under development" defined to include more than code?

Because the definition of testing given in the lecture says the system under development is your code but also companion data, specifications, models and designs. Once those artefacts are inside the system under test, testing them becomes meaningful, and that is what allows validation to start before implementation. It is also what makes an executable specification such as a Gherkin feature file, or an executable model such as a Petri Net, a legitimate object of verification rather than mere documentation.

What does the exam consist of, and what should you prepare besides the artifacts?

A discussion of the tasks you completed, which can be one big task producing a software artifact, possibly evolving into a scientific paper, or a set of smaller tasks, explicitly not all of them, assuming roughly 90 to 100 hours of work in addition to the labs. Besides the artifacts you should prepare the links to other parts of the course, since the discussion explicitly covers them: for each task, be ready to say which other module it connects to and how.

Why does the course insist that scale eventually forces complexity, and where does that resurface later?

Because among the sources of complexity, scale is the one you cannot design away: as the system grows, elements of complexity become unavoidable. It resurfaces in Part C, where the number of states becomes too large to write a transition relation extensionally and intensional specification languages such as Petri Nets are introduced, and again in Part D, where aggregate computing abandons the single-device abstraction altogether and programs the ensemble, precisely because per-device control does not scale.