The course opens by re-grounding everything in the discipline that contains it. Software engineering is an engineering discipline that is concerned with all aspects of software production, from the early stages of system specification through to maintaining the system after it has gone into use. Two halves of that sentence carry the weight.
“Engineering discipline” means that engineers make things work: they apply theories, methods and tools where these are appropriate. It is engineering as the science of the artificial — we are not discovering nature, we are building artefacts that did not exist. And engineering is about getting results of the required quality within schedule and budget; the qualifier is not decoration, it is the definition.
“All aspects of software production” means that software engineering is not just the technical process of writing software. It also includes software project management and the development of the tools, methods and theories that support development. The architect, as we will see in Chapter 2, lives exactly on that boundary.
Everything in this course is an answer to one question: how do we build software of the required quality, on schedule and on budget, when the system is too big for one person to hold in their head? Architecture is the discipline of the structures that make that possible.
The course distinguishes four levels, and the distinction matters because they change at very different speeds.
| Level | Definition | Rate of change |
|---|---|---|
| Principles | Important and general truths that are central to successful software development | Change slowly |
| Methods | General guidelines that govern the execution of some activity — rigorous, systematic | ↓ |
| Techniques | More technical and mechanical, more specific than methods | ↓ |
| Methodologies | Methods + techniques: promoting a certain approach to a problem by preselecting the methods and techniques to be used | ↓ |
| Tools | Developed to support the application of techniques, methods and methodologies | Change quickly |
Be able to place a concrete item on this ladder. “Domain-Driven Design” is a methodology (methods + techniques with a preselected approach); “separation of concerns” is a principle; “ArchUnit” is a tool; “write the interface definition first” is a technique.
Software development is a creative activity, and creativity carries a natural tendency to be neither precise nor accurate. Rigor — precision and exactness — is the necessary complement to creativity in every engineering activity: it is what lets us repeatedly produce reliable products, control their costs and increase our confidence in their reliability.
Rigor comes in degrees. Its highest degree is formality: requiring the software process to be driven by mathematical laws. Formality, when possible and useful, is the basis for mechanising the process.
Traditionally there is only one phase of software development where a formal approach is systematically used: programming. Programs are formal objects — written in a language whose syntax and semantics are fully defined, they are formal descriptions that can be automatically manipulated by compilers and executed by interpreters. The lesson of the module is that rigor and formality should be applied throughout the process, not only when we type code: doing so has a positive effect on reliability and verifiability, but also on maintainability, reusability, portability, understandability and interoperability.
This principle returns twice later in the course in a very concrete form: as fitness functions (Chapter 5), which mechanise architectural governance, and as executable specifications in Gherkin (Chapter 13), which mechanise acceptance criteria. Both are the same move: make an informal statement formal enough that a machine can check it.
Separation of concerns allows us to deal with different aspects of a problem separately, so that we can concentrate on each individually, in order to master inherent complexity. It is common-sense practice in everyday life, promoted to an engineering principle.
Why it is unavoidable: developing a software product requires many decisions — about the features of the product, about the development process, about economic and financial matters — and those decisions are very often strongly related and interdependent. Taking all the issues into account at the same time, by the same people, is not possible in practice. The only way to master the complexity of a project is to separate the different concerns: isolate the issues that are not closely related, and consider each one together with only the relevant details of the related issues.
The module also names the price: by separating concerns we may miss global optimisations that would only be possible by taking the parts together. And separation of concerns often becomes separation of responsibilities: different cooperating roles, such as analysts and programmers. (Chapter 4 will show that this innocent-looking consequence is exactly what Conway’s law is about.)
A fundamental application of the principle is to separate problem-domain concerns from implementation concerns: problem-domain properties hold in general, regardless of the implementation environment, because they concern the domain. Keep this sentence: Part II of the course (Domain-Driven Design) is an entire methodology built on top of it, and the Clean Architecture of Chapter 4 is its structural enforcement.
Modularity is the capability of dividing a complex system into simpler pieces called modules; a system composed of modules is called modular. Modularity lets separation of concerns be applied in two phases: first dealing with the details of each module in isolation, ignoring the details of the others; then dealing with the overall characteristics of all modules and their relationships, in order to integrate them into a coherent system.
Which of the two phases comes first names the design strategy:
| Capability | What it means |
|---|---|
| Decomposability | Decomposing a complex system into simpler pieces — the divide et impera principle |
| Composability | Composing a complex system from existing modules, bottom-up from elementary components |
| Understandability | Understanding the system in terms of its pieces |
| Modifiability | Modifying the system by modifying only a small number of its pieces |
The key design principle for modules is pursuing the goal of high cohesion and low coupling.
Why the pair matters: modular structures with high cohesion and low coupling let us see modules as black boxes when describing the overall structure of a system, and deal with each module separately when describing or analysing its functionality. That is the practical realisation of separation of concerns, and an effective means to achieve composability, decomposability, understandability and modifiability.
Abstraction is a fundamental technique for understanding and analysing complex problems in any engineering domain. In applying abstraction we identify the important aspects of a phenomenon and ignore its details — a special case of separation of concerns, where we separate the concern of the important aspects from the concerns of the less important details.
What counts as a detail that may be ignored depends on the purpose of the abstraction: there may be many different abstractions of the same reality, each providing a view that serves a specific purpose. In engineering, models are the result of an abstraction.
Abstraction permeates the whole of programming: the languages we use are abstractions built on top of the hardware, and the programs we write are abstractions related to some application domain. It applies to processes too — the comment in the header of a procedure is an abstraction of the effect of the code; the key factors used in cost estimation are abstractions of a project.
Anticipation of change is the ability to anticipate how and where changes are likely to occur. It is very important today because software evolves constantly, so a special effort is required to predict the shape of future change. Changes should be isolated in specific portions of the software, so that a change is restricted to those small portions.
Consequently, anticipation of change is the basis for the modularisation strategy: you do not decompose at random, you decompose along the lines where you expect change.
Reusability is strongly affected by anticipation of change. A component is reusable if it can be directly reused to produce a new product; more realistically it may need some changes first. In this sense reusability is low-grain evolvability — evolvability at component level. It also requires tools for managing versions and revisions in a controlled manner.
The principle of generality, quoted in full in the slides: “Every time you are asked to solve a problem, try to focus on the discovery of a more general problem that may be hidden behind the problem at hand. It may happen that the generalized problem is not more complex — indeed, it may be even simpler — than the original problem. Moreover, it is likely that the solution to the generalized problem has more potential for being reused.”
The caveat is explicit: the generalised solution may be more costly — in speed of execution, memory, time needed to adapt it — so an evaluation is necessary. Generality is nevertheless the principle that lets us develop general tools and packages for the market: it is why spreadsheets, databases and word processors exist as products.
Incrementality characterises a process that proceeds in a stepwise fashion, in increments: we try to reach the goal by successively closer approximations, each an increment over the previous one. Applied to software, the application is produced as the result of an evolutionary process.
One way of applying it is identifying useful early subsets of an application that can be developed and delivered to customers to get early feedback, letting the application evolve in a controlled manner.
The motivation is blunt: in most practical cases there is no way of getting all the requirements right before an application is developed. Requirements emerge as the application — or part of it — becomes available for practical experimentation. The sooner we receive feedback about usefulness, the easier it is to incorporate the required changes. Incrementality is intertwined with anticipation of change, and is one of the cornerstones on which evolvability is based. Intermediate stages may constitute prototypes of the end product.
The seven principles, in the order the module lists them: rigor and formality, separation of concerns, modularity, abstraction, anticipation of change, generality, incrementality. Note the internal dependencies you can quote: modularity is the size-dimension of separation of concerns; anticipation of change is the basis for the modularisation strategy; incrementality is intertwined with anticipation of change; abstraction is a special case of separation of concerns.
Software processes are the activities involved in producing a software system; software process models are abstract representations of those processes. Given the heterogeneity of software and of organisations, there is no single universally applicable software process — but every software process must include, in some form, four fundamental activities.
| Activity | Definition | Detail from the module |
|---|---|---|
| Software specification | The functionality of the software and the constraints on its operation must be defined | Requirements engineering is the process of developing a specification; specifications communicate the customer’s system needs to the developers |
| Software development | The software meeting the specification must be produced | Design and implementation transform a requirements specification into an executable system — from what to how |
| Software validation | The software must be validated to ensure it does what the customer wants | Checking that the system conforms to its specification and meets the real needs of its users |
| Software evolution | The software must evolve to meet changing customer needs | Changes are continuous; development and maintenance form a continuum |
Processes also include supporting activities such as software configuration management and project planning.
A software process model — also called a Software Development Life Cycle (SDLC) model — is a simplified representation of a software process, describing how software processes are organised. The module surveys the main families.
Sequential, plan-driven process. All activities must be planned and scheduled before starting the project; each activity is a separate phase arranged in linear order. Each phase produces documents that need to be approved before the next phase begins.
Verification and Validation model — an extension of the waterfall. Like the waterfall, all requirements are gathered at the start and cannot be changed, but for every phase there is an associated testing phase, planned in parallel. Strong discipline, good for software where failures are unacceptable; not ideal for complex projects or projects with unclear or changing requirements.
Developing an initial implementation, exposing it to user feedback, and evolving it through new versions. The initial implementation covers the most important functionality; subsequent increments expand on the previous ones until everything is implemented. Each iteration passes through requirements, design, coding and testing.
More effective because stakeholders and developers interact earlier, and efficient because developers only focus on what is important and bugs are fixed as they arise — but it requires a clear and complete definition of the whole system from the beginning.
The process starts by implementing a small set of software requirements and releasing it for user feedback; these are then enhanced iteratively in the evolving versions until the system is completed. Like the incremental model it produces results at the early stages; a good choice for large software that can easily be broken down into modules.
Rapid Application Development — based on iterative development and prototyping with little planning involved: requirements gathered through focus groups and workshops, reuse of software components, informal communication, functional modules developed in parallel for faster delivery.
Accommodates changing requirements, reduces development time and increases component reusability; more complex to manage. Good for systems that must be produced in a short time and have known requirements.
Risk-driven iterative process model that delivers projects in loops. Each cycle: address the highest-risk problem and determine the objective and alternate solutions; evaluate the alternatives and identify the risks involved and possible solutions; develop a solution and verify whether it is acceptable; plan for the next cycle. A good choice for projects with unclear needs, or still in research and development.
Integration and configuration — the general process model for reuse-based development, based on integrating and configuring existing components rather than writing everything from scratch. This is the process-level counterpart of the generality and reusability principles.
The agile model encourages continuous iterations of development and testing. Each incremental part is developed over an iteration, and each iteration is designed to be small and manageable so it can be completed within a few weeks. Each iteration focuses on implementing a small set of features completely, involving customers in the development process and minimising documentation by using informal communication. It is modern, widespread in the software development community, and a good choice for projects with changing requirements.
| We value… | …over |
|---|---|
| Individuals and interactions | processes and tools |
| Working software | comprehensive documentation |
| Customer collaboration | contract negotiation |
| Responding to change | following a plan |
Principle 11 — best architectures emerge from self-organising teams — is the hinge between this chapter and the rest of the course. Chapter 4 will show the same claim from the other side, as Conway’s law and the Inverse Conway Manoeuvre: team structure and architecture are the same object seen twice.
Various agile methods have been developed. Some focus on practices — the main example being Extreme Programming (XP). Others focus on managing the flow of work — the main examples being the Scrum and Kanban frameworks. Some others seek to cover the full development life cycle.
Among agile software development practices the module names three, and the first one is where the course goes next:
To all aspects of software production, from early system specification through to maintaining the system after it has gone into use. This explicitly includes non-technical activities such as software project management, and the development of tools, methods and theories that support development. And, because it is an engineering discipline, it commits us to results of the required quality within schedule and budget.
A method is a general guideline governing the execution of some activity, rigorous and systematic. A technique is more technical and mechanical, and more specific. A methodology is methods + techniques, promoting a certain approach to solving a problem by preselecting which methods and techniques to use. Tools are developed to support the application of techniques, methods and methodologies. Principles change slowly, tools change quickly.
Formality: requiring the software process to be driven by mathematical laws. It matters because formality, where possible and useful, is the basis for the mechanisation of the process. Traditionally programming is the only phase where a formal approach is systematically used, because programs are formal objects with fully defined syntax and semantics and can be automatically manipulated by compilers and interpreters.
We may miss global optimisations that would only be possible by taking the parts together. There is also an organisational consequence: separation of concerns tends to become separation of responsibilities, with different cooperating roles (analysts, programmers), which introduces communication boundaries.
Cohesion: a module has high cohesion if all of its elements are strongly related — statements, procedures, declarations grouped in the same module for a logical reason, cooperating to achieve the common goal that is the function of the module. It is an internal property of a module. Coupling: a module has low coupling if it has minimised its dependencies on other modules; coupling measures the interdependence between two modules, so it is an external property.
Decomposability (breaking a complex system into simpler pieces, divide et impera), composability (building a complex system from existing modules), understandability (understanding the system in terms of its pieces) and modifiability (modifying the system by modifying only a small number of pieces).
Because changes should be isolated in specific portions of the software, so that any change is restricted to those small portions. Deciding where to draw module boundaries is therefore deciding where you expect change to happen. This also explains reusability: a component is reusable if it can be reused directly to produce a new product, so reusability is low-grain evolvability — evolvability at component level.
That in most practical cases there is no way of getting all the requirements right before an application is developed. Requirements emerge as the application, or part of it, becomes available for practical experimentation. The sooner we get customer feedback about usefulness, the easier it is to incorporate the required changes; intermediate stages may serve as prototypes of the end product.
Specification (defining functionality and operational constraints — requirements engineering), development (producing software that meets the specification, i.e. going from what to how), validation (checking the system conforms to its specification and meets the real needs of users) and evolution (changing existing systems to meet new requirements). Supporting activities such as configuration management and project planning sit alongside them.
The V model keeps the waterfall’s up-front, frozen requirements but pairs every development phase with an associated testing phase, planned in parallel — strong discipline, good where failure is unacceptable. The spiral model is instead risk-driven and iterative: each loop addresses the highest-risk problem, determines objectives and alternative solutions, evaluates alternatives and risks, develops and verifies a solution, then plans the next cycle. It suits projects with unclear needs or in R&D.
Individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; responding to change over following a plan. Note the form: the items on the right still have value; the items on the left have more value.
Domain-Driven Design, Test-Driven Development and Behaviour-Driven Development. It matters because DDD — introduced by Evans in 2004 under the motto “Tackling Complexity in the Heart of Software” — is the methodology on which the whole second part of this course, and the route to microservices, is built.