Definitions & Goals for Distributed Systems

1. Prologue

Distributed systems are everywhere: the Web, cloud computing, IoT, peer-to-peer networks, online banking, multiplayer gaming. But what exactly is a distributed system? This lesson establishes a solid conceptual foundation by examining the most influential definitions and the engineering goals that have shaped the field over the last decades.

You will see that there is no single definition — different communities emphasise different aspects. The computer scientist looks at what users observe; the computer engineer looks at how the system is built; the middleware architect looks at how components communicate. Each perspective is valid and each reveals something essential.

2. Classic Definitions

User’s View (Computer Scientist Definition)

A distributed system is a collection of independent computers that appears to its users as a single coherent system. — Tanenbaum & van Steen, 2017

This is an observational, a posteriori definition: it describes what the user sees. The user does not care about the multiplicity of machines, the network topology, or the location of data. What matters is that the system “just works” as one coherent whole. This definition captures the illusion of unity that well-designed distributed systems provide.

Engineer’s View (Computer Engineer Definition)

A distributed system is a collection of autonomous computational entities conceived as a single coherent system by its designer. — Tanenbaum & van Steen, 2017

This is a constructive, a priori definition: it describes how the system is designed. The key word is conceived: the designer intentionally builds multiple autonomous entities with the explicit goal that they work together coherently. This perspective emphasises that distributedness is a design choice, not an accident.

3. Remarks on the Definitions

Several important observations cut across both definitions:

• Multiplicity of components — a distributed system is made of many independent or autonomous computational entities (computers, microprocessors, sensors, etc.).
• No assumption on individual nature — the definitions do not focus specifically on either computational processes or computing devices. The components can be heterogeneous in structure, behaviour, and capability.
• Need for coherence — despite multiplicity and heterogeneity, the system must appear or be conceived as a single coherent system.
• Heterogeneity is the default — the challenge is to achieve coherence over heterogeneity, not to enforce homogeneity.

4. Message-Passing Definition

A distributed system is one in which components located at networked computers communicate and coordinate their actions only by passing messages. — Coulouris et al., 2012

This definition foregrounds three essential features:

• Physical distribution — components are located at different networked computers, not in the same address space or even the same room.
• Interaction through messages — the only means of communication is message passing over a network. No shared memory, no shared clock.
• Uncoupling in control — each component makes its own decisions; coordination is achieved exclusively through message exchange.

5. Middleware Architecture

How do we make many different computers work together as one coherent system? The classic architectural answer is middleware.

graph TB
    subgraph "Machine A"
        AppA["Application A"]
        MidA["Middleware"]
        OSA["OS + Hardware"]
    end
    subgraph "Machine B"
        AppB["Application B"]
        MidB["Middleware"]
        OSB["OS + Hardware"]
    end
    AppA --> MidA
    MidA --> OSA
    AppB --> MidB
    MidB --> OSB
    OSA <==> OSB

The middleware layer extends over multiple machines and offers every application the same interface. It achieves two things:

• Collaboration (coherence) — by enabling meaningful interaction between autonomous components through well-defined communication mechanisms (message formats, protocols, semantics).
• Amalgamation (uniformity) — by hiding differences in technology, structure, and behaviour behind a common shared interface, just as an operating system hides hardware differences from applications.

6. Pitfalls of Distributed Systems

Building a distributed system looks easy — hardware, software, and networking components are readily available. However, distribution introduces problems rather than solving them at first glance. L. Peter Deutsch identified eight false assumptions that first-time distributed system developers make.

These false assumptions relate to properties unique to distributed systems: network reliability, security, heterogeneity, topology, latency, bandwidth, transport cost, and administrative domains. In non-distributed (centralised) systems, these problems typically do not surface because the entire system runs under a single administrative domain with predictable hardware and no network in the critical path.

7. Goal: Resource Availability

Resources in any real-world setting are physically distributed: printers sit in different rooms, storage servers live in different datacentres, sensors are scattered across a geographic area. A fundamental reason to build a distributed system is to make these distributed resources available to users as if they belonged to a single system.

What is a resource?

Anything that could be connected to a computational system and that anyone could legitimately use: printers, scanners, storage devices, distributed sensors, databases, compute nodes, network bandwidth, and so on.

By enabling interaction between users and resources, distributed systems become enablers of sharing, information exchange, and collaboration. A prime example is grid computing, where computational resources across institutional boundaries are pooled to solve large-scale problems.

8. Goal: Distribution Transparency

Physical distribution is often not a feature from the user’s perspective. Users want to use resources without caring where those resources are physically located or how they are implemented internally.

Transparency is the hiding of non-relevant properties of a system’s components and structure from the user’s perception. By hiding irrelevant details, distributed systems provide users with a higher level of abstraction.

9. Types of Transparency

There are several distinct types of transparency, each targeting a different aspect of distribution. Explore them in the interactive widget below.

These seven types of transparency are not independent — access transparency is foundational to most others, and failure transparency often depends on replication transparency. The ISO Reference Model for Open Distributed Processing (RM-ODP) formalises these into a standard taxonomy.

10. Access & Location Transparency

Access Transparency

Different resources represent data differently, have different internal structures, and expose different access protocols. Access transparency hides all of this heterogeneity behind a uniform, homogeneous interface. Whether a file is stored on a Linux server, a Windows NAS, or a cloud object store, the user accesses it through the same API.

Examples: NFS (Network File System) provides access transparency by making remote files appear as local files; SQL databases hide storage engine differences behind a standard query language.

Location Transparency

The physical location of a resource is often irrelevant to its use. Location transparency hides where a resource is physically situated. Resources are accessed through logical identifiers (names) that are not bound to physical network addresses.

The classic example is the URL: https://example.com/paper.pdf identifies a resource without revealing which server, datacentre, or country it lives in. The DNS system resolves the logical name to a physical address transparently.

11. Migration, Relocation & Replication Transparency

Migration Transparency

Resources and users can be mobile. A resource might move from one node to another, or a user might change their point of attachment. Migration transparency hides changes of location so that the system maintains coherence and functionality without disruption. Modern examples include virtual machine live migration in cloud environments and mobile users roaming between cell towers.

Relocation Transparency

While migration transparency handles the fact that a resource can move, relocation transparency handles the fact that a resource is moving right now. It is a specialised, online version of migration transparency: resources remain accessible even while they are changing location. This is significantly harder because both the old and new locations may need to cooperate during the transition.

Replication Transparency

Replication is used to improve performance (local copies reduce latency and bandwidth consumption) and fault tolerance (redundancy masks failures). Replication transparency hides the existence of multiple copies from users: all replicas share the same name and are kept in sufficiently consistent state that users perceive a single resource. Users should never need to know they are accessing a replica.

12. Concurrency & Failure Transparency

Concurrency Transparency

In a distributed system, users and resources work autonomously and concurrently. Two users might try to access the same resource at the same time. Concurrency transparency hides the fact that resources are shared among concurrent users. The system manages access policies and ensures that concurrent accesses do not corrupt resource state — and does so transparently.

The fundamental problem is consistency under concurrency: when multiple users read and write the same data simultaneously, the system must guarantee that the resource remains in a consistent state. This is typically achieved through locking, transactions, or optimistic concurrency control, all hidden from the user.

Failure Transparency

In a distributed system, anything can fail: a network link, a disk, a server process, a power supply. As Leslie Lamport famously noted: “You know you have a distributed system when the crash of a computer you’ve never heard of stops you from getting any work done.”

Failure transparency aims to mask failures and hide the recovery of resources from users. This is perhaps the hardest type of transparency because of the latency problem: how do you distinguish between a dead resource and a very slow one? Is “silence” from a resource caused by processing delay, by a deliberate choice to not respond, by resource failure, or by network failure? Without a bound on message delivery time, these are indistinguishable.

13. Degree of Transparency

Hiding distribution is not always the best idea. There are situations where awareness of distribution is valuable:

• Time zones — a user moving between time zones might trigger a download that appears to end before it started if the system hides location changes.
• Download source selection — knowing whether a file server is in the US, Japan, or Europe helps a user choose the closest source for a large download (unless the resource is replicated such that location does not matter).
• Performance vs. information — location awareness can improve performance; the optimal degree of transparency depends on the application.

14. Goal: Openness

An open distributed system is one that can work with a number and variety of components that is not determined once and for all at design time. Open systems are designed to be extensible and are fundamentally unpredictable in terms of what components may join later.

How to Design for Unpredictability

Designing for openness requires predictable items that all components agree on:

• Standard rules for service syntax and semantics
• Standard message interchange formats and protocols
• Interface Definition Languages (IDLs) to specify how interfaces are defined

IDLs capture syntax (the signatures of operations) but often do not specify semantics or the protocol (the expected order of interactions). This is a limitation: two components may agree on the interface syntax but still interoperate incorrectly if they expect different interaction sequences.

Non-Functional Properties of Open Systems

15. Goal: Scalability

A distributed system is scalable if it can handle growth along any relevant dimension without breaking.

Dimensions of Scalability (Neuman, 1994)

A system scales up when:

• The number of users and resources grows (size scalability)
• The geographical distribution of users and resources extends (geographical scalability)
• The system spans a growing number of distinct administrative domains (administrative scalability)

What Hinders Scalability?

Centralisation is the enemy of scalability:

Sometimes centralisation is necessary (security requirements, normative constraints, or optimal theoretical efficiency), but it should be avoided whenever possible.

Decentralised vs. Centralised Algorithms

The trouble with centralised algorithms in distributed systems (Raynal, 2013):

• Data must flow from the whole network to and from a single location, overloading the network
• Any transmission problem affects the entire algorithm

Features of decentralised algorithms (Kshemkalyani & Singhal, 2011):

16. Scaling Techniques

Three fundamental techniques for achieving geographical scalability (Neuman, 1994):

1. Hiding Communication Latency

The basic idea: avoid wasting time waiting for remote responses. Use asynchronous communication whenever possible: send a request, continue working, and handle the response when it arrives via an interrupt or callback. This prevents the application from stalling.

Limitation: Some interactions are inherently synchronous (a web user waiting for a page). In these cases, code shipping is an alternative — send executable code to the client (e.g., JavaScript form validation) so the interaction can proceed locally without round trips.

sequenceDiagram
    participant C as Client
    participant S as Server
    C->>S: Submit form
    Note right of C: (a) Server checks form
Round-trip latency for each error
    S-->>C: Error - retry
    C->>S: Resubmit
    activate S
    S-->>C: OK
    deactivate S

sequenceDiagram
    participant C as Client
    participant S as Server
    Note right of C: (b) Client checks form
via shipped JavaScript
    C->>C: Validate locally
    C->>S: Submit valid form
    activate S
    S-->>C: OK
    deactivate S

2. Distribution

Take a component, split it into parts, and spread the parts across the system. The classic example is the Domain Name System (DNS):

• The DNS namespace is hierarchically organised into a tree of domains
• Domains are divided into non-overlapping zones
• Each zone is served by a single (authoritative) server
• The naming service is distributed across many machines without centralisation

graph TD
    Root["."]
    com["com"]
    org["org"]
    it["it"]
    example["example.com"]
    unibo["unibo.it"]
    apice["apice.unibo.it"]
    Root --> com
    Root --> org
    Root --> it
    com --> example
    it --> unibo
    unibo --> apice

3. Replication

When performance degrades, replicate components across the system to increase availability and reduce latency. Copies are placed near potential users. Caching is a special form of replication, but with a key difference:

17. Goal: Situatedness

Situatedness is the property of a system being immersed in its environment: capable of (timely) perceiving and producing environmental change, and suitably dealing with environmental events. Mobile, adaptive, and pervasive computing systems have made situatedness a key concern.

Context-Awareness: Space & Time

Any non-trivial system needs to know where it is working and when, in order to perform its function effectively. This is the spatio-temporal context. Examples: a GPS-guided drone, a smart traffic system that adjusts signals based on time of day and congestion, a disaster response system that coordinates rescue teams based on their locations.

Context-Awareness: Environment

Beyond space and time, systems need awareness of the broader environment: its nature, structure, available resources, and potential hazards. This includes understanding the physical environment (temperature, humidity, obstacles) and the social environment (people, organisations, policies).

Knowledge-Intensive Environments (KIE)

A particularly relevant class is knowledge-intensive environments: environments where large amounts of distributed knowledge are essential for system activity. Systems must access, understand, and potentially inject knowledge while interacting locally within the working environment.

Situatedness & Distributed Systems

Physical distribution of computational systems is essential to cope with the distributed nature of many working environments. When requirements mandate situated computation within a distributed physical environment, situated distributed systems are the only way out. Examples:

• Disaster recovery scenarios (rescue teams, sensors, drones)
• Environmental monitoring (distributed sensors collecting data)
• Crowd steering and management
• Live event coordination

Openness of distributed situated systems is essential to deal with the unpredictability of complex environments. Scalability allows them to cope with working environments of growing complexity.

18. Conclusion & Open Questions

Lessons Learned

• There are several converging definitions for distributed systems, each highlighting different aspects (user view, engineer view, message-passing view).
• Distributed systems are easy to build — which means they are easy to misbuild. Deutsch’s pitfalls remind us that assumptions that work in centralised systems break in distributed ones.
• The five design goals — resource availability, transparency, openness, scalability, and situatedness — capture what makes a distributed system good, not just functional.
• Modern concerns increasingly emphasise robustness, reliability, and fault tolerance alongside the classical goals.

Open Questions

Check Your Understanding