Chapter 1 ended on a promise: the cluster is the computer. This chapter asks the concrete question that comes first — what kind of hardware do we need? — and the answer starts from a limitation that has nothing to do with software.
Big data is big: it does not fit a single drive, and it does not fit a single (typical) machine. Big data also requires a lot of computing resources, so the obvious first reflex — scale only the disk, with a NAS or a SAN — solves the wrong half of the problem. You end up with the capacity, but every byte you want to compute on must still travel across a wire to reach a processor. Scaling only the disk means a lot of data transfer.
The table the course uses to make this concrete is worth memorising, because it shows two quantities growing at very different speeds:
| Year | Size | Speed | Time to read the whole disk |
|---|---|---|---|
| 1990 | 1.3 GB | 4.4 MB/s | 5 minutes |
| 2014 | 1 TB | 100 MB/s | 3 hours |
| 2015 | 1 TB | 600 MB/s | 30 minutes |
| 2025 | 4 TB | 6 GB/s | 11 minutes |
The arithmetic behind the last column is just capacity divided by transfer rate:
time to read = capacity / transfer speed
1990 1.3 GB / 4.4 MB/s = 302 s ~ 5 minutes
2014 1 TB / 100 MB/s = 10 486 s ~ 3 hours
2015 1 TB / 600 MB/s = 1 748 s ~ 30 minutes
2025 4 TB / 6 GB/s = 683 s ~ 11 minutesRead the column, not the rows. Between 1990 and 2014 the disk got roughly 800 times bigger and only about 23 times faster, so the time to read it grew from five minutes to three hours. SSDs bought some of that back in 2015 and again in 2025 — but even in 2025, with a very fast drive, a single disk still needs eleven minutes just to hand over its own contents once. Multiply by a dataset that spans thousands of such disks and the conclusion writes itself: one machine reading its own storage sequentially is not a plausible plan.
The four rows of the table as presets, plus free inputs. The last line is the scale-out move: the same data spread over several machines, each reading its own share in parallel.
From here the deck opens the fork that organises the rest of the course:
Five moves an infrastructure team might make. Which axis is each one on?
Scale-up buys you a bigger computer; scale-out buys you more computers. Scale-up is always simpler — you change nothing about your software — which is exactly why it is the first thing everybody tries, and why the interesting question is where its ceiling is. The next two sections are about that ceiling.
Symmetric MultiProcessing is the architecture of the machine you are reading this on. It is adopted by traditional notebooks and workstations, and its defining property is that several processors share the same RAM, the same I/O bus and the same disk(s). Each processor has its own cache, but from the memory downwards everything is common property.
This is the natural home of scale-up: to make an SMP machine faster you add a processor, add RAM, add a disk. And it works, until it meets two hard walls:
The second limit is the interesting one, because it is structural rather than commercial. In an SMP machine the shared bus is a single resource that every added processor makes more contended. Scale-up does not merely stop helping at some point — past that point, the very thing that makes SMP convenient (one memory, one bus, one disk, therefore no distributed programming) is what caps it.
The alternative is stated in one sentence: a shared-nothing architecture is a group of computing nodes, working together to solve a common goal, where each (group of) computing unit has its own resources, and (groups of) computing units communicate with the others through the interconnect.
Nothing in that definition mentions a specific technology, and that is deliberate: shared-nothing is a family, not a product. Both architectures the course goes on to describe — clusters and MPP systems — are shared-nothing. What distinguishes them is what the nodes are made of and what the interconnect is.
Shared-nothing converts a hardware problem into a software problem. There is no bus to saturate because there is no shared memory — but there is now no shared memory, so processes cannot simply read each other’s variables. Everything they need from one another must be sent as a message over the interconnect. This is precisely why Chapter 1 promised that frameworks would “hide system-level details”: the details being hidden are the ones this architecture creates.
A cluster is a shared-nothing architecture where:
That last clause is the whole trade. You may add as many nodes as you can afford, from any vendor you like, and nobody can stop you; what you cannot do is make the network between them as fast as a memory bus. Every design decision in the rest of this course — where to place a block, where to run a task, when to shuffle — is a consequence of that single fact.
Compute nodes are physically stored on racks:
So the cluster is not a flat bag of machines: it is a hierarchy — data centers contain racks, racks contain nodes — and moving data up that hierarchy gets progressively more expensive. Hold on to this picture, because two of the most important mechanisms in the course are direct answers to it: HDFS replica placement puts copies on different racks on purpose (Chapter 3), and the scheduler tries to run each task on the node that already holds its data (Chapter 5).
Clusters are usually made with commodity hardware, and the deck defines the word carefully, because it is routinely misread. A commodity is something — a good, a product — that:
The examples given are gold and electricity: an ounce of gold is an ounce of gold whoever sold it to you. The pros follow directly — no vendor lock-in, and interchangeability / interconnectivity. The cons follow just as directly: you need to deal with failures.
Commodity is not the same as low-end. The slides are explicit: cheap components with a high failure rate can be a false economy. Commodity means interchangeable and vendor-neutral, not bargain-bin. If your nodes fail twice as often, you pay for it in replication, in re-executed tasks and in operations staff — and those costs are not on the invoice.
Massively Parallel Processing is also a shared-nothing architecture — this is the point students most often miss — but the resemblance stops at the definition:
So both architectures are shared-nothing, and both scale out. The difference is who owns the design. A cluster is assembled by you out of interchangeable parts; an MPP system is delivered by a vendor as a coherent product — with the performance guarantees and the constraints that implies.
Symmetric MultiProcessing. Several processors share the same RAM, the same I/O bus and the same disk(s). Adopted by traditional notebooks and workstations. Shares: everything below the caches. Scales by: scale-up. Stops at: the physical number of devices that can be mounted or plugged in, and the bus bottleneck.
Cluster. Shared-nothing, where every node is a system on its own (an SMP machine), typically connected through Ethernet or InfiniBand. Shares: nothing but the network. Scales by: scale-out, without limit and without vendor lock-in. Costs: scalability comes at the price of network speed, and you must deal with failures yourself. Usually built from commodity hardware.
MPP. Also shared-nothing, but the nodes are proprietary hardware modules, identical to each other, with a minimal OS — not independent SMPs — joined by proprietary, high-bandwidth interconnects. Shares: nothing but a very fast proprietary fabric. Scales by: scale-out in fixed increments, up to a fixed capacity (Blue Gene: increments of 1024 modules, up to 65,536). Costs: vendor lock-in on both hardware and software.
| Pros with an MPP | Pros with a cluster |
|---|---|
|
No implementation challenges — it is sold as-is (but you need the space to put it and the power to run it) High-speed message passing between computing nodes Specialized hardware and software — better performances and reliability; ideal for single, vertical solutions (HPC supercomputers, data warehousing applications like Teradata) |
Infinite scaling Cheaper Can be implemented on-premise or in-cloud Generalist hardware and software — commodity hardware, no vendor lock-in; ideal for heterogeneous applications |
The classic trap: MPP is not the opposite of shared-nothing. Both clusters and MPP systems are shared-nothing architectures. Sort them on three axes instead — who makes the nodes (any vendor vs one vendor), how far they scale (unlimited vs fixed capacity in fixed increments), and what the interconnect is (Ethernet/InfiniBand vs proprietary high-bandwidth). And remember the shape of the choice: MPP is ideal for a single vertical solution, a cluster for heterogeneous applications.
Having a single large cluster is tantalizing to many organizations, and for good reasons: no data silos, simpler governance. One place to look, one set of permissions, one operations team.
And yet, the deck says, multiple clusters are inevitable within medium-large enterprise settings. The reasons are worth reading as five separate arguments, because they come from five different departments:
| Reason | Why it forces a second cluster |
|---|---|
| Resiliency | Every cluster sits within a single point of failure due to geography. However well engineered, one site is one flood, one fire, one power grid. |
| Software development | Mitigate the risk of impacting critical production environments, by isolating configuration, integration or evolution testing and deployment. |
| Workload isolation | Hardware resources tuned for specific workloads, and less resource contention between them. |
| Legal separation | Some data cannot legally sit next to other data, or cannot leave a jurisdiction. |
| Storage / compute separation | Separate storage from compute so the two can scale differently — especially in cloud implementations. |
The last row quietly announces the direction of the whole field, and Chapter 12 will pick it up. In a classical Hadoop cluster, storage and compute live on the same nodes precisely so that computation can be moved to the data. Separating them undoes that arrangement on purpose, because in the cloud, elastic compute and cheap durable object storage grow at different rates and are billed differently. Two ideas that look contradictory are both right — in different economies.
Clusters and MPP are not the only shapes on the map. The deck lists the others in one breath: grid computing, Non-Uniform Memory Access (NUMA), Distributed Shared Memory (DSM), Network of Workstation (NoW), Constellations.
The comparison that actually matters for this course, though, is with High-Performance Computing. HPC generally refers to massively parallel systems specifically devoted to solving computationally-intensive tasks: scientific simulations, weather forecasts, 3D modeling — and the usage of GPUs is increasing. Big Data systems, conversely, are mostly data-intensive.
That one-word difference — computation-intensive versus data-intensive — explains every divergence between the two worlds, and the deck proves it with two machines at the same institution:
| Cluster @PoliTo | Cores | RAM | Storage | GPUs | Interconnect |
|---|---|---|---|---|---|
| Big Data computing cluster | 1400+ physical cores | 15 TB | 8 PB | 6 nVidia V100 | — |
| HPC computing cluster | 2900+ physical cores | 27 TB | 256 TB | 32 nVidia V100 | 100Gb/s Infiniband EDR |
Read the two rows against each other. The HPC cluster has roughly twice the cores, nearly twice the RAM and more than five times the GPUs — but the Big Data cluster has more than thirty times the storage (8 PB against 256 TB). Same institution, same budget order of magnitude, opposite shapes. HPC buys FLOPS; Big Data buys bytes and the bandwidth to reach them.
Hardware settled, the deck turns to the second question: what kind of software do we need? The answer is organised by the reference architecture published by NIST, which is not a product diagram but a division of roles. Five of them, and each is a different kind of actor.
In short:
The Framework Provider is itself layered, and the layering is the part worth memorising because the rest of the course walks up it:
| Layer of the Framework Provider | Responsibility | Where it appears in this course |
|---|---|---|
| Infrastructure frameworks | Support the underlying computing, storage and networking functions required to implement the overall system; associated with physical or virtual infrastructure resources | This chapter; the cloud in Chapter 12 |
| Data platform frameworks | Manage the organization and distribution of data (file systems, databases, etc.) | HDFS and Parquet in Chapter 3, NoSQL in Chapter 11 |
| Processing frameworks | How data will be processed in support of Big Data applications (e.g., batch or streaming framework) | MapReduce and Spark, Chapters 4–9; streaming, Chapter 10 |
It also handles the communication with the Big Data Application Provider and the assignment of physical resources to the respective activities.
The Big Data Application Provider executes a specific set of operations — a pipeline or workflow — that must meet the requirements established by the System Orchestrator as well as security and privacy requirements. Its activities are Collection, Preparation, Analytics, Visualization, Access. Each activity:
There may be multiple and differing instances of each activity, or a single program may perform multiple activities; and each of the functions can run on a separate Big Data Framework Provider, or all can use a common one.
The System Orchestrator integrates the required application activities into an operational vertical system; configures and manages the other components of the architecture to implement one or more workloads; monitors workloads and the system to verify that quality requirements are met; and elastically assigns and provisions additional physical or virtual resources. It is performed by human and/or software components.
At the two ends of the picture sit the parties that are not part of your system at all:
The deck also overlays real tools on the diagram, which is the fastest way to make the abstraction concrete: KAFKA at ingestion, HDFS as the data platform, YARN managing infrastructure resources, MAP-REDUCE / SPARK as the processing frameworks, and AIRFLOW on top as the orchestrator. Every one of those names is a chapter of this course.
Do not confuse the System Orchestrator with the Big Data Framework Provider. The Framework Provider supplies general resources and services and assigns physical resources to activities; the Orchestrator decides which programs run, when and how, monitors whether quality requirements are being met, and elastically provisions more resources. One offers capability, the other exercises judgement — and the Orchestrator may well be a human being.
Alongside the NIST roles, the deck presents a more implementation-flavoured reference (from the Microsoft architecture guide): the logical components that fit into a big data architecture. The caveat comes with it — individual solutions may not contain every item. This is a menu, not a checklist.
| Component | What it does |
|---|---|
| Data sources | Data stores, static files, real-time sources |
| Data storage | Distributed file store (data lake) or database |
| Batch processing | Long-running batch jobs to filter, aggregate, and prepare the data for analysis |
| Analytical data store | Serve the processed data in a structured format that can be queried using analytical tools |
| Analysis and reporting | Traditional OLAP and BI tools, interactive exploration, analytical notebooks |
| Real-time message ingestion | Capture and store real-time messages for stream processing; act as a buffer for messages; support scale-out processing, reliable delivery and message queuing semantics |
| Stream processing | Filter, aggregate and prepare the data for analysis; the processed stream data is then written to an output sink |
| Orchestration | Automation of repeated data processing operations, encapsulated in workflows — e.g., transforming source data, moving data between multiple sources and sinks, loading into an analytical data store, pushing results to a dashboard |
Notice that the list contains two processing paths — batch processing and stream processing — each with its own entry point (data storage on one side, real-time message ingestion on the other). That duplication is not an accident of the diagram: it is the question the next section answers.
| Batch analyses | Stream analysis |
|---|---|
|
Analytical algorithms are launched over large amounts of stored data (at rest) The typical analytical scenario Complex algorithms may take minutes/hours Run whenever needed (on-demand) |
Analytical algorithms are continuously running (non-stop) over potentially infinite data, as soon as it is collected Results in near real-time Useful for monitoring purposes One-pass computation |
And then the question that produces the last two architectures of the chapter: can we do both on the same data?
In a Lambda architecture, all data coming into the system goes through two paths:
The same events flow down both. The hot path answers now and may be approximate; the cold path answers later and is authoritative. The answer you serve is a blend of the two, with the batch result eventually superseding the streaming one.
In a Kappa architecture, all data flows through a single path, using a stream processing system:
Lambda used to be the easiest solution — but it needs the parallel development and maintenance of two parallel pipelines: same goal, different languages, different frameworks.
With Kappa, modeling everything under the streaming paradigm is not trivial. But the deck lines up the counter-arguments:
“Broad maturation of streaming systems combined with robust frameworks for unbounded data processing will in time allow for the relegation of the Lambda Architecture to the antiquity of big data history where it belongs.”
— Tyler Akidau, Slava Chernyak and Reuven Lax, Streaming Systems. The What, Where, When, and How of Large Scale Data Processing, O’Reilly, 2018
Both architectures map directly onto the components of the previous section: in Lambda the components split into a cold path and a hot path, while in Kappa a single unique path carries everything.
The logical components of section 9, lit up per path.
The slides draw this overlay as a picture on the components diagram rather than as a list. The widget above reconstructs it from the definitions given for each component and each path; treat the exact assignment of a borderline component as a reading aid, not as an examinable fact. What is examinable is the shape: two paths in Lambda, one in Kappa, and replay standing in for the batch layer.
State the difference in one sentence each. Lambda: all data goes through two paths — a hot path for timely but potentially less accurate results, a cold path for less timely but more accurate results — at the cost of developing and maintaining two parallel pipelines with different languages and frameworks. Kappa: all data flows through a single path using a stream processing system, and re-computing the entire dataset is done by replaying the stream. The argument in favour of Kappa is that a well-designed streaming system is a strict superset of batch functionality, with exactly-once semantics and strong consistency making its result identical to the batch result.
Because it solves capacity without solving access: scaling only the disk means a lot of data transfer. The data still has to travel to a processor to be computed on. The disk table makes the underlying problem visible: capacity has grown far faster than transfer speed, so a single 1 TB disk at 100 MB/s needs about three hours simply to hand over its own contents once. More storage attached to the same machine multiplies that problem instead of dividing it.
Scale-up (vertical scaling): adding more resources — processors, RAM, disks — or upgrading machines, for instance buying a more expensive and robust server, or buying specialized hardware. Scale-out (horizontal scaling): adding more machines. Scale-up makes one computer bigger and requires no change to your software; scale-out makes the number of computers larger and requires distributed software.
First, the physical number of devices that can be mounted or plugged in — a chassis has a finite number of sockets and slots. Second, the BUS bottleneck: in SMP several processors share the same RAM, the same I/O bus and the same disks, and the bus cannot go over a certain speed. The second limit is the structural one, because it means each additional processor increases contention on a resource that does not grow.
Yes — MPP is also a shared-nothing architecture, and this is the point most often confused. The differences: in a cluster every node is a system on its own (an SMP machine) connected by Ethernet or InfiniBand, with unlimited scalability and no vendor lock-in; in MPP the nodes are proprietary hardware modules, identical to each other, with a minimal OS (not independent SMPs), connected by proprietary high-bandwidth interconnects, with limited (fixed) capacity and vendor lock-in on both hardware and software — for example IBM Blue Gene, which scales in increments of 1024 modules up to 65,536.
A commodity is something that can be bought from a wide range of vendors, at more or less the same price, without being able to distinguish where it comes from — the examples given are gold and electricity. The pros are no vendor lock-in and interchangeability/interconnectivity; the con is that you need to deal with failures. What it does not mean is low-end: commodity is not the same as cheap, and cheap components with a high failure rate can be a false economy.
Compute nodes sit on racks — 8 to 64 nodes per rack, with many racks per cluster, and racks grouped into data centers. The rule is that intra-rack bandwidth is much greater than inter-rack bandwidth, and more generally that increasing networking layers decreases the overall bandwidth. So the cost of moving a byte depends on how far it has to travel through the hierarchy — which is why replica placement and task scheduling are both topology-aware in later chapters.
A single large cluster is tantalizing because it gives no data silos and simpler governance, but multiple clusters are inevitable because of: resiliency (every cluster sits within a single point of failure due to geography); software development (isolating configuration, integration or evolution testing and deployment mitigates the risk of impacting critical production environments); workload isolation (hardware tuned for specific workloads, less resource contention); legal separation; and the need to separate storage from compute so they can scale differently, especially in cloud implementations.
HPC generally refers to massively parallel systems specifically devoted to solving computationally-intensive tasks — scientific simulations, weather forecasts, 3D modeling — with increasing use of GPUs. Big Data systems are mostly data-intensive. The two PoliTo clusters show the consequence: the HPC cluster has more cores (2900+ vs 1400+), more RAM (27 TB vs 15 TB), far more GPUs (32 vs 6) and a 100Gb/s Infiniband EDR interconnect, while the Big Data cluster has 8 PB of storage against the 256 TB of the HPC one.
Infrastructure frameworks support the underlying computing, storage and networking functions required to implement the overall system, and are associated with physical or virtual infrastructure resources. Data platform frameworks manage the organization and distribution of data — file systems, databases and so on. Processing frameworks define how data will be processed in support of Big Data applications, for example a batch or streaming framework.
The Orchestrator is “the puppet master”: it decides which programs should be run, when and how; integrates the required application activities into an operational vertical system; configures and manages the other components of the architecture to implement one or more workloads; monitors workloads and the system to verify that quality requirements are met; and elastically assigns and provisions additional physical or virtual resources. It may be performed by humans and/or software. The Framework Provider, in contrast, supplies the general resources and services (and the assignment of physical resources to activities) that the application is built on.
Collection, Preparation, Analytics, Visualization, Access. What is special is their independence: each activity is specific to the application, and can be implemented by independent stakeholders and deployed as a stand-alone service. There may be multiple differing instances of each activity or a single program performing several of them, and each function may run on a separate Big Data Framework Provider or share a common one.
Batch analyses: algorithms launched over large amounts of stored data (at rest); the typical analytical scenario; complex algorithms may take minutes or hours; run whenever needed, on-demand. Stream analysis: algorithms continuously running non-stop over potentially infinite data as soon as it is collected; results in near real-time; useful for monitoring purposes; one-pass computation. The two architectures answer the question of doing both on the same data.
Lambda: all data entering the system goes through two paths — a hot path for timely yet potentially less accurate data in real time, and a cold path for less timely but more accurate data. Kappa: all data flows through a single path using a stream processing system; as in a Lambda speed layer, all event processing is performed on the input stream and persisted as a real-time view, and to re-compute the entire data set you simply replay the stream. The strongest argument for Kappa is that Lambda requires the parallel development and maintenance of two pipelines with the same goal but different languages and frameworks, whereas well-designed streaming systems provide a strict superset of batch functionality — with exactly-once semantics and strong consistency, the result of a streaming job equals that of a batch job — and unified frameworks such as Google Cloud Dataflow and Apache Flink let you write both kinds of job once.