Chapters 3 and 4 gave us a place to put bytes and a way to compute over them. This chapter is about everything in between: how the data is organised once it lands, and who decides where and when each task actually runs.
The deck opens with a map of the whole flow, and it is worth reading as a picture of the entire course. On the left, data enters either as near-real-time streams or through ETL. Streams pass through an event-driven service bus (e.g., Kafka) and a streaming engine (e.g., Spark Streaming, Storm) into real-time processing. In parallel, batch processing feeds analytics (e.g., Spark, Cloudera, Flink) and interactive applications (e.g., NoSQL, NewSQL). And underneath all of it, holding everything together, sits the Data Lake — a Distributed File System plus DBMSs.
Use that picture as a table of contents for the rest of these notes. The batch branch is Chapters 5 to 9; the streaming branch, with Kafka and the event bus, is Chapter 10; the interactive branch with NoSQL and NewSQL is Chapter 11; and the governance question that this whole diagram raises — who keeps it from becoming a mess — is Chapter 12.
The Data Lake started as a deliberately minimal proposal:
In the original idea, the Data Lake was just a central repository system for storage, processing and analysis of raw data, in which the data is kept in its original format and is processed to be queried only when needed.
The appeal is obvious: no schema to agree on before ingestion, no transformation cost paid for data nobody may ever query, no loss of information from an early normalisation. The catch is equally obvious in hindsight. Soon it became clear that — in the absence of organization and governance — the Data Lake could easily become a Swamp (Couto, Borges, Ruiz, Marczak, Prikladnicki, A mapping study about data lakes: An improved definition and possible architectures, Proc. SEKE, Lisbon, 2019).
A swamp is a lake in which the data is still all there and nobody can find, trust or interpret any of it. Deferring the schema does not delete the work of understanding the data; it just moves that work to the moment of the query, and multiplies it by the number of people who ask.
The remedy is structure. To facilitate data governance, the Data Lake can be partitioned into areas, representing the processing state of the data. Note the criterion carefully — the areas are not divided by subject or by owner, but by how far along the pipeline a dataset has travelled.
| Area | What it contains |
|---|---|
| Landing | Where data are stored immediately after ingestion |
| Working | Temporary tables and configuration data |
| Harbor | Data (completely or partially) processed from the landing area |
| Discovery | Data created by data scientists for experimental purposes; it serves as a safe sandbox, where data is only taken from (and not published to) other areas |
| Access | Data ready to be accessed by reporting and visualization tools |
Francia, Gallinucci, Golfarelli, Leoni, Rizzi, Santolini, “Making data platforms smarter with MOSES”, Future Generation Computer Systems 125 (2021): 299–313.
The Discovery area is the interesting one. Its rule is a one-way valve: data scientists may read from any area, but what they produce stays inside Discovery and is never published back. That single constraint is what lets exploration be genuinely free without contaminating the datasets that reporting depends on — the experiment cannot leak into production by accident.
A common reference today is the medallion architecture, which compresses the same idea into three named layers:
| Layer | Definition |
|---|---|
| Bronze | Where data are stored immediately after ingestion; a common practice here is to apply only minimal transformations on the data (e.g., type conversions) |
| Silver | Contains data processed from the bronze layer; it represents the point at which raw data is filtered, cleaned and “dressed up” (augmented) by joining across one or many other tables |
| Gold | Contains data ready to be accessed by reporting and visualization tools; it is purpose-built to solve explicit intended business goals |
Strengholt, P., Building Medallion Architectures: Designing with Delta Lake and Spark, O’Reilly Media, 2025.
The deck asks the question directly: is the medallion architecture just a “revamped” version of the 3-layer data warehouse architecture? The correspondence is exact:
| Data Lake area | Medallion level | Data warehouse layer |
|---|---|---|
| Raw area | Bronze | Source layer |
| Harbor area | Silver | Reconciled layer |
| Access area | Gold | Warehouse layer |
Golfarelli, M. and Rizzi, S., Data Warehouse design: Modern principles and Methodologies, McGraw-Hill, New York, 2009.
And the answer the slides give is honest: “Perhaps. Ultimately, there is no universal number of areas/levels/layers.” The basic philosophy is common; then you specialize it based on the complexities within your organization and data pipelines.
If asked to compare the Data Lake with the Data Warehouse, do not fall into the “lake = new, warehouse = old” trap. The layered organisation is the same idea in both: raw → reconciled → ready-to-serve. What genuinely differs is when the structure is imposed — the warehouse transforms on ingestion (schema-on-write), the lake keeps the original format and transforms on query (schema-on-read) — and the discipline required to stop the second one from becoming a swamp.
Chapter 4 described map and reduce tasks as though they simply happened. The deck now asks the three questions that were quietly postponed:
The answer is a component with a specific job description. The Resource Negotiator:
Apache YARN — Yet Another Resource Negotiator — is Hadoop’s cluster resource management. It was introduced in Hadoop 2 to improve the MapReduce implementation, and — this is the part that matters historically — it is general enough to support other distributed computing paradigms.
YARN provides APIs for requesting and working with cluster resources, and these APIs are typically used by distributed computing frameworks (e.g., MapReduce, Spark), NOT by user code. That single design decision is why Spark could later arrive and take over the computation layer without anyone having to replace the cluster underneath: YARN does not know what a map task is. It knows about containers with CPU and memory, and it will hand them to whoever asks.
YARN provides its core services via two types of long-running daemons:
| Daemon | Scope | Responsibility |
|---|---|---|
| Resource Manager (RM) | Global — one per cluster | The ultimate authority that arbitrates resources among all the applications |
| Node Manager (NM) | Per-node slave | Responsible for containers. Monitors the resource usage of containers (e.g., CPU, memory) and reports them to the RM |
A container is an abstract entity, used to execute an application-specific process with a constrained set of resources. It is the unit of everything in YARN: allocation, isolation, accounting and failure.
Upon request from a client, the RM finds a NM that can launch the application master process (AMP) in a container. The AMP:
The most-missed point about YARN is why the AMP exists at all. If the RM scheduled every task of every job directly, it would become a bottleneck and a single point of failure for progress. Instead the RM does one coarse thing per application — start an AMP — and the per-application AMP does the fine-grained work of requesting, launching and monitoring containers. The RM arbitrates between applications; the AMP manages within one.
Where a container is placed is not an arbitrary choice. YARN exploits cluster topology and data block replication to apply the data locality principle:
When computations involve a large set of data, it is cheaper (i.e. faster) to move code to data rather than data to code.
A map task is a few hundred kilobytes of JAR; its input split is 128 MB. Shipping the former is free, shipping the latter is the job. The following cases respect the order the resource manager prefers:
| # | Placement | What it costs |
|---|---|---|
| 1 | Process and data on the same node | No network at all: a local disk read |
| 2 | Process and data on different nodes of the same rack | One hop over the fast intra-rack link |
| 3 | Process and data on different racks of the same data center | Crosses the aggregation layer, where bandwidth is scarcer |
| 4 | Process and data on different racks of different data centers | The worst case: wide-area transfer of a whole split |
Pick where the free container is, relative to the node holding the input split, and see how MapReduce classifies the placement.
Data locality is the point where all three previous chapters meet. Chapter 2 said intra-rack bandwidth is much greater than inter-rack bandwidth — that is why the ladder has these rungs. Chapter 3 said replication is topology-aware and the NameNode knows which DataNodes hold each block — that is what makes the choice possible. Chapter 4 said the optimal split size equals the DFS block size — that is what makes one task correspond to one local block. Remove any of the three and locality stops working.
Application execution consists of three movements: the client submits the application, the RM bootstraps the AMP instance, and the AMP instance manages the application execution. In eight steps:
Advance through the sequence and watch which component is acting at each moment. Notice how quickly the Resource Manager drops out of the picture.
Follow the client through those eight steps. It talks to the RM exactly once, in step 1, and after step 3 it never talks to it again — all progress reporting goes directly to the AMP. Follow the RM and the pattern is the same: it launches one container per application and then steps aside. This is the architecture of a system designed for thousands of concurrent tasks: the global authority is deliberately kept out of every hot path.
Deciding which application gets the next free container is a policy question, not a mechanism one, so YARN provides a choice of schedulers and configurable policies.
Simple to understand, no configuration needed. Applications are served in the order they arrive.
Not suitable for shared clusters. One long job at the head of the queue starves everything behind it — a five-hour ETL blocks a five-second query, which is intolerable the moment more than one team uses the cluster.
Reserves a fixed amount of capacity to each job. The cluster is carved into guaranteed shares, so a small interactive job always has somewhere to run, regardless of what else is queued.
The cost of a guarantee is rigidity: capacity reserved for a queue that is currently idle is capacity nobody is using.
Dynamically balances the available resources between all running jobs. Rather than reserving shares up front, it continuously redistributes: with one job running, that job gets everything; when a second arrives, resources are rebalanced between them.
Now the implementation of the paradigm from Chapter 4. Recall the definition by Dean and Ghemawat (Google): MapReduce is a programming model and an associated implementation for processing and generating large data sets — and Hadoop MapReduce is an open-source implementation of the MapReduce programming model.
A MapReduce program, referred to as a job, consists of:
Each MapReduce job is divided by the system into smaller units called tasks — map tasks and reduce tasks. The tasks are scheduled using YARN and run on nodes in the cluster, and if a task fails, it will be automatically rescheduled to run on a different node.
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private static final IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
context.write(key, new IntWritable(sum));
}
}Compare this signature with the abstract one from Chapter 4: reduce (k2, list(v2)) → list(k3, v3). The Iterable<IntWritable> values is the list(v2) — the framework has already collected, shuffled and sorted every value emitted for this key, from every map task in the cluster, before this method is called even once.
public class WordCount {
public static void main(String[] args) throws Exception {
Job job = new Job(new Configuration(), "WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
}
}Those three listings are the entire what. There is not one line about which machine runs what, how the intermediate pairs travel, or what happens when a node dies mid-job. All of that is the how, and it is the subject of the next section. This is the founding compromise of Chapter 1 made concrete: you write forty lines of domain logic, and you accept that you do not control the execution.
The important idea behind MapReduce is separating the what of distributed processing from the how. We have seen the former; now the latter.
The developer launches the job on the client’s JVM, which contacts the YARN RM to submit the application. The execution framework takes care of everything else: it transparently handles all aspects of distributed code execution, on clusters ranging from a single node to a few thousand nodes.
| Step | What happens |
|---|---|
| 1 | The MapReduce program calls either the submit() or waitForCompletion() method to create the Job. |
| 2 | The Job contacts the RM to get an application ID. |
| 3 | The Job copies to the DFS the resources needed to run the job: configuration file, JAR file, input splits metadata. The job JAR is copied with a default replication factor of 10, so that NMs can easily access it. |
| 4 | The application is submitted to the RM. |
Chapter 3 established that the default replication factor is 3. The job JAR gets 10 because its access pattern is the opposite of a data block: instead of one task reading one block, potentially hundreds of Node Managers need this same small file at almost the same instant. More replicas means more nodes that can serve it in parallel, and a much better chance that each NM finds a copy locally. It is replication used for read throughput, not for durability.
| Step | What happens |
|---|---|
| 5a | The YARN Scheduler (called by the RM) allocates a container on a NM to host the AMP — called MRAppMaster in the reference picture. |
| 6 | The AMP is launched in the container. |
| 7 | Input split data are identified on the DFS. |
At this point the AMP evaluates the amount of resources required to run the job, and asks itself two questions: how many containers are necessary? and is the parallelization worth the overhead? For a small enough input, the answer to the second question is no — starting containers across a cluster costs more than simply running the job where it stands.
Step 8: the AMP requests containers for map (and reduce) tasks. Two rules govern the order:
The RM tries to honor the data locality constraint for map tasks, distinguishing three cases:
| Case | Definition |
|---|---|
| Data-local | The task is run on the same node that the input split resides on |
| Rack-local | The task is run on a different node in the same rack that the input split resides on |
| Neither | Some tasks are neither data-local nor rack-local |
Both scheduling rules in step 8 have a reason worth stating. Maps first, with higher priority, because no reducer can finish before every mapper has emitted its pairs — allocating reducers early would just hold containers idle. But reduce tasks start at 5% of map completion, not at 100%, because the shuffle can begin as soon as some map output exists: overlapping the transfer with the remaining map work hides most of the network cost. It is a deliberate compromise between not wasting containers and not serialising the two phases.
| Step | What happens |
|---|---|
| 9a | The AMP starts the containers by contacting the NMs, in accordance with the assignments made by the RM. |
| 9b | The container is initialized to run the map (or reduce) task. |
| 10 | The task loads the required resources: configuration file, JAR file, input data. |
| 11 | The map (or reduce) task is run. Code execution is isolated within the container: any bug or failure will not affect the NM. |
Jobs may take from tens of seconds to hours to run, so a job and each of its tasks carry a status, which includes:
The three reduce bands are a favourite exam detail, and they explain a phenomenon everybody who has watched a Hadoop console has seen: a reduce sitting at 33% for a very long time. That is not the reduce function being slow — it is the shuffle, i.e. the network transfer of map output, which is the expensive part. The user-defined reduce logic does not begin until the last third of the bar.
Chapter 3 said that on commodity hardware failure is the norm rather than the exception. Here is the compute-side counterpart of that statement. Note that who detects the failure depends on what failed:
| Failure | Detected and managed by | What happens |
|---|---|---|
| Failure of a worker | the AMP | The tasks assigned to this worker will have to be redone — and in case of a reduce failure, the map tasks that produced its input may have to be redone as well. The AMP sets the status of each failed task to idle and reschedules them on a worker when one becomes available. |
| Failure of the AMP | the RM | Job history is used to recover the state of the tasks that were already run. Worst case: the entire map-reduce job must be restarted. |
Recall from Chapter 4 that map output is intermediate output stored on the local disk, deliberately not in HDFS with replication, because that would be overkill. The consequence appears exactly here: when a worker dies, the map output it was holding dies with it. If a reducer needed those pairs, they no longer exist anywhere, and the corresponding map tasks must be re-executed to regenerate them. The storage decision and the recovery cost are two sides of one trade-off.
Stepping back, this is the complete list of what the MapReduce “runtime” does on your behalf:
| Concern | What it means |
|---|---|
| Scheduling | Assigns workers to map and reduce tasks |
| Data distribution | Gets data to the workers |
| Synchronization | Gathers, sorts and shuffles intermediate data |
| Errors and faults | Detects container failures and restarts |
And everything happens on top of a DFS.
Hadoop’s MapReduce is the result of many years of development — and that is precisely the problem. Follow the dates backwards:
In the meanwhile, a lot changed:
Against that backdrop, two families of limitation stand out.
| Hardware resources are not adequately exploited | The paradigm is strict |
|---|---|
|
|
The deck puts the two data-processing models side by side. In MapReduce, every iteration is an HDFS read, the computation, and an HDFS write; the next iteration reads it all back. Every interactive query re-reads the input from HDFS from scratch. The caption is blunt: slow due to data replication and disk I/O. Remember that an HDFS write is not one disk write — with replication factor 3, it is three, two of them across the network.
In Spark, the input is read once with one-time processing, and then every iteration and every query runs from what is already in memory: 10–100× faster than network and disk.
Set the number of iterations of an algorithm and compare how many times each model touches the distributed file system.
“Why is Hadoop MapReduce slow for iterative algorithms?” has a two-part answer, and most people give only the first. Part one: results are persisted to HDFS between jobs, so an n-iteration algorithm pays 2n distributed-storage operations, each write replicated three times. Part two: the paradigm itself is strict — an algorithm that does not decompose into map and reduce must be expressed as a chain of separate jobs, and it is the chain, not any single job, that creates those round trips. Spark attacks both: RDDs can stay cached in RAM, and the API is not limited to two functions.
That is the bridge to the next chapter. Everything in Chapter 6 — resilient distributed datasets, lazy evaluation, the DAG, caching — is an answer to one of the bullet points above.
Originally the Data Lake was just a central repository system for storage, processing and analysis of raw data, in which data is kept in its original format and is processed to be queried only when needed. It becomes a swamp in the absence of organization and governance: the data is all still there, but nobody can find, trust or interpret it. Deferring the schema does not remove the work of understanding the data — it moves that work to query time and multiplies it by the number of people asking.
Landing (data stored immediately after ingestion), Working (temporary tables and configuration data), Harbor (data completely or partially processed from the landing area), Discovery (data created by data scientists for experimental purposes), Access (data ready to be accessed by reporting and visualization tools). The areas represent the processing state of the data. Discovery is special because it is a safe sandbox where data is only taken from, and not published to, other areas — a one-way valve that lets experimentation be free without contaminating anything downstream.
Raw area = Bronze = Source layer; Harbor area = Silver = Reconciled layer; Access area = Gold = Warehouse layer. Bronze holds data immediately after ingestion with only minimal transformations such as type conversions; Silver is where raw data is filtered, cleaned and augmented by joining across one or many other tables; Gold is purpose-built to solve explicit intended business goals. Is medallion just a revamped 3-layer warehouse? “Perhaps” — there is no universal number of areas, levels or layers; the basic philosophy is common and you specialize it based on the complexities of your organization and pipelines.
It has a global view of the cluster resources; decides where and when each unit of work is allocated and with how many resources (CPU and RAM); uses a scheduling policy to manage concurrency; avoids over-instantiation of processes on the same worker; and handles fault tolerance. YARN was introduced in Hadoop 2 to improve the MapReduce implementation but is general enough to support other distributed computing paradigms, because its APIs for requesting cluster resources are used by frameworks (MapReduce, Spark) and not by user code. YARN allocates containers with CPU and memory; it does not know what a map task is.
The Resource Manager is global, one per cluster, and is the ultimate authority that arbitrates resources among all applications. The Node Manager is a per-node slave responsible for containers — abstract entities used to execute an application-specific process with a constrained set of resources — and it monitors their CPU and memory usage and reports to the RM. On a client request, the RM finds a NM that can launch the AMP in a container; the AMP controls the execution of one application, requests further containers dynamically to run the distributed computation, and offloads the burden from the RM. The RM arbitrates between applications, the AMP manages within one.
When computations involve a large set of data, it is cheaper (i.e. faster) to move code to data rather than data to code. Preference order: (1) process and data on the same node; (2) process and data on different nodes of the same rack; (3) process and data on different racks of the same data center; (4) process and data on different racks of different data centers. It works because block replication is topology-aware, so the scheduler has three candidate nodes per block rather than one.
(1) The client submits the application including the specifications to launch the AMP. (2) The RM negotiates a container in which to start the AMP and launches it. (3) The AMP registers with the RM on boot-up; after asking the RM for details, the client can communicate directly with its own AMP. (4) The AMP negotiates appropriate resource containers. (5) On successful allocation the AMP launches the container by giving the launch specification to the NM. (6) The application code in the container reports progress and status to its AMP. (7) The client communicates directly with the AMP for status and progress. (8) On completion the AMP deregisters with the RM and shuts down, letting its container be repurposed. The client contacts the RM only in step 1 and, after step 3, deals exclusively with its AMP.
(1) The program calls submit() or waitForCompletion() to create the Job. (2) The Job contacts the RM to get an application ID. (3) The Job copies to the DFS the resources needed to run the job — configuration file, JAR file, input splits metadata. (4) The application is submitted to the RM. The JAR is copied with a default replication factor of 10 so that Node Managers can easily access it: unlike a data block read by one task, this small file is needed almost simultaneously by many NMs, so extra replicas buy parallel read throughput and a better chance of a local copy — replication for performance, not durability.
Requests for map tasks are made first and with higher priority because no reducer can complete before the mappers have emitted their pairs, so allocating reducers early would hold containers idle. But reduce tasks are requested once 5% of the map tasks have completed, rather than 100%, because the shuffle can begin as soon as some map output exists — overlapping the network transfer with the remaining map work hides most of its cost.
The progress of a reduce is an estimate of the proportion of the reduce input processed, divided into 0–33% shuffling, 34–66% sorting, 67–100% reducing. This is why a reduce can appear stuck at around 33%: that is the shuffle, i.e. the network transfer of map output from every mapper, and the user-defined reduce function does not start running until the final third.
A worker failure is detected and managed by the AMP: the tasks assigned to that worker must be redone, the AMP sets each failed task to idle and reschedules it on a worker when one becomes available. An AMP failure is detected and managed by the RM: job history is used to recover the state of the tasks already run, and in the worst case the entire job must be restarted. A reduce failure can force map re-execution because map output is intermediate data written to the local disk, not to HDFS with replication — when the node holding it dies, those pairs no longer exist anywhere and the corresponding map tasks must regenerate them.
Hardware resources are not adequately exploited: every MapReduce job ends by saving results to HDFS which is slow; there is no possibility to keep data in RAM; the multi-core architecture is not easily exploited; the end result is that it is slow. The paradigm is strict: everything has to fit into map and reduce so programming is not straightforward; complex algorithms take multiple jobs and passes on hard disk; and it is not suitable for iterative algorithms or interactive processing. In MapReduce every iteration reads from and writes to HDFS (slow due to data replication and disk I/O), whereas Spark reads once and then works from memory, 10–100× faster than network and disk.