The course opens with the only argument nobody disputes: there is a lot of data. Between 100 and 200 ZB are generated worldwide every year, and — the point that really matters — data grows faster than computing power. That single sentence is the reason this course exists: if the two curves grew at the same rate, a bigger machine would always be the answer.
Two reference figures from the slides give a sense of scale:
| Source | Rate | Yearly figure |
|---|---|---|
| Facebook (2012 data) | Storage increases 500 TB/day (estimated more than 8× today) | 180 PB/year |
| SKA telescope | 3 TB every second = 260 PB/day | To be compressed and reduced to 600 PB/year |
Note the last cell: the SKA does not store 260 PB a day. It compresses and reduces the raw torrent down to 600 PB a year. Reduction at the source is not a detail, it is part of the architecture — an idea that returns in the acquisition stage of the lifecycle later in this chapter.
Big Data is not a size, it is a mismatch: data volume grows faster than the processing capability of the hardware you can reasonably buy. Everything else in the course — distributed storage, MapReduce, Spark, streaming, NoSQL — is a strategy for living with that mismatch.
The term big data comes in two flavours, and the course insists on the distinction because they behave very differently:
When does data become big? Hard to say. The slides sketch it as a plane with data volume on one axis and IOPS (Input/Output Operations Per Second) on the other: there is a region covered by normal processing capability, and everything beyond it is the big data region. You cross the border either by having too much data, or by demanding too many operations per second on it, or both.
The slides collect three definitions that all say the same thing in different words — all of them relative to available technology, none of them absolute:
| Source | Definition |
|---|---|
| Teradata Magazine, 2011 | “Big data exceeds the reach of commonly used hardware environments and software tools to capture, manage, and process it within a tolerable elapsed time for its user population.” |
| The McKinsey Global Institute, 2012 | “Big data refers to data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” |
| Wikipedia | “Big data is data sets that are so voluminous and complex that traditional data processing application softwares are inadequate to deal with them.” |
All three definitions are relative: they define big data with respect to the commonly used hardware and the typical software of their moment. What was big data in 2011 may be a laptop job today. This is exactly why the adjective (“big data tools”) is more useful than the noun.
The most popular characterisation is the list of V’s. The course presents four of them, while noting that the number keeps growing depending on who is counting.
Volume — large quantity of data. The obvious one, and the one that motivates distributed storage (Chapter 3) and distributed processing (Chapters 4–9).
Velocity — refers to the speed of data production… and to the speed of consumption and analysis. Both halves matter: a stream that arrives fast but may be analysed tomorrow is a very different system from one that must answer within seconds. Chapter 10 is entirely about this V.
Variety — structured, unstructured, multimedia. Variety is what breaks the single-schema assumption of a relational warehouse, and it is the driver behind schemaless storage (Chapter 11) and multi-layer platforms (Chapter 12).
Veracity — refers to the trustworthiness of data. Data can be inconsistent, incomplete, ambiguous, obsolete, false, approximate. Veracity is the V that survives no matter how much hardware you throw at the problem: it is handled with metadata, provenance and cleaning, not with more nodes.
“How many V’s?” is a slide in the deck with no fixed answer: different sources add Value, Variability, Visualisation and more. Learn the four above, and be aware that the list is marketing as much as taxonomy.
We have always known that data is powerful. So what changed? The slides give three answers.
New data sources drive an exponential growth of data, and growth opens new analytical opportunities. It is not only that there is more of the same data: there are kinds of data that simply did not exist before (sensors, clickstreams, social posts, wearables).
The second answer is a change in where the value sits. The quote from an Amazon employee makes it concrete:
“We sold more items today that didn’t sell at all yesterday than we sold today of all the items that did sell yesterday.”
The most value does not come from the small set of highly popular items, but from the long list of niche items. Put together, the individually-insignificant data is actually the most valuable — the inverse of the 80-20 Pareto rule. And you can only exploit a long tail if you can afford to keep and process all of it.
In some cases, more data is not just more: it is better. Two examples from the slides:
The deck lists success stories across domains: crime prevention in Los Angeles, diagnosis and treatment of genetic diseases, investments in the financial sector, generation of personalised advertising, astronomical discoveries, injury prevention of football players.
A more structured way to read them is the “today’s challenge / new data / what’s possible” table: in each row, a new data source is what turns an old problem into a solvable one.
| Today’s challenge | New data | What’s possible |
|---|---|---|
| Healthcare: expensive office visits | Remote patient monitoring | Preventive care, reduced hospitalization |
| Manufacturing: in-person support | Product sensors | Automated diagnosis, support |
| Location-Based Services | Real time location data | Geo-advertising, traffic, local search |
| Public Sector: standardized services | Citizen surveys | Tailored services, cost reductions |
| Retail: one size fits all marketing | Social media | Sentiment analysis, segmentation |
Is there a limit? Privacy can become an issue, and ethical questions must be asked. The slides cite four cautionary cases: the AOL search data leak (NYT, 2006), the de-anonymisation of the Netflix dataset against IMDB (Wired, 2007), Why Johnny Can’t Browse The Internet In Peace (Forbes, 2012), and How Companies Learn Your Secrets (NYT, 2012). Two of these are re-identification stories: data released as anonymous stopped being anonymous once it was joined with another dataset.
Big Data is not just about storage or analysis. Data follows a typical lifecycle, and each step poses its own challenges. The reference is a 2012 community white paper, Challenges and Opportunities with Big Data. The cycle has six stages, and the most important ones are acquisition and processing (the extraction–integration–analysis block).
| Stage | What it involves |
|---|---|
| Acquisition | Selection: big data does not have to be too big — understand which data is actually valuable. Filtering & compression: raw data is often too voluminous to store it all. Collect metadata: metadata is fundamental to understand, measure and control the data; provenance tracing enables trustworthiness, reproducibility, debugging. |
| Extraction | Transformation, normalization: pull out the required information and express it in a structured form suitable for analysis — often application dependent (different features are extracted from an MRI image and from an image of the Solar System). Cleaning, error handling: recognise inaccurate data (faulty sensors), unreliable data (biased opinions), obsolete data. |
| Integration | Discover the relationships between datasets. Standardization, entity resolution, conflict management. There is a tendency to adopt pay-as-you-go techniques: semi-automatic integration based on mappings, iteratively refined by the user as new mappings are discovered — as an alternative to a structured integration approach, which is too heavy. |
| Analysis | Exploration (approach the data explorative-ly, gain understanding through real-time interactions), analytics (traditional BI and OLAP, or data mining / machine learning / AI; develop scalable and fast performing algorithms), delivery (model and represent results so non-expert users can explore them further). |
| Interpretation | Do not rush to conclusions: it is not difficult to collect big data, it is hard to extract valuable knowledge. Fundamentals: domain knowledge, data provenance. Always verify results, account for the assumptions made, use small artificial samples to verify expectations. |
| Decision | Requires strong managerial skills: evolve the business strategy on the acquired knowledge, verify the impact of the decision on newly acquired data, provide feedback for continuous improvement. |
Two stages are quietly the hardest, and neither is “run the algorithm”. Acquisition decides what you will never be able to analyse (what you discard is gone), and interpretation decides whether the numbers you obtained mean anything at all.
Within the analysis stage, the slides distinguish three scenarios. They differ in latency, in the amount of data touched, and — importantly — in their I/O and transactional profile:
| Scenario | Computation | Latency | I/O profile |
|---|---|---|---|
| Analytics (batch) | Simple/complex computations over large amounts of stored data. Collect, transform and model data to discover useful information and support decision-making | minutes, hours | Append-only I/O, not necessarily persistent data, no ACID transactions |
| Interactive (near-real-time) | Simple computations over small/large amounts of stored data. Process data to quickly affect the environment (e-commerce, search engines, booking) | milliseconds, seconds | Read/write I/O, persistent data, (soft) ACID transactions |
| Streaming (near-real-time) | Simple computations over small amounts of continuously incoming data. Monitor and analyse data on the fly via time windows | near real-time | Stream I/O, possibly persistent data for analytics, no ACID transactions |
Read the workload, pick the scenario. Each verdict quotes the discriminating feature from the table above.
Orthogonal to the scenario is the goal of the analysis. The four levels build on each other:
| Goal | Question answered | Definition from the slides |
|---|---|---|
| Descriptive | What happened? | Gives insights into the past; simply signals that something is wrong or right |
| Diagnostic | Why did it happen? | Gives in-depth insights into a particular problem; measures historical data against other data |
| Predictive | What is likely to happen? | Uses the findings of descriptive and diagnostic analytics to predict what is likely to happen |
| Prescriptive | What should we do? | Prescribes what action to take to eliminate a future problem, or to take full advantage of a promising trend |
Keep the two axes separate. Scenario (batch / interactive / streaming) is about how data arrives and how fast you must answer. Goal (descriptive / diagnostic / predictive / prescriptive) is about what kind of question you are asking. A prescriptive analysis can be batch, and a descriptive one can be streaming.
The lifecycle raises three technical challenges that no single machine can solve:
And the solution the whole field converged on is stated in two lines:
Hide system-level details from the developers. No more race conditions, lock contention, etc.; no need to become hardcore techies.
Separate the what from the how. The developer specifies the computation that needs to be performed; the execution framework (the “runtime”) handles the actual execution. The cluster/datacenter is the computer.
Every framework in this course is an instance of that compromise. In MapReduce (Chapter 4) the developer writes two functions and the runtime does scheduling, data distribution, synchronisation and fault handling. In Spark (Chapter 6) the developer writes a chain of transformations and the runtime builds and optimises the execution plan. The price is always the same: you give up control over how, and you must learn to express your problem in the framework’s shape.
The history is short and worth memorising, because the names in it are the tools of the rest of the course.
Notice the shape of this timeline: storage first (2003), computation second (2004). That order is not an accident and it is the order of this course too — HDFS in Chapter 3, MapReduce in Chapter 4. A distributed processing paradigm only makes sense once the data is already distributed, because the whole trick is to move the code to the data.
The deck closes on the professional angle: (Big) Data Analyst, Architect, Engineer, Scientist. The main figure that emerged with big data is the data scientist — “the sexiest job of the 21st century” (Harvard Business Review, 2013), “2015’s hottest profession” (Mashable), “officially the best job in America” (Forbes, 2016).
Conclusions from the slides: we live in the era of Big Data, with a wide range of availability in different areas, big opportunities to solve big problems, and great demand for big-data-skilled professionals. Data can create value — the challenge is how to manage and use it; methodological aspects are important; and it is a rapidly evolving area.
| Module | Hours | Content |
|---|---|---|
| CSE + DTM Module 1 Theoretical background on Big Data |
20 hours | HW infrastructures, SW architectures · Data structures (Parquet) and storage services (DFS, NoSQL) · Batch applications: MapReduce programming paradigm, Apache Spark, SQL-on-Hadoop · Streaming algorithms and applications |
| CSE Module 2 In-depth practice + Big Data in the Cloud |
30 hours | Main focus: hands-on Apache Spark applications and non-trivial MapReduce programming |
| DTM Module 2 Handling Big Data in the Cloud + guided practice |
30 hours | Main focus: cloud platforms, including taxonomies, billing models, on-prem vs cloud · Industrial seminars (CSE students invited) |
Oral examination with questions on all (theoretical and practical) aspects of the course. A single session of about 30 minutes with both teachers, each covering his own module. Seminars are included. Interaction during lectures/labs is considered in the final evaluation. Exams must be in presence; a grade cannot be refused more than once. There are no scheduled dates — booking is self-service with a minimum of one week notice. Be prepared: if you fail or refuse, you wait one month to try again.
Project (mandatory, pre-arranged with the teacher, groups up to 2 people, guidelines on GitHub): vote(project) ∈ [0,2]; once discussed, the project is final.
Oral: discussion of the project (~15 min) plus questions on all aspects of the course (~30 min) on the same day; vote(oral) ∈ [0,30].
Final vote = vote(project) + vote(oral).
Find a suitable dataset — one or more tables/collections of data, overall in the order of GBs. Write a Spark program to run a not-so-complex analytical pipeline, with focus on proper design and performance tuning; verify and discuss results. Evaluation: 0 points for sufficient projects, 1–2 additional points based on technical complexity, performance evaluation, and creativity.
The CSE project is graded on design and performance tuning, not on dataset size or algorithmic novelty — “a not-so-complex analytical pipeline” is the wording. Chapters 6–9 (Spark, Spark SQL, tuning, non-trivial designs) are the ones that earn those extra points.
Four business questions, four analysis goals. Click a question to see which level of analytics it belongs to and why.
The bibliography of the course, grouped by topic — useful when a chapter of these notes leaves you wanting more:
| Topic | Books |
|---|---|
| Hadoop and Spark | Learning Spark. Lightning-Fast Data Analytics (2nd ed.), Damji, Wenig, Das, Lee; O’Reilly, 2020 · Hadoop. The Definitive Guide (4th ed.), Tom White; O’Reilly, 2015 · Spark: The Definitive Guide, Chambers & Zaharia; O’Reilly, 2018 |
| Algorithms and architectures | Designing Data-Intensive Applications, Martin Kleppmann; O’Reilly, 2017 · Architecting Modern Data Platforms, Kunigk, Buss, Wilkinson, George; O’Reilly, 2019 · Mining of Massive Datasets (2nd ed.), Leskovec, Rajaraman, Ullman; Cambridge University Press, 2014 |
| Streaming | Streaming Data — Understanding the real-time pipeline, Psaltis; Manning, 2017 · Streaming Systems, Akidau, Chernyak, Lax; O’Reilly, 2018 |
| Cloud | Cloud Computing for Science and Engineering, Foster & Gannon; MIT Press, 2017 · Designing Cloud Data Platforms, Zburivsky & Partner; Simon and Schuster, 2021 |
| NoSQL | Database Internals, Alex Petrov; O’Reilly, 2019 · Next Generation Databases. NoSQL, NewSQL, and Big Data, Guy Harrison; Apress, 2015 |
Because every accepted definition is relative to the available technology. Teradata speaks of data that “exceeds the reach of commonly used hardware environments and software tools… within a tolerable elapsed time for its user population”; McKinsey of size “beyond the ability of typical database software tools”. The frontier also has two dimensions, not one: data volume and IOPS. A modest dataset queried at an extreme rate can be a big data problem, and a huge dataset scanned once a month may not be.
As a noun (“we have big data”) the boundary between normal and big data is vague. As an adjective (“we use big data tools”) the meaning is much more specific and refers to identifiable things: a big data architecture (e.g., Lambda), big data tools (e.g., Apache Spark), a big data paradigm (e.g., Map-Reduce).
Volume (large quantity of data), Velocity (speed of production and of consumption/analysis), Variety (structured, unstructured, multimedia), Veracity (trustworthiness: data can be inconsistent, incomplete, ambiguous, obsolete, false, approximate). Volume, velocity and variety are attacked with infrastructure and tools; veracity is not — it is addressed with metadata, provenance, cleaning and domain knowledge.
The most value does not come from the small set of highly popular items, but from the long list of niche items: put together, the individually-insignificant data is actually the most valuable — the inverse of the 80-20 Pareto rule. It needs big data technology because exploiting the tail requires keeping and scanning everything: the moment you sample or aggregate away the rare items, the tail disappears and only the head is left.
Acquisition → Extraction → Integration → Analysis → Interpretation → Decision (and back to acquisition). The most important challenges are in acquisition and in processing, where processing means the extraction–integration–analysis block.
It is a technique used in the integration stage: semi-automatic integration based on mappings, iteratively refined by the user as new mappings are discovered. It is the alternative to the classic structured integration approach, which is considered too heavy for the volume and variety of big data. The trade-off is that you accept a partially-integrated dataset now instead of a fully-integrated one much later.
Analytics (batch): simple/complex computations over large amounts of stored data, minutes to hours; append-only I/O, not necessarily persistent data, no ACID transactions. Interactive (near-real-time): simple computations over small/large stored data, milliseconds to seconds; read/write I/O, persistent data, soft ACID transactions. Streaming (near-real-time): simple computations over small amounts of continuously incoming data, analysed on the fly via time windows; stream I/O, possibly persistent data for later analytics, no ACID transactions.
Descriptive (what happened — insights into the past, signals that something is wrong or right) → Diagnostic (why it happened — in-depth insights into a particular problem, measuring historical data against other data) → Predictive (what is likely to happen — built on the findings of the first two) → Prescriptive (what to do — prescribe an action to eliminate a future problem or exploit a promising trend).
1. Hide system-level details from the developers: no more race conditions, lock contention, and no need to become a hardcore techie. 2. Separate the what from the how: the developer specifies the computation to be performed, and the execution framework (“runtime”) handles the actual execution — the cluster/datacenter is the computer.
2003: Google publishes about its cluster architecture and distributed file system (GFS). 2004: Google publishes about MapReduce, a programming model built on top of GFS (C++, closed source). 2006: Apache and Yahoo! produce Hadoop and HDFS, the open-source Java implementations, evolved from Lucene/Nutch. The order matters because the processing paradigm presupposes the distributed storage: MapReduce derives its efficiency from running each task where its data block already lives.
1. Data cannot fit a single machine, so many computers must work together toward a common goal. 2. Parallelization: how to organise and assign work units to workers, and how to handle communication between work units and between applications/services. 3. Fault tolerance: how to avoid data loss when disks or machines break, and what to do when a machine breaks or the network goes down mid-processing.
A dataset in the order of GBs, and a Spark program running a not-so-complex analytical pipeline with focus on proper design and performance tuning, with results verified and discussed. Grading: 0 points for a sufficient project, plus 1–2 additional points based on technical complexity, performance evaluation and creativity; the final vote is vote(project) + vote(oral) with vote(oral) ∈ [0,30].