Скачать или смотреть 09 - Batch Processing Big Data: Apache Hadoop vs. Apache Spark

This video tells the story of two revolutionary technologies, Hadoop and Spark, that completely changed how companies handle massive amounts of data.

The Data Deluge and Hadoop
The "data deluge" of the early 2000s, caused by the internet's explosion, created a problem: a single computer could no longer store or process all the information. The solution was to spread data across many computers.
Apache Hadoop was the first framework designed to tame this big data. Its core functions were:
Storage: Using the Hadoop Distributed File System (HDFS) to make clusters of computers act like one enormous hard drive.
Processing: Using the powerful MapReduce model, which employs a "divide and conquer" strategy to chop up a massive problem into smaller pieces, work on them simultaneously, and then combine the results.
Hadoop was a game-changer—reliable and scalable—but it had a fatal flaw: it was not always fast, especially for iterative tasks or real-time queries.

Spark: The Successor
The need for speed led to Apache Spark, which is orders of magnitude faster than Hadoop.
The Secret: Unlike Hadoop's MapReduce, which read data from the slow disk after every operation, Spark keeps the data in the computer's lightning-fast memory (RAM) while working on it.
Technology: Spark uses a Resilient Distributed Dataset (RDD), which is a collection of data spread across many machines. It achieves reliability by saving the "recipe" (the lineage of instructions) instead of constantly writing to the disk, allowing it to quickly recreate lost data.

Spark is the clear successor and the next generation of big data processing. Today, Spark is the "hidden engine powering our modern world," used by companies like Netflix for movie recommendations, Goldman Sachs to analyze financial data, and Spotify for personalized playlists.