Скачать или смотреть Bài 5 Các bước viết mã MapReduce Java MapReduce Application, Big Data Management

playlist:    • ISIT312 Big Data Management  

github: https://github.com/hoanglong1712/tai-...

2025 04 05 11 38 08

Java MapReduce Application

Java MapReduce Application
Outline
Building blocks of MapReduce program
Word Count: The "Hello, World" of MapReduce
Hadoop datatype objects
Input and output formats
Java code of Driver
Java code of Mapper
Java code of Reducer
Java code of ToolRunner
Setting up a local running environment
Combiner API
Partitioner API
Miscellaneous
2/53
Building blocks of MapReduce program
Structure of MapReduce Java Application
3/53
Building blocks of MapReduce program
Mapper, Reducer, Combiner, and Partitioner classes correspond to their
counterparts in the MapReduce model
The Driver or ToolRunner in a MapReduce program represents the client
program
These classes implement the MapReduce logic
The main method of a MapReduce program is in the Driver or ToolRunner
The code of the two is very standard
-
-
4/53
Building blocks of MapReduce program
An elementary MapReduce program consists only of a Mapper class, a
Reducer class and a Driver
As the main method is contained in the Driver, sometimes (but not
always) it is convenient to make Mapper and Reducer as inner classes in
Driver, which contains routine codes
5/53
Building blocks of MapReduce program: Driver
Driver is the program which sets up and starts a MapReduce application
Driver code is executed on the client; this code submits the application
to the ResourceManager along with the application's configuration
Driver can submit the job asynchronously (in a non-blocking fashion) or
synchronously (waiting for the application to complete before
performing another action)– the second way will be used in our
examples
Driver can also configure and submit more than one application; for
instance, running a workflow consisting of multiple MapReduce
applications
6/53
Building blocks of MapReduce program: Mapper
Mapper Java class contains a map() method
Its object instance iterates through the input to execute a map()
method, using the InputFormat and its associated RecordReader
The number of HDFS blocks for the file determines the number of input
splits, which, in turn, determines the number of Mapper objects (or Map
tasks) in a MapReduce application (also see the previous lecture)
Mappers do most of the heavy lifting in data processing in MapReduce,
as they read the entire input file for the application
Mappers can also include setup and cleanup code to run in any given
object lifespan
Setup is called before the map() method; cleanup is called after it
7/53
Building blocks of MapReduce program: Reducer
Reducer runs against a partition and each key and its associated values
are passed to a reduce() method inside Reducer class
Reduce's InputFormat matches Mapper's OutputFormat
While Mapper usually do the data preparation, for example, filtering and
extracting), Reducer usually contains the main application logic
The runtime of Reducer instances is usually faster (and much faster in
some cases) than the runtime of Mapper instances
For example, summation, counting, and averaging operations are implemented
in Reducers
-
8/53
Java MapReduce Application
Outline
Building blocks of MapReduce program
Word Count: The "Hello, World" of MapReduce
Hadoop datatype objects
Input and output formats
Java code of Driver
Java code of Mapper
Java code of Reducer
Java code of ToolRunner
Setting up a local running environment
Combiner API
Partitioner API
Miscellaneous
9/53
Word Count: The "Hello, World" of MapReduce
WordCount: Read a text file and count occurrences of each word
Consider a text document containing a fragment of the works of
Shakespeare
The input format is TextInputFormat
After the text is read, the input to Map task, i.e. the process running
Mapper is the following
O Romeo, Romeo! wherefore art thou Romeo?
Deny thy father, and refuse thy name
Romeo and Juliet

(0, 'O Romeo , Romeo ! wherefore art thou Romeo ?')
(45, 'Deny thy father, and refuse thy name')
Romeo and Juliet
10/53
Word Count: The "Hello, World" of MapReduce
The output of the Map task is the following
Note that if you want to filter out some trivial words such as "a", "and", ...
then it can be done in a Mapper
('O', 1)
('Romeo', 1)
('Romeo', 1)
('wherefore', 1)
('art', 1)
('thou', 1)
('Romeo', 1)
('Deny', 1)
('thy', 1)
('father', 1)
('and', 1)
('refuse', 1)
('thy', 1)
('name', 1)
Key-Value pairs
11/53
Word Count: The "Hello, World" of MapReduce
Before sending data to Reduce task, there is a shu!e-and-sort stage
Shu!e-and-sort is usually hidden from a programmer
The following is the input to Reduce task

('and', [1])
('art', [1])
('Deny', [1])
('father', [1])
('name', [1])
('O', [1])