Скачать или смотреть Understanding the Difference Between JobManager and JobMaster in Apache Flink

Confused about the terms `JobManager` and `JobMaster` in Apache Flink? Discover the differences and understand the latest terminology changes implemented in Flink's architecture.
---
This video is based on the question https://stackoverflow.com/q/63669973/ asked by the user 'Tom' ( https://stackoverflow.com/u/4443784/ ) and on the answer https://stackoverflow.com/a/63673309/ provided by the user 'David Anderson' ( https://stackoverflow.com/u/2000823/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Confused with JobManager and JobMaster

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding the Difference Between JobManager and JobMaster in Apache Flink

If you're new to Apache Flink, you might find yourself swimming in a sea of terminology and concepts. One area that often leads to confusion among newcomers is the distinction between the JobManager and the JobMaster. This guide aims to clarify these terms and explain how they relate to the architecture of Flink.

The Confusion: JobManager vs. JobMaster

Many Flink users report feeling baffled by the dual terminology: “JobManager” is frequently mentioned in documentation while “JobMaster” appears in the source code. What do these terms mean, and are they interchangeable?

This confusion often stems from a recent transition in Flink’s architecture, particularly following a change known as FLIP-6, which overhauled how these concepts are defined and implemented.

Short Answer: What You Need to Know

To put it simply:

The JobManager referred to in documents is not a single component. It comprises multiple elements:

Dispatcher: Manages the scheduling of tasks and execution plans.

ResourceManager: Manages resources specific to the cluster framework.

Blob Server: Handles binary large object storage.

JobMasters: A separate instance for each job that is executed.

In the codebase, you won't find a class named JobManager. Instead, you'll encounter the JobMaster, which is a distinct implementation focused on the execution of a single job.

Detailed Explanation of the Terms

To further clarify, let's look at some key points regarding the evolution of these terms:

Historical Evolution

Over the years, the concept of JobManager has evolved significantly.

The monolithic JobManager structure was refactored by the introduction of FLIP-6, breaking the functionality down into several components.

Terminology Changes

With the merging of FLIP-6, Flink documents introduced the term Flink Master

This term was intended to reflect the new architecture but created further confusion.

In the latest release (1.11), Flink reverted back to the term JobManager

However, the term JobMaster remains in the code for the component that manages job execution.

Current Understanding

The JobManager now broadly refers to the entire framework that includes the Dispatcher, ResourceManager, and JobMasters.

The JobMaster specifically refers to the processing unit responsible for individual jobs in the architecture.

Summary

The confusion between JobManager and JobMaster stems from historical naming conventions and architectural changes implemented in Apache Flink. Understanding these definitions is crucial for navigating Flink’s documentation and codebase effectively. Here’s a quick recap:

JobManager: An umbrella term that refers to multiple components managing jobs within the cluster.

JobMaster: A specific class responsible for executing individual jobs.

Understanding these distinctions helps new users grasp the complexities of Apache Flink’s architecture, leading to more effective development and deployment of streaming jobs.

If you're still feeling a bit lost, remember: this terminology can sometimes be tricky, but with practice, it will make much more sense!