Скачать или смотреть Spark Architecture | Driver | Executor | Job | Stages & Task

Saprk Architecture

Spark Driver Responsibilty:
1. requests resources (CPU, memory, etc.) from the cluster manager for Spark’s executors
2. Transforms all the Spark operations into DAG computations, schedules
3. Distributes their execution as tasks across the Spark executors.
4. Once the resources are allocated, it communicates directly with the executors.

Spark Session
implicit val spark: SparkSession = SparkSession
.builder()
.getOrCreate()

Spark executor
1. A Spark executor runs on each worker node in the cluster.
2. The executors communi‐ cate with the driver program and are responsible for executing tasks on the workers.

Example of spark-submit
spark-submit --master yarn-cluster \
--driver-cores 2 \
--driver-memory 2G \
--num-executors 10 \
--executor-cores 5 \
--executor-memory 2G \
--conf spark.dynamicAllocation.minExecutors=5 \
--conf spark.dynamicAllocation.maxExecutors=30 \
--conf spark.dynamicAllocation.initialExecutors=10 \ # same as --num-executors 10
--class com.spark.sql.jdbc.SparkDFtoOracle2 \
Spark-hive-sql-Dataframe-0.0.1-SNAPSHOT-jar-with-dependencies.jar