Скачать или смотреть Spark Performance Tuning | Practical Performance Optimization | Spark Developer

Dynamic Allocation Property set to true by default is false also dynamic initial and max executors should be set:
spark.dynamicAllocation.enabled= true
spark.dynamicAllocation.minExecutors=10,
spark.dynamicAllocation.maxExecutors= 30,

Executor and Driver memory need to be set:
spark.executor.memory=2g,
spark.driver.memory=2g,

Example of spark-submit
spark-submit --master yarn-cluster \
--driver-cores 2 \
--driver-memory 2G \
--num-executors 10 \
--executor-cores 5 \
--executor-memory 2G \
--conf spark.dynamicAllocation.minExecutors=5 \
--conf spark.dynamicAllocation.maxExecutors=30 \
--conf spark.dynamicAllocation.initialExecutors=10 \ # same as --num-executors 10
--class com.spark.sql.jdbc.SparkDFtoOracle2 \
Spark-hive-sql-Dataframe-0.0.1-SNAPSHOT-jar-with-dependencies.jar