Скачать или смотреть REAL TIME PERFORMANCE TUNING IN SPARK | INTERVIEW QUESTIONS #2

00:00 Introduction
09:14 Spark Performance optimization
12:35 Spark Performance tips
13:01 Hive performance optimization

============================================================
SPARK INTERVIEW QUESTIONS PART - 2
===========================================================

Performance Related :

7. how to manage executor memory an executor core?
https://spoddutur.github.io/spark-not...


*Cluster Config:*
10 Nodes
16 cores per Node
64GB RAM per Node
Let’s assign 5 core per executors = --executor-cores = 5 (for good HDFS throughput)
Leave 1 core per node for Hadoop/Yarn daemons = Num cores available per node = 16-1 = 15
So, Total available of cores in cluster = 15 x 10 = 150 core
Number of available executors = (total cores/num-cores-per-executor) = 150/5 = 30
Leaving 1 executor for ApplicationManager = --num-executors = 29
Number of executors per node = 30/10 = 3
Memory per executor = 64GB/3 = 21GB
Counting off heap overhead = 7% of 21GB = 3GB. So, actual --executor-memory = 21 - 3 = 18GB

8. Spark performance tuning in ur project( explain about what are all the performance issue u faced)

Serialization
Writting data in to database with streaming logic.
Memory out of bound..etc

9. what are all the hive performance tuning done in ur project. explain about that
map side join (broadcast variables)
Partitions
Denormalizing data (normalization data )
RDBMS world
Input format (avro parqut etc)
vectoraization. 64km


10.if u face long running job in spark, how will you handle this performance?

read wheaterh
filter null
transformation cecious transformation stage1DF
stage1DF.persist(DISK_MEMORY)
us = table action
india = table action
UK = table action

First we need to check the yarn logs why its running for long time.
Then check the data if its more.
and check the DAG in yarn and find out which stage its taking more time.
based on the issue we have to handle the issue by increasing the memory or partitions or writting the better code.

11. How to integrate Hive and Spark?
Using spark 2.X version
sparkSession ss;
sc.sql ("hive query goes here")


Whatsapp Group Link: https://chat.whatsapp.com/D5h5Q68LT0p...

Please Read the group's Description after joining.

Java Basic Training in Tamil :    • Java Basics in Tamil | Tamilboomi  

Hadoop Learning in Tamil :    • Big data and Hadoop Introduction in Tamil ...  

Spark Learning in Tamil :    • Introduction to Spark | Spark in Tamil #1  

Career Training :    • 3 Points to make your resume better in Tam...  

Course Feedback :    • 20 Years Experience Hadoop Learning | #Mew...  

For More :
Whatsapp : +91 9619663272
Site : tamilboomi.com
Facebook : www.facebook.com/tamilboomiofficial
www.facebook.com/aruthecoach
Email : [email protected]
Twitter : www.twitter.com/aruthecoach