Скачать или смотреть Hadoop Rack Awareness

Hadoop Rack Awareness

Amazon Affiliates:
▶ JBL Quantum 200 Over-Ear- Gaming Headset - https://amzn.to/3iqempR
▶ Check out my list of recommended books on Hadoop/Spark/Storm/DataScience/Kafka/Cassandra/Mongodb -- http://bit.ly/2FaP1MF *****
▶ Check out my selection of books on Python/DeepLearning/MachineLearning/TensorFlow http://bit.ly/2LvtxhX *****
▶ Check out my list of DSLR and other recommended Gear http://bit.ly/2qXbgkp

Rack Awareness
Rack Awareness in Haoop

To get maximum performance out of Hadoop, it is important to configure Hadoop so that it knows the topology of your network. If your cluster runs on a single rack, then there is nothing more to do, since this is the default. for multirack clusters, you need to map nodes to racks. This allows Hadoop to prefer within-rack transfers (where there is more bandwidth available) to off-rack transfers when placing MapReduce tasks on nodes.

HDFS will also be able to place replicas more intelligently to trade off performance and resilience.

Why Rack awareness?
In Hadoop, rack awareness is required for below reasons:
• To improve data high availability and reliability.
• Improve the performance of the cluster.
• To improve network bandwidth.
• Avoid losing data if entire rack fails though the chance of the rack failure is far less than that of node failure.
• To keep bulk data in the rack when possible.
• An assumption that in-rack id’s higher bandwidth, lower latency.

To use rack awareness, we essentially provide Hadoop with a script that it consults on file write, which tells it which nodes live in which racks.

Hadoop can then make an intelligent decision about spreading the blocks across two or more racks, and in this case Hadoop will write a single block to one node in one rack, and then it will write all subsequent copies of that block to a second rack.
So now we have no single point of failure. We can lose any single rack, and we still have the ability to recreate the file and run processing on it.