Скачать или смотреть Unleashing the Power of Hadoop Cluster: 10 Must-Know Commands for Single Node Clusters

Video introduces the 10 Must know Hadoop Commands along with the process to access the Kaggle dataset using Python library.

Dataset can be located at
https://www.kaggle.com/datasets/stefa...

Hadoop shell command reference at
https://hadoop.apache.org/docs/r1.2.1...

Top 10 Hadoop Fs commands : All commands below start with hdfs dfs
hdfs is the main command to manage all the components of HDFS.
dfs is the sub command to manage files in HDFS
sudo -u hdfs hdfs dfs : Makes the user as hdfs

mkdir /dir = make directory
chown -R group:user /dir = change ownership
ls /dir |grep word = Global search

copyFromLocal or hdfs dfs -put=:to copy files or directories from local filesystem into HDFS
copyToLocal or hdfs dfs -get = to copy files or directories from HDFS to local filesystem.
rm -R -skipTrash /dir :delete directory recursively
tail : can be used to preview last 1 KB of the file
cat : can be used to print the whole contents of the file on the screen

hdfs fsck dir/file -files -blocks -locations (FS consistency check)
help stat
df :to get the current capacity and usage of HDFS.
du :to get the size occupied by a file or folder.

chmod : change the permissions of files.
owner : rwx, while members of group & others have r-x.
rwx stands : read, write and execute

The supporting playlists are
Python Data Engineering Playlist
   • Learn to Data Engineer and Problem Solve: ...  
Python Ecosystem of Libraries
   • Mastering the Python Ecosystem: Must-Know ...  
ChatGPT and AI Playlist
   • Learn about AI Language Models and Reinfor...  
AWS and Python AWS Wrangler
   • Building a Powerful Data Pipeline with AWS...  

PS: Got a question or have a feedback on my content. Get in touch
By leaving a Comment in the video
@mail [email protected]
@twitter Handle is @KQrios
@medium   / about  
@github https://github.com/Kamalabot