30 Data Skipping and Z-Ordering in Delta Lake Tables | Optimize & Data Compaction Delta Lake Tables

Video explains - What is the impact of data skipping on jobs? How z-ordering in delta lake works ? How to optimize delta lake tables?

Chapters
00:00 - Introduction
00:31 - What is Data Skipping and Z-Ordering in Delta Lake?
03:34 - Z-Ordering for more than 1 column/Multidimensional Z-ORDER
04:38 - Delta Lake Table Optimization with Example
11:59 - Multi Column Z-Ordering in Delta Lake Table
14:43 - Impact of Partitioning with Z-Ordering
16:24 - Selective Z-Ordering with Partition filters
17:57 - Auto Compaction in Delta Lake Table

For Local PySpark Jupyter Lab setup just run the command - docker pull jupyter/pyspark-notebook
Python Basics - https://www.learnpython.org/
GitHub URL for code - https://github.com/subhamkharwal/pysp...
Delta Lake Optimization Documentation - https://docs.delta.io/latest/optimiza...

The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.

New video in every 3 days ❤️

#spark #pyspark #python #dataengineering