Скачать или смотреть SQL pivot table rows to columns example explained | sql Databricks tutorial | SQL interview question

Hey everyone, pivot table is one of the highy used functions in data engineering field. You can find pivot is implemented at a lot of places to create columns from the repeating row values in a dataframe. Basically, pivot converts long spark dataframes into wider ones. Also, it is applicable on sql tables as well. A sql table can be pivoted to convert rows into columns.

Pivot as a concept is very useful, a lot of data engineering interviewers and interview panelist toss atleast one question related to sql pivot function or pyspark pivot function. These are the questions you can expect in mid-senior level as well for the companies like Infosys, TCS, Accenture, Yash Technologies and a lot of other organizations. SQL pivot is one of the most asked pyspark developer interview question. This is infact a most frequently asked data engineering interview question and we're trying to crack that.

This video explain how to implement pivot function on a table using both sql and pyspark. I'm using Databricks community edition platform to run and perform both spark and sql pivot function. Databricks makes it super easy to practive all these things as it's free to use and don't need any installations on your local machine to run python, Spark or SQL.

In this sql query I firstly spin up a cluster to run python, spark and SQL then create a dataframe using pyspark. I've written a pyspark query too to convert our dataframe into the a pivoted one. Later I'm converting the spark dataframe to a Temp View databricks delta table so to be able to run the sql query on databricks using %sql databricks magic command in a new code cell.

Below are the key points that I'm covering in the above video.