Скачать или смотреть Run Pandas as Fast as Spark - 10 useful practical functions

The aim of this video is to provide steps with the most used functions for managing DataFrames in Spark and their analogues in Pandas-on-Spark. Note that the only difference in syntax between Pandas-on-Spark and Pandas is just the import pyspark.pandas as ps line.

The Pandas API on Spark is a total game changer. Spark integrates a Pandas API so you can run Pandas on top of Spark.
we can obtain a wide range of benefits:
If you use Pandas but you are not familiar with Spark, you can work with Spark right away, with no learning curve.
You can have a single codebase for everything: small data and big data. A single machine and distributed machines.
You can run your Pandas code faster.

Let's switch between Pandas, Pandas-on-Spark, and Spark.
The first thing we need to know is what exactly we are working with. When working with Pandas, we use the class pandas.core.frame.DataFrame. When working with the pandas API in Spark, we use the class pyspark.pandas.frame.DataFrame . Both are similar, but not the same. The main difference is that the former is in a single machine, whereas the latter is distributed.

This tutorial can be fully replicated on your side with standard Jupyter Notebook installed. Just follow the instructions from the begining.

So, with this video we will cover the following Pandas and Spark functions on dataframes (do not forget SQL, it is also needed a little bit!):
0:00 - Intro and importing PySpark libraries
2:57 - Import libraries and set a Spark session
4:28 - Load data and construct Pandas and Spark dataframes
6:48 - Function: SELECT
8:17 - Function: DROP DUPLICATES
9:41 - Function: FILTER
12:34 - Function: COUNT
14:28 - Function: DISTINCT
15:38 - Function: SORT
17:18 - Function: GROUP BY (for aggregation)
18:30 - Function: REPLACE
19:46 - Function: JOIN
20:36 - Outro and good bye!

Be notified, that before executing all steps explained in this tutorial, you must install PySpark module to your Python environment with pip install pyspark in your terminal.

Official PySpark documentation: https://spark.apache.org/docs/latest/...

#pyspark #sql #dataframe