Скачать или смотреть Python || Pyspark || Tamil || Union() || UnionAll() || Functions

Python || Pyspark || Tamil || Union() || UnionAll() || Functions


Union() and unionAll() transformations are used to merge two or more Dataframes
of the same schema or structure

union() & UnionAll method merges two dataframes and returns the new data frame with
all the rows from two data frames regardless of duplicate data.

To Remove duplicates use distinct() function


from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql import dataframe
from pyspark.sql.functions import col
from pyspark.sql.functions import *

spark = SparkSession \
.builder \
.appName("Simple") \
.master("local[*]") \
.getOrCreate()

data1 = [('Ria', 'F', '20', 'IT'),
('Suma', 'F', '30', 'HR'),
('Kavi', 'M', '40', 'Payroll'),
('Barathi', 'X', '50', 'IT'),
('Varman', 'M', '100', 'Payroll')]

data2 = [('Ryan', 'M', '20', 'IT'),
('Suman', 'M', '20', 'HR'),
('Varman', 'M', '100', 'Payroll'),
('Rathi', 'F', '20', 'IT'),
('Barathi', 'X', '50', 'IT')]

schema = ['Name', 'Gender', 'Sal', 'Dep']

df3 = spark.createDataFrame(data1, schema)
df3.show()
df4 = spark.createDataFrame(data2, schema)
df4.show()

df5 = df3.unionAll(df4)
df6 = df3.union(df4)

df5.show()
df5.dropDuplicates().show()
df5.distinct().show()