40. struck type vs map type in pyspark | what is the difference between struct type and map type

Azure Databricks #spark #pyspark #azuredatabricks #azure
In this video, I discussed How to use struct type & map type in pyspark.

1.Struct Type function in pyspark
2. Map Type function in pyspark
3. Struct Type vs Map type in pyspark

Create dataframe:
======================================================
from pyspark.sql.types import *
data=[(1,('A-424','Noida','India')),(2,('M.15','Unnao','India'))]
schema = StructType([
StructField('AddId', IntegerType(), True),
StructField('Address', StructType([
StructField('Add1', StringType(), True),
StructField('City', StringType(), True),
StructField('Country', StringType(), True)
]))
])
dfST=spark.createDataFrame(data,schema)
display(dfST)
-----------------------------------------------------------------------------------------------------------------------
df1=dfST.select('*',dfST.Address.Country.alias('Address1'))
display(df1)
---------------------------------------------------------------------------------------------------------------------
data1=[(1,{'Laptop':'Apple',"Mobile":"OnePlus","HeadPhones":"boat"}),(2,{'Laptop':'Apple',"Mobile":"OnePlus"})]
from pyspark.sql.types import StructField, StructType, StringType, MapType
schema1 = StructType([
StructField('EmpId', IntegerType(), True),
StructField('Items', MapType(StringType(),StringType()),True)
])

dfMT=spark.createDataFrame(data1,schema1)
display(dfMT)
-------------------------------------------------------------------------------------------------------------------
from pyspark.sql.functions import explode
display(dfMT.select('*',explode(dfMT.Items)))
============================================================

37. schema comparison in pyspark | How to Compare Two DataFrames in PySpark | pyspark interview:
   • 37. schema comparison in pyspark | Ho...  


Learn PySpark, an interface for Apache Spark in Python. PySpark is often used for large-scale data processing and machine learning.

Azure Databricks Tutorial Platlist:
   • Azure Databricks Tutorial  

Azure data factory tutorial playlist:
   • Azure Data factory (adf)  

ADF interview question & answer:
   • adf interview questions and answers f...  

1. pyspark introduction | pyspark tutorial for beginners | pyspark tutorial for data engineers:
   • 1. pyspark introduction | pyspark tut...  

2. what is dataframe in pyspark | dataframe in azure databricks | pyspark tutorial for data engineer:
   • 2. what is dataframe in pyspark | dat...  

3. How to read write csv file in PySpark | Databricks Tutorial | pyspark tutorial for data engineer:
   • 3. How to read write csv file in PySp...  

4. Different types of write modes in Dataframe using PySpark | pyspark tutorial for data engineers:
   • 4. Different types of write modes in ...  

5. read data from parquet file in pyspark | write data to parquet file in pyspark:
   • 5. read data from parquet file in pys...  

6. datatypes in PySpark | pyspark data types | pyspark tutorial for beginners:
   • 6. datatypes in PySpark | pyspark dat...  

7. how to define the schema in pyspark | structtype & structfield in pyspark | Pyspark tutorial:
   • 7. how to define the schema in pyspar...  

8. how to read CSV file using PySpark | How to read csv file with schema option in pyspark:
   • 8. how to read CSV file using PySpark...  

9. read json file in pyspark | read nested json file in pyspark | read multiline json file:
   • 9. read json file in pyspark | read n...  

10. add, modify, rename and drop columns in dataframe | withcolumn and withcolumnrename in pyspark:
   • 10. add, modify, rename and drop colu...  

11. filter in pyspark | how to filter dataframe using like operator | like in pyspark:
   • 11. filter in pyspark | how to filter...  

12. startswith in pyspark | endswith in pyspark | contains in pyspark | pyspark tutorial:
   • 12. startswith in pyspark | endswith ...  

13. isin in pyspark and not isin in pyspark | in and not in in pyspark | pyspark tutorial:
   • 13. isin in pyspark and not isin in p...  

14. select in PySpark | alias in pyspark | azure Databricks #spark #pyspark #azuredatabricks #azure
   • 14. select in PySpark | alias in pysp...  

15. when in pyspark | otherwise in pyspark | alias in pyspark | case statement in pyspark:
   • 15. when in pyspark | otherwise in py...  

16. Null handling in pySpark DataFrame | isNull function in pyspark | isNotNull function in pyspark:
   • 16. Null handling in pySpark DataFram...  

17. fill() & fillna() functions in PySpark | how to replace null values in pyspark | Azure Databrick:
   • 17. fill() & fillna() functions in Py...  

18. GroupBy function in PySpark | agg function in pyspark | aggregate function in pyspark:
   • 18. GroupBy function in PySpark | agg...  

19. count function in pyspark | countDistinct function in pyspark | pyspark tutorial for beginners:
   • 19. count function in pyspark | count...  

20. orderBy in pyspark | sort in pyspark | difference between orderby and sort in pyspark:
   • 20. orderBy in pyspark | sort in pysp...  

21. distinct and dropduplicates in pyspark | how to remove duplicate in pyspark | pyspark tutorial:
   • 21. distinct and dropduplicates in py...