scd2 in spark | Lec-24

In this video I have talked about slowly changing dimension type 2.

Directly connect with me on:- https://topmate.io/manish_kumar25

Discord channel:-   / discord  

SCD Data:-

customer_dim_data = [

(1,'manish','arwal','india','N','2022-09-15','2022-09-25'),
(2,'vikash','patna','india','Y','2023-08-12',None),
(3,'nikita','delhi','india','Y','2023-09-10',None),
(4,'rakesh','jaipur','india','Y','2023-06-10',None),
(5,'ayush','NY','USA','Y','2023-06-10',None),
(1,'manish','gurgaon','india','Y','2022-09-25',None),
]

customer_schema= ['id','name','city','country','active','effective_start_date','effective_end_date']

customer_dim_df = spark.createDataFrame(data= customer_dim_data,schema=customer_schema)

sales_data = [

(1,1,'manish','2023-01-16','gurgaon','india',380),
(77,1,'manish','2023-03-11','bangalore','india',300),
(12,3,'nikita','2023-09-20','delhi','india',127),
(54,4,'rakesh','2023-08-10','jaipur','india',321),
(65,5,'ayush','2023-09-07','mosco','russia',765),
(89,6,'rajat','2023-08-10','jaipur','india',321)
]

sales_schema = ['sales_id', 'customer_id','customer_name', 'sales_date', 'food_delivery_address','food_delivery_country', 'food_cost']

sales_df = spark.createDataFrame(data=sales_data,schema=sales_schema)


https://spark.apache.org/docs/latest/...

https://spark.apache.org/docs/latest/...

For more queries reach out to me on my below social media handle.

Follow me on LinkedIn:-   / manish-kumar-373b86176  
Follow Me On Instagram:-   / competitive_gyan1  
Follow me on Facebook:-   / manish12340  

My Second Channel --    / @competitivegyan1  

Interview series Playlist:-    • Interview Questions and answers  


My Gear:-
Rode Mic:-- https://amzn.to/3RekC7a
Boya M1 Mic-- https://amzn.to/3uW0nnn
Wireless Mic:-- https://amzn.to/3TqLRhE
Tripod1 -- https://amzn.to/4avjyF4
Tripod2:-- https://amzn.to/46Y3QPu
camera1:-- https://amzn.to/3GIQlsE
camera2:-- https://amzn.to/46X190P
Pentab (Medium size):-- https://amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- https://amzn.to/3RpmIS0
Mobile:-- https://amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- https://amzn.to/3Ns5Okj
Mouse+keyboard combo -- https://amzn.to/3Ro6GYl
21 inch Monitor-- https://amzn.to/3TvCE7E
27 inch Monitor-- https://amzn.to/47QzXlA
iPad Pencil:-- https://amzn.to/4aiJxiG
iPad 9th Generation:-- https://amzn.to/470I11X
Boom Arm/Swing Arm:-- https://amzn.to/48eH2we

My PC Components:-
intel i7 Processor:-- https://amzn.to/47Svdfe
G.Skill RAM:-- https://amzn.to/47VFffI
Samsung SSD:-- https://amzn.to/3uVSE8W
WD blue HDD:-- https://amzn.to/47Y91QY
RTX 3060Ti Graphic card:- https://amzn.to/3tdLDjn
Gigabyte Motherboard:-- https://amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- https://amzn.to/4avkgSK
Liquid cooler:-- https://amzn.to/472S8mS
Antec Prizm FAN:-- https://amzn.to/48ey4Pj