This INCREDIBLE trick will speed up your data processes.

In this video we discuss the best way to save off data as files using python and pandas. When you are working with large datasets there comes a time when you need to store your data. Most people turn to CSV files because they are easy to share and universally used. But there are much better options out there! Watch as Rob Mulla, Kaggle grandmaster, discusses some alternative ways of saving data files: pickle, parquet and feather files. I run some benchmarks to show that you can save time, space and keep the important metadata about your files in the process!

Timeline
00:00 Intro
00:49 Creating our Data
02:08 CSVs
04:39 Setting dtypes for CSVs
06:15 Pickle Files
07:16 Parquet ❤️
09:07 Feather
10:31 Other Options
11:02 Benchmarking
12:19 Takeaways
12:43 Outro

Code Gist: https://gist.github.com/RobMulla/7384...
Follow me on twitch for live coding streams:   / medallionstallion_  

Other Videos:
Speed up Pandas:    • Make Your Pandas Code Lightning Fast  
Efficient Pandas Dataframes:    • Speed Up Your Pandas Dataframes  
Inroduction to Pandas:    • A Gentle Introduction to Pandas Data ...  
Exploritory Data Analysis Video:    • Exploratory Data Analysis with Pandas...  
Audio Data in Python:    • Audio Data Processing in Python  
Image Data in Python:    • Image Processing with OpenCV and Python  


* Youtube: https://youtube.com/@robmulla?sub_con...
* Discord:   / discord  
* Twitch:   / medallionstallion_  
* Twitter:   / rob_mulla  
* Kaggle: https://www.kaggle.com/robikscube

#python #code #datascience #pandas