How to work with big data files (5gb+) in Python Pandas!

In this video, we quickly go over how to work with large CSV/Excel files in Python Pandas. Instead of trying to load the full file at once, you should load the data in chunks. This is especially useful for files that are a gigabyte or larger. Let me know if you have any questions :).

Source code on Github:
https://github.com/KeithGalli/Data-Sc...

Raw data used (from Kaggle):
https://www.kaggle.com/datasets/mkech...

I want to start uploading data science tips & exercises to this channel more frequently. What should I make videos on??

-------------------------
Follow me on social media!
Instagram |   / keithgalli  
Twitter |   / keithgalli  
TikTok |   / keithgalli  

-------------------------
If you are curious to learn how I make my tutorials, check out this video:    • How to Make a High Quality Tutorial V...  

Practice your Python Pandas data science skills with problems on StrataScratch!
https://stratascratch.com/?via=keith

Join the Python Army to get access to perks!
YouTube - https://www.youtube.com/channel/UCq6X...
Patreon -   / keithgalli  

*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.

-------------------------
Video timeline!
0:00 - Overview
1:25 - What not to do.
2:16 - Python code to load in large CSV file (read_csv & chunksize)
8:00 - Finalizing our data