Django + Celery + Selenium to Scrape Anything with Python

🚀 Sign up for Bright Data Now https://brdta.com/justin

Selenium with Bright Data can scrape nearly any website regardless of Captcha or Login required. In this series, you are going to use Django and Celery to ensure you can scrape on your schedule and store the data with excellent structure. We will be tracking Amazon product price fluctuations.

Topics Covered:
⦿ Integrating Django with Celery for scheduled and background task processing
⦿ First-hand experience with Selenium in Jupyter for live browser automation
⦿ Advanced proxy scraping techniques using Bright Data and Selenium including handling complex challenges like Amazon Captchas
⦿ Building helper functions to streamline your scraping module
⦿ Combining Django and Jupyter for an enhanced data-driven development experience
⦿ Data parsing with BeautifulSoup4 to effectively find and extract the data you need
⦿ Efficient data storage using Django models
⦿ Offloading heavy scraping tasks with Celery for improved application performance
⦿ Automating and scheduling scraping tasks within the Django admin using Celery
⦿ Plus, much more in-depth knowledge and practical tips!

Recommended to Watch
⦿ Try Django 3.2 (or similar):    • Try Django 3.2 - Python Web Developme...  
⦿ 30 Days of Python (up to about day 15):    • 30 Days of Python ~ Python 3.8+  
⦿ Some HTML / CSS Experience is ideal

Resources
⦿ Code: https://github.com/codingforentrepren...
⦿ Subscribe: https://cfe.sh/youtube
⦿ Django: https://djangoproject.com
⦿ Bright Data: https://brdta.com/justin

🚀 Next up? Web Scraping with Python + AI our Free Udemy Course: https://www.udemy.com/course/smarter-...

00:00:00 - Welcome
00:02:23 - Tools
00:06:07 - Rapid Fire Django + Celery Integration
00:19:06 - Setup Project
00:21:23 - First Look at Selenium with Jupyter
00:25:08 - Proxy Scraping with Bright Data + Selenium
00:32:08 - Helper Functions for Bright Data Module
00:35:55 - Django + Jupyter Integration
00:39:00 - Parsing Data with BeautifulSoup4
00:52:02 - Amazon Captcha + Prepare Parser Helper Functions
00:58:30 - Saved Scraped Data with Django Models
01:10:51 - Celery Task to Offload Scrape-to-Save
01:22:29 - Schedule Scraping Events with Celery in the Django Admin
01:33:16 - Thank you and next steps