How to scrape dynamic content with Puppeteer and Node.js

In this video, I'll show you how you can scrape dynamic content websites with Puppeteer. and Node.js.

In this first part, we discuss how to make selectors to find elements in the page, how to close popups, how to do auto scrolling and get the links for lazy-loaded images.

00:00 - Intro
00:52 - Create project
02:41 - Setting up environment variable
05:37 - What we are going to do
07:49 - How to automatically close a popup modal
13:00 - How to find a specific button on the page
16:10 - How to make a function to wait a few seconds
18:48 - Puppeteer click bug
24:03 - Puppeteer click bug workaround
24:50 - Getting logs from Puppeteer "evaluate" function (also works for $eval)
28:51 - Selecting the list of images
30:18 - Lazy-loaded images explained
31:23 - Auto scrolling the page
33:03 - Finding the scrollbars via javascript code in Chrome browser
38:15 - Finding the scrollbars via node.js code
39:43 - Implementing scrolling behaviour
45:35 - Executing the code


On the next video:

- Get the links from the pictures
- Scrape images slider