Скачать или смотреть How to Successfully Scrape DOM Nodes Using Puppeteer: Avoiding Undefined Returns

Discover how to effectively retrieve DOM nodes with Puppeteer, ensuring you avoid the common issue of receiving undefined results during scraping.
---
This video is based on the question https://stackoverflow.com/q/64216965/ asked by the user 'Marco' ( https://stackoverflow.com/u/14396625/ ) and on the answer https://stackoverflow.com/a/64218825/ provided by the user 'Vaviloff' ( https://stackoverflow.com/u/2715393/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Puppeteer evaluate returns undefined for an array of DOM nodes

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Introduction

When working with Puppeteer, a powerful Node.js library for web scraping, developers often face the challenge of extracting data from complex DOM structures. One common issue that arises is attempting to return an array of DOM nodes directly from the page.evaluate function, which results in undefined outputs. If you've experienced this, you're not alone!

In this guide, we'll explore how to effectively scrape DOM nodes, particularly focusing on a scenario where we want to gather table cells identified by specific classes. By the end, you will understand how to avoid the undefined results and successfully retrieve your desired data.

Understanding the Problem

What is Puppeteer?

Puppeteer is a Node.js library that provides a high-level API for controlling headless Chrome or Chromium browsers. It’s commonly used for web scraping, automated testing, and taking screenshots of web pages. However, when you try to fetch DOM nodes using document.querySelectorAll, you may encounter issues when trying to return the data directly.

The Issue at Hand

When executing the following code:

[[See Video to Reveal this Text or Code Snippet]]

You might see an unexpected result: the result variable returns undefined. This behavior occurs because DOM nodes are complex objects that cannot be serialized directly by the page.evaluate function.

Steps to Retrieve Data Correctly

1. Understanding Serialization

To successfully return data from page.evaluate, we must keep in mind that only certain types of data can be serialized across the boundary between the browser context and the Node.js context. These include basic data types such as strings, numbers, and arrays—but not complex objects.

2. Solution: Return Simple Objects

To solve the issue and retrieve data as expected, we need to modify the function to return simple objects. For example, instead of returning the DOM nodes directly, we can return an array of their text values. Here’s how you can do it:

[[See Video to Reveal this Text or Code Snippet]]

3. Explanation of the Solution

Array.from: This method converts the NodeList (the result of querySelectorAll) into an array, making it easier to manipulate.

map: This function iterates over each element in the array, allowing us to extract only the text content from each cell.

innerText: This property fetches the text content of the cells, which is a simple string that can be easily serialized.

4. Output the Result

With the modified code, the result variable now contains an array of strings representing the text from each <td> element with the specified class. You can log or process this array as needed:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

Puppeteer is a powerful tool for web scraping, but it requires an understanding of how data is serialized. By knowing that we cannot directly return complex DOM nodes, we can adjust our approach to return simple, serializable objects instead.

Next time you find yourself struggling with undefined returns in Puppeteer, remember to extract the data you need into a simple format like text values. Happy scraping!