Скачать или смотреть How to Extract Text from a Div Using Python Selenium

Discover how to use Python and Selenium to effectively scrape text from div elements on web pages, even when IDs or classes are absent.
---
This video is based on the question https://stackoverflow.com/q/68899024/ asked by the user 'exec85' ( https://stackoverflow.com/u/12470444/ ) and on the answer https://stackoverflow.com/a/68899065/ provided by the user 'Prophet' ( https://stackoverflow.com/u/3485434/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: how to get div text with python selenium?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Extract Text from a Div Using Python Selenium

In the world of web scraping, extracting specific elements from a webpage is a common task. However, it can become tricky when the elements you're targeting do not have an ID or class attribute. This post tackles a specific scenario: how to get the number "950" from a div that lacks both an ID and a class using Python and Selenium.

Understanding the Problem

You may encounter a situation where you need to interact with a webpage that houses thousands of elements structured in a complex hierarchy. An example problem could be the need to extract a price (like "950") nested deep within a div element that has neither a unique ID nor a class to simplify your search.

Here's a snippet of the HTML structure we are dealing with:

[[See Video to Reveal this Text or Code Snippet]]

The Solution: Using XPath with Selenium

To extract the desired text, you can use Selenium's capabilities to navigate and query DOM elements using XPath. XPath is a powerful query language that can be used to select nodes from an XML document, including HTML documents.

Steps to Extract the Text

Set up Selenium: Ensure you have Selenium installed. If you haven’t done that yet, you can install it using pip:

[[See Video to Reveal this Text or Code Snippet]]

Initialize the WebDriver: Set up the WebDriver for your preferred browser. This could be Chrome, Firefox, etc.

Identify the Element with XPath: Use the following command to fetch the text "950".

[[See Video to Reveal this Text or Code Snippet]]

This XPath expression clearly indicates the path to retrieve the text from the targeted div.

The first div selects the outer container identified by its unique class player-hover-box.

The nested /div/div[2] traverses to locate the second inner div that contains the desired text.

Handling Multiple Elements

If the webpage contains multiple similar structures and you need to find your target price among them, you must modify the XPath. This could involve:

Finding unique attributes of sibling elements.

Selecting elements based on their relative position or by using additional context.

Example Usage

Here’s how your full Python script might look:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

Using XPath with Selenium makes it possible to extract text from complex HTML structures where elements lack identifiers. By following the steps outlined in this post, you can confidently retrieve essential data points, such as prices or other vital stats, from a variety of web pages.

If you find yourself frequently needing to extract data from web pages, becoming familiar with Selenium and XPath is a valuable skill in your web scraping toolkit. Happy coding!