Скачать или смотреть Mastering Selenium: How to Extract Text and Href Attributes in Python

Learn how to efficiently use Python's Selenium to extract text and href attributes from nested HTML elements. This guide walks you through the process step-by-step.
---
This video is based on the question https://stackoverflow.com/q/69338260/ asked by the user 'psmith' ( https://stackoverflow.com/u/2895197/ ) and on the answer https://stackoverflow.com/a/69341522/ provided by the user 'pmadhu' ( https://stackoverflow.com/u/16452840/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Python selenium - get text and href

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Mastering Selenium: How to Extract Text and Href Attributes in Python

When working with web automation and scraping using Python's Selenium, one common task you might encounter is extracting text and links from nested HTML elements. Let's dive into a specific problem that many developers face, and I'll guide you through the solution step-by-step.

The Problem

Imagine you're dealing with a series of nested div tags containing text and hyperlinks as shown below:

[[See Video to Reveal this Text or Code Snippet]]

Your goal is to retrieve the following elements for each div:

Text1

Text2 (the text inside the hyperlink)

Text3

Text4

The href attribute from the <a> tag

While you can easily get Text1, Text2, Text3, and Text4, extracting the href value requires a bit of extra work. Let’s explore how to achieve this.

The Solution

Here's how you can extract the desired information using Selenium.

Step 1: Set Up Your Environment

Make sure you have the following prerequisites before using the Selenium WebDriver:

Python installed on your system

Selenium package installed: Run pip install selenium in your terminal or command prompt.

A browser driver, such as ChromeDriver, if you are using Chrome.

Step 2: Write the Code

Below is a code snippet that demonstrates how to extract all required pieces of information from the nested divs:

[[See Video to Reveal this Text or Code Snippet]]

Step 3: Explanation of the Code

elements = driver.find_elements_by_xpath("//div[@ class='class1']/*"): This line locates all immediate child elements within the div with class class1. This includes class2, which contains the necessary nested information.

Looping Through the Elements:

For each element e, e.text retrieves the plain text within that element.

link = e.find_element_by_xpath(".//a").get_attribute("href"): This line fetches the href attribute of the hyperlink nested within each div. The period (.) at the start of the XPath expression ensures that the search is conducted relative to the current div.

Error Handling: The try-except block ensures that if an <a> tag is absent in an element, the code gracefully skips it and continues execution.

Final Output

The code returns all required texts and URLs in this format:

Text1

https://somelink

Text2

Text3

Text4

Conclusion

Extracting text and href attributes from nested HTML elements using Python and Selenium is a straightforward task once you know the right approach. With the above steps, you should be well on your way to mastering this aspect of web scraping and automation. Happy coding!