Скачать или смотреть How to reliably extract URLs contained in URLs with Python

Extracting URLs from text is a common task in web scraping, data analysis, and text processing. In this tutorial, we'll explore how to reliably extract URLs contained within other URLs using Python. We'll use the re (regular expressions) module to accomplish this task.
Prerequisites:
Step 1: Import Required Libraries
First, you need to import the re module for regular expression operations.
Step 2: Define a Function to Extract URLs
Create a function that takes a text input and returns a list of URLs found within that text. We'll use a regular expression pattern to match URLs. This pattern will match standard HTTP and HTTPS URLs.
Step 3: Test the Function
Now, let's test our extract_urls function with some sample text.
Step 4: Handling URLs Within URLs
If you want to extract URLs that are contained within other URLs, you can use a modified version of the extract_urls function. This version will first extract the outer URLs and then, for each outer URL, extract the inner URLs. You can use a loop for this purpose.
Step 5: Test the Modified Function
Let's test the modified function with text containing URLs within URLs.
Step 6: Handle Relative URLs
If you need to handle relative URLs as well, you can combine the urllib.parse module to convert relative URLs into absolute URLs. Here's a modified version of the function:
In this version, you need to provide the base_url to resolve relative URLs.
Step 7: Test with Relative URLs
Let's test the modified function with a relative URL.
This tutorial demonstrates how to reliably extract URLs contained within URLs using Python. You can use these functions to extract URLs from text, including those nested within other URLs and handle relative URLs.
ChatGPT