Скачать или смотреть How to Save Spider Data in Scrapy: Exporting Links to CSV or JSON

Learn how to save your list of scraped links from a Scrapy spider to a CSV or JSON file using simple methods.
---
This video is based on the question https://stackoverflow.com/q/76790345/ asked by the user 'mugndhn' ( https://stackoverflow.com/u/22265845/ ) and on the answer https://stackoverflow.com/a/76790611/ provided by the user 'Alexander' ( https://stackoverflow.com/u/17829451/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Scrapy: How to save a list from our spider class in a file when scraping

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Scrapy: Saving Scraped Links to a File

Scraping the web is a common task for many developers, and Scrapy is one of the most powerful frameworks for this purpose. While collecting data through a spider, you often need to save that data for further processing or analysis. In this guide, we’ll tackle a common problem: How do you save a list of webpage links fetched by your spider to a CSV or JSON file after scraping?

Understanding the Problem

When using Scrapy, you might have a collection of links stored in a list within your spider class. In this case, you have links saved in self.links, and you want to save this list as a file after scraping is completed. This can help you perform additional data processing on the followed pages later on.

The Solution: Using Feed Exports in Scrapy

Scrapy offers a built-in feature called Feed Exports that makes saving scraped data straightforward. Instead of manually writing the data to a file, you can simply yield a dictionary from your parse method, and Scrapy will handle the rest.

Step-by-step Instructions

Modify Your Spider Class:
Update your spider class to define custom settings for Feed Exports.

Yielding Data:
Make sure you yield a dictionary containing your scraped data within the parse method.

Example Implementation

Here's how to adapt the spider to save the links into a CSV file named items.csv:

[[See Video to Reveal this Text or Code Snippet]]

Alternative: Using Command-Line Options

You can also run your spider from the command line and specify the output file type without changing the spider code. Just use the -o option to define the output file:

[[See Video to Reveal this Text or Code Snippet]]

Saving as JSON

If you prefer saving your links in a JSON file instead, you simply need to modify the custom settings:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

Saving scraped data in Scrapy is a breeze with the Feed Exports feature. By following the simple steps outlined above, you can easily export your list of web links to a CSV or JSON file.

This will not only help you keep your data organized but also make it easier for future processing and analysis. With this knowledge in hand, you're well on your way to becoming a Scrapy expert!