Скачать или смотреть Scrapy Python Tutorial : Yelp Business data | XPATH and 'response.meta.get' for multi level scraping

Writing a spider to scrape Yelp business information.



Sometimes some of the data needs to be passed to the another callback function while using the scrapy framework. In this video we need to get some data from the main listings page AND go to the details page and get the telephone number and website for Scrapy to output to a CSV using FEEDS as the export.



So, we use the meta property of the scrapy request to pass our data and get it with "response.meta.get" in the 'fetch_details' method.

I am using Python 3.8 with Scrapy 2.1 and Atom as my IDE/Editor.

Video Chapters #
___________________
0:20 Introduction
1:49 start_urls
3:48 begin the scrapy spider
5:00 XPATH(or CSS) selectors
7:08 Scrapy Shell (and Fordbidden by robots.txt)
11:34 selectors : tag and class
15:50 pages may not be processed in synch
17:50 example of out of synch output
18:54 'meta'
28:56 response.meta.get
30:34 successful test run
32:00 inspecting the test results
34:53 csv output, in synch
37:47 gratuitous flames
37:53 next_page can be used - just un comment it (remove #)
38:16 don't forget to....

Many Scrapy web scraping tutorials, especially the ""quotes" and "books to scrape" ones will not show how to pass variables in this way :

⦿ yield Request(absolute_url, callback=self.fetch_detail, meta={'link': link, 'logo_url': logo_url, 'lcompanyname':lcompanyname})

This allows you to have your method called "parse", and send (using yield & 'meta') to the next function - eg "parse_details" or "fetch_details" - where you can then send all of the collected data to the same row in the output.

➤ https://docs.scrapy.org/en/latest/top...

Note!
After 1.7, Request.cb_kwargs became the preferred way for handling user information, leaving Request.meta for communication with components like middlewares and extensions.

*Meta is still used widely and you can find it on some intermediate/advanced tutorials and Stackoverflow.

** Don't confuse this with " Extracting keywords from metatag using scrapy"

⦿ If you need to know how to phrase this concept then Scrapy documentation says :

"In some cases you may be interested in passing arguments to those callback functions so you can receive the arguments later, in the second callback."

I show 2 ways, and illustrate why it's best to use "meta" with yield Request when using callback and passing variables between methods.

This is useful when collecting data from a main page and a detail page.

start url = "https://www.yelp.com/search?find_desc..."

➤ Scrapy reference : https://docs.scrapy.org/en/latest/top...
➤ About Scrapy - https://towardsdatascience.com/a-mini...
➤ A Minimalist End-to-End Scrapy Tutorial (Part II) - https://towardsdatascience.com/a-mini...
➤ About XPATH and using the dot "." https://stackoverflow.com/questions/5...

➤ Download my Yelp Python Scrapy code : https://github.com/RGGH/Scrapy5
All my code on GitHub is available at : https://github.com/RGGH

Buy Dr Pi a Coffee...or Tea! : https://www.buymeacoffee.com/DrPi ☕️☕️☕️
else: 🤬

Disclaimer:
This video is for educational use only - please do not abuse websites using this code - if you do then you may get banned from the website you are scraping.

(Apologies for re-uploading this, I saw 2 typos in the original, and it was not the correct aspect ratio in places. My bad).


See you around yeah?
Dr P.