This repository contains educational example scrapers for popular web scraping targets using the ScrapFly web scraping API and Python.
Most Scrapers use a simple web scraping stack:
- Python version 3.10+
- Scrapfly's Python SDK for sending HTTP request, bypass blocking and parsing the HTML using the built-in parsel selector.
- asyncio for writing concurrent code using the async/await syntax.
- JMESPath and nested-lookup for JSON parsing when needed.
- loguru for logging.
To learn more about web scraping see our full tutorials on how to scrape these targets (and many others) see the scrapeguide directory.
This repository contains educational reference material to illustrate how accessible web scraping can be and the provided programs are not intented to be used in web scraping production. That being said, Scrapfly team is constantly updating and improving all of this code for optimal experience.
Scrapfly does not offer legal advice and as always, consult a lawyer when creating programs that interact with other people's websites though here's a good general intro of what NOT to do:
- Do not store PII (personally identifiable information) of EU citizens who are protected by GDPR.
- Do not scrape and repurpose entire public datasets which can be protected by database protection laws in some countries.
- Do not scrape at rates that could damage the website and scrape only publicly available data.
- Install the required libraries:
$ pip install scrapfly-sdk[all] jmespath loguru nested-lookup - Export your ScrapFly API key
- On Mac:
$ export SCRAPFLY_KEY="YOUR SCRAPFLY KEY"- On Windows:
$ setx SCRAPFLY_KEY "YOUR SCRAPFLY KEY"- cd into the scraper directory and run the code:
$ cd ./example-scraper
$ python run.pyThe following is the list of supported websites grouped by type.
- Domain.com.au
- Idealista.com
- Homegate.ch
- Immobilienscout24.de
- Immoscout24.ch
- Immowelt.de
- Realestate.com.au
- Realtor.com
- Redfin.com
- Rightmove.co.uk
- Seloger.com
- Zillow.com
- Zoopla.co.uk
The aliexpress.com scraper can scrape the following data:
- Product pages for a specific product data.
- Search pages for product listing data.
- Product reviews.
View sample data
For the full guide, refer to our blog article How to Scrape Aliexpress.com (2023 Update)
The amazon.com scraper can scrape the following data:
- Product pages for a specific product data.
- Search pages for product listing data.
- Product reviews.
View sample data
For the full guide, refer to our blog article How to Scrape Amazon.com Product Data and Reviews
The bing.com scraper can scrape the following data:
- Search pages for search results (SERPs) data.
- Search pages for keyword data.
- Search page for rich snippet data.
View sample data
For the full guide, refer to our blog article How to Scrape Bing Search with Python
The booking.com scraper can scrape the following data:
- Hotel pages for a specific hotel data.
- Search pages for hotel listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Booking.com (2023 Update)
The crunchbase.com scraper can scrape the following data:
- Company pages for a specific company data.
- Investor pages for a specific investor data.
- Search pages for hotel listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Crunchbase Company and People Data (2023 Update)
The domain.com.au scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Domain.com.au Real Estate Property Data
The ebay.com scraper can scrape the following data:
- Product pages for a specific product data.
- Search pages for product listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Ebay using Python
The etsy.com scraper can scrape the following data:
- Product pages for a specific product data.
- Shop pages for a specific shop data.
- Search pages for product listings data.
View sample data
For the full guide, refer to our blog article
The fashionphile.com scraper can scrape the following data:
- Product pages for product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Fashionphile for Second Hand Fashion Data
The glassdoor.com scraper can scrape the following data:
- Company search for company page URLs (overiew, jobs, reviews, salaries).
- Job pages for a specific job data.
- Salary pages for a specific company salaries data.
- Review pages for a specific company reviews data.
View sample data
For the full guide, refer to our blog article How to Scrape Glassdoor (2023 update)
The goat.com scraper can scrape the following data:
- Product pages for product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Goat.com for Fashion Apparel Data in Python
The homegate.ch scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Homegate.ch Real Estate Property Data
The idealista.com scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
- Provinces pages for search pages URLs.
View sample data
For the full guide, refer to our blog article How to Scrape Idealista.com in Python - Real Estate Property Data
The immobilienscout24.de scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Immobilienscout24.de Real Estate Data
The immoscout24.ch scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Immoscout24.ch Real Estate Property Data
The immowelt.de scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Immowelt.de Real Estate Data
The indeed.com scraper can scrape the following data:
- Job pages for a specific job data.
- Search pages for job listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Indeed.com (2023 Update)
The instagram.com scraper can scrape the following data:
- User pages for a specific user data.
- Post Pages for a specific post data.
View sample data
For the full guide, refer to our blog article How to Scrape Instagram
The leboncoin.fr scraper can scrape the following data:
- product pages for a specific product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Web Scrape Leboncoin.fr using Python
The nordstorm.com scraper can scrape the following data:
- Product pages for product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Nordstrom Fashion Product Data
The realestate.com.au scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Realestate.com.au Property Listing Data
The realtor.com scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
- Feed pages for newly added propery listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Realtor.com - Real Estate Property Data
The redfin.com scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Redfin Real Estate Property Data in Python
The rightmove.co.uk scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape RightMove Real Estate Property Data with Python
The seloger.com scraper can scrape the following data:
- Property pages for a specific real estate property data.
- Search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Seloger.com - Real Estate Listing Data
The stockx.com scraper can scrape the following data:
- Product pages for a specific product data.
- Search pages for product listings data.
View sample data
For the full guide, refer to our blog article How to Scrape StockX e-commerce Data with Python
The threads.net scraper can scrape the following data:
- User pages for a specific user data.
- Theads Pages for a specific thread data.
View sample data
For the full guide, refer to our blog article How to scrape Threads by Meta using Python (2023-08 Update)
The tripadvisor.com scraper can scrape the following data:
- Holtel pages for a specific hotel data.
- Holtes data in a specific location.
- Search pages for hotel listings data.
View sample data
For the full guide, refer to our blog article How to Scrape TripAdvisor.com (2023 Updated)
The trustpilot.com scraper can scrape the following data:
- Company pages for a specific company data.
- Reviews data in a specific company reviews.
- Search pages for company listings data.
View sample data
For the full guide, refer to out blog article How to Scrape Trustpilot.com Reviews and Company Data
The twitter.com scraper can scrape the following data:
- Twitter tweet pages for a specific tweet data.
- Twitter user pages for a specific user data.
View sample data
For the full guide, refer to our blog article How to Scrape X.com (Twitter) using Python (2023-11 Update)
The vestiairecollective.com scraper can scrape the following data:
- Product pages for product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Vestiaire Collective for Fashion Product Data
The g2.com scraper can scrape the following data:
- Review pages for company reviews.
- Search pages for company data.
- Alternative pages for company alternatives.
View sample data
For the full guide, refer to our blog article How to Scrape G2 Company Data and Reviews
The walmart.com scraper can scrape the following data:
- Product pages for product data.
- Search pages for product listing data.
View sample data
For the full guide, refer to our blog article How to Web Scrape Walmart.com (2023 Update)
The wellfound.com scraper can scrape the following data:
- Company pages for company data.
- Search pages for job listing data.
View sample data
For the full guide, refer to our blog article How to Scrape Wellfound Company Data and Job Listings
The yellowpages.com scraper can scrape the following data:
- Business pages for business data.
- Search pages for business listing data.
View sample data
For the full guide, refer to our blog article How to Scrape YellowPages.com Business Data and Reviews (2023 Update)
The yelp.com scraper can scrape the following data:
- Business pages for business data.
- Review pages for business data.
- Search pages for business listing data.
View sample data
For the full guide, refer to our blog article How to Web Scrape Yelp.com (2023 update)
The zillow.com scraper can scrape the following data:
- Zillow property pages for a specific real estate property data.
- Zillow search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Zillow Real Estate Property Data in Python
The zoominfo.com scraper can scrape the following data:
- Company pages for company data.
- Directory pages for company page URLs.
- Company pages for company FAQs data.
View sample data
For the full guide, refer to our blog article How to Scrape Zoominfo Company Data (2023 Update)
The zoopla.co.uk scraper can scrape the following data:
- Zoopla property pages for a specific real estate property data.
- Zoopla search pages for real estate property listings data.
View sample data
For the full guide, refer to our blog article How to Scrape Zoopla Real Estate Property Data in Python