Skip to content

MNwake/WebScraper

Repository files navigation

🛒 Discount Product Scraper

Welcome to the Discount Product Scraper! 🎉
This application is designed to help users find products with massive discounts across various online retailers.
Currently, it is under development, but it already demonstrates powerful capabilities and a scalable architecture. 💻

🚀 Features

  • Slow Scraping Technology:
    The app carefully scrapes data from supported websites, adhering to rate limits and minimizing detection risk.
    This ensures reliable and consistent results for users.

  • Multi-Site Support:
    Initial support for Home Depot is implemented, and more websites are in progress:

    • 🔨 Northern Tool
    • 🛠️ Ace Hardware
    • 📦 Amazon
    • 🛒 Sam's Club
  • Massive Discount Finder:
    Scans for products with major discounts and organizes them for easy user access.

  • Proxies Support:
    Incorporates proxy handling for more efficient scraping.

  • Dynamic ZIP Code Search:
    Allows scraping results to be tailored based on location.

✨ Features Coming Soon

  • 🚧 Expanded Retailer Support:
    Full implementation for Ace Hardware, Amazon, Northern Tool, and Sam's Club.

  • 📊 Enhanced Data Presentation:
    Improved user interfaces to display the results in an intuitive and actionable format.

  • 🌐 Global Search Support:
    Adding functionality for international searches across retailers.

⚙️ How to Test

To test the application, you will need to set the following environment variables:

  • OPENAI_API_KEY
  • GOOGLE_API_KEY
  • SEARCH_ENGINE

These keys allow the application to use OpenAI and Google services for enhanced functionality.


🎯 Note: This application is still a work in progress, and contributions or feedback are welcome. 🚀
Stay tuned for exciting updates and new features!

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published