Scrapy project

All

27 repositories

scrapy
Public
Scrapy, a fast high-level web crawling & scraping framework for Python.
python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python
Python
•
BSD 3-Clause "New" or "Revised" License
•11k•53k•452•217•Updated Nov 25, 2024Nov 25, 2024
scrapy.org
Public
The scrapy.org website
html hacktoberfest
HTML
•139•60•1•1•Updated Nov 25, 2024Nov 25, 2024
itemadapter
Public
Common interface for data container classes
python metadata python3 scrapy hacktoberfest python-dataclasses python-attrs
Python
•
BSD 3-Clause "New" or "Revised" License
•13•62•6•4•Updated Nov 19, 2024Nov 19, 2024
protego
Public
A pure-Python robots.txt parser with support for modern conventions.
python robots-txt hacktoberfest robots-parser
DIGITAL Command Language
•
BSD 3-Clause "New" or "Revised" License
•28•56•5•1•Updated Nov 15, 2024Nov 15, 2024
scrapyd
Public
A service daemon to run Scrapy spiders
Python
•
BSD 3-Clause "New" or "Revised" License
•569•3k•7•0•Updated Nov 11, 2024Nov 11, 2024
parsel
Public
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
css python xml scraping selectors xpath lxml hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•146•1.2k•29•12•Updated Oct 16, 2024Oct 16, 2024
w3lib
Public
Python library of web-related functions
python hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•104•393•11•4•Updated Oct 16, 2024Oct 16, 2024
itemloaders
Public
Library to populate items using XPath and CSS with a convenient API
hacktoberfest
Python
•
BSD 3-Clause "New" or "Revised" License
•16•45•17•4•Updated Oct 16, 2024Oct 16, 2024
cssselect
Public
CSS Selectors for Python
css python selectors hacktoberfest
Python
•
Other
•61•291•17•4•Updated Oct 16, 2024Oct 16, 2024
queuelib
Public
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
python python3 persistent hacktoberfest queues non-persistent
Python
•
BSD 3-Clause "New" or "Revised" License
•54•270•3•2•Updated Oct 16, 2024Oct 16, 2024
scrapyd-client
Public
Command line client for Scrapyd server
Python
•
BSD 3-Clause "New" or "Revised" License
•146•770•5•0•Updated Oct 11, 2024Oct 11, 2024
form2request
Public
Python 3.8+ library to build HTTP requests out of HTML forms
Python
•
BSD 3-Clause "New" or "Revised" License
•3•4•1•0•Updated Jul 15, 2024Jul 15, 2024
loginform
Public
Fill HTML login forms automatically
Python
•80•270•11•3•Updated Apr 24, 2024Apr 24, 2024
xtractmime
Public
https://mimesniff.spec.whatwg.org/ implementation for Python
Python
•
BSD 3-Clause "New" or "Revised" License
•2•14•0•0•Updated Jan 16, 2024Jan 16, 2024
quotesbot
Public
This is a sample Scrapy project for educational purposes
Python
•
MIT License
•777•1.3k•0•6•Updated Nov 29, 2023Nov 29, 2023
booksbot
Public
A crawler for http://books.toscrape.com
Python
•941•40•0•4•Updated Aug 8, 2023Aug 8, 2023
scrapely
Public
A pure-python HTML screen-scraping library
HTML
•272•1.9k•26•5•Updated Apr 4, 2022Apr 4, 2022
scrapy-bench
Public
A CLI for benchmarking Scrapy.
python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
Python
•
MIT License
•15•31•6•2•Updated Feb 24, 2021Feb 24, 2021
scrapy-itemloader
Public archive
[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
Python
•
BSD 3-Clause "New" or "Revised" License
•7•6•2•0•Updated May 5, 2020May 5, 2020
scurl
Public
Performance-focused replacement for Python urllib
python cython chromium gurl urlparse
Python
•
Apache License 2.0
•6•21•10•0•Updated Oct 2, 2018Oct 2, 2018
url-chromium
Public
url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
chromium gurl
C++
•
BSD 3-Clause "New" or "Revised" License
•2•2•0•0•Updated Aug 7, 2018Aug 7, 2018
base-chromium
Public
base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
C++
•
BSD 3-Clause "New" or "Revised" License
•3•7•0•0•Updated Jul 31, 2018Jul 31, 2018
dirbot
Public
Scrapy project to scrape public web directories (educational) [DEPRECATED]
Python
•1.1k•1.6k•0•0•Updated Oct 27, 2017Oct 27, 2017
scrapy-bench-speedcenter
Public
Codespeed for scrapy-bench
Python
•2•2•0•0•Updated Aug 28, 2017Aug 28, 2017
pypydispatcher
Public
A fork of http://pydispatcher.sourceforge.net/ with PyPy support
Python
•
Other
•4•16•1•0•Updated Jul 3, 2017Jul 3, 2017
slybot
Public
58•224•5•0•Updated Apr 27, 2015Apr 27, 2015
gsoc2014-integration-tests
Public
GSoC2014 - Scrapy Integration tests project
Shell
•3•3•0•0•Updated Mar 18, 2014Mar 18, 2014