integration for web-poet's support on additional requests and Meta

[BurnzZ]

Requires the following to be merged in web-poet:

• implementation of additional requests web-poet#22
• add from_bytes_dict alternative constructor for HttpResponseHeaders web-poet#33

This is built on top of

• Use the new HttpResponse that replaces ResponseData in web_poet #67

Progress

• Testing across a wide-variety of scenarios/cases
  • contextvars and Twisted
  • scrapy-poet and scrapy integration
• Docs
  • update doc links using intersphinx-mapping when web-poet is released.
• update error handling after Raise exceptions when receiving 400-5xx responses web-poet#38 is merged
  • ensure that Scrapy doesn't redirect on HEAD requests (reference)
• Tests
  • Add test to verify that additional requests work both with an asyncio and a non-asyncio Twisted reactor.
• Changelog
• Use dont_filter for additional requests
• Document that spider middlewares are not followed for additional requests, while downloader middlewares are.
• Once there is a web-poet release with the new page input classes, like HttpResponse:
  • Update minimum dependencies web-poet>=0.2.0 and scrapy>=2.6.0 in setup.py and tox.ini

Fixes #65.

[codecov]

Codecov Report

Merging #62 (c0eddf3) into master (53e5b92) will increase coverage by 0.29%.
The diff coverage is 100.00%.

❗ Current head c0eddf3 differs from pull request most recent head 98ce454. Consider uploading reports for the commit 98ce454 to get more accurate results

@@            Coverage Diff             @@
##           master      #62      +/-   ##
==========================================
+ Coverage   97.46%   97.76%   +0.29%     
==========================================
  Files           9       10       +1     
  Lines         395      448      +53     
==========================================
+ Hits          385      438      +53     
  Misses         10       10              

Impacted Files	Coverage Δ
scrapy_poet/api.py	100.00% <100.00%> (ø)
scrapy_poet/downloader.py	100.00% <100.00%> (ø)
scrapy_poet/middleware.py	100.00% <100.00%> (ø)
scrapy_poet/page_input_providers.py	95.00% <100.00%> (+1.00%)	⬆️
scrapy_poet/utils.py	100.00% <100.00%> (ø)

[kmike]

👍 +1 to merge it after making sure it works with the latest web-poet (scrapinghub/web-poet#42 is merged).

[Gallaecio]

Closing and reopening to trigger tests with the latest web-poet master.

As for merging, we still need to wait for a web-poet release (0.2?), then update requirements here, re-run tests, and then merge.

[kmike]

@Gallaecio could you please check scrapinghub/web-poet#51 before making scrapy-poet release with these changes?

[Gallaecio]

3398dc7 is not as trivial as the commit message may suggest:

• Added the right minimum versions of dependencies to setup.py (lowest versions for which tests pass)
• Had tox test those minimum versions (I have just realized that I need to update GitHub Actions accordingly in a follow-up commit)
• Made Scrapy a dependency in line with update dep requirement: scrapy>=2.6.0 scrapy-plugins/scrapy-zyte-api#29
• Removed tldextract as a dependency, not needed since url-matcher integration with scrapy-poet #56 (although it is still available, as it is a dependency of Scrapy)
• mypy now runs with the minimum version of Python and dependencies.

[kmike]

👍, +1 to merge.

[BurnzZ]

Got one minor change left and I believe we're good to go. 🚀

[BurnzZ]

As per scrapinghub/web-poet#51 (comment), we should probably make this into 0.3.0, as well as updating the CHANGELOG and tox.ini.

EDIT: I just realized we're not using contextvars in scrapy-poet so we can set it to 0.2.0.

[Gallaecio]

Yes, I went through the same path. It was when I went to apply the contextvar name change to the patch when I realized we do not need 0.3.0 here.

[kmike]

Thanks @BurnzZ and @Gallaecio, that's a big milestone!