Handle item overrides

[BurnzZ]

Fixes the case when the following code below results in a ProviderDependencyDeadlockError. In this example, the issue stems from the product: Product dependency that's declared. However, scrapy-poet sees ProductPage as the one that provides it which causes the deadlock error.

import attrs
from zyte_common_items import Product
from web_poet import handle_urls, WebPage

@handle_urls("example.com")
@attrs.define 
class ProductPage(WebPage[Product]):
    product: Product

    def name(self) -> str:
        return f"(modified) {self.product.name}"

The use case for this is when items like Product are produced by a provider and not a page object. For example, scrapy-zyte-api's ZyteApiProvider (reference). The ProductPage in this scenario tries to override the item produced by ZyteApiProvider.

---

TODO:

• add more test cases to handle different scenarios
• verify if this is the best implementation to go

[BurnzZ]

Increasing this value allows ZyteApiProvider (as well as any other item provider in the future) to run before scrapy-poet's ItemProvider. This ensures any item dependency in page objects are available.

[BurnzZ]

Passing the prev_instances allows the PO to gain access to the item produced by a provider (as long as that provider was called earlier).

@attrs.define 
class ProductPage(WebPage[Product]):
    product: Product

In the above example, before the ProductPage instance can be made, the product: Product instance should first be created. This is where this passing of new prev_instances parameter is useful.

[codecov]

Codecov Report

Merging #164 (3536421) into master (5339d63) will increase coverage by 0.14%.
The diff coverage is 100.00%.

❗ Current head 3536421 differs from pull request most recent head 92d1f1e. Consider uploading reports for the commit 92d1f1e to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #164      +/-   ##
==========================================
+ Coverage   85.64%   85.78%   +0.14%     
==========================================
  Files          14       14              
  Lines         801      809       +8     
==========================================
+ Hits          686      694       +8     
  Misses        115      115              

Files Changed	Coverage Δ
scrapy_poet/downloadermiddlewares.py	100.00% <ø> (ø)
scrapy_poet/commands.py	35.64% <100.00%> (ø)
scrapy_poet/injection.py	98.52% <100.00%> (+0.05%)	⬆️
scrapy_poet/page_input_providers.py	100.00% <100.00%> (ø)

[BurnzZ]

Scrapy 2.11 changes broke this since stats was missing.

[wRAR]

Ouch, I've already fixed it in master before looking here.

[wRAR]

But it's good to see we did it in (kinda) the same way!

[BurnzZ]

Notice that only JoeyPage had touched the data and KangarooPage didn't because the earlier PO had higher priority.

[Gallaecio]

Conflicts resolved, but john-kurkowski/tldextract#305 breaks tests through url-matcher’s get_domain, will fix that now.

To-do:

• Freeze url-matcher in min as well. (it is already the case 🤦 )

[Gallaecio]

I need to investigate a potential issue with DummyResponse not working on code that relies on this branch.

Never mind, it was caused by an outdated test expectation.

[kmike]

Thanks @BurnzZ and @Gallaecio!