-
Notifications
You must be signed in to change notification settings - Fork 762
MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
class NotMatchesListRegexDecideRule and MatchesListRegexDecideRule are used by DecideRuleSequence for finding candidates for crawling, I found them a little confusing in the beginning , but this is what I have figured out.
class NotMatchesListRegexDecideRule inherits MatchesListRegexDecideRule
and returns the opposite(in terms of regex evaluation):
protected boolean evaluate(CrawlURI object) {
* return ! super.evaluate(object);*
}
Again, the opposite in terms of "regex evaluation" not "decision(ACCEPT,REJECT,NONE)" the decision by default for both is "ACCEPT", so you have the set the decision manually.
so:
if you want to accept a positive pattern match: add
MatchesListRegexDecideRule with <property name="decision" value="ACCEPT"/>
if you want to reject a positive positive pattern match: add
MatchesListRegexDecideRule with <property name="decision" value="REJECT"/>
and:
if you want to accept a negative pattern match: add
NotMatchesListRegexDecideRule with <property name="decision" value="ACCEPT"/>
if you want to reject a negative pattern match: add
NotMatchesListRegexDecideRule with <property name="decision" value="REJECT"/>
To be honest I think the existing of NotMatchesListRegexDecideRule is a bit disturbing, as you can use MatchesListRegexDecideRule for almost every thing.
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse