NewsSiteMapParserBolt: do not detect feeds as sitemaps

[sebastian-nagel]

If a news feed uses the sitemaps namespace it is erroneously detected as sitemap which causes that it's processed as sitemap (without being properly parsed) and not as feed. One example feed:

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.drudge.com/~d/styles/itemcontent.css"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:wordzilla="http://www.cadenhead.org/workbench/wordzilla/namespace" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" version="2.0">

[silentninja]

In addition to checking for a sitemap tag match which we are doing now, can we also check for the presence of blacklisted tags, such as <rss>, and short-circuit the org.commoncrawl.stormcrawler.news.ContentDetector#getFirstMatch function to return as no matches found?
This would prevent identifying feeds as sitemaps

[sebastian-nagel]

Hi @silentninja, thanks for the suggestions: yes, it would be possible to implement contradictory patterns in the ContentDetector. In doubt, I would first go with a positive pattern which has a higher priority than the sitemap namespace(s), same as done for the news sitemap namespace and the <sitemapindex pattern. This solution wouldn't require changes in the ContentDetector.

[silentninja]

@sebastian-nagel Thanks for your response. While I understand the reasoning behind reusing the existing logic, I believe it’s cleaner and more intuitive to ensure positive matches are sitemaps and do not include feeds.

Mixing the two could lead to false assumptions that all positive matches are valid sitemaps, which could introduce confusion or errors down the line. Keeping feeds separate from sitemap matches ensures better clarity and avoids potential misclassification. For reference, the current approach in NewsSiteMapDetectorBolt highlights a potential pitfall where any positive match is assumed to be a sitemap

An alternate solution if we don't want to edit the ContentDetector would be to use a separate ContentDetector for detecting feeds and returning early if a match is found