The CSPAN scrapers 'insert_count' field is incorrect.

[onyxfish]

Each scraper is reporting thousands of inserts--I think the value actually being stored is the total number of documents in the database when the scraper completes.

I'm not sure if this is affecting other scrapers. I need to test with a clean database to get a clearer picture.

[chaunceyt]

this is an interesting issue. Are you spawning a process for each scraper? If so, I need to rethink how I calculate the count.

[onyxfish]

For posterity, it was discussed in chat that the scheduler is indeed spawning additional processes. This may make including this data infeasible. Awaiting a determination on this.