Skip to content

cefimenda/scraper

Repository files navigation

Scavenger

Information:

Scavenger is a scraper built on Puppeteer that collects audition listings from the two most popular websites in the artist community - www.backstage.com, www.playbill.com

I have started building this project when I first started learning javascript and over time implemented the new tools that I learned into it. Hence on examination it is clear that there are conflicting styles or different methodologies used. (i.e. parts that use vanilla Ajax calls vs Jquery Ajax calls.)

Currently tools utilized:

-A mysql database (with sequelize) to hold audition data,

-A mongo database (with mongoose) to hold comments specific to auditions provided by users,

-Puppeteer to run a headless chromium instance that navigates pages and collects data,

-An express server.

Weaknesses and known issues:

-Audition filtering is done on the client side, making filtering slow and taxing for the client side device.

Planning on restructuring api end points to get the filtering and heavy lifting done through db queries.

-UI responsivity issue on XL screen sizes, causing the loading and audition cards to overlap with the fixed filterbar.

Looking into adding pagination so that the filters bar will no longer need to be fixed on the screen since height of auditions will not be larger than the height of the filterbar.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published