Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ld Json Schema Scraper #50

Open
wants to merge 102 commits into
base: master
Choose a base branch
from
Open

Ld Json Schema Scraper #50

wants to merge 102 commits into from

Commits on Apr 28, 2021

  1. support jamie oliver scraping

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    6fdf15a View commit details
    Browse the repository at this point in the history

  2. fix some tests. add defaultSetDescription to BaseScraper

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    ecbe2bd View commit details
    Browse the repository at this point in the history

  3. fix more tests

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    cda11c4 View commit details
    Browse the repository at this point in the history

  4. rename

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    8669a34 View commit details
    Browse the repository at this point in the history

  5. fix SimplyRecipes tests

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    e451674 View commit details
    Browse the repository at this point in the history

  6. fix yummly. update puppeteer to latest version

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    a005ae9 View commit details
    Browse the repository at this point in the history

  7. fix thespruceeats tests

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    b39a9f6 View commit details
    Browse the repository at this point in the history

  8. theSpruceEast - add recipe description

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    4db4c1d View commit details
    Browse the repository at this point in the history

  9. fix woolworths

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    55d8d9f View commit details
    Browse the repository at this point in the history

  10. closetCooking add description

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    27e38f9 View commit details
    Browse the repository at this point in the history

  11. try to change travis node version

    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    00ba239 View commit details
    Browse the repository at this point in the history

  12. revert travis node version to stable 

    add logs
    @shaniqwa
    shaniqwa committed Apr 28, 2021
    Configuration menu
    Copy the full SHA
    6b217c2 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2021

  1. more logs

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    caa38c9 View commit details
    Browse the repository at this point in the history

  2. logs

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    baf8a87 View commit details
    Browse the repository at this point in the history

  3. log

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    f6ff471 View commit details
    Browse the repository at this point in the history

  4. log

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    affc05a View commit details
    Browse the repository at this point in the history

  5. log

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    70314e3 View commit details
    Browse the repository at this point in the history

  6. skip should fetch the expected recipe test if server is not responding

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    f04e3a4 View commit details
    Browse the repository at this point in the history

  7. add ambitiouskitchen description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    20583c3 View commit details
    Browse the repository at this point in the history

  8. add recipetineats description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    4736af1 View commit details
    Browse the repository at this point in the history

  9. add GimmeDelicious description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    b47e379 View commit details
    Browse the repository at this point in the history

  10. add tastOfHome description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    1e07117 View commit details
    Browse the repository at this point in the history

  11. add SeriousEats description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    71e1029 View commit details
    Browse the repository at this point in the history

  12. add therecipecritic description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    7045020 View commit details
    Browse the repository at this point in the history

  13. add myrecipes description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    f426a0f View commit details
    Browse the repository at this point in the history

  14. bonappetit add descriptions and tags

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    7a7e1ae View commit details
    Browse the repository at this point in the history

  15. tastesbetterfromscratch add description and tags

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    5821206 View commit details
    Browse the repository at this point in the history

  16. kitchen stories - add description and tags

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    dd2f66a View commit details
    Browse the repository at this point in the history

  17. foodandwine add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    972b953 View commit details
    Browse the repository at this point in the history

  18. NomNomPaleo add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    60c7803 View commit details
    Browse the repository at this point in the history

  19. budgetbytes add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    7d6bc4a View commit details
    Browse the repository at this point in the history

  20. melskitchencafe add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    077b462 View commit details
    Browse the repository at this point in the history

  21. melskitchencafe add tags

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    7d6c137 View commit details
    Browse the repository at this point in the history

  22. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    7ea10ed View commit details
    Browse the repository at this point in the history

  23. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    b03707a View commit details
    Browse the repository at this point in the history

  24. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    01a2b1f View commit details
    Browse the repository at this point in the history

  25. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    def84dd View commit details
    Browse the repository at this point in the history

  26. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    616e69a View commit details
    Browse the repository at this point in the history

  27. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    41dbaee View commit details
    Browse the repository at this point in the history

  28. description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    cce53ee View commit details
    Browse the repository at this point in the history

  29. allRecipes description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    0422101 View commit details
    Browse the repository at this point in the history

  30. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    97a9611 View commit details
    Browse the repository at this point in the history

  31. add description

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    caf42c3 View commit details
    Browse the repository at this point in the history

  32. descriptions

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    b9b9b95 View commit details
    Browse the repository at this point in the history

  33. more descriptions

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    9f378cb View commit details
    Browse the repository at this point in the history

  34. comment fix

    @shaniqwa
    shaniqwa committed Apr 29, 2021
    Configuration menu
    Copy the full SHA
    74317f1 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2021

  1. remove moment js

    @shaniqwa
    shaniqwa committed May 3, 2021
    Configuration menu
    Copy the full SHA
    83ec05e View commit details
    Browse the repository at this point in the history

  2. remove log

    @shaniqwa
    shaniqwa committed May 3, 2021
    Configuration menu
    Copy the full SHA
    8b3c584 View commit details
    Browse the repository at this point in the history

  3. fix tags

    @shaniqwa
    shaniqwa committed May 3, 2021
    Configuration menu
    Copy the full SHA
    d84d2e3 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2021

  1. Merge pull request #1 from shaniqwa/jamie-oliver 

    Jamie oliver
    @shaniqwa
    shaniqwa authored Jul 8, 2021
    Configuration menu
    Copy the full SHA
    75c85e9 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2021

  1. fix some tests. add defaultSetDescription to BaseScraper

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    a2364ec View commit details
    Browse the repository at this point in the history

  2. fix more tests

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    47879bd View commit details
    Browse the repository at this point in the history

  3. fix woolworths

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    668bf36 View commit details
    Browse the repository at this point in the history

  4. try to change travis node version

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    5fe3af0 View commit details
    Browse the repository at this point in the history

  5. revert travis node version to stable 

    add logs
    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    d729e97 View commit details
    Browse the repository at this point in the history

  6. more logs

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    0234e5c View commit details
    Browse the repository at this point in the history

  7. logs

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    1dec3a6 View commit details
    Browse the repository at this point in the history

  8. log

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    c33596e View commit details
    Browse the repository at this point in the history

  9. skip should fetch the expected recipe test if server is not responding

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    8dd9980 View commit details
    Browse the repository at this point in the history

  10. add myrecipes description

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    d929b2b View commit details
    Browse the repository at this point in the history

  11. add description

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    46b0672 View commit details
    Browse the repository at this point in the history

  12. comment fix

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    81d3a44 View commit details
    Browse the repository at this point in the history

  13. remove moment js

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    01f4700 View commit details
    Browse the repository at this point in the history

  14. fix tags

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    f7ef513 View commit details
    Browse the repository at this point in the history

  15. rebase

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    f465c30 View commit details
    Browse the repository at this point in the history

  16. remove console.log

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    2b85e8e View commit details
    Browse the repository at this point in the history

  17. delete duplicate function

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    cc60ea8 View commit details
    Browse the repository at this point in the history

  18. rename file- typo

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    192b19e View commit details
    Browse the repository at this point in the history

  19. fix tests

    @shaniqwa
    shaniqwa committed Aug 7, 2021
    Configuration menu
    Copy the full SHA
    9782144 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2021

  1. add ld+json default scraper. apply to bon appetit and taste of home a… 

    …s well to fix broken tests
    @shaniqwa
    shaniqwa committed Aug 8, 2021
    Configuration menu
    Copy the full SHA
    9b1669d View commit details
    Browse the repository at this point in the history

  2. bug fixs

    @shaniqwa
    shaniqwa committed Aug 8, 2021
    Configuration menu
    Copy the full SHA
    b6023ec View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2021

  1. minor fixes

    @shaniqwa
    shaniqwa committed Aug 9, 2021
    Configuration menu
    Copy the full SHA
    86aa661 View commit details
    Browse the repository at this point in the history

  2. woolworths should use the ld json schema, but it's still not working … 

    …since server response is 403
    @shaniqwa
    shaniqwa committed Aug 9, 2021
    Configuration menu
    Copy the full SHA
    e651c01 View commit details
    Browse the repository at this point in the history

  3. remove woolworths test

    @shaniqwa
    shaniqwa committed Aug 9, 2021
    Configuration menu
    Copy the full SHA
    d776a87 View commit details
    Browse the repository at this point in the history

  4. fix handling unsupported / invalid urls

    @shaniqwa
    shaniqwa committed Aug 9, 2021
    Configuration menu
    Copy the full SHA
    f4533ac View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2021

  1. html decode - add regex for removing specific short cods 

    fix tests: mels kitchen now using ld json scraper
add expected results to all ld json scraper test cases
fix bug/edge case handling in description and servings
    @shaniqwa
    shaniqwa committed Aug 15, 2021
    Configuration menu
    Copy the full SHA
    fbad66d View commit details
    Browse the repository at this point in the history

  2. fix test

    @shaniqwa
    shaniqwa committed Aug 15, 2021
    Configuration menu
    Copy the full SHA
    ee6379c View commit details
    Browse the repository at this point in the history

  3. Merge pull request #2 from shaniqwa/JSON+LD 

    add Json+ld scraper
    @shaniqwa
    shaniqwa authored Aug 15, 2021
    Configuration menu
    Copy the full SHA
    daa4b0a View commit details
    Browse the repository at this point in the history

  4. add support for HowToSection. add test for a website requested in iss… 

    …ue (pav-bhaji-recipe-mumbai the longest recipe ever!!)
    @shaniqwa
    shaniqwa committed Aug 15, 2021
    Configuration menu
    Copy the full SHA
    ba11390 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #3 from shaniqwa/JSON+LD 

    add support for HowToSection. add test for a website requested in iss…
    @shaniqwa
    shaniqwa authored Aug 15, 2021
    Configuration menu
    Copy the full SHA
    2ca3b61 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2021

  1. fix typo

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    00ae10b View commit details
    Browse the repository at this point in the history

  2. add logs

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    36be742 View commit details
    Browse the repository at this point in the history

  3. Throw errors

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    7485a33 View commit details
    Browse the repository at this point in the history

  4. update puppeteer to latest version

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    bd18d28 View commit details
    Browse the repository at this point in the history

  5. don't use puppeteer in json scraper

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    89e26fb View commit details
    Browse the repository at this point in the history

  6. remove logs, fix test

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    1a85858 View commit details
    Browse the repository at this point in the history

  7. fix tests

    @shaniqwa
    shaniqwa committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    d89b6d9 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2021

  1. Merge pull request #4 from shaniqwa/JSON+LD 

    Json+ld
    @shaniqwa
    shaniqwa authored Aug 21, 2021
    Configuration menu
    Copy the full SHA
    48c8413 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2021

  1. if no recipe schema was found, do not throw error. return page title,… 

    … image & description meta data
    @shaniqwa
    shaniqwa committed Dec 13, 2021
    Configuration menu
    Copy the full SHA
    d53f5cc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into JSON+LD 

    * master:
    @shaniqwa
    shaniqwa committed Dec 13, 2021
    Configuration menu
    Copy the full SHA
    97afad1 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2021

  1. fix all tests

    @shaniqwa
    shaniqwa committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    bf64b78 View commit details
    Browse the repository at this point in the history

  2. fix 404 handling

    @shaniqwa
    shaniqwa committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    0aac237 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2023

  1. fix test. make expected test to match 80%. more scrapers now use defa… 

    …ult ld json
    @shaniqwa
    shaniqwa committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    caa800f View commit details
    Browse the repository at this point in the history

  2. more fixes

    @shaniqwa
    shaniqwa committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    5e7223e View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2023

  1. change to a proper module. update dependencies

    @shaniqwa
    shaniqwa committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    e0c1e34 View commit details
    Browse the repository at this point in the history

  2. remove puppeteer

    @shaniqwa
    shaniqwa committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    28ddf8a View commit details
    Browse the repository at this point in the history

  3. fix main path

    @shaniqwa
    shaniqwa committed Jun 3, 2023
    Configuration menu
    Copy the full SHA
    8e2de13 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. fix files

    @shaniqwa
    shaniqwa committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    72286a3 View commit details
    Browse the repository at this point in the history

  2. fix file path import

    @shaniqwa
    shaniqwa committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    fc39ded View commit details
    Browse the repository at this point in the history

  3. fix

    @shaniqwa
    shaniqwa committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    c357254 View commit details
    Browse the repository at this point in the history

Commits on Dec 25, 2023

  1. remove console.logs

    @shaniqwa
    shaniqwa committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    d6e7250 View commit details
    Browse the repository at this point in the history

  2. bump version

    @shaniqwa
    shaniqwa committed Dec 25, 2023
    Configuration menu
    Copy the full SHA
    d4acc04 View commit details
    Browse the repository at this point in the history