-
-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add freecodecamp scraper #815
Conversation
f904ef4
to
74445e3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good ; thanks for the test, it's nice to have.
I am requesting changes mostly because of the category/warehouse_path that @kelson42 must answer first.
If you can make a minor release about output
param, that would be nice as this is a new scraper.
dispatcher/backend/src/common/schemas/offliners/freecodecamp.py
Outdated
Show resolved
Hide resolved
I guess I missed the question, sorry. But if the question is: do we need a dedicated (zimfarm) category, IMHO yes. |
@rgaudin Good catch on output folder and languages list! I'm a bit puzzled about output flag, I see no reason to keep it in the list of scraper flags and force it to a '/output' value since it is then setup properly at zimfarm/dispatcher/backend/src/utils/offliners.py Lines 47 to 51 in 576565b
I confirmed via slack with Kelson that we have to create both a new category and a new folder, because he expects us to quickly create many FCC ZIMs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK then ; I'll walk you through creating those folders
There is a reason why the output flag is kept in each scraper flags ; it is "mandatory" to avoid the UI / API having mixed feeling about this weird flag. I reverted the change. |
Rationale
Changes
ScheduleCategory
andOffliner
enumsDockerImageName
for freecodecamp has been set toopenzim/freecodecamp
categories
,warehouse_paths
andoffliners
insrc/contstants.js
POST /tasks/{task_id}
operation needs aworker_name
set as query parameterTests