Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OGA audio file are mirrored in a novid ZIM #1571

Closed
kelson42 opened this issue Dec 23, 2021 · 11 comments
Closed

OGA audio file are mirrored in a novid ZIM #1571

kelson42 opened this issue Dec 23, 2021 · 11 comments
Assignees
@pavel-karatsiuba
Labels
bug good first issue wikimedia Direct impact on Wikimedia content scraping
Milestone
1.12.0

Comments

@kelson42
Copy link
Collaborator

https://en.wikipedia.org/wiki/Laissez-faire
@kelson42 kelson42 added bug good first issue wikimedia Direct impact on Wikimedia content scraping labels Dec 23, 2021
@dcshzj
Copy link

dcshzj commented Jan 11, 2022

Hello! I was exploring this issue and I found that the audio file is not downloaded for the "novid" format. This is the snip of the download log for a normal download of the Laissez-faire article:

Normal download log 
[log] [2022-01-11T02:34:44.976Z] Downloading a total of [46] files...
[info] [2022-01-11T02:34:44.977Z] Worker [0] processing batch of [10] files
[info] [2022-01-11T02:34:44.978Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-validated.png?6232c]
[info] [2022-01-11T02:34:44.978Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Wiktionary-logo-en-v2.svg/40px-Wiktionary-logo-en-v2.svg.png]
[info] [2022-01-11T02:34:44.978Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/8/8a/Loudspeaker.svg/11px-Loudspeaker.svg.png]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/audio-ltr.svg?74d04]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://upload.wikimedia.org/wikipedia/en/thumb/8/8a/OOjs_UI_icon_edit-ltr-progressive.svg/10px-OOjs_UI_icon_edit-ltr-progressive.svg.png]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-icon-hl.svg?5e652]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-icon.svg?c5368]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/external-link-ltr-icon.svg?48e54]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/skins.vector.styles.legacy/images/user-avatar.svg?b7f58]
[info] [2022-01-11T02:34:44.979Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-problematic.png?f3177]
[info] [2022-01-11T02:34:44.985Z] Worker [1] processing batch of [10] files
[info] [2022-01-11T02:34:44.986Z] Downloading [https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-wordmark-en.svg]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/2/23/Icons-mini-file_acrobat.gif]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/5/5d/Checker-16x16.png]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/portal-separator.png?4ab04]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/34px-Wikiquote-logo.svg.png]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-temp-icon-hl.svg?f3db6]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/magnify-clip-rtl.svg?38fd5]
[info] [2022-01-11T02:34:44.986Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/search.svg?bbf78]
[info] [2022-01-11T02:34:44.987Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/external-ltr.svg?59558]
[info] [2022-01-11T02:34:44.987Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/document-ltr.svg?0bf90]
[info] [2022-01-11T02:34:44.995Z] Worker [2] processing batch of [11] files
[info] [2022-01-11T02:34:44.995Z] Downloading [https://upload.wikimedia.org/wikipedia/en/thumb/0/01/A_coloured_voting_box.svg/16px-A_coloured_voting_box.svg.png]
[info] [2022-01-11T02:34:44.995Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-temp-icon.svg?195f5]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/extensions/UniversalLanguageSelector/resources/images/cog-sprite.svg?c3fa1]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/f/f3/Emblem-money.svg/16px-Emblem-money.svg.png]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/watch-icon.svg?6ab6f]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/bullet-icon.svg?d4515]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/skins/MonoBook/resources/images/bullet.svg]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-silver-star.png?70a8c]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/watch-icon-hl.svg?9bf90]
[info] [2022-01-11T02:34:44.996Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-digitaldocument.png?d1c50]
[info] [2022-01-11T02:34:45.004Z] Worker [3] processing batch of [12] files
[info] [2022-01-11T02:34:45.004Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/2006_AEGold_Proof_Obv.png/28px-2006_AEGold_Proof_Obv.png]
[info] [2022-01-11T02:34:45.004Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-proofread.png?e81f9]
[info] [2022-01-11T02:34:45.004Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/tab-normal-fade.png?1cc52]
[info] [2022-01-11T02:34:45.004Z] Downloading [https://en.wikipedia.org/static/images/project-logos/enwiki-2x.png]
[info] [2022-01-11T02:34:45.004Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/ftp-ltr.svg?c5643]
[info] [2022-01-11T02:34:45.005Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/chat-ltr.svg?e0649]
[info] [2022-01-11T02:34:45.005Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/video.svg?a3758]
[info] [2022-01-11T02:34:45.005Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/Yellow_flag_waving.svg/75px-Yellow_flag_waving.svg.png]
[info] [2022-01-11T02:34:45.005Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/f/f9/En-us-laissez-faire.ogg]
[info] [2022-01-11T02:34:45.005Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Capitalismlogo.svg/16px-Capitalismlogo.svg.png]
[info] [2022-01-11T02:34:45.013Z] Worker [4] processing batch of [3] files
[info] [2022-01-11T02:34:45.013Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/tab-separator.png?09d4b]
[info] [2022-01-11T02:34:45.014Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/mail.svg?8a79c]
[info] [2022-01-11T02:34:45.014Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/Sound-icon.svg/45px-Sound-icon.svg.png]
[info] [2022-01-11T02:34:45.851Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/magnify-clip-ltr.svg?8330e]
[info] [2022-01-11T02:34:46.585Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-golden-star.png?ed948]
[info] [2022-01-11T02:34:46.587Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/arrow-down.svg?9426f]
[log] [2022-01-11T02:34:47.267Z] Progress downloading files [10/46] [21.7%]
[log] [2022-01-11T02:34:47.271Z] Progress downloading files [20/46] [43.5%]
[log] [2022-01-11T02:34:47.271Z] Progress downloading files [30/46] [65.2%]
[log] [2022-01-11T02:34:47.407Z] Progress downloading files [40/46] [87.0%]
[log] [2022-01-11T02:34:47.411Z] No files to retry
[log] [2022-01-11T02:34:47.411Z] Done with Downloading a total of [46] files

This is the download log with the "novid" format option enabled:

Novid format download log 
[log] [2022-01-11T02:35:03.816Z] Downloading a total of [45] files...
[info] [2022-01-11T02:35:03.818Z] Worker [0] processing batch of [10] files
[info] [2022-01-11T02:35:03.818Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-validated.png?6232c]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Wiktionary-logo-en-v2.svg/40px-Wiktionary-logo-en-v2.svg.png]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/8/8a/Loudspeaker.svg/11px-Loudspeaker.svg.png]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/audio-ltr.svg?74d04]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://upload.wikimedia.org/wikipedia/en/thumb/8/8a/OOjs_UI_icon_edit-ltr-progressive.svg/10px-OOjs_UI_icon_edit-ltr-progressive.svg.png]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-icon-hl.svg?5e652]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-icon.svg?c5368]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/external-link-ltr-icon.svg?48e54]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/skins.vector.styles.legacy/images/user-avatar.svg?b7f58]
[info] [2022-01-11T02:35:03.819Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-problematic.png?f3177]
[info] [2022-01-11T02:35:03.829Z] Worker [1] processing batch of [10] files
[info] [2022-01-11T02:35:03.829Z] Downloading [https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-wordmark-en.svg]
[info] [2022-01-11T02:35:03.829Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/2/23/Icons-mini-file_acrobat.gif]
[info] [2022-01-11T02:35:03.829Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/5/5d/Checker-16x16.png]
[info] [2022-01-11T02:35:03.829Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/portal-separator.png?4ab04]
[info] [2022-01-11T02:35:03.830Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/34px-Wikiquote-logo.svg.png]
[info] [2022-01-11T02:35:03.830Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-temp-icon-hl.svg?f3db6]
[info] [2022-01-11T02:35:03.830Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/magnify-clip-rtl.svg?38fd5]
[info] [2022-01-11T02:35:03.830Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/search.svg?bbf78]
[info] [2022-01-11T02:35:03.831Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/external-ltr.svg?59558]
[info] [2022-01-11T02:35:03.831Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/document-ltr.svg?0bf90]
[info] [2022-01-11T02:35:03.842Z] Worker [2] processing batch of [11] files
[info] [2022-01-11T02:35:03.842Z] Downloading [https://upload.wikimedia.org/wikipedia/en/thumb/0/01/A_coloured_voting_box.svg/16px-A_coloured_voting_box.svg.png]
[info] [2022-01-11T02:35:03.842Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/unwatch-temp-icon.svg?195f5]
[info] [2022-01-11T02:35:03.842Z] Downloading [https://en.wikipedia.org/w/extensions/UniversalLanguageSelector/resources/images/cog-sprite.svg?c3fa1]
[info] [2022-01-11T02:35:03.842Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/f/f3/Emblem-money.svg/16px-Emblem-money.svg.png]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/watch-icon.svg?6ab6f]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/bullet-icon.svg?d4515]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/skins/MonoBook/resources/images/bullet.svg]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-silver-star.png?70a8c]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/watch-icon-hl.svg?9bf90]
[info] [2022-01-11T02:35:03.843Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-digitaldocument.png?d1c50]
[info] [2022-01-11T02:35:03.852Z] Worker [3] processing batch of [11] files
[info] [2022-01-11T02:35:03.853Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/2006_AEGold_Proof_Obv.png/28px-2006_AEGold_Proof_Obv.png]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-proofread.png?e81f9]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/tab-normal-fade.png?1cc52]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/static/images/project-logos/enwiki-2x.png]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/ftp-ltr.svg?c5643]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/chat-ltr.svg?e0649]
[info] [2022-01-11T02:35:03.853Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/video.svg?a3758]
[info] [2022-01-11T02:35:03.854Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/Yellow_flag_waving.svg/75px-Yellow_flag_waving.svg.png]
[info] [2022-01-11T02:35:03.854Z] Downloading [https://en.wikipedia.org/w/extensions/WikimediaBadges/resources/images/badge-golden-star.png?ed948]
[info] [2022-01-11T02:35:03.854Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/arrow-down.svg?9426f]
[info] [2022-01-11T02:35:03.860Z] Worker [4] processing batch of [3] files
[info] [2022-01-11T02:35:03.860Z] Downloading [https://en.wikipedia.org/w/skins/Vector/resources/common/images/tab-separator.png?09d4b]
[info] [2022-01-11T02:35:03.860Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/mail.svg?8a79c]
[info] [2022-01-11T02:35:03.860Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/Sound-icon.svg/45px-Sound-icon.svg.png]
[info] [2022-01-11T02:35:04.767Z] Downloading [https://en.wikipedia.org/w/resources/src/mediawiki.skinning/images/magnify-clip-ltr.svg?8330e]
[info] [2022-01-11T02:35:05.017Z] Downloading [https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/Capitalismlogo.svg/16px-Capitalismlogo.svg.png]
[log] [2022-01-11T02:35:05.977Z] Progress downloading files [10/45] [22.2%]
[log] [2022-01-11T02:35:05.978Z] Progress downloading files [20/45] [44.4%]
[log] [2022-01-11T02:35:05.992Z] Progress downloading files [30/45] [66.7%]
[log] [2022-01-11T02:35:06.267Z] Progress downloading files [40/45] [88.9%]
[log] [2022-01-11T02:35:06.268Z] No files to retry
[log] [2022-01-11T02:35:06.269Z] Done with Downloading a total of [45] files

There is one less file downloaded (En-us-laissez-faire.ogg), which is correct as it is the spoken version of the article at the bottom. However, I also noticed that in both cases, the "listen" link at the top of the article is clickable, but nothing happens when a user clicks on it.

Do you have any pointers on how to proceed with this issue? I would love to help out as a new contributor. Thank you!

@kelson42
Copy link
Collaborator Author

kelson42 commented Jan 11, 2022 •
edited
Loading

Hmm.... it is there available http://library.kiwix.org/wikipedia_en_all_maxi/A/Laissez-faire (ZIM file is https://mirror.download.kiwix.org/zim/wikipedia/wikipedia_en_all_maxi_2021-12.zim) and see the tags http://library.kiwix.org/wikipedia_en_all_maxi/M/Tags. This is done with novid as well. Here is the recipe of the Zimfarm https://farm.openzim.org/recipes/wikipedia_en_all_maxi. If novid is specified no audio/video file should be download at all. A clickable link pointing to HTTP 404 is wrong as well, maybe both problems are linked... or not.

@ghost
Copy link

ghost commented Feb 5, 2022

Hello, I am new here and want to contribute in your project I would like to ask what i should learn to contribute here and please also tell me some resources ....

@Aadish01
Copy link

Aadish01 commented Mar 7, 2022

Hello , is this issue still open or assigned to someone ?

@kelson42
Copy link
Collaborator Author

kelson42 commented Mar 7, 2022

yes, this is open

@Aadish01
Copy link

Aadish01 commented Mar 7, 2022

Can you please tell which file needs to be fixed ??

@stale
Copy link

stale bot commented Jul 10, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

@stale stale bot added the stale label Jul 10, 2022
@kelson42 kelson42 added this to the 1.12.0 milestone Jul 10, 2022
@stale stale bot removed the stale label Jul 10, 2022
@stale
Copy link

stale bot commented Sep 21, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

@stale stale bot added the stale label Sep 21, 2022
@kelson42 kelson42 modified the milestones: 1.13.0, 1.12.0 Dec 31, 2022
@stale stale bot removed the stale label Dec 31, 2022
@kelson42 kelson42 self-assigned this Dec 31, 2022
@kelson42 kelson42 modified the milestones: 1.12.0, 1.13.0 Dec 31, 2022
@kelson42 kelson42 modified the milestones: 1.13.0, 1.12.0 Jan 26, 2023
@kelson42
Copy link
Collaborator Author

kelson42 commented Feb 1, 2023

It seems to me this might be a duplicate of #890

@kelson42
Copy link
Collaborator Author

kelson42 commented Feb 1, 2023 •
edited
Loading

So here we have in the HTML scraped from Wikipedia something like:

<a href="//upload.wikimedia.org/wikipedia/commons/7/7e/Fr-Laissez-faire.oga" class="internal" title="Fr-Laissez-faire.oga">listen</a>

If include (so no special --format, then the link should point to the data in the ZIM

If not-included, then the link should be removed, but content "listen" should be kept (like for a read/dead internal link).

listen

Obviously we should ideally do something smarter than that (don't display "listen" with the logo before), but this is an other topic.

@kelson42 kelson42 modified the milestones: 1.12.0, 1.13.0 Feb 1, 2023
@kelson42
Copy link
Collaborator Author

kelson42 commented Feb 1, 2023

Fixed by #1763

@kelson42 kelson42 closed this as completed Feb 1, 2023
@kelson42 kelson42 modified the milestones: 1.13.0, 1.12.0 Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pavel-karatsiuba pavel-karatsiuba

Labels
bug good first issue wikimedia Direct impact on Wikimedia content scraping
Projects
None yet
Milestone
1.12.0
Development

No branches or pull requests
4 participants
@kelson42 @dcshzj @pavel-karatsiuba @Aadish01