10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed, GCFAprendeLibre] #14

holta · 2023-01-19T01:18:23Z

@tim-moody please review before merging, to confirm this is sufficiently correct?

holta · 2023-01-19T01:36:53Z

FYI this PR is nothing more than an attempt at cleaning up the metadata for these three ~10GB ZIM files:

Long story short, this PR is nothing more than the automated results arising from running:

cd /opt/iiab-share/iiab-content/catalogs
./mk-zim-cat-item.py cest-pas-sorcier_fr_top_2021-01.zim --source https://s3.us-east-2.wasabisys.com/iiab-zims/
./mk-zim-cat-item.py crashcourse_en_top_2021-01.zim --source https://s3.us-east-2.wasabisys.com/iiab-zims/
./mk-zim-cat-item.py teded_en_top_2021-01.zim --source https://s3.us-east-2.wasabisys.com/iiab-zims/
mv cest-pas-sorcier_fr_top_2021-01.json crashcourse_en_top_2021-01.json teded_en_top_2021-01.json zim-cat-fragments
./mk-zim-cat.py

Concluding Questions:

Is the above automated process good enough?
Mangled and ugly metadata (below) unfortunately appears in each ZIM file's .json fragment — but seems to likely originate from the ZIM files themselves — so probably we're stuck with it for now?

"tags": "y;o;u;t;u;b;e;_;v;i;d;e;o;s;:;y;e;s;_ftindex:no;_pictures:yes;_videos:yes;_details:yes",

holta · 2023-01-19T01:44:23Z

ASIDE: How important is the following issue?

mk-zim-cat-item.py does not do maxi, etc. properly #12

holta · 2023-01-19T01:46:44Z

Recap screenshots showing the original problem: (Admin Console ends up showing ~10 TB erroneously, when it should show ~10 GB for these 3 ZIM files)

holta · 2023-01-19T01:49:14Z

Another way of viewing the proposed (harmless???) SIDE EFFECTS of this PR:

holta · 2023-01-19T07:37:43Z

Note that metadata associated with these 3 ZIM files is already problematic even prior to this PR — as seen in this kiwix-serve screenshot — if you look closely:

So I'd assume this PR does not make the situation any worse? Hopefully @tim-moody can confirm?

tim-moody · 2023-01-19T14:20:10Z

I don't see anything wrong with the revised version, except the date, which is probably technically correct as in when the zim was created, but not in sync with the content.

The addition of the funny youtube tag and the first videos tag is not clear to me. Might have been passed as an argument at zim creation and kiwix mangled it. Hard for me to think that kiwix knows this is youtube material and added it.

The important thing is that size is now correct and publisher is now correct, both used by the catalog display. Also, we need the pictures and details tags that are set by kiwix and were missing.

Kiwix reports the path relative to the zims directory, so the ../library is common across all zims and isn't really used.

Is the above automated process good enough?

Well, it works for me, and I was the only one who used it. The problems identified in this ticket were produced by not using it.

But it is hardly ready for mass consumption, so if there are an increasing number of zim creators who can't use it, someone could improve it.

holta · 2023-01-19T14:35:17Z

@georgejhunt can you recommend merging this PR experimentally — or another course of action if further metadata fixes are required?

holta · 2023-01-19T17:09:55Z

ASIDE: @deldesir asks if we can learn from @georgejhunt about any crucial-or-critical metadata hand-fixes that might be advisable in general?

Specifically, do the 3 Basic Electricity ZIM files below illustrate any metadata tips & tricks we should learn from here?

holta · 2023-01-19T19:23:25Z

ASIDE: @tim-moody you're probably aware but just FYI @deldesir did a quick survey of metadata across all 9 of our IIAB Catalog ZIM files:

"Educatonal" is misspelled at https://github.com/iiab-share/iiab-content/blob/main/catalogs/zim-cat-fragments/gcflearn_es_all_maxi_2021-03.json#L10
Logo (favicon) is missing in some cases like GCFLearnFree ES above. Also missing from these:
- Basic Electricity
- NIH Rare Diseases

tim-moody · 2023-01-19T19:29:13Z

"Educatonal" is misspelled

I probably mistyped it. Can be fixed by hand.

Not sure that we use favicon since we don't use kiwix as a front end

tim-moody · 2023-01-19T20:02:50Z

looks right

holta · 2023-01-19T20:10:24Z

Not sure that we use favicon since we don't use kiwix as a front end

Isn't this logo/favicon automatically extracted when new menudefs are auto-created, to showcase new ZIM files on IIAB's main page? (Or so I thought, maybe I've got this all wrong, apologies if so!)

tim-moody · 2023-01-19T20:16:00Z

Isn't this logo/favicon automatically extracted when new menudefs are auto-created

not sure if we are still able to do this with the new catalog. in any event I always assumed that if we create a zim we would also create its menu def

tim-moody · 2023-01-19T20:23:34Z

anyway, you're right someone should check if we are doing this with the new catalog and implement it if not. These logos look to be 48px x 48, which is a little small, but better than nothing. probably @deldesir could figure it out.

holta · 2023-01-20T19:23:01Z

@tim-moody after this PR (or similar) is merged, can users see the new ZIM catalog by logging back in to Admin Console, and/or by clicking "Reindex Kiwix Content", "Refresh Kiwix Catalog" or similar ?

(Or is a fresh install of IIAB & Admin Console likely necessary?)

tim-moody · 2023-01-20T21:38:26Z

just refresh catalog

holta · 2023-01-21T17:01:50Z

I tried contacting @georgejhunt directly for his recommendations here. No luck reaching him so far, since Wednesday. He should have good ideas. In any case, I suggest we move forward with this PR allowing for community testing — and amend it later wherever George and users can offer further improvements.

tim-moody · 2023-01-21T17:54:34Z

I agree

holta · 2023-01-22T14:35:05Z

Looks great.
Hopefully @deldesir and others can confirm.
Fresh installs of all 4 ZIM files can't hurt to verify!

tim-moody · 2023-01-22T15:41:03Z

Fresh installs of all 4 ZIM files can't hurt to verify!

would verify that source url not broken

holta · 2023-01-22T16:46:55Z

Fresh installs of all 4 ZIM files can't hurt to verify!

would verify that source url not broken

First fresh install doesn't seem to work: @tim-moody is it unrelated that jobs are marked "SCHEDULED" but never begin?

I rebooted and that does not help. Waiting 10min also did not help. FYI this is Debian 12 (currently in pre-release freeze, and generally reliable in other regards). Feel free to login to 10.8.0.38 if that helps understand why it's stuck?

tim-moody · 2023-01-22T16:51:53Z

can't login

holta · 2023-01-22T16:54:02Z

http://10.8.0.46/admin (Ubuntu 22.04) appears to have the very same problem.

Can you log into either? (Or if not, any idea why?)

holta · 2023-01-22T16:57:54Z

can log into admin but not ssh as my key not supported

Your ssh key should be supported on both VM's (mine works). Any idea what's happening?

tim-moody · 2023-01-22T17:02:00Z

mine doesn't on either

holta · 2023-01-22T17:05:24Z

mine doesn't on either

Hopefully when you're logged in as iiab-admin you can diagnose (why your ssh key is suddenly not working?)

holta · 2023-01-22T17:27:58Z

Just FYI iiab-cmdrv.service appears to be running fine on both VM's:
(below is from Debian 12 VM 10.8.0.38; Ubuntu 22.04 VM 10.8.0.46 is equivalent)

root@box:~# systemctl status iiab-cmdsrv
● iiab-cmdsrv.service - Provides the IIAB Command Server
     Loaded: loaded (/etc/systemd/system/iiab-cmdsrv.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-01-22 11:49:52 EST; 35min ago
    Process: 687 ExecStart=/opt/admin/cmdsrv/iiab-cmdsrv3.py --daemon (code=exited, status=0/SUCCESS)
   Main PID: 897 (iiab-cmdsrv3.py)
      Tasks: 9 (limit: 2316)
     Memory: 86.7M
        CPU: 8.761s
     CGroup: /system.slice/iiab-cmdsrv.service
             └─897 /usr/bin/python3 /opt/admin/cmdsrv/iiab-cmdsrv3.py --daemon

Jan 22 11:55:37 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-ADM-CONF.
Jan 22 11:55:37 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-VARS.
Jan 22 11:55:37 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-ANS.
Jan 22 11:55:37 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-IIAB-INI.
Jan 22 11:55:37 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-ZIM-STAT.
Jan 22 11:55:38 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-OER2GO-STAT.
Jan 22 11:55:38 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-OSM-VECT-STAT.
Jan 22 11:55:38 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-SPACE-AVAIL.
Jan 22 11:55:38 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-EXTDEV-INFO.
Jan 22 11:55:48 box IIAB-CMDSRV[897]: IIAB-CMDSRV : Received CMD Message GET-JOB-STAT {"last_rowid":1}.

tim-moody · 2023-01-22T17:54:07Z

adm cons doesn't think kiwix is installed, so is waiting for it

if you install from a preset kiwix will be installed, but otherwise you have to check it in configure and ico

holta · 2023-01-22T18:11:41Z

On 10.8.0.38 installing Kiwix and then restarting iiab-cmdsrv got things moving, thanks for solving that!
And then it moved TED-Ed from /libary/working/zims as soon as the 78MB nih_rarediseases_en_all_maxi_2020-12.zim was done downloading i.e. when the download of teded_en_top_2021-01.zim was only 3% complete!

Any idea why this is happening?

FYI /library/zims/content/teded_en_top_2021-01.zim continues to expand (presumably as a result of the wget command keeping the file handle, despite the destination having moved!) Is this A-Ok ??

tim-moody · 2023-01-22T18:11:47Z

what I said was with regard to .46. I see that .38 now has kiwix installed and the zim downloads have started and in one case succeeded.

tim-moody · 2023-01-22T18:17:40Z

The only thing I can think of is that the restart logic doesn't handle job dependencies properly.

tim-moody · 2023-01-22T18:19:12Z

Also though jobs 3 and 4 say succeeded, the zim was not added, and the job output even says that. On checking I see that rare diseases did get added, but not to menu

holta · 2023-01-22T18:24:04Z

Also though jobs 3 and 4 say succeeded, the zim was not added, and the job output even says that. On checking I see that rare diseases did get added, but not to menu

I noticed that.

Weird that neither ZIM file appears on the IIAB home page so far.

(Certainly I can manually force these later, i.e. after the 10GB TED-Ed is downloaded, I can run iiab-make-kiwix-lib or click Install Content > Reindex Kiwix Content -- and then run iiab-update-menus)

Hopefully these kinds of things are a very rare occurrence arising from systemctl restart iiab-cmdsrv ?

tim-moody · 2023-01-22T18:26:10Z

Hopefully these kinds of things are a very rare occurrence arising from systemctl restart iiab-cmdsrv ?

That's what I'm hoping. Just looking at the restart code to see if anything jumps out. Is it possible that wget continued even when you restart cmdsrv? i.e. there was no reboot

holta · 2023-01-22T18:29:46Z

Is it possible that wget continued even when you restart cmdsrv?

The only thing I did was systemctl restart iiab-cmdsrv to try to force both wget's to begin.

I did not reboot. (Should I have rebooted instead?)

tim-moody · 2023-01-22T18:32:11Z

restart assumes a cold start after a shutdown or crash

tim-moody · 2023-01-22T18:35:54Z

teded download is marked STARTED in the db, seems like we had a job that was still running at the OS level, but not at the cmdsrv level. don't know what that would do.

holta · 2023-01-22T18:40:44Z

Weird that neither ZIM file appears on the IIAB home page so far.

Just FYI neither ZIM file appeared on the IIAB main page http://10.8.0.38 in the end.

And just 1 of 2 appears at http://10.8.0.38/kiwix/ in the end.

(I can manually force the fixing of all the small issues above, as mentioned earlier, so I'm only mentioning this as an FYI.)

holta · 2023-01-22T18:42:00Z

teded download is marked STARTED in the db, seems like we had a job that was still running at the OS level, but not at the cmdsrv level. don't know what that would do.

Time to repeat the experiment on 10.8.0.46 == Ubuntu 22.04 just for kicks?

(Do you want to do that...or should I?)

tim-moody · 2023-01-22T18:43:19Z

reboot .46 first as I have cmdsrv running manually

holta · 2023-01-22T18:47:00Z

I rebooted .46
I ran ./runrole kiwix to install kiwix-tools successfully.

I don't know how to check the DB for individual jobs' status (do you want to do this, then restart the VM to see how it goes?)

tim-moody · 2023-01-22T18:48:15Z

there might be a logical flaw in cmdsrv 3634ff if the job is the highest number, has no dependencies, but is itself dependent.

holta · 2023-01-22T18:51:51Z

there might be a logical flaw in cmdsrv 3634ff

Is 3634ff a particular commit? Am I looking in the wrong repo if so (am not seeing it!)

tim-moody · 2023-01-22T18:55:51Z

sorry is a line number and I don't know how you do the fancy L# url

holta · 2023-01-22T19:01:48Z

These lines below? (Click ellipsis on left, if you want either kind of URL...)

tim-moody · 2023-01-22T19:02:10Z

https://github.com/iiab/iiab-admin-console/blob/master/roles/cmdsrv/files/iiab-cmdsrv3.py#L3566

is the function

yeah was just doing it

tim-moody · 2023-01-22T19:51:39Z

I ran ./runrole kiwix to install kiwix-tools successfully.

since this was after reboot, cmdsrv may not know about it unless you restart it and refresh admin

holta · 2023-01-22T20:04:14Z

I ran ./runrole kiwix to install kiwix-tools successfully.

since this was after reboot, cmdsrv may not know about it unless you restart it and refresh admin

I rebooted .46 (Ubuntu 22.04).

The behavior is identical to .38 (Debian 12).

(In each case, zim_install_move.sh teded_en_top_2021-01.zim is run prematurely, triggered by a different ZIM file download's completing.)

tim-moody · 2023-01-22T20:54:56Z

This is how it should work if kiwix is installed using Admin Console even after the zim downloads are scheduled.

And they were properly added to the home menu.

deldesir · 2023-01-23T21:08:56Z

FYI, I removed all ZIM files via admin console (Install content > Manage content > Remove selected content) and redownload them using Admin console again. I monitored the commands jobs and all succeed. I can access all the ZIM via Kiwix. No service restart was needed.

tim-moody · 2023-01-23T23:03:26Z

@deldesir perfect. thanks for confirming.

holta · 2023-02-10T22:44:57Z

2. And then it moved TED-Ed from /libary/working/zims as soon as the 78MB nih_rarediseases_en_all_maxi_2020-12.zim was done downloading i.e. when the download of teded_en_top_2021-01.zim was only 3% complete!

Above issue is confirmed fixed! Thanks to @tim-moody's:

PR fix job restart iiab/iiab-admin-console#523

10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py

f57dce3

holta requested a review from tim-moody January 19, 2023 01:18

holta requested a review from georgejhunt January 19, 2023 01:37

holta changed the title ~~10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py~~ 10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed] Jan 19, 2023

Typo in gcflearn_es_all_maxi_2021-03.json & iiab-zim-cat.json

ee75e8f

holta changed the title ~~10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed]~~ 10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed, GCFAprendeLibre] Jan 20, 2023

holta merged commit 5733f5a into iiab-share:main Jan 22, 2023

holta mentioned this pull request Feb 10, 2023

fix job restart iiab/iiab-admin-console#523

Merged

10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed, GCFAprendeLibre] #14

10 TB -> 10 GB: Fix metadata w/ mk-zim-cat-item.py & mk-zim-cat.py [C'est pas sorcier, CrashCourse, TED-Ed, GCFAprendeLibre] #14

Conversation

holta commented Jan 19, 2023

holta commented Jan 19, 2023 • edited Loading

holta commented Jan 19, 2023

holta commented Jan 19, 2023

holta commented Jan 19, 2023

holta commented Jan 19, 2023

tim-moody commented Jan 19, 2023

holta commented Jan 19, 2023

holta commented Jan 19, 2023 • edited Loading

holta commented Jan 19, 2023

tim-moody commented Jan 19, 2023

tim-moody commented Jan 19, 2023

holta commented Jan 19, 2023

tim-moody commented Jan 19, 2023 • edited Loading

tim-moody commented Jan 19, 2023

holta commented Jan 20, 2023 • edited Loading

tim-moody commented Jan 20, 2023

holta commented Jan 21, 2023 • edited Loading

tim-moody commented Jan 21, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023 • edited Loading

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023 • edited Loading

holta commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

tim-moody commented Jan 22, 2023

tim-moody commented Jan 22, 2023 • edited Loading

holta commented Jan 22, 2023 • edited Loading

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023 • edited Loading

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023

tim-moody commented Jan 22, 2023

tim-moody commented Jan 22, 2023

holta commented Jan 22, 2023 • edited Loading

tim-moody commented Jan 22, 2023

deldesir commented Jan 23, 2023

tim-moody commented Jan 23, 2023

holta commented Feb 10, 2023

holta commented Jan 19, 2023 •

edited

Loading

holta commented Jan 19, 2023 •

edited

Loading

tim-moody commented Jan 19, 2023 •

edited

Loading

holta commented Jan 20, 2023 •

edited

Loading

holta commented Jan 21, 2023 •

edited

Loading

holta commented Jan 22, 2023 •

edited

Loading

tim-moody commented Jan 22, 2023 •

edited

Loading

tim-moody commented Jan 22, 2023 •

edited

Loading

holta commented Jan 22, 2023 •

edited

Loading

tim-moody commented Jan 22, 2023 •

edited

Loading

holta commented Jan 22, 2023 •

edited

Loading