Allow building an Endless Key home directory #117
Comments
We want to be able to ship different content for Kolibri and the Endless Key app. This commit adds support for installing their content to independent locations, side-by-side. #117
We want to be able to ship different configuration for Kolibri and the Endless Key app. This commit adds support for installing their configuration to independent locations, side-by-side. #117
We want to be able to ship different configuration for Kolibri and the Endless Key app. This commit adds support for installing their configuration to independent locations, side-by-side. #117
|
I wrote a draft PR based on my initial understanding of the task description. But looking back at the project description and UX flow, I'm not sure how things are supposed to work: are we going to have a pre-defined list of pre-loaded content written-out in image configuration INI files, or have the user select the starter pack to be downloaded? The flow in the project spec has the starter pack selection screen, but isn't part of the idea to avoid the initial download for (some of) the Endless OS users? |
|
When I last spoke with @dylanmccall about this, I think the intention was to make the image builder use the content manifest files directly instead of converting to the image builder's INI format. However, I think we can start with the INI format if that's easier. I could see adding a management command to kolibri-explore-plugin that just does the initial collection download immediately rather than the current method where it's driven by the frontend. Then the image builder configuration would be as simple as specifying the desired content pack. That would take a bit of work in the explore plugin, though. |
|
@dbnicholson Thanks, that helps clarify things a bit. However, on the current UX flow, for the English version of the app we're still expecting the user to select a starter pack, which to me seems to indicate that the image builder should not download / pre-load any content for the Endless Key app, since the user will choose it when they first run the app. That said, I don't know how well that model fits with the target audience of Endless OS, so I think we should check with @jofilizola. With that said, you are right that we can start with pre-loading some content using the image builder configuration INI files, so I'll go ahead and create a test configuration and test my changes, and mark them ready for review once I'm happy with the code. |
|
Oh, I didn't realize the starter pack would be chosen at runtime. I guess one way forward would be preloading the content from all starter packs in the image. A slimmer alternative would be downloading all the channels and all the thumbnails from all the collections. You'd still need to download the actual content at runtime, but you could skip some of the other download steps. |
|
I think we can punt on the content right now. I think the most important things we needed to setup an Endless Key Kolibri homedir are:
That's the bare minimum to have a system managed homedir supporting the explore plugin since it can already handle all the downloading at runtime. |
|
I had a quick call with @jofilizola earlier today and she clarified the intended behavior: the plan is to preload all Endless Key content into certain target images. There will still be a selection screen for a start pack (which maybe we should start calling something else), but that selection will only affect what appears in the discovery page. The library will show all the Endless Key content. |
We want to be able to ship different content and configuration for Kolibri and the Endless Key app. This commit adds support for installing the content and configuration for each app to separate locations, so they can be used side-by-side. #117
We want to be able to ship different content and configuration for Kolibri and the Endless Key app. This commit adds support for installing the content and configuration for each app to separate locations, so they can be used side-by-side. #117
The Endless Key app needs a different set of plugins than traditional Kolibri. #117
I've got the list from https://github.com/endlessm/eos-administration/blob/master/ansible/roles/kolibri/templates/setup-kolibri.sh.j2#L109-L116 minus
So essentially the same as what is done in https://github.com/endlessm/eos-administration/blob/master/ansible/roles/kolibri/files/download-apps.sh.
It looks like these paths already defaults to the right place if the options are not set. See https://github.com/endlessm/kolibri-explore-plugin/blob/master/kolibri_explore_plugin/views.py#L25-L28 and https://github.com/endlessm/kolibri-explore-plugin/blob/master/kolibri_explore_plugin/collectionviews.py#L33-L35 I have some draft code addressing all these points, pending testing. |
|
I have a draft PR ready for an initial round of review, however, I'm hitting the following problem when trying to enable |
I think this is because Kolibri 0.15.7 is being used: The current explore plugin releases need the 0.16 and the error is because the plugin API changed. You'll have to go the app route and install from a github wheel like https://github.com/learningequality/kolibri/releases/download/v0.16.0-beta5/kolibri-0.16.0b5-py2.py3-none-any.whl. This might be too messed up to carry on right now. The idea here is that you want to use a Kolibri version less than or equal to the one that will be in the Flatpak. Otherwise you risk running database migrations that code in the flatpak doesn't know about. However, neither the Kolibri flatpak nor the Endless Key flatpak are actually using Kolibri 0.16 yet. See endlessm/endless-key-flatpak#9. What would be a little more appropriate, particularly since |
|
In that case, I think running the flatpak may be the best approach, especially since we maybe be installing the Kolibri and Endless Key flatpaks side-by-side, and they may be shipping different versions of kolibri. Let's move this back to the backlog for now and focus on the remaining bits to have the flatpak published. At that point, we can come back to this and try to do the right thing. |
We are going to use the kolibri command from the flatpak app, so we need to be running from ostree. #117
|
I got back to this for a bit and wrote the changes to use kolibri from the flatpak, but I'm now hitting an error from (I'm running |
The |
The Endless Key app needs a different set of plugins than traditional Kolibri. #117
I'm still hitting this error when trying to call |
|
I also looked into adjusting the permissions of KOLIBRI_HOME for Endless Key, and found out that it is already addressed by endlessm/eos-kolibri@d9d4419. However, we do not create a separate system user for Endless Key, so its KOLIBRI_HOME will be chown'ed to |
We want to be able to ship different content and configuration for Kolibri and the Endless Key app. This commit adds support for installing the content and configuration for each app to separate locations, so they can be used side-by-side. #117
The Endless Key app needs a different set of plugins than traditional Kolibri. #117
|
I believe what I have written now is doing the right thing. In addition to what is described on the last comments above, I'm also allowing the user to specify in the configuration which collections (artist, explorer etc) they want to preload, although I'm not entirely sure if we'll use it or not -- depends on how we end up defining the preloading strategy. I can easily drop that part if we don't want it. But I am hitting a problem when preloading the content, where In my tests it always fails on that same channel, I've opened a draft PR with my changes. |
Trying with collections that do not include that channel (athlete + spanish) got me to avoid that problem, although the build failed for an unrelated reason (good old |
I imagine the channel database is corrupted on the content server. Let me take a look. |
|
I deleted the channel on the server, so the next time you run this it should be imported again. |
Great, thanks! I'll give it a try. That may have happened because I aborted one build while it was running the hook that instructs the server to download some channels, but it is just a guess. Let's see if it builds successfully now. In any case, I was able to test a build with the athlete + spanish collections, and it looks like things are working.
There are a few things we may want to address separately though.
|
Same error, with the same channel ID: |
And it looks like the channel was re-imported (other channels just jumped from 0 straight to 100%): |
Instead of loading every channel from every collection JSON from the Endless Key app, allow the image configuration to list which collections to include. #117
We want to preload the whole Endless Key library in Endless OS images. We also need to ensure the content is available in our Kolibri cache server, if we are pulling the content from it. #117
Instead of loading every channel from every collection JSON from the Endless Key app, allow the image configuration to list which collections to include. #117
|
I believe the import problem is unrelated to the changes. These hooks are executed unless two things are true: A. |
|
I built this locally with the One potential issue is that we're doing the installation with 0.15.7, which is very old even for the nearly dead 0.15 series. I'm going to add on a couple commits that:
|
We still need to select a content pack so that the tags can be applied at runtime. endlessm/kolibri-explore-plugin#863
The product decision was to disable the generated channel launchers for Endless Key, so these will go away. |
This will avoid trying to download and install the same channel more than once. #117
|
I've created #121 addressing the deduplication of the list of channels to be downloaded. Once that is merged I can test with eos5.0 and backport. |
Will those changes also prevent number 4 above? Or do we need to remove a mime type handler somewhere else? |
Oh, for some reason I thought we'd filed that one away somewhere, but I can't find it. I made a separate ticket for it over here: endlessm/endless-key-flatpak#102. It's a minor fix we need to do in the Endless Key desktop app, independent of ostree stuff. |
|
This work needs to be backported to the eos5.0 branch, although it would be nice to give it a test on master first. |
We want to preload the whole Endless Key library in Endless OS images. We also need to ensure the content is available in our Kolibri cache server, if we are pulling the content from it. #117
Instead of loading every channel from every collection JSON from the Endless Key app, allow the image configuration to list which collections to include. #117
This will avoid trying to download and install the same channel more than once. #117
|
I have backported this to eos5.0 and tested a 5.0 build. Things seem to be working, but it still tried to "download" content when I selected the Explorer pack (which was already preloaded). However, it was very fast (probably about 1 minute), and looking at the logs it seems only the tagging was happening at that time. It would be nice to fix the UX so it does not look like to the user that there is still some download happening. Here is the config I used for testing: And here are the logs: Since the changes here look correct, I've pushed them to eos5.0. |
|
This is followed-up by endlessm/kolibri-explore-plugin#890 |
The image builder knows how to create a predefined Kolibri home at
/var/lib/kolibriwith a selection of content (described somewhere in our wondrous web of INI files). It generates a database file for Kolibri to pick up right where the image builder left off; no database migrations to worry about. We should change a lot of this. First, let's just get it working: where the image builder hard-codes/var/lib/kolibri, we want it to optionally use/var/lib/endless-key.The text was updated successfully, but these errors were encountered: