Adding full-text search

[Inbefortus]

In Kiwix iOS it is possible to search text within articles, and if I'm not mistaken, it is also possible on PC. Unfortunately, this feature was removed in Kiwix Android a while ago. :(

This feature would allow us users to dive even deeper into Wikipedia, it would enhance search significantly, and also enhance navigation by quite a bit.

Personally, I find it extremely beneficial. I can't think of any disadvantages.

The question is, of course, whether this is feasible in terms of programming.

@Jaifroid I think you once said that this is not implementable at the moment, but I could be wrong. Regardless, what do you think?

[Jaifroid]

@Inbefortus It's something we're looking to implement via recent work that was undertaken, and reported upstream here: kiwix/kiwix-js#769 (this is the port of the libzim library to WASM = WebAssembly). There are a few blockers on the PR kiwix/kiwix-js#766, notably kiwix/kiwix-js#513, where large ZIM files over 2GB cannot currently be handled by the port of libzim.

So it's a bit slow, but we're making progress...

If we can't make progress on the 2GB limit, then another possible angle would be to compile Xapian directly to WebAssembly and bind it to our current implementations. This would overcome the 2GB limit, but would mean implementing things directly in JS that others have already solved in C++, and that's hard work...

[Inbefortus]

@Jaifroid Very cool that you are already working on it. Please let me know if I can be of any help, I would be willing to take a closer look at this once you have something out.

[Jaifroid]

@Inbefortus Just to let you know that this is now implemented in the PWA and the Electron app (see Releases), and the browser-extension versions (Kiwix JS).

[Inbefortus]

@Jaifroid Yes, I know, I happened upon it by accident and was pleasantly surprised :-) Congratulations on the successful implementation, you and your team! That must have been quite a ride.

Sadly, Full-Text Search has ceased to function as a result of a recent update, as it is no longer triggered. There are only title search results.

• https://farm.openzim.org/pipeline/99092c6329a4e53250041b36 is the ZIM I use.

Before, it worked flawlessly. Is there something I'm overlooking?

[Jaifroid]

@Inbefortus If you look at the bottom of Configuration, there is an API panel. It should give you some clue as to why Full-Text search isn't available. It could be the ZIM doesn't have a full-text index? This is what it looks like if the ZIM has FT index and the system/browser supports it:

And this is what it looks like if the ZIM has no FT index (e.g., the endless ZIMs don't have one in order to produce a smaller ZIM):

Finally, here's what happens if the browser/platform doesn't support it:

If you see "no_atomics" it means the browser doesn't support that type of WASM. I'm working on extending support if possible to older browsers. It also doesn't work on Android (yet).

[Jaifroid]

@Inbefortus I downloaded Wikipedia 0.8 and opened it with the PWA on Windows and Linux, also in the Electron app on both OSs. The full-text index appears to be working for me:

[Inbefortus]

@Jaifroid The issue appears to be installing Kiwix JS PWA rather than doing so directly in the browser (in this case, Samsung Internet Browser).

Installation:

Browser:

Interesting. I've always used the installation, and I'm still puzzled as to why it used to work but now doesn't. Whatever. I'm glad it's working again. From now on, I will only use the browser version if necessary.

[Jaifroid]

Hmm, that's a bit strange, if it's the same browser and same OS. It shouldn't make any difference whether installed or not. It's not supposed to work on Android at all (hence the "no_sharedArrayBuffer" designation). Is this Android or Chrome OS?

Your feedback is very useful because it may in fact be a false detection of lack of support in the case of the installed app, and I'd like to ensure the app doesn't artificially prevent usage of full-text search when in fact it is supported!

[Jaifroid]

I'm still puzzled as to why it used to work but now doesn't. Whatever. I'm glad it's working again. From now on, I will only use the browser version if necessary

This would be explained by my putting in code to prevent an anomalous situation in Android phones (at least, my own Android phone) which appeared to support sharedArrayBuffer, but actually couldn't load the full-text index... Clearly that code is not universally true, so I need to revisit it.

[Inbefortus]

@Jaifroid

Is this Android or Chrome OS?

My device is still running Android 10.

it may in fact be a false detection of lack of support in the case of the installed app

That, I believe, is exactly the case here!