Do not search for exhaustive SNIPPET. #697
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`MSet::snippet` is more complex than it seems. Xapian does some kind of complex algorithm to find the best text subset to select. It does this by calculating the score/ranking of each term in the text. To do so, it has to evaluate the terms in the context of the whole mset and so, load "lot" of data from the database. The perfect is the enemy of the good. By removing the SNIPPET_EXHAUSTIVE flag, xapian evaluate less and return (far more) quicker. (https://xapian.org/docs/apidoc/html/classXapian_1_1MSet.html#a4797ae2295f88e49a9f76e3b89c21d88aea6a34a9c66720a44d5969ed47ca8edb) Generated snippet is different, but still valid. Do not search for exhaustive SNIPPET
Codecov Report
@@ Coverage Diff @@
## master #697 +/- ##
=======================================
Coverage 84.61% 84.61%
=======================================
Files 98 98
Lines 4308 4310 +2
Branches 1873 1869 -4
=======================================
+ Hits 3645 3647 +2
Misses 662 662
Partials 1 1
Continue to review full report at Codecov.
|
|
Great! This deserves a dedicated (patch) release IMO. |
It seems it has even more importance in the snippet generation speed. SNIPPET_BACKGROUND_MODEL ask xapian to also compute score for non-query terms. By removing it, we compute score only for query terms and it is far better.
|
It seems that by removing |
veloman-yunkan
approved these changes
May 16, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Working on kiwix/libkiwix#769 (which is about suggestion), I've found this important improvement (about search).
MSet::snippetis more complex than it seems.Xapian does some kind of complex algorithm to find the best text subset to
select. It does this by calculating the score/ranking of each term in the
text. To do so, it has to evaluate the terms in the context of the whole
mset and so, load "lot" of data from the database.
The perfect is the enemy of the good.
By removing the SNIPPET_EXHAUSTIVE flag, xapian evaluate less and return
(far more) quicker.
(https://xapian.org/docs/apidoc/html/classXapian_1_1MSet.html#a4797ae2295f88e49a9f76e3b89c21d88aea6a34a9c66720a44d5969ed47ca8edb)
Generated snippet is different, but still valid.
Do not search for exhaustive SNIPPET
On my computer with a zim file on a external usb drive (for low IO) and with a clean fs cache, a search for
homedrop from 87s to 15s (!!)Generated snippets are different but the quality doesn't seem really impacted :
Original snippet (slow) :
New snippet (quick):