knowhere-v1.3.9

Release : knowhere-v1.3.9

What's Changed

• Split queries in IVF and BF by @hhy3 in #702

Full Changelog: v1.3.8...v1.3.9

knowhere 2.0.0

Knowhere-2.0

At Knowhere 2.0, our aim is to be more accessible by lowering the barriers to learning and contribution. This involves restructuring our code for improved readability and scalability. Also, the previous version of our code has been relocated to the Knowhere-1.x branch.

Inheritance

Knowhere 1.x inherited its structure from FAISS. However, as more indexes were integrated, this structure became difficult to maintain and understand. Knowhere 2.0 reworks this by adapting a one-layer FLAT inheritance structure.

In Knowhere 2.0, all indexes inherit from the IndexNode class, which has a clean structure. Furthermore, any new joining index in the future only needs to implement this IndexNode class.

API

Knowhere 2.0 has simplified its interfaces by eliminating unused legacy interfaces and combining duplicate ones.

Config

Knowhere 1.x uses JSON as its configuration format, which requires custom code for serialization, deserialization, validation, and more. This can sometimes make it difficult to understand and extend the code.

In Knowhere 2.0, all you need to do is write a configuration class that inherits from the BaseConfig class for each index. Knowhere will take care of the rest of the work, including serialization, deserialization, and validation.

Here is an example code:

class SimpleConfig : public BaseConfig {
 public:
    KNOHWERE_DECLARE_CONFIG(BaseConfig) {
        KNOWHERE_CONFIG_DECLARE_FIELD(k)
            .set_default(15)
            .description("search for top k similar vector.")
            .for_all();
    }
};

Error code

Instead of throwing multiple exceptions, Knowhere 2.0 will catch all exceptions and convert them into error codes. This will enhance the library's usability and readability, while also ensuring the caller's safety as Knowhere continues to expand.

Next step

As previously mentioned, our goal with Knowhere 2.0 is to be more open. As a result, we are working to improve the contribution experience. For instance, we are still in the process of creating a simple and straightforward set of APIs for third-party contributors who are interested in helping us expand our SIMD support.

In our future updates, we will prioritize support for more indexes, such as GPU-based and SCANN, which are critical areas for our development efforts.

knowhere-v1.3.8

Release : knowhere-v1.3.8

What's Changed

• Use param radius and range_filter for range search by @cydrain in #635
• Rename need_filter to range_filter_exist for better readibility by @cydrain in #639
• Fixed DiskANN bitsetView filter problem @hhy3 in #658
• FIxed too many threads problem when we do Bruteforce by @hhy3 in #662
• Remove useless memory malloc in diskann by @cqy123456 in #667

Full Changelog: v1.3.7...v1.3.8

knowhere-v1.3.7

Release : knowhere-v1.3.7

What's Changed

• Support show knowhere version by @cydrain in #607
• Enable debug log by @cydrain in #609
• Fix BruteForce::Search memory leak when metric type invalid by @cydrain in #611
• Fix range search benchmark wrong with IP metric type by @cydrain in #632
• Replace diskann distance function by faiss distance function by @cqy123456 in #634

Full Changelog: v1.3.6...v1.3.7

knowhere-v1.3.6

Release : knowhere-v1.3.6

knowhere-v1.3.5

Release : knowhere-v1.3.5

knowhere-v1.3.4

Release : knowhere-v1.3.4

Main Changes

• Faster Index build for DiskANN and HNSW. For SIFT1M data, DiskANN improved from 699s to 275s, HNSW improved from 318s to 294s
• Faster Search for HNSW serial execution improved ~50% at 90% recall, parallel improved ~3 times QPSs.
• Fix potential aio exceeding OS limit problem by introducing an aio pool.
• Fixed aio io_getevents crash problem
• Add Thread pool for all indexes to fix the omp threads number problem.

Chang List

• Print error message when the function abnormal returns by @cqy123456 in #491
• Add ut for AsyncIndex by @smellthemoon in #492
• [skip e2e] Add deps for code checker into build image by @jennyli-z in #495
• [skip e2e]Add python2 for cpp lint by @jennyli-z in #497
• [skip e2e]Add python2 for cpp lint by @jennyli-z in #498
• [skip e2e]Update ut to 20.04 by @jennyli-z in #488
• Support metric IP for diskann range search by @cqy123456 in #496
• Re-organize knowhere benchmark by @cydrain in #499
• Add benchmark ssnpp by @cydrain in #501
• Remove assemble and disassemble util func by @xige-16 in #502
• Update benchmark SSNPP by @cydrain in #503
• Add float test for benchmark SSNPP by @cydrain in #504
• [skip e2e] Update sift rane search benchmark logs by @cydrain in #505
• Remove setting thread omp logs by @cydrain in #506
• Update HNSW range search benchmark by @cydrain in #507
• Add benchmark qps for search and range search by @cydrain in #508
• Remove HNSW range search parameter range_k, use ef instead by @cydrain in #509
• Update SSNPP benchmark by @cydrain in #510
• Update SSNPP benchmark test name by @cydrain in #511
• Add sift range search benchmark with multi radius by @cydrain in #512
• [skip e2e] Update benchmark readme by @cydrain in #515
• Pass CMAKE_PREFIX_PATH to externla project by @jiaoew1991 in #516
• Add ctpl as ThreadPool and use it in HNSW search by @liliu-z in #513
• Release free mem in diskann aligned_free function by @cqy123456 in #520
• Make the Query thread pool global by @liliu-z in #517
• Add thread pool to diskann search by @liliu-z in #522
• Support Feder for DiskANN by @cydrain in #521
• Remove stats in hnsw by @liliu-z in #527
• Fix HNSW/DiskANN feder memory leak by @cydrain in #529
• Refactor knowhere range search to accept 2-side boundaries by @cydrain in #528
• Update release for current branch by @liliu-z in #534
• Use elkan's algo to accelerate KMeans by @hhy3 in #537
• Add prefetch in hnsw by @hhy3 in #546
• Optimize hnsw search by @hhy3 in #552
• Optimize hnsw search algo by @hhy3 in #553
• [skip e2e]Delete Unit Test on windows by @Bennu-Li in #561
• Add visited list pool in hnsw by @hhy3 in #559
• Replace diskann compute function with faiss's compute function by @cqy123456 in #568
• Fix aio io_getevents by @hhy3 in #567
• Add Thread Pool for all indexes in Knowhere by @liliu-z in #570
• Fix thread safety in hnsw by @hhy3 in #575

New Contributors

• @xige-16 made their first contribution in #502
• @Bennu-Li made their first contribution in #561

Full Changelog: v1.3.2...v1.3.4

knowhere-v1.3.3

Release : knowhere-v1.3.3

Fixes

• Print error message when the function abnormal returns by @cqy123456 in #491
• Support metric IP for diskann range search by @cqy123456 in #496
• Fix the memory not get freed after Prepared issue by @cqy123456 in #520

Full Changelog: v1.3.2...v1.3.3

knowhere-v1.3.2

Release : knowhere-v1.3.2

knowhere-v1.3.1

Release : knowhere-v1.3.1

We added some fixes for in this release:

1. Set default value for DiskANN param.
2. Support range search for Feder.
3. Stop building Openblas every time but use the pre-installed lib to save the compilation time.