Has anyone used this package for Vector Simularity Search via RediSearch?

[jonny7737]

I have constructed commands like 'FT.INFO', 'FT.CREATE' and 'FT.ADD' using the 'send_object()' method. These all work correctly as far as I can tell. The right stuff gets added to the right index.

The problem comes when I attempt a search: I have not been able to construct a working 'FT.SEARCH' command. They all fail with a redis error.

This code correctly creates the index:

        await redis.send_object([
          'FT.CREATE',
          'conversations',
          'SCHEMA',
          'chatId',
          'TEXT',
          'WEIGHT',
          '1.0',
          'role',
          'TEXT',
          'request',
          'TEXT',
          'WEIGHT',
          '1.0',
          'embedding',
          'NUMERIC',
          'SORTABLE'
        ]);

This code correctly adds docs to the index:

    result = await redis.send_object([
      'FT.ADD',
      'conversations',
      const Uuid().v4(),
      '1.0',
      'FIELDS',
      'chatId',
      chatId,
      'role',
      role,
      'request',
      request,
      'embedding',
      embedding
    ]);

Using RedisInsight-v2, I can see that the 'FT.ADD' added the correct data.

But this code always fails with a redis error:

    final searchResults = await redis.send_object([
      'FT.SEARCH',
      'conversations',
      embedding,
      'LIMIT',
      '0',
      resultsCount.toString(),
      'SORTBY',
      '1',
      'SCORE',
      'ASC'
    ]);

I have tried many variations on this command and they all fail with the same error.
RedisError(Unknown argument SCORE at position 6 for

)

Any help is greatly appreciated.

[ra1u]

Nice to hear your feedback on how you use this library.

I am not sure that number of parameters for SORTBY is correct.
From https://redis.io/commands/ft.search/
Can you check that you have correct number of elements and that the attribute that you want to sort over exists.

From your post is clear what is not working, but I do not understand what you want to achieve. Can you explain what you want to do and why do you believe that current program should work.

Kind regards, Luka.

[jonny7737]

The project is an extended learning exercise. Exploring ways to extend the usefulness of OpenAI APIs for chat conversations that contain a large amount of information. Redis (and RediSearch) support Vector Similarity search which will allow the search to determine the relevant information to pass to the OpenAI API. The process is: Receive a user request Retrieve an embedding for the request - embedding is an array of doubles Store the embedding and other meta-data in redis Request a response from OpenAI for the request Retrieve an embedding for the response Store the embedding and other meta-data in redis Return to step 1 At some point the request sent to OpenAI in step 4 will approach the limit of the API - token limit. At this point, between step 2 and step 3, I need to perform a VSS (Vector Similarity Search) via redis and redisearch to identify the relevant parts of the conversation and use those parts to construct the next request.

Can you check that you have correct number of elements and that the attribute that you want to sort over exists.

Using RedisInsight-v2: I believe the data is there. I do not understand your question regarding the correct number of elements. The data in the field labeled ‘embedding’ in the image below, is ~20 KB. Does this cause any concern for the redis package?  --EDIT - sorry for the image reference. forgot it will not show here. Thank you for looking into my questions.

…

On May 27, 2023, at 2:23 PM, Luka Rahne ***@***.***> wrote: Nice to hear your feedback on how you use this library. I am not sure that number of parameters for SORTBY is correct. From https://redis.io/commands/ft.search/ Can you check that you have correct number of elements and that the attribute that you want to sort over exists. From your post is clear what is not working, but I do not understand what you want to achieve. Can you explain what you want to do and why do you believe that current program should work. Kind regards, Luka. — Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6S42VX2CFGISFDOTRJDV3XIJIC7ANCNFSM6AAAAAAYRKXDPE>. You are receiving this because you authored the thread.

[jonny7737]

After looking into more current documentation for FT.SEARCH, i have removed the '1' following 'SORTBY'. The previous error has been replaced with a very cryptic:
RedisError(Syntax error at offset 0 near )

[ra1u]

Error you have received is from Redis server not from this library. RedisError is type from redis-dart when redis server returns valid error.
https://redis.io/docs/reference/protocol-spec/#resp-errors

We have difficulty helping you because you still did not explain what is that you want to do. Particularly with command FT.SEARCH you claim it fails, but it is unclear why your command is constructed this way and why it has so many arguments. It is unclear what values some of arguments have.

My advice is that you start with cli utility and figure out from there, what should be the appropriate command to execute for your usecase . Try to find expert advice related to particular commands in redis forum. My redis skills does not go much beyond understanding redis protocol.

[jonny7737]

thank you.