[bug]: Broken after upgrade - get_context_docs - TypeError: float() argument must be a string or a real number, not 'list'

[ga-it]

Describe the bug
a Context Chat query against the existing vector database after upgrade produces a 500 error

To Reproduce
Steps to reproduce the behavior:

1. Perform upgrade to 3.1.0 from 3.0.1
2. Perform context chat query from Nextcloud assistant
3. Receive 500 error based on failed embed query

Context Chat Backend logs (if applicable, from the docker container)

```

llama_print_timings: load time = 483.19 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 305.76 ms / 13 tokens ( 23.52 ms per token, 42.52 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 470.34 ms / 14 tokens
TRACE: X:52532 - ASGI [4] Send {'type': 'http.response.start', 'status': 500, 'headers': '<...>'}
INFO: X:52532 - "POST /query HTTP/1.1" 500 Internal Server Error
TRACE: X:52532 - ASGI [4] Send {'type': 'http.response.body', 'body': '<4577 bytes>'}
TRACE: X:52532 - ASGI [4] Raised exception
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/message_logger.py", line 84, in call
raise exc from None
File "/usr/local/lib/python3.11/dist-packages/uvicorn/middleware/message_logger.py", line 80, in call
await self.app(scope, inner_receive, inner_send)
File "/usr/local/lib/python3.11/dist-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/app/context_chat_backend/ocs_utils.py", line 75, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 214, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/starlette/concurrency.py", line 39, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 943, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/context_chat_backend/controller.py", line 123, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/app/context_chat_backend/controller.py", line 361, in _
return execute_query(query)
^^^^^^^^^^^^^^^^^^^^
File "/app/context_chat_backend/controller.py", line 330, in execute_query
return process_context_query(
^^^^^^^^^^^^^^^^^^^^^^
File "/app/context_chat_backend/chain/one_shot.py", line 61, in process_context_query
context_docs = get_context_docs(user_id, query, vectordb, ctx_limit, scope_type, scope_list)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/context_chat_backend/chain/context.py", line 29, in get_context_docs
return user_client.similarity_search(query, k=ctx_limit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/langchain_community/vectorstores/chroma.py", line 350, in similarity_search
docs_and_scores = self.similarity_search_with_score(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/langchain_community/vectorstores/chroma.py", line 439, in similarity_search_with_score
query_embedding = self._embedding_function.embed_query(query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/langchain_community/embeddings/llamacpp.py", line 130, in embed_query
return list(map(float, embedding))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: float() argument must be a string or a real number, not 'list'
TRACE: X:52532 - HTTP connection lost

</details>


**Setup Details (please complete the following information):**
 - Nextcloud Version: 30.0.1 - 30.0.1.2
 - AppAPI Version: 4.0.0
 - Context Chat PHP Version 3.1.0
 - Context Chat Backend Version 3.1.0
 - Nextcloud deployment method: Docker
 - Context Chat Backend deployment method: manual, remote

[kyteinsky]

Hi again, here too I'd say to use the latest release. The embedding backend was changed to a different model that is multilingual as opposed to the instructor model we had, which was english-only. We also plan to use postgresql instead of chromadb and migrate to a different schema soon. These will be automatically done when a later release is used else the necessary changes would have to be done manually.

In this case, it seems the embedder has been corrected to use llama so you'd have to get rid of the older embeddings since two different embeddings don't play together. I'm not 100% sure this is the case until I see the config.yaml and some more logs from the same request.

[ga-it]

Oh wow. This is a nightmare.

Our Chroma embeddings database is 314gb:

314G Oct 28 03:45 chroma.sqlite3

As per my discussion with @marcelklehr in #49 duplicated embeddings on group folders result in massive cost.

And for that reason I made the following point "If possible, when this is implemented provide a migration utilising current embeddings to a shared embeddings model to prevent yet another crawl!"

On group folders containing TBs of documents and with many users, the crawl time projections are months with associated GPU and energy requirements.

Fulltextsearch has a similar issue - nextcloud/fulltextsearch#878

It seems critical that the architecture for both incorporate some sort of abstraction layer between the data and the users - such as some sort of role based security which users would then access embeddings and indexing through. This would also then have the benefit that on changes to permissions, recrawling would not need to take place as user access would be immediately turned off through access to roles in the abstraction layer. Take that advice as from the layman from where it comes, but the immediate prospect is we lose months of embedding time for this change and any future changes to the embedding model which has a cost and functionality hit.

Fundamentally, even with an abstraction layer, changes to the embedder in future are likely to result in the same issue. But with the duplicated embedding process, the cost will be multiplied by the number of users with access to the group folders.

[marcelklehr]

The embedding backend was changed to a different model that is multilingual as opposed to the instructor model we had, which was english-only.

@kyteinsky We did not release this yet, though, right? So why would @ga-it run into errors due to this?

[marcelklehr]

duplicated embeddings on group folders result in massive cost.

We are aware of this and have a plan to mitigate this.

[kyteinsky]

@kyteinsky We did not release this yet, though, right? So why would @ga-it run into errors due to this?

@ga-it is using the master branch code, we talked over email, he promised to use latest releases.

let's continue in #49 since the discussion is unrelated to this issue. Closing this since it is not from a released version.