Chromadb: InvalidDimensionException: Embedding dimension 1024 does not match collection dimensionality 384

[divrajput]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

class Chroma_database:
    def __init__(self,CHROMA_DATA_PATH= "patient_chroma",embedding_model_name="dunzhang/stella_en_1.5B_v5"):
        
        model_kwargs = {"device": "cuda"}
        encode_kwargs = {"normalize_embeddings": True}
        self.embedding_function = SentenceTransformerEmbeddings(model_name=embedding_model_name, model_kwargs=model_kwargs, encode_kwargs=encode_kwargs)
        self.CHROMA_DATA_PATH = CHROMA_DATA_PATH
        # EMBED_MODEL = "all-MiniLM-L6-v2"
        self.COLLECTION_NAME = "demo_docs"
        self.persistent_client = chromadb.PersistentClient(path=self.CHROMA_DATA_PATH)
        self.collection = self.persistent_client.get_or_create_collection(name=self.COLLECTION_NAME,metadata={"hnsw:space": "cosine"}
                                                                          )
        
    def add_collection(self,json_file_path= "/home/azureuser/div_rag_exp/exp_1/dataframe/all_data_embedding.json"):
        with open(json_file_path, 'r') as json_file:
            chunk_dict = json.load(json_file)
        print(chunk_dict)
        self.collection.add(
            documents=chunk_dict["documents"],
            # embeddings=chunk_dict["embeddings"],
            metadatas=chunk_dict["metadatas"],
            ids=chunk_dict["ids"]
        )
        return
    def __call__(self):
        langchain_chroma = Chroma(
            client=self.persistent_client,
            collection_name=self.COLLECTION_NAME,
            embedding_function=self.embedding_function  # Use the variable containing the collection name
        )
        print("There are", langchain_chroma._collection.count(), "in the collection")
        return langchain_chroma
# # To add document  
# CHROMA_DATA_PATH= "exp1_patient_chroma"  
# langchain_chroma_obj = Chroma_database(CHROMA_DATA_PATH = CHROMA_DATA_PATH)
# langchain_chroma = langchain_chroma_obj.add_collection()
# langchain_chroma


CHROMA_DATA_PATH= "exp1_patient_chroma"  
langchain_chroma = Chroma_database(CHROMA_DATA_PATH = CHROMA_DATA_PATH)()
query = "What is family Medical history?"
langchain_chroma.similarity_search_with_score(query,k=4)

Description

I am using "dunzhang/stella_en_1.5B_v5" embedding model and also will be using many custom embedding model. So I do not want to add pre defined embedding to my chroma collection and at the similarity search time, I want to embedd and find similarity.

Error :
InvalidDimensionException: Embedding dimension 1024 does not match collection dimensionality 384

Please Provide me solution how I can use multiple embedding models through on chroma db collection.

System Info

langchain==0.2.11
langchain-chroma==0.1.2
langchain-community==0.2.10
langchain-core==0.2.24
langchain-text-splitters==0.2.2

[dosubot[bot]]

To resolve the dimension mismatch error and use multiple embedding models with a single ChromaDB collection, you need to ensure that the embeddings from different models have the same dimensionality. Here is a step-by-step approach to achieve this:

1. Ensure Embedding Dimensionality Consistency: When using multiple embedding models, make sure that all models produce embeddings of the same dimension. If they do not, you will need to transform the embeddings to a common dimensionality. This can be done using techniques like PCA (Principal Component Analysis) or by training a small neural network to map embeddings from one space to another.

2. Transform Embeddings: If your embedding models produce different dimensions, you can use a transformation function to map them to a common space. Here is an example using PCA:

   from sklearn.decomposition import PCA
   import numpy as np
   
   def transform_embeddings(embeddings, target_dim):
       pca = PCA(n_components=target_dim)
       transformed_embeddings = pca.fit_transform(embeddings)
       return transformed_embeddings
   
   # Example usage
   embeddings_model_1 = np.random.rand(10, 1024)  # Embeddings from model 1 (1024 dimensions)
   embeddings_model_2 = np.random.rand(10, 384)   # Embeddings from model 2 (384 dimensions)
   
   target_dim = 256  # Target dimension for all embeddings
   transformed_embeddings_1 = transform_embeddings(embeddings_model_1, target_dim)
   transformed_embeddings_2 = transform_embeddings(embeddings_model_2, target_dim)

3. Add Transformed Embeddings to ChromaDB: Once you have transformed the embeddings to a common dimensionality, you can add them to the ChromaDB collection.

   from langchain_community.vectorstores import Chroma
   from langchain_core.documents import Document
   
   # Assuming `transformed_embeddings_1` and `transformed_embeddings_2` are the transformed embeddings
   vectorstore = Chroma.from_texts(
       collection_name="langchain_store",
       texts=["text1", "text2", "text3", "text4", "text5", "text6"],
       embedding=None,
       metadatas=[{"source": "model_1"}] * 3 + [{"source": "model_2"}] * 3,
       embeddings=transformed_embeddings_1.tolist() + transformed_embeddings_2.tolist()
   )

4. Persisting Data: If you want to save the data to disk, you can initialize the Chroma client and specify the directory for persistence.

   # save to disk
   db2 = Chroma.from_documents(docs, embedding_function, persist_directory="./chroma_db")
   docs = db2.similarity_search(query)
   
   # load from disk
   db3 = Chroma(persist_directory="./chroma_db", embedding_function=embedding_function)
   docs = db3.similarity_search(query)
   print(docs[0].page_content)

5. Using Docker for Chroma: You can also run Chroma in a Docker container and connect to it from your application.

   # create the chroma client
   import uuid
   import chromadb
   from chromadb.config import Settings
   
   client = chromadb.HttpClient(settings=Settings(allow_reset=True))
   client.reset()  # resets the database
   collection = client.create_collection("my_collection")
   for doc in docs:
       collection.add(
           ids=[str(uuid.uuid1())], metadatas=doc.metadata, documents=doc.page_content
       )
   
   # tell LangChain to use our client and collection name
   db4 = Chroma(
       client=client,
       collection_name="my_collection",
       embedding_function=embedding_function,
   )
   query = "What did the president say about Ketanji Brown Jackson"
   docs = db4.similarity_search(query)
   print(docs[0].page_content)

By ensuring that all embeddings have the same dimensionality before adding them to the ChromaDB collection, you can avoid dimension mismatch errors and successfully use multiple embedding models with a single collection. Additionally, the ChromaDB library provides various methods to handle embeddings, metadata, and persistence, ensuring smooth integration and operation ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[arshren]

The default model you are using produces 384-dimensional embeddings, but your collection is configured for 1024 dimensions. Switch to a model that produces 1024-dimensional embeddings and the issue will be resolved.

I solved my issue by using
sentence_transformer_ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="BAAI/bge-large-en-v1.5")
earlier, I was using
sentence_transformer_ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2"), which generates embedding with dimension 384 and is also the default embedding used in Chroma