Getting OutOfMemoryError: CUDA out of memory. I have limited GPU resources I am currently testing graph based RAG solution on google colab tesla T4 GPU.

[KaifAhmad1]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

chain = Neo4jVectorChain(graph=graph, embeddings=embeddings, verbose=True)

graph_result = chain.run("How can we enhance the specificity and efficiency of CRISPR/Cas9 gene-editing technology to minimize off-target effects and increase its potential for therapeutic applications?")

chain = GraphCypherQAChain.from_llm(
    cypher_llm=llm,
    qa_llm=llm,
    graph=graph,
    verbose=True,
    return_intermediate_steps=True,
    validate_cypher=True
)

graph_result = chain.run("How can we enhance the specificity and efficiency of CRISPR/Cas9 gene-editing technology to minimize off-target effects and increase its potential for therapeutic applications?")

Description

/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py:392: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.3` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.


> Entering new GraphCypherQAChain chain...
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
[<ipython-input-40-4ff3ab735a16>](https://localhost:8080/#) in <cell line: 1>()
----> 1 graph_result = chain.run("How can we enhance the specificity and efficiency of CRISPR/Cas9 gene-editing technology to minimize off-target effects and increase its potential for therapeutic applications?")

44 frames
[~/.cache/huggingface/modules/transformers_modules/Deci/DeciLM-7B/c3aef1e1ef8bb412da34c8dd500db2d9cd6f5ec9/transformers_v4_35_2__modeling_llama.py](https://localhost:8080/#) in forward(self, hidden_states)
    102     def forward(self, hidden_states):
    103         input_dtype = hidden_states.dtype
--> 104         hidden_states = hidden_states.to(torch.float32)
    105         variance = hidden_states.pow(2).mean(-1, keepdim=True)
    106         hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)

OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 14.75 GiB of which 5.06 MiB is free. Process 3356 has 14.74 GiB memory in use. Of the allocated memory 14.61 GiB is allocated by PyTorch, and 2.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

System Info

cuda = 12.1
pip = 24.0
python = 3.10.10

[dosubot[bot]]

🤖

Hey @KaifAhmad1, great to see you diving into another challenging adventure with LangChain! How's everything going on your end?

Based on the error message you provided, it seems that the GPU memory is being exhausted during the execution of your code. This is a common issue when working with large models or large amounts of data on GPUs with limited memory.

From the context provided, there are a couple of similar issues that have been resolved in the LangChain repository:

1. Issue #16978 suggests several solutions to this problem, including reducing the batch size, using gradient accumulation, using a smaller model, freeing up GPU memory, and using a GPU with more memory.

2. Issue #5513 suggests clearing the PyTorch cache using the following code snippet:

import torch
torch.cuda.empty_cache()

You can try these solutions to see if they help resolve your issue. However, please note that the flan-t5-xxl model may inherently require more memory than is available, and that LangChain prompts may be biased towards OpenAI models, which could necessitate editing the prompts or creating a custom pipeline for open-source models.

If these solutions do not resolve your issue, it would be helpful to have more information about the specific code you're running and the exact error message. This will allow us to provide a more specific solution.

Sources

• OutOfMemoryError: CUDA out of memory. Tried to allocate 104.22 GiB. GPU 0 has a total capacty of 14.75 GiB of which 8.83 GiB is free. Process 252083 has 5.91 GiB memory in use. Of the allocated memory 5.63 GiB is allocated by PyTorch, and 156.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
• Flan-t5-xxl doesnot work with openapi_agent
• libs/langchain/langchain/chains/graph_qa/hugegraph.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[beyondguo]

How to set max sequence length for embedding models? If I have a very long document in my docs, it's easily to trigger OOM error.