Help with Python Imports

[frederickbrown]

With some AI assistance I am writing a LangChain app to search some chat logs for content related to a search term. From the error messages when first trying to get this script to run, I updated some of the imports to the LangChain-Community versions. Now the script seems to work although I am still getting some error messages about deprecated usage. Attached is the script and the output. search.txt is the script and out.txt is the output. Any help would be appreciated.

fred b
search.txt
out.txt

[dosubot[bot]]

Hey @frederickbrown! I'm here to help you with your Python import issues. Let's take a look at your script and output files.

I found some relevant information regarding deprecated imports and functions in the LangChain framework that might help you resolve the warnings you're encountering.

1. Deprecated Imports: Ensure that you are using the correct imports from the langchain_community package instead of the deprecated ones from langchain. For example, if you are using UnstructuredMarkdownLoader, it should be imported as follows:

   from langchain_community.document_loaders import UnstructuredMarkdownLoader

2. Deprecated Functions and Classes: Some functions and classes now require explicit arguments or have been replaced. For instance:

   • Functions like VectorStoreToolkit and FlareChain now require an explicit LLM to be passed as an argument.
   • The LLMChain is deprecated in favor of RunnableSequence.
   • The RetrievalQA is deprecated in favor of create_retrieval_chain.

3. Method Changes: Methods such as BaseLanguageModel.predict and Chain.__call__ are deprecated in favor of invoke and ainvoke.

4. Suppressing Warnings: If you need to suppress deprecation warnings temporarily, you can use the suppress_langchain_deprecation_warning context manager:

   from langchain_core._api.deprecation import suppress_langchain_deprecation_warning
   
   with suppress_langchain_deprecation_warning():
       # Code that triggers deprecation warnings

For a detailed list of deprecated imports and their replacements, you can refer to the LangChain documentation on deprecations ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To update your script and resolve the deprecation warnings, you need to replace the deprecated RetrievalQA and RetrievalQAWithSourcesChain classes with the create_retrieval_chain function. Here's how you can modify your script:

import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings, OpenAI
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_community.document_loaders import TextLoader

# Load environment variables from .env file
load_dotenv()

# Get the OpenAI API key
openai_api_key = os.getenv("OPENAI_API_KEY")

# Ensure the API key is available
if not openai_api_key:
    raise ValueError("OpenAI API key not found in .env file")

# Directory containing your chat log files
log_directory = "./test"

# Process each file
documents = []
for filename in os.listdir(log_directory):
    if filename.endswith(".txt"):
        file_path = os.path.join(log_directory, filename)
        loader = TextLoader(file_path)
        documents.extend(loader.load())

# Split the documents into chunks
text_splitter = CharacterTextSplitter(chunk_size=1048, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings and store in Chroma
embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(texts, embeddings)

# Initialize the language model
llm = OpenAI()

# Define the system prompt
system_prompt = (
    "Use the given context to answer the question. "
    "If you don't know the answer, say you don't know. "
    "Use three sentence maximum and keep the answer concise. "
    "Context: {context}"
)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)

# Create the question-answer chain
question_answer_chain = create_stuff_documents_chain(llm, prompt)

# Create the retrieval chain
chain = create_retrieval_chain(docsearch.as_retriever(), question_answer_chain)

# Query your conversations
query = "What did we discuss about Boolean functions?"
result = chain.invoke({"input": query})
print(result)

This script replaces the deprecated classes with the recommended create_retrieval_chain function and uses invoke instead of run to execute the chain ^[1][2].

To continue talking to Dosu, mention @dosu.

[frederickbrown]

@dosu

That runs beautifully. Thank you. Can you now point me to a reference that could explain how to specify exactly what is returned from the search query. You version return an over abundance of information?

[dosubot[bot]]

Could you provide more details on what specific information you would like to retrieve from the search query? Are you looking to limit the number of sentences, filter by certain keywords, or perhaps format the output in a specific way? This will help me guide you to the right resources or suggest modifications to your script.

To continue talking to Dosu, mention @dosu.

[frederickbrown]

@dosu

Well, I'd be happy if the search just return the names of the files that the search terms were found in. For the moment I can do without the quoted text.

How would I modify the script to do that, just return the the file names of the files containing content related to the search term?

[dosubot[bot]]

To modify the script to return only the names of the files containing content related to the search term, you can adjust the retrieval process to focus on identifying the source documents. Here's how you can do it:

import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.document_loaders import TextLoader

# Load environment variables from .env file
load_dotenv()

# Get the OpenAI API key
openai_api_key = os.getenv("OPENAI_API_KEY")

# Ensure the API key is available
if not openai_api_key:
    raise ValueError("OpenAI API key not found in .env file")

# Directory containing your chat log files
log_directory = "./test"

# Process each file and keep track of file names
documents = []
file_names = []
for filename in os.listdir(log_directory):
    if filename.endswith(".txt"):
        file_path = os.path.join(log_directory, filename)
        loader = TextLoader(file_path)
        documents.extend(loader.load())
        file_names.append(filename)

# Split the documents into chunks
text_splitter = CharacterTextSplitter(chunk_size=1048, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings and store in Chroma
embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(texts, embeddings)

# Query your conversations
query = "What did we discuss about Boolean functions?"
retriever = docsearch.as_retriever()
results = retriever.retrieve(query)

# Extract and print the file names of the documents containing the search term
found_files = set()
for result in results:
    # Assuming each result has a 'source' attribute that contains the file path
    source_path = result.metadata.get('source', '')
    file_name = os.path.basename(source_path)
    found_files.add(file_name)

print("Files containing the search term:")
for file in found_files:
    print(file)

This script assumes that each result from the retriever has metadata that includes the source file path. Adjust the script as needed based on the actual structure of your data and the metadata available.

To continue talking to Dosu, mention @dosu.