Is that possible to combine Query_engine or query_engine_tools with Autogen Agents?

[mraguth]

I did see a notebook about Autogen + Llamaindex. However, I use Query_engine most for all my analysis. Is that possible to combine query_engine ouput with Autogen and allow Autogen agent to take my llamaindex output for further analysis? Thoughts or sample code will be greatly appreicated.

https://microsoft.github.io/autogen/docs/notebooks/agentchat_group_chat_with_llamaindex_agents

[dosubot[bot]]

Yes, it is possible to combine Query_engine or query_engine_tools with Autogen Agents. Below is a sample code demonstrating how to achieve this:

from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    load_index_from_storage,
)

from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

# Load or build the indices
try:
    storage_context = StorageContext.from_defaults(persist_dir="./storage/lyft")
    lyft_index = load_index_from_storage(storage_context)

    storage_context = StorageContext.from_defaults(persist_dir="./storage/uber")
    uber_index = load_index_from_storage(storage_context)

    index_loaded = True
except:
    index_loaded = False

if not index_loaded:
    # Load data
    lyft_docs = SimpleDirectoryReader(input_files=["./data/10k/lyft_2021.pdf"]).load_data()
    uber_docs = SimpleDirectoryReader(input_files=["./data/10k/uber_2021.pdf"]).load_data()

    # Build index
    lyft_index = VectorStoreIndex.from_documents(lyft_docs)
    uber_index = VectorStoreIndex.from_documents(uber_docs)

    # Persist index
    lyft_index.storage_context.persist(persist_dir="./storage/lyft")
    uber_index.storage_context.persist(persist_dir="./storage/uber")

# Create query engines
lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
uber_engine = uber_index.as_query_engine(similarity_top_k=3)

# Define query engine tools
query_engine_tools = [
    QueryEngineTool(
        query_engine=lyft_engine,
        metadata=ToolMetadata(
            name="lyft_10k",
            description="Provides information about Lyft financials for year 2021. Use a detailed plain text question as input to the tool.",
        ),
        return_direct=False,
    ),
    QueryEngineTool(
        query_engine=uber_engine,
        metadata=ToolMetadata(
            name="uber_10k",
            description="Provides information about Uber financials for year 2021. Use a detailed plain text question as input to the tool.",
        ),
        return_direct=False,
    ),
]

# Setup ReAct Agent
llm = OpenAI(model="gpt-3.5-turbo-0613")
agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)

# Example query
response = agent.chat("What was Lyft's revenue growth in 2021?")
print(str(response))

This code sets up a ReAct agent that uses query engine tools to answer questions about Lyft and Uber's financials for the year 2021. The return_direct=False parameter ensures that the agent processes the tool's output before returning a response ^[1][2].

Additionally, you can explore the MultiDocumentAgentsPack class from LlamaIndex, which provides a more structured way to handle multiple documents and agents. Here is an example of how you can use it:

from llama_index.core import Document
from llama_index.packs.multi_document_agents.base import MultiDocumentAgentsPack

# Load documents
docs = [
    Document(text="Content of Lyft 2021 financial report", title="Lyft 2021"),
    Document(text="Content of Uber 2021 financial report", title="Uber 2021")
]

doc_titles = ["Lyft 2021", "Uber 2021"]
doc_descriptions = ["Lyft financials for year 2021", "Uber financials for year 2021"]

# Initialize MultiDocumentAgentsPack
multi_doc_agents_pack = MultiDocumentAgentsPack(docs, doc_titles, doc_descriptions)

# Example query
response = multi_doc_agents_pack.run("What was Lyft's revenue growth in 2021?")
print(str(response))

This approach leverages the MultiDocumentAgentsPack to set up a multi-document agents architecture, making it easier to manage and query multiple documents ^[3][4].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To resolve the error "TypeError: Can't instantiate abstract class TripSpecialistAgent with abstract methods achat, chat, chat_history, reset," you need to ensure that all abstract methods from the BaseChatEngine class are implemented in the TripSpecialistAgent class.

Here is an example of how you can define the TripSpecialistAgent class by implementing all the required abstract methods:

from llama_index.legacy.chat_engine.types import BaseChatEngine, AGENT_CHAT_RESPONSE_TYPE, StreamingAgentChatResponse
from llama_index.legacy.core.llms.types import ChatMessage
from typing import List, Optional

class TripSpecialistAgent(BaseChatEngine):
    def __init__(self, llm, system_message, description):
        self.llm = llm
        self.system_message = system_message
        self.description = description
        self._chat_history = []

    def reset(self) -> None:
        """Reset conversation state."""
        self._chat_history = []

    def chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> AGENT_CHAT_RESPONSE_TYPE:
        """Main chat interface."""
        # Implement your chat logic here
        response = self.llm.chat(
            system_message=self.system_message,
            user_message=message
        )
        return AGENT_CHAT_RESPONSE_TYPE(response=response)

    def stream_chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> StreamingAgentChatResponse:
        """Stream chat interface."""
        # Implement your streaming chat logic here
        response = StreamingAgentChatResponse(response="This is a streaming response from TripSpecialistAgent.")
        return response

    async def achat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> AGENT_CHAT_RESPONSE_TYPE:
        """Async version of main chat interface."""
        # Implement your async chat logic here
        response = await self.llm.achat(
            system_message=self.system_message,
            user_message=message
        )
        return AGENT_CHAT_RESPONSE_TYPE(response=response)

    async def astream_chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> StreamingAgentChatResponse:
        """Async version of main chat interface."""
        # Implement your async streaming chat logic here
        response = StreamingAgentChatResponse(response="This is an async streaming response from TripSpecialistAgent.")
        return response

    @property
    def chat_history(self) -> List[ChatMessage]:
        return self._chat_history

    @property
    def metadata(self):
        return {
            "name": "trip_specialist",
            "description": self.description
        }

This implementation ensures that all abstract methods from the BaseChatEngine class are provided, thus resolving the instantiation error ^[1][2].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu... no luck. Waste of time... more errors than bfore:
ModuleNotFoundError: No module named 'llama_index.legacy'
#from llama_index.legacy.core.chat_engine.types import BaseChatEngine, AGENT_CHAT_RESPONSE_TYPE, StreamingAgentChatResponse

[dosubot[bot]]

To resolve the ModuleNotFoundError: No module named 'llama_index.legacy' error, you should use the correct import paths for the BaseChatEngine and related types from the current version of the LlamaIndex framework. Here is the corrected code:

from typing import Any, List, Dict, Optional, Tuple
from llama_index.core.tools import QueryEngineTool, BaseTool
from llama_index.core.agent.custom.simple import CustomSimpleAgentWorker
from llama_index.core.chat_engine.types import BaseChatEngine, AGENT_CHAT_RESPONSE_TYPE, StreamingAgentChatResponse
from llama_index.core.llms.types import ChatMessage
from llama_index.llms.openai import OpenAI

class SummarizeTool(BaseTool):
    def __init__(self, llm):
        self.llm = llm

    def summarize(self, text: str) -> str:
        # Implement the summarization logic using the LLM
        summary = self.llm.summarize(text)
        return summary

    @property
    def description(self):
        return "A tool to summarize text responses."

class TripSpecialistAgent(BaseChatEngine):
    def __init__(self, llm, system_message, description):
        self.llm = llm
        self.system_message = system_message
        self.description = description
        self._chat_history = []

    def reset(self) -> None:
        """Reset conversation state."""
        self._chat_history = []

    def chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> AGENT_CHAT_RESPONSE_TYPE:
        """Main chat interface."""
        response = self.llm.chat(
            system_message=self.system_message,
            user_message=message
        )
        return AGENT_CHAT_RESPONSE_TYPE(response=response)

    def stream_chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> StreamingAgentChatResponse:
        """Stream chat interface."""
        response = StreamingAgentChatResponse(response="This is a streaming response from TripSpecialistAgent.")
        return response

    async def achat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> AGENT_CHAT_RESPONSE_TYPE:
        """Async version of main chat interface."""
        response = await self.llm.achat(
            system_message=self.system_message,
            user_message=message
        )
        return AGENT_CHAT_RESPONSE_TYPE(response=response)

    async def astream_chat(
        self, message: str, chat_history: Optional[List[ChatMessage]] = None
    ) -> StreamingAgentChatResponse:
        """Async version of main chat interface."""
        response = StreamingAgentChatResponse(response="This is an async streaming response from TripSpecialistAgent.")
        return response

    @property
    def chat_history(self) -> List[ChatMessage]:
        return self._chat_history

    @property
    def metadata(self):
        return {
            "name": "trip_specialist",
            "description": self.description
        }

Make sure you have the correct imports and that the llama_index package is installed and up to date. This should resolve the ModuleNotFoundError and ensure that the TripSpecialistAgent class is correctly implemented with all required methods ^[1][2].

If you continue to face issues, @logan-markewich, it seems like this conversation has hit a few roadblocks and could really use your expertise.

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu.. after changing the imports, it is working...however defining a agent seems like very long process based on your 'tripspecialistagent' :-(

from llama_index.core.base.llms.types import (
    ChatMessage,
    ChatResponseAsyncGen,
    ChatResponseGen,
)

[dosubot[bot]]

Yes, defining an agent can seem lengthy, but LlamaIndex provides more streamlined ways to define agents. You can use the ReActAgent and AgentRunner classes to simplify the process. Here is an example of how you can define an agent more concisely:

from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI
from llama_index.core.agent import ReActAgent

# Define a sample Tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers and return the result."""
    return a * b

multiply_tool = FunctionTool.from_defaults(fn=multiply)

# Initialize LLM
llm = OpenAI(model="gpt-3.5-turbo-0613")

# Initialize ReAct agent
agent = ReActAgent.from_tools([multiply_tool], llm=llm, verbose=True)

This approach abstracts away the need to manually define tools and their metadata. You can also use the AgentRunner class to automatically pick the best agent depending on the LLM:

from llama_index.core.agent import AgentRunner

agent = AgentRunner.from_llm([multiply_tool], llm=llm, verbose=True)

Additionally, you can define more complex agents with planning capabilities or use other agents as tools, providing a flexible and powerful way to manage your LLM applications ^[1].

To continue talking to Dosu, mention @dosu.

[gkzhb]

I think you can use LLamaIndexConversableAgent llama_index_agent with llamaindex chat engine.
The LLamaIndexConversableAgent relies on llama_index_agent's chat and achat methods which are defined in BaseChatEngine class in llamaindex.

So you can try to replace query engine with chat engine. Their differences are mentioned here.

If you don't want to have chat history side effects or the lib of query engine doesn't provide chat engine implementation, you can create a new class eg. ChatEngineFromQueryEngine to implement chat and achat methods with query engine.

A simple way to do this is to create a chat engine with the query engine. Every time calling ChatEngineFromQueryEngine's chat and achat methods, pass in empty chat_history to the chat engine.
Refer to llamaindex BaseIndex.as_chat_engine() for how to create a chat engine with the query engine.