CaptainAgent PR Part 1: Adding blog post and document

website/blog/2024-11-15-CaptainAgent/index.mdx

@@ -0,0 +1,133 @@
+---
+title: "Introducing CaptainAgent for Adaptive Team Building"
+authors:
+  - LinxinS97
+  - jialeliu
+  - jieyuz2
+tags: [LLM, GPT, AutoBuild]
+---
+![Illustration of how CaptainAgent build a team](img/overall.png)
+
+**TL;DR**
+- We introduce CaptainAgent, an agent equipped with the capability to adaptively assemble a team of agents through retrieval-selection-generation process to handle complex tasks via the [`nested chat`](https://ag2ai.github.io/ag2/docs/tutorial/conversation-patterns#nested-chats) conversation pattern in AG2.
+- CaptainAgent supports all types of `ConversableAgents` implemented in Autogen.
+
+# Introduction
+
+Given an ad-hoc task, dynamically assembling a group of agents capable of effectively solving the problem is a complex challenge. In many cases, we manually design and select the agents involved. In this blog, we introduce **CaptainAgent**, an intelligent agent that can autonomously assemble a team of agents tailored to meet diverse and complex task requirements.
+We design a two-step workflow for CaptainAgent to build a high quality team and solve the problem:
+- (**Step 1**) CaptainAgent will first break down the task, list several roles needed for each subtask, and then create a team of agents accordingly by retrieval, selection, and generation. Each of these will be equipped with predefined tools retrieved from the tool library.
+![Building workflow](img/build.png)
+- (**Step 2**) this team of agents will attempt to solve the subtask via conversation with the free-form tool using. Once it's done, a reflector LLM will provide CaptainAgent with a reflection report for it to decide whether to adjust the team or subtask instruction or to terminate and output the results.
+![Building workflow](img/chat.png)
+
+We also provide an agent library and a tool library for CaptainAgent to choose from when building the team. In the following section, we demonstrate how to use CaptainAgent with or without the provided library.
+
+# Using CaptainAgent without libraries
+CaptainAgent can serve as a drop-in replacement for the general AssistantAgent. To use CaptainAgent, we need to instantiate a CaptainAgent and a UserProxy, then start a chat. To do that we just need to configure to the settings of group chat instantiated by CaptainAgent.
+Without the agent library and tool library, CaptainAgent will automatically generate a set of agents into a group chat.
+
+```python
+from autogen.agentchat.contrib.captain_agent import CaptainAgent
+
+general_llm_config = {
+    "temperature": 0,
+    "config_list": autogen.config_list_from_json("OAI_CONFIG_LIST", filter_dict={"model": ["gpt-4-1106-preview"]}),
+}
+
+nested_mode_config = {
+    "autobuild_init_config": {
+        "config_file_or_env": "OAI_CONFIG_LIST",
+        "builder_model": "gpt-4-1106-preview",
+        "agent_model": "gpt-4-1106-preview",
+    },
+    # this is used to configure the autobuild building process
+    "autobuild_build_config": {
+        "default_llm_config": {"temperature": 1, "top_p": 0.95},
+        "code_execution_config": {"timeout": 300, "work_dir": "groupchat", "last_n_messages": 1},
+        "coding": True,
+    },
+    "group_chat_config": {"max_round": 15},
+    "group_chat_llm_config": general_llm_config.copy(),
+    "max_turns": 3,
+}
+
+## build agents
+captain_agent = CaptainAgent(
+    name="captain_agent",
+    llm_config=general_llm_config,
+    nested_mode_config=nested_mode_config,
+)
+user_proxy = CaptainUserProxyAgent(
+    name="user_proxy",
+    code_execution_config={"use_docker": False},
+)
+query = "Let's play game of 24. Given 4 numbers, you need to use +, -, *, / to get 24. The numbers are 2, 2, 7, 12."
+result = user_proxy.initiate_chat(captain_agent, message=query)
+```
+
+# Using CaptainAgent with libraries
+To use CaptainAgent with libraries, we just need to specify the path to the tool library and agent library. The two libraries are independent, so you can choose to use one of the libraries or both.
+The tool library we provide requires subscribing to specific apis, please refer to the [docs](https://ag2ai.github.io/ag2/docs/topics/captainagent) for details. The following example does not necessarily requires tool usage.
+
+```python
+from autogen.agentchat.contrib.captain_agent import CaptainAgent
+
+general_llm_config = {
+    "temperature": 0,
+    "config_list": autogen.config_list_from_json("OAI_CONFIG_LIST", filter_dict={"model": ["gpt-4-1106-preview"]}),
+}
+
+nested_mode_config = {
+    "autobuild_init_config": {
+        "config_file_or_env": "OAI_CONFIG_LIST",
+        "builder_model": "gpt-4-1106-preview",
+        "agent_model": "gpt-4-1106-preview",
+    },
+    # this is used to configure the autobuild building process
+    "autobuild_build_config": {
+        "default_llm_config": {"temperature": 1, "top_p": 0.95},
+        "code_execution_config": {"timeout": 300, "work_dir": "groupchat", "last_n_messages": 1},
+        "coding": True,
+        "library_path": "captainagent_expert_library.json"
+    },
+    "autobuild_tool_config": {
+        "tool_root": "default",  # this will use the tool library we provide
+        "retriever": "all-mpnet-base-v2",
+    },
+    "group_chat_config": {"max_round": 10},
+    "group_chat_llm_config": general_llm_config.copy(),
+    "max_turns": 3
+}
+
+## build agents
+captain_agent = CaptainAgent(
+    name="captain_agent",
+    llm_config=general_llm_config,
+    nested_mode_config=nested_mode_config,
+)
+user_proxy = CaptainUserProxyAgent(
+    name="user_proxy",
+    code_execution_config={"use_docker": False},
+)
+query = 'Find the stock price of Microsoft in the past 1 year and plot a line chart to show the trend. Save the line chart as "microsoft_stock_price.png".'
+result = user_proxy.initiate_chat(captain_agent, message=query)
+```
+
+# Further Reading
+For a detailed description of how to configure the CaptainAgent, please refer to the [document](https://ag2ai.github.io/ag2/docs/topics/captainagent).
+
+Please refer to our [paper](https://arxiv.org/pdf/2405.19425) for more details about CaptainAgent and the proposed new team-building paradigm: adaptive build.
+
+If you find this blog useful, please consider citing:
+```
+@misc{song2024adaptiveinconversationteambuilding,
+      title={Adaptive In-conversation Team Building for Language Model Agents},
+      author={Linxin Song and Jiale Liu and Jieyu Zhang and Shaokun Zhang and Ao Luo and Shijian Wang and Qingyun Wu and Chi Wang},
+      year={2024},
+      eprint={2405.19425},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2405.19425},
+}
+```

website/blog/authors.yml

@@ -18,9 +18,9 @@ yiranwu:
 
 jialeliu:
   name: Jiale Liu
-  title: Undergraduate student at Xidian University
-  url: https://leoljl.github.io
-  image_url: https://github.com/LeoLjl/leoljl.github.io/blob/main/profile.jpg?raw=true
+  title: PhD student at Pennsylvania State University
+  url: https://github.com/LeoLjl
+  image_url: https://github.com/leoljl.png
 
 thinkall:
   name: Li Jiang

website/docs/topics/captainagent/_category_.json

@@ -0,0 +1,4 @@
+{
+    "label": "Captain Agent",
+    "collapsible": true
+}

website/docs/topics/captainagent/agent_library.mdx

@@ -0,0 +1,39 @@
+# Agent Library
+A simple agent in agent library requires three fields:
+- description: This describes the functionality of the agent.
+- system_message: This provides the system message of the agent for initialization.
+- name: The name of the agent.
+
+An example of the agent library looks like:
+```
+[
+    "description": "The Python_Programming_Expert specializes in using Python's pandas and numpy libraries to manipulate large data sets, particularly focusing on creating and analyzing a new 'STEM' feature from educational datasets, and works collaboratively in a multidisciplinary team.",
+    "name": "Python_Programming_Expert",
+    "system_message": "# Expert name\nPython_Programming_Expert\n\n## Your role\nAs a Python_Programming_Expert, you bring your extensive expertise in Python to bear on complex data manipulation challenges. Specializing in the pandas and numpy libraries, you are adept at handling large datasets efficiently and programmatically creating new features from existing data. Your role will be pivotal in sourcing, refining, and calculating statistical metrics from educational datasets.\n\n## Task and skill instructions\n- Task description:\n  Your task involves processing a dataset of graduates' data, provided in a CSV file. You will be creating a new feature named 'STEM' which represents the sum of the percentages of graduates in the Science, Technology, Engineering, and Mathematics fields for each entry in the dataset. Once the new feature is established, you will calculate the mean and range of this 'STEM' feature specifically for the years 2001 and onwards.\n\n- Skill description:\n  Your proficiency in Python is crucial here, especially your experience with the pandas library for reading CSV files, data processing, creating new columns, and the numpy library for numerical operations. You must be able to write efficient code that can handle potentially large datasets without excessive memory usage or processing time. Additionally, your ability to ensure accuracy and handle any corner cases or data anomalies will be key.\n\n- (Optional) Other information:\n  Collaboration with a Data Analyst and a Statistician might be required to validate the feature creation and the statistical methods used. Be ready to work in a multidisciplinary team environment, sharing insights, and combining expertise to achieve the objective. Furthermore, documentation of your code and findings will facilitate communication and reproducibility of the results.\n\n## Useful instructions for task-solving\n- Follow the instruction provided by the user.\n- Solve the task step by step if you need to.\n- If a plan is not provided, explain your plan first.\n- If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n- When you find an answer, verify the answer carefully. \n- Include verifiable evidence in your response if possible.\n    \n## How to use code?\n- Suggest python code (in a python coding block) or shell script (in a sh coding block) for the Computer_terminal to execute.\n- When using code, you must indicate the script type in the code block.\n- Do not suggest incomplete code which requires users to modify.\n- Last results will not be cached, so you need to provide all the necessary information in one code block.\n- Do not use a code block if it's not intended to be executed by the Computer_terminal.\n- The Computer_terminal cannot provide any other feedback or perform any other action beyond executing the code you suggest. \n- The Computer_terminal can't modify your code.\n- Use 'print' function for the output when relevant. \n- Check the execution result returned by the user.\n- Do not ask users to copy and paste the result.\n- If the result indicates there is an error, fix the error and output the code again. \n- If you want the Computer_terminal to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. "
+]
+```
+
+We provide an example in `notebook/captainagent_expert_library.json`.
+
+## Adding advanced agents
+We also support adding agents with advanced capability to the library, aside from agents with different system message. Just need to add a `model_path` field and any other arguments that needs to pass while initialization. For example, to add a WebSurferAgent:
+
+```
+[
+    {
+        "name": "WebServing_Expert",
+        "description": "A helpful assistant with access to a web browser. Ask them to perform web searches, open pages, navigate to Wikipedia, answer questions from pages, and or generate summaries.",
+        "system_message": "",
+        "agent_path": "autogen/agentchat/contrib/web_surfer/WebSurferAgent",
+        "browser_config": {
+            "viewport_size": 5120,
+            "downloads_folder": "coding",
+            "request_kwargs": {
+                "headers": {
+                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0"
+                }
+            }
+        }
+    }
+]
+```

website/docs/topics/captainagent/tool_library.mdx

@@ -0,0 +1,59 @@
+# Tool Library
+In CaptainAgent, tools are in the form of python functions. The agents can write code to import functions and call them according to their needs. This can significantly enhance the functionality and capabilities of the agents.
+
+We provide a list of tools that comes with the release of CaptainAgent.
+
+## Using the Built-in Tool Library
+### Install dependencies
+First install the requirements for running tools via pip. The requirements file is located in `autogen/agentchat/contrib/captainagent/tools/requirements.txt`.
+
+### Subscribe to Certain APIs
+To use the provided built-in tools, it is required to obtain a Bing Search API key and RapidAPI key.
+For Bing API, you can read more about how to get an API on the [Bing Web Search API](https://www.microsoft.com/en-us/bing/apis/bing-web-search-api) page.
+For RapidAPI, you can [sign up](https://rapidapi.com/auth/sign-up) and subscribe to these two links([link1](https://rapidapi.com/solid-api-solid-api-default/api/youtube-transcript3), [link2](https://rapidapi.com/420vijay47/api/youtube-mp3-downloader2)).
+These two apis have free billing options and there is no need to worry about extra costs for small scale runs.
+
+Whenever you run the tool-related code, remember to export the api keys to system variables.
+```bash
+export BING_API_KEY=""
+export RAPID_API_KEY=""
+```
+or
+```python
+import os
+os.environ["BING_API_KEY"] = ""
+os.environ["RAPID_API_KEY"] = ""
+```
+
+Then you are good to go. Feel free to try out the examples provided in the CaptainAgent notebook.
+
+## What is in the Tool Library
+The tool library consists of three types of tools: math, data_analysis and information_retrieval. Its folder layout is as follows. Each 'py' file implements a tool.
+
+```
+tools
+├── README.md
+├── data_analysis
+│   ├── calculate_correlation.py
+│   └── ...
+├── information_retrieval
+│   ├── arxiv_download.py
+│   ├── arxiv_search.py
+│   └── ...
+├── math
+│   ├── calculate_circle_area_from_diameter.py
+│   └── ...
+└── tool_description.tsv
+```
+
+Tools can be imported from `tools/{category}/{tool_name}.py` with exactly the same tool name.
+`tool_description.tsv` contains descriptions of tools for retrieval.
+
+## How Tool Library works
+When an agent's description is provided, a retriever will retrieve `top_k` tool candidates from the library based on the semantic
+similarity between agent description and tool description. The tool description is same as that in `tool_description.tsv`.
+
+After candidates are retrieved, the agent's system message will be updated with the tool candidates' information and how to call the tools.
+A user proxy with the ability to execute the code will be added to the nested chat. Under the hood, this is achieved by leveraging the
+[User Defined Functions](/docs/topics/code-execution/user-defined-functions) feature. A `LocalCommandLineCodeExecutor` equipped with all the functions serves as
+code executor for the user proxy.