Merge pull request #18 from microsoft/experimental

Upgrade to version 0.1.3

docs/_toc.yml

@@ -12,6 +12,11 @@ parts:
     - file: quickstart/quick_start_2
     - file: quickstart/virtualhome
 
+  - caption: FAQ
+    numbered: false
+    chapters:
+      - file: faq/faq
+
   - caption: 📚Tutorials
     chapters:
     - file: tutorials/basic_tutorial

docs/faq/faq.md

@@ -0,0 +1,47 @@
+# FAQ
+
+### Difference to Libraries like TextGrad
+
+TextGrad is both a library and an optimizer algorithm. Currently, we support three optimizers:
+
+- OPRO: [Large Language Models as Optimizers](https://arxiv.org/abs/2309.03409)
+- TextGrad: [TextGrad: Automatic "Differentiation" via Text](https://arxiv.org/abs/2406.07496)
+- OptoPrime: [Our proposed algorithm](https://arxiv.org/abs/2406.16218) -- using the entire computational graph to perform parameter update. It is 2-3x
+  faster than TextGrad.
+
+Using our framework, you can seamlessly switch between different optimizers:
+
+```python
+optimizer1 = OptoPrime(strange_sort_list.parameters())
+optimizer2 = OPRO(strange_sort_list.parameters())
+optimizer3 = TextGrad(strange_sort_list.parameters())
+```
+
+Here is a summary of the optimizers:
+
+|                                   | Computation Graph | Code as Functions | Library Support | Supported Optimizers      | Speed       | Large Graph |
+|-----------------------------------|-------------------|-------------------|------------------|---------------------------|-------------|-------------|
+| OPRO                              | ❌                 | ❌                 | ❌            | OPRO                      | ⚡️          | ✅      |
+| TextGrad                          | ✅                 | ❌                 | ✅            | TextGrad                  | 🐌          | ✅      |
+| Trace  | ✅                 | ✅                 | ✅            | OPRO, OptoPrime, TextGrad | ⚡  | ✅      |
+
+The table evaluates the frameworks in the following aspects:
+
+- Computation Graph: Whether the optimizer leverages the computation graph of the workflow.
+- Code as Functions: Whether the framework allows users to write actual executable Python functions and not require
+  users to wrap them in strings.
+- Library Support: Whether the framework has a library to support the optimizer.
+- Speed: TextGrad is about 2-3x slower than OptoPrime (Trace). OPRO has no concept of computational graph, therefore is very fast.
+- Large Graph: OptoPrime (Trace) represents the entire computation graph in context, therefore, might have issue with graphs that have more than hundreds of operations. TextGrad does not have the context-length issue, however, might be very slow on large graphs.
+
+We provide a comparison to validate our implementation of TextGrad in Trace:
+
+<p align="center">
+    <img src="https://github.com/microsoft/Trace/blob/main/docs/images/compare_to_textgrad3.png" alt="drawing" width="100%"/>
+</p>
+
+To produce this table, we ran the TextGrad pip-installed repo on 2024-10-30, and we also include the numbers reported in the TextGrad paper.
+The LLM APIs are called around the same time to ensure a fair comparison. TextGrad paper's result was reported in 2024-06.
+
+### Difference to Libraries like AutoGen, AG2, OpenAI Swarm, Llama Stack
+

opto/optimizers/optoprime.py

@@ -1,18 +1,18 @@
+
 from typing import Any, List, Dict, Union, Tuple
 from dataclasses import dataclass, asdict
-from opto.trace.nodes import ParameterNode, Node, MessageNode
-from opto.optimizers.optimizer import Optimizer
-
-from opto.trace.propagators import TraceGraph, GraphPropagator
 from textwrap import dedent, indent
-from opto.trace.propagators.propagators import Propagator
-from opto.optimizers.buffers import FIFOBuffer
 import autogen
 import warnings
 import json
-
 import re
 import copy
+from opto.trace.nodes import ParameterNode, Node, MessageNode
+from opto.trace.propagators import TraceGraph, GraphPropagator
+from opto.trace.propagators.propagators import Propagator
+from opto.optimizers.optimizer import Optimizer
+from opto.optimizers.buffers import FIFOBuffer
+from opto.utils.llm import AutoGenLLM
 
 
 def get_fun_name(node: MessageNode):
@@ -252,7 +252,7 @@ class OptoPrime(Optimizer):
     def __init__(
         self,
         parameters: List[ParameterNode],
-        config_list: List = None, # autogen config_dict
+        LLM: AutoGenLLM = None,
         *args,
         propagator: Propagator = None,
         objective: Union[None, str] = None,
@@ -267,11 +267,7 @@ def __init__(
     ):
         super().__init__(parameters, *args, propagator=propagator, **kwargs)
         self.ignore_extraction_error = ignore_extraction_error
-        if config_list is None:
-            config_list = autogen.config_list_from_json("OAI_CONFIG_LIST")
-        if filter_dict is not None:
-            config_list = autogen.filter_config_list(config_list, filter_dict)
-        self.llm = autogen.OpenAIWrapper(config_list=config_list)
+        self.llm = LLM or AutoGenLLM()
         self.objective = objective or self.default_objective
         self.example_problem = ProblemInstance.problem_template.format(
             instruction=self.default_objective,
@@ -510,13 +506,13 @@ def call_llm(
         messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}]
 
         try:  # Try tp force it to be a json object
-            response = self.llm.create(
+            response = self.llm(
                 messages=messages,
                 response_format={"type": "json_object"},
                 max_tokens=max_tokens,
             )
         except Exception:
-            response = self.llm.create(messages=messages, max_tokens=max_tokens)
+            response = self.llm(messages=messages, max_tokens=max_tokens)
         response = response.choices[0].message.content
 
         if verbose:

opto/optimizers/utils.py

@@ -0,0 +1,13 @@
+def print_color(message, color=None, logger=None):
+    colors = {
+        'red': '\033[91m',
+        'green': '\033[92m',
+        'yellow': '\033[93m',
+        'blue': '\033[94m',
+        'magenta': '\033[95m',
+        'cyan': '\033[96m'
+    }
+    print(f"{colors.get(color, '')}{message}\033[0m")  # Default to no color if invalid color is provided
+
+    if logger is not None:
+        logger.log(message)