From 1152766ce3f08236d1c0e8c80c8f6ae5f0369d48 Mon Sep 17 00:00:00 2001 From: chinganc Date: Mon, 24 Jun 2024 11:54:57 -0700 Subject: [PATCH 1/2] Update --- docs/index.html | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/docs/index.html b/docs/index.html index 017cc76..09832c0 100644 --- a/docs/index.html +++ b/docs/index.html @@ -324,40 +324,44 @@

Trace Graph

-

A workflow can have many components. Trace creates a unified representation of all the components through a user-defined graph. - This graph is created by using two Trace primitives (node and bundle) to decorate the workflow. - These primitives represent heterogenous node types in the graph and operations on. Usages of these objects are automatically traced in the graph. +

A workflow can have many components. Trace creates a unified representation of all the components through a user-defined computational graph, called the Trace graph. + This directed acyclic graph is created by using two Trace primitives (node and bundle) to decorate the workflow, which represent node and operations in the graph. Usages of these objects are automatically traced and added to the Trace graph.

Trace primitive - can be used to wrap over Python objects. The example below shows how different types of Python objects can be included in the Trace graph. + can be used to wrap over Python objects as nodes in the graph. The example below shows how different types of Python objects can be included in the Trace graph. Nodes can be marked as trainable, which allows the optimizer to change the content of the node. 

                                         
     from trace import node
     w = node(3)
     x = node({"learning_rate": 1e-3})
-    y = node("You are a helpful assistant.")
+    y = node("You are a helpful assistant.", trainable=True)
     z = node([2, 5, 3])

-

Similarly, allows us to represent Python functions as a node. We can describe what the function is doing, or allowing the optimizer to change the content of this function.

+

Similarly, allows us to represent Python functions as an operator in the graph. We can describe what the function is doing, and optionally let the optimizer to change the content of this function by setting the trainable flag.

                                     
     import math
     from trace import bundle
 
+    @bundle()
+    def cbrt(x):
+        """ Return the cube root of x. """
+        return math.cbrt(x)
+
     @bundle(trainable=True)
     def retrieve_doc(x):
         metric = 'cos_sim'
         return http.api_call(x, metric)

-

Taking the perspective of primitives and making the graph construction completely automatic offers immense flexibility. +

The perspective of primitives makes the graph construction completely automatic, which offers immense flexibility to users. Consider a typical prompt-based LLM task. A program needs to first query the LLM to get a response, and then some post-processing method must be written to extract and verify the LLM response. - The design space of this task is how to construct the best query (prompt) and how to extract the answer from the response.

-

However, these two tasks are tightly coupled. The complexity of the extraction code determines how simple the prompt can be designed and vice versa. + +

The design space of this task is how to construct the best query (prompt) and how to extract the answer from the response. However, these two tasks are tightly coupled. The complexity of the extraction code determines how simple the prompt can be designed and vice versa. Moreover, LLM behaviors are stochastic. When subtle shifts happen in LLM's response, new post-processing code must be written to account for the change.

@@ -425,23 +429,23 @@

Trace Graph

-

Trace implicitly constructs a computation graph through Python operators and functions. The graph - represents the actual execution of the user-defined workflow (not the same as the original program). - The original program can have complicated logic, but the actual execution graph (Minimal Subgraph) contains - the necessary information for an LLM-based optimizer to change the workflow. For an example program: +

Trace constructs a computational of the user-defined computational workflow. The graph + is an abstraction of the workflow's execution, which might not be the same as the original program. + The original program can have complicated logic, but the computational graph contains + the necessary information for an optimizer to change the workflow. For an example program: 

                                         
     from trace import node
 
     x = node(-1.0, trainable=True)
-    a = bar(x) # automatically traced
-    b = unknown_func()  # not traced
+    a = bar(x) # code with complex logic abstracted as an operator by bundle
+    b = unknown_func()  # not applied by bundle
     y = a + b
     z = a * y
     z.backward(feedback="Output should be larger.", visualize=True)
-

If we visualize during the backward pass, we obtained a partial graph of how the program is run. Trace +

If we visualize during the backward pass, Trace returns to the optimizer a partial graph (Minimal Subgraph) of how the program is run. Trace allows user to design which part of the workflow they want to present to the LLM optimizer. Users can choose to present as much information (i.e., a complete picture of the workflow) or as little information (i.e., only the most crucial part of the workflow) as possible. From 2a1e137981a1c69cafc36d47317f481b8226d0c9 Mon Sep 17 00:00:00 2001 From: chinganc Date: Mon, 24 Jun 2024 12:20:05 -0700 Subject: [PATCH 2/2] update --- docs/index.html | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/index.html b/docs/index.html index 678a0f7..a22dc15 100644 --- a/docs/index.html +++ b/docs/index.html @@ -429,10 +429,10 @@

Trace Graph

-

Trace constructs a computational of the user-defined computational workflow. The graph +

Trace constructs a computational graph of the user-defined computational workflow. The graph is an abstraction of the workflow's execution, which might not be the same as the original program. The original program can have complicated logic, but the computational graph contains - the necessary information for an optimizer to change the workflow. For an example program: + the necessary information for an optimizer to update the workflow. For an example program: 

                                         
@@ -440,7 +440,7 @@ 
Trace Graph

 
     x = node(-1.0, trainable=True)
     a = bar(x) # code with complex logic abstracted as an operator by bundle
-    b = unknown_func()  # not applied by bundle
+    b = traced_function()  # traceable codes
     y = a + b
     z = a * y
     z.backward(feedback="Output should be larger.", visualize=True)
@@ -453,7 +453,7 @@ 
Trace Graph

                                     

                                         image
                                     

-                                    
An optimizer works this execution graph presented by Trace. We present an initial design of an optimizer that
+                                    
An optimizer works with this Trace graph presented by Trace. In the paper, we present an initial design of an optimizer that
                                         represents this execution graph as a code debugging report, and ask an LLM to change part of the graph that is marked as
                                         trainable=True according to feedback.
                                     

@@ -485,7 +485,7 @@ 
Trace Graph

     #Feedback:
     Output should be larger.
-

This debug report is presented to an LLM-based optimizer and LLM optimizer proposes changes to the variables. +

This debug report is presented an LLM and the LLM is asked to propose changes to the variables. Note that this report may look like the actual program, but is not the same as the python program. Even though any user can directly present the FULL python program to an LLM and ask it to change, it would be (1) Difficult to control LLM to only change the relevant/necessary part; (2) Hard to flexibly specify which part of the workflow LLMs should focus on.