Thoughts and ideas for Assistants

[pietz]

Over the past days I've spent a lot of time building what I would consider a fairly complex multi assistant system with marvin Applications and I wanted to share a few thoughts, insights and shortcomings of the current state. This is not meant as criticism but food for thought and I would be very interested in getting some feedback.

First of all, I see marvin with all its features as a high level OpenAI API that makes things simpler and semantically closer to how developers think. For example, the concept and execution of the AI function is just 🤌. I definitely had this moment of "Yes, that's what the interaction always should have been". Anyway, for Assistants and Applications, I'm not entirely sure that the same can be said about the current implementation. I found myself going back to the native OpenAI SDK because it was more straight forward.

Assistants vs. Applications

I'm not convinced the naming and mental model for Applications is great. I don't think they should be called Applications or even exist in the first place. I think a nicer and cleaner way to go about it, would be to allow assistants to have an optional State parameter, which would then also add the necessary update_state tool and additional prompt. I see what you wanted to do with the application idea, but I think calling it an assistant better describes what it is and what it does.

assistant_with_state = Assistant(
    ...
    state=ToDoState()
)

Persistence

Applications add the idea of persistence and state, but they don't persist anything really. The state exists in the runtime of the variable but once that's gone, so is the state. Loading the respective thread doesn't bring back the state either. It's just gone. I found this cool project, which adds the persistence by using the currently unused metadata field of the assistant. It feels like a hack and is limited to 16 512 byte entries, but I believe it would be a great addition to the marvin assistant/application. On top of that, it should be considered to add an "easy" way of connecting the state to a local database or JSON file to really persist the information the application collected.

assistant_with_state_and_storage = Assistant(
    ...
    state=ToDoState(),
    storage="data.json" # ":metadata:" or None
)
assistant.say("I need to visit the store")

{
    "thread_CE3QsgNJIelGq4vedG5ZxUGA": {
        "todos": [
            {
                "name": "Visit the store",
                "done": false
            }
        ]
    }
}

Real World vs. Toy App

If you build a small local demo using the interactive chat() function or CLI tool that marvin provides, the experience is excellent. However, I believe that most real world applications won't benefit from that. For example, I'd be much more interested in integrating marvin applications/assistants into web servers like fastAPI. It's not impossible or even super complicated to do that now, but I think if my vision of marvin is correct, it should be made a lot easier. Imagine something like:

assistant = Assistant(...)
assistant.serve()

# Created '/add' and '/run' endpoints
# Uvicorn running on http://127.0.0.1:8000 

This would be dope. It would have generated endpoints like this:

@app.post("/add")
async def add(message: str = Form(...)):
    ...
    return message

@app.get("/run")
async def run():
    ...
    return StreamingResponse(...)

Assistants & Thread

I think for the OpenAI SDK it makes a lot of sense to separate the assistant from the thread, but in a higher level tool like marvin, I'm not sure that's true. Have you considered baking an optional thread_id into the assistant? I think that would make the interaction a bit cleaner.

assisant_with_thread = Assistant(
    ...
    thread_id
)
assistant.say("Hi")

Alright, that's it for now! Happy to discuss these ideas!