Potential API features

[iboB]

Let's compose a list of potential API features. This is not bound to the MVP some of them might be distant future ones. The goal of this is to have the use cases in mind in order to design the API so that it allows for them.

Let's have the different ideas as separate responses in the discussion tree, so that we can comment on them separately

[iboB]

Seamless API use in terms of provider (whether it's local/edge or remote/server)

[iboB]

We already have this. Having all API calls being asynchronous with a callback is enough, right?

[iboB]

We no longer have this, lol. We should implement the API in a manner most suitable for the target language and platform.

[iboB]

Backtracking. "Ok you generated this, but I want something else" or "what if instead of X, you had Y in the context at that point"

[iboB]

Maybe we can solve this by having a backtracking op? This possible and relatively easy as long as we have not squashed the context in an infinite context scenario.

Alternatively we may save an instance's state at a given point (the problem with this is that we have to know where the "split" is at the moment it happens)

[zah]

This is also useful for implementing various solution search algorithms that are often described as "Tree of Thoughts (ToT)". What the API needs to offer here is efficiency, allowing the key-value caches at various checkpoints in the conversation to be effectively reused.

[iboB]

#132

[tzanko-matev]

Set of APIs (building blocks) for doing RAG