About gptel scope & modular extension development

[daedsidog]

TL;DR at the end.

Some time ago, my Emacs config file contained code which allowed gptel to highlight sections of buffers and conveniently passed them down to the model prompts. With great help from @karthink, this code was improved and refined down to gptel-context.el, to introduce what is now known as the gptel contexts feature.

This addition was not limited to gptel-contexts.el alone, but also had to have entries for it in the gptel-transient.el file for the transient menus, the main gptel.el file for custom variables & settings, and the backend files for proper context inclusion.

This cemented the contexts feature as an inseparable part of gptel, much like the dedicated chat buffer. At the time, this did not seem to be an issue, because I thought model context was a very crucial system for many workflows.

After this, I've had many ideas on how to expand on the context feature, but they were outside of the scope of the level of simplicity that gptel tried to strive for, so I decided to make them into an extension, and so Evedel was born, which attempted to add improvements and expansions to the contexts idea I originally had in mind.

Evedel, as of right now, is almost completely independent from gptel, because all it uses is gptel-request. It does not even use the original contexts feature of gptel. In fact, calling it a gptel extension is dubious, which is something I would like to change.

Evedel reached a stage of development where I would like to better integrate it with gptel: adding various gptel dedicated chat buffer integrations, being able to add transient menu entries to it, etc. I am unsure how to best approach this, and what to do with the original contexts of gptel, which--after Evedel is better integrated into gptel--will be made obsolete.

In an ideal world, at least in my opinion, gptel should offer at its core a unified API that is callable from Emacs through gptel-request, while everything else should be a modular extension. So, in this world, the chat buffer would be a core extension shipped with gptel. Developers could create these components to gptel which would be loaded independently. The current contexts feature should also be loaded as a default extension, as users are supposedly expect it by now. The core components could still be hosted in the main repository.

Recently, a lot of discussions & issues were raised which concern the contexts feature, such as adding RAG (#492, #434, #322, #377, #305) , which is something I planned to eventually tackle in the extension. If those features become core gptel features, it might be harder for me to integrate Evedel in gptel down the road.

TL;DR:

1. Is this restructure, or something similar to it, likely to happen in the future, or is the current level of modularity satisfactory?
2. What's the best approach to integrate extensions right now? For example, the contexts: should I just replace them altogether, or shim them along with the original contexts?
3. How would I replace the gptel contexts? There are custom variables, transient entries, the context file itself, key bindings, stuff in each backend file, etc., all that would need addressing. It doesn't seem like an easy path to me.
4. How would I shim the gptel contexts? It would seem to me that I would need to keep my own version of gptel-transient.el, and keep it updated with gptel whenever there are breaking changes. Easier than the above point, but still quite tedious.
5. Even in my "ideal world", on would still be interested to add extensions to extensions: for example, adding more features to the chat. The best approach I see there would be to fork the "chat component", keep a separate version, and add features. Users can then load that component instead of the default chat, which means that changes to the default chat component won't be breaking to the extension.

[karthink]

@daedsidog thanks for the detailed questions! Yes, there are plans afoot for better composition. Sorry for the delay, I'm currently held up with some other projects. Will respond in detail soon!

[karthink]

@daedsidog sorry, I was held up trying to implement #514 -- which you may find interesting. Answering your questions with a detailed plan is the next TODO on my gptel agenda.

[daedsidog]

Thanks. There's no rush, as there's plenty of stuff to do that don't entail gptel integration right now.

#514

I'll give it a look.

[karthink]

I read your post in full, but will respond to the TL;DR questions as it summarizes the issues well. For reference, here is an export of my notes on the gptel project. The items are sorted in priority order, and the top 3 items are usually what I'm working on. (This could be a github projects page, but I find it much easier to work with Org -- which makes collaboration difficult. I'll keep this page updated though.)

In an ideal world, at least in my opinion, gptel should offer at its core a unified API that is callable from Emacs through gptel-request, while everything else should be a modular extension. So, in this world, the chat buffer would be a core extension shipped with gptel. Developers could create these components to gptel which would be loaded independently. The current contexts feature should also be loaded as a default extension, as users are supposedly expect it by now. The core components could still be hosted in the main repository.
[...]
Is this restructure, or something similar to it, likely to happen in the future, or is the current level of modularity satisfactory?

The current level of modularity is not satisfactory! I share your plan and want to turn gptel into a collection of modules you can require as needed. This will also make contributing to gptel easier. Doing this requires figuring out the dependency order, which is tricky.

The first step is clear though: place all the gptel-request code into a single gptel-request.el file that is independent of gptel.el and contains no UI code. Then extensions that only need gptel-request can require only this file. Note that this is currently item 4 in the project plan above. Once the WIP tool-use branch is merged, it should be easy to do.

That leaves the following:

• gptel-context.el: Functions for adding context to gptel requests
• gptel-rewrite.el: Functions for rewriting text in place
• gptel-rag.el: Adapter to external RAG providers, may be folded into gptel-context if this is small (Data)
• gptel-transient.el: Transient menus for gptel-send
• gptel.el: Core user-facing commands (gptel, gptel-send) and chat interface.
• gptel-complete.el: completion-at-point functionality

All of these are UI-focused libraries (except for gptel-rag.el).

The problem is that except for gptel-request.el, which lies "below" all of these, it's not easy to draw clear separations between the rest of the code.

For example, gptel-transient currently includes the menu code for managing context. If we move this part of the code into gptel-context, it will have to be loaded when running gptel-menu. The only way to solve this is to make each feature provide its own transient prefix, but then we can't have an integrated menu like gptel-menu right now -- it will be like the rewrite menu, which has to be accessed from gptel-menu as a sub-menu. This also means loading any of the above files (except gptel-request.el) will involve loading transient.el.

All of these also don't have to be part of gptel -- although that makes drawing clear boundaries even more necessary.

I'm okay doing things one at a time -- I will split gptel-request into its own library after merging the tool-use branch.

What's the best approach to integrate extensions right now? For example, the contexts: should I just replace them altogether, or shim them along with the original contexts?

Hmm, I would start by seeing if you can reuse the underlying data structure -- which in this case is only gptel-context--alist and the functions that act on it. If not, we can try to figure out why. See also item 12 in my project plan link above -- this would be like the annotation feature you already use in Evedel.

I'll respond to your other questions (on RAG, replacing parts of gptel's UI etc) soon -- but I just wanted to mention that gptel's directives can be much more general than system message strings now. The entire context feature can be implemented as a single gptel directive. Are you aware of these changes?

[daedsidog]

For reference, here is an export of my notes on the gptel project. The items are sorted in priority order, and the top 3 items are usually what I'm working on. (This could be a github projects page, but I find it much easier to work with Org -- which makes collaboration difficult. I'll keep this page updated though.)

This looks incredibly useful for aligning expectations. Cheers.

The current level of modularity is not satisfactory! I share your plan and want to turn gptel into a collection of modules you can require as needed. This will also make contributing to gptel easier. Doing this requires figuring out the dependency order, which is tricky.

I'm glad to hear that my sanity check has passed.

The first step is clear though: place all the gptel-request code into a single gptel-request.el file that is independent of gptel.el and contains no UI code. Then extensions that only need gptel-request can require only this file. Note that this is currently item 4 in the project plan above. Once the WIP tool-use branch is merged, it should be easy to do.
[...]
The problem is that except for gptel-request.el, which lies "below" all of these, it's not easy to draw clear separations between the rest of the code.

For example, gptel-transient currently includes the menu code for managing context. If we move this part of the code into gptel-context, it will have to be loaded when running gptel-menu. The only way to solve this is to make each feature provide its own transient prefix, but then we can't have an integrated menu like gptel-menu right now -- it will be like the rewrite menu, which has to be accessed from gptel-menu as a sub-menu. This also means loading any of the above files (except gptel-request.el) will involve loading transient.el.

All of these also don't have to be part of gptel -- although that makes drawing clear boundaries even more necessary.

I'm okay doing things one at a time -- I will split gptel-request into its own library after merging the tool-use branch.

This looks much more palatable to me than what I had in mind, although, is this structure definitive? For example, to me, it is gptel-request that is the "core" of project, not the chatting interface (despite growing from there).

I am not in any way saying that it is wrong, or that I have a better design for it, because I truly am lost when it comes to strategizing the best interface. If I were to be left to my own devices, I would elect that gptel.el does absolutely nothing but define "command autoloads" for each individual component. I don't know how to answer questions like: should you load all the functionality if you open the gptel menu, or just the invoked parts? My gut feeling tells me the former. I still feel that the chatting interface deserves its own file, though.

In terms of scope, I am glad to see gptel be balkanized into its "independent" components. Having each component use the external functions provided by other components will be a great working example on how to develop extensions.

At the end of the day, I'm not even sure myself that I know what exactly I would want to have, as a developer. I want to be able to entirely replace existing gptel components, or just partially modify them. This could probably be a matter of having the cake and eating it too.

A concrete example: when loading Evedel, I want to truncate the existing contexts feature, and add more transient menu options in the chat interface, allowing users to config my extension from the existing menu. An alternative approach would be to reprogram Evedel to use the existing gptel-contexts.el features, but this introduces a problem: I don't want the user to be able to manually remove contexts, because it will mess with the bookkeeping done by my extension.

Hmm, I would start by seeing if you can reuse the underlying data structure -- which in this case is only gptel-context--alist and the functions that act on it. If not, we can try to figure out why. See also item 12 in my project plan link above -- this would be like the annotation feature you already use in Evedel.

Like I said above, I could probably reuse the data structure, but it's the peripheral features of gptel-contexts.el that I am concerned with. I would want the contexts to be "under the hood", with the user having no ability to interface with them. I could bind the data structures as I am constructing a request, but that sounds like it would defeat the purpose & just be expensive.

Annotations can be nested, not all of them create contexts, and the text is determined when I want to process them, I bookkeep only the annotations overlays, so I don't see how I could leverage the contexts data structure here. And, again, even if I could, it would "take control" of the contexts, preventing the user from adding/removing from them, unnecessarily.

I know I am probably being unclear, but I feel like talking about this more will help reveal the problems I am having, so it's helpful still.

I'll respond to your other questions (on RAG, replacing parts of gptel's UI etc) soon -- but I just wanted to mention that gptel's directives can be much more general than system message strings now. The entire context feature can be implemented as a single gptel directive.

Are you aware of these changes?

Not entirely, as (thankfully :) ) the amount of new features recently has been staggering and hard to keep track of. I recently started leveraging more of gptel's features (like gptel-rewrite) instead of reinventing the wheel, and will, for example, wait for the RAG utilities, whereas before I planned on implementing it myself.

[karthink]

I'm okay doing things one at a time -- I will split gptel-request into its own library after merging the tool-use branch.

This looks much more palatable to me than what I had in mind, although, is this structure definitive? For example, to me, it is gptel-request that is the "core" of project, not the chatting interface (despite growing from there).

gptel-request is indeed the core of the project. I don't think it matters what the name is though? If you want to use gptel-request (the API function), you just run (require 'gptel-request) -- this way nothing in gptel that is irrelevant to your extension is loaded.

For users of gptel, it is important to be able to do (use-package gptel :config ...) and have everything work.

I am not in any way saying that it is wrong, or that I have a better design for it, because I truly am lost when it comes to strategizing the best interface. If I were to be left to my own devices, I would elect that gptel.el does absolutely nothing but define "command autoloads" for each individual component.

Autoloads can be defined anywhere in the project, they don't need to be in gptel.el. If I created a gptel-chat.el, there would be nothing at all in gptel.el except (require 'gptel-chat), which is redundant.

I don't know how to answer questions like: should you load all the functionality if you open the gptel menu, or just the invoked parts? My gut feeling tells me the former. I still feel that the chatting interface deserves its own file, though.

I have more thoughts on this but it will take a while to explain, I'll get to it later. The short version is that presenting the functionality and loading the functionality are independent so there are actually two separate questions to answer here.

In terms of scope, I am glad to see gptel be balkanized into its "independent" components. Having each component use the external functions provided by other components will be a great working example on how to develop extensions.

At the end of the day, I'm not even sure myself that I know what exactly I would want to have, as a developer. I want to be able to entirely replace existing gptel components, or just partially modify them. This could probably be a matter of having the cake and eating it too.

You and me both. We can figure this out as we go. I'm keeping modularity in mind now as I add new features. BTW, I'm planning to add a couple of new UI features to gptel, but that's it. All other features in the future are only going to be refinements. For example, users want to specify files to include in the context as links in the Org buffer. There is no chat UI change here, only a change in the behavior of the chat buffer.

gptel-rewrite is almost a package on its own, and is part of gptel only for historical reasons. If I write entire new features like gptel-complete, they'll be separate packages (like gptel-quick). gptel-rag is just going to be an adapter that RAG-specific packages can plug into.

A concrete example: when loading Evedel, I want to truncate the existing contexts feature, and add more transient menu options in the chat interface, allowing users to config my extension from the existing menu. An alternative approach would be to reprogram Evedel to use the existing gptel-contexts.el features, but this introduces a problem: I don't want the user to be able to manually remove contexts, because it will mess with the bookkeeping done by my extension.
[...]

Hmm, I would start by seeing if you can reuse the underlying data structure -- which in this case is only gptel-context--alist and the functions that act on it. If not, we can try to figure out why. See also item 12 in my project plan link above -- this would be like the annotation feature you already use in Evedel.

Like I said above, I could probably reuse the data structure, but it's the peripheral features of gptel-contexts.el that I am concerned with. I would want the contexts to be "under the hood", with the user having no ability to interface with them. I could bind the data structures as I am constructing a request, but that sounds like it would defeat the purpose & just be expensive.

Two separate issues here:

• replacing gptel's UI components with Evedel UI components, and
• retrofitting Evedel's behavior into gptel's data structures.

I think the first one is easy with a little bit of coordination. The second might require more thinking -- although I don't understand the issue with let-binding gptel-context--alist around calls to gptel-request. This seems like the normal way to do things in elisp to me?

Are you aware of these changes?

Not entirely, as (thankfully :) ) the amount of new features recently has been staggering and hard to keep track of.

This is why I need to write that (developer's) manual! I think it's going to take me over a month to manage that, but the result should pay off for a long time afterwards. Ironically I don't think this is something LLMs can help with, since only I can explain the intent of the design, as opposed to just describing what functions do.

I recently started leveraging more of gptel's features (like gptel-rewrite) instead of reinventing the wheel, and will, for example, wait for the RAG utilities, whereas before I planned on implementing it myself.

Yeah, I need to address RAG integration plans and your other questions from the opening post, I'll do that separately.