Roadmap: External Apps as Buffers

[AndrewECarlson]

First, thank you for all the work you have put into this project!

I have been thinking a lot recently about how we could get the kind of GUI extensibility that emacs has while staying inside a neovim type paradigm, when I came across the emacs-application-framework. It doesn't seem very active, but I think thats because the emacs community already has a variety of solutions to this problem.

I was looking into this related issue where you mentioned being open to other widgets, and this type of approach seems like it would minimize the maintainance burden of custom widgets while maximizing the flexibility.

Would you be interested in integrating something similar?

That project depends on having an X server, but it should be possible to make it cross platform per this stackoverflow question

It seems like that approach to GUI plug ins would fit really well with neovim. You could take an app that has neovim embedded or vim like bindings and embed it in neovim! That way you can keep all of your vim like thinking in one context.

It opens the door to a lot of interesting applications like...

• viewing various markups better
• adding toolbars for feature discoverability ( if someone wanted that)
• using jupyter notebooks in vim
• browsing the web in vim

This would also allow you to use vim as a limited tiling window manager so for development work where thats useful you can have that while still using a traditional desktop environment for things that don't play nicely with traditional floating window managers. I would also see this feature seeing a lot of use in windows where tiling window managers are less of an option.

Furthermore this seems like it would be something of a natural next step after terminal mode effectively made vim a terminal multiplexor. Sorry for rambling a bit at the end. I have been mulling this over for a little while now.

[justinmk]

@AndrewECarlson did you review tabgrid proposal? Nvim 0.4 also has "multigrid" (which is the foundation for "tabgrid").

I would avoid new "frameworks" and platforms-on-platforms, they generally just add entropy.

[AndrewECarlson]

@justinmk I'm not sure that solves quite the same issue, but I may be misunderstanding that proposal. That is for tying multi multiple desktop windows together into a single vim instance and switching between them by switching tabs and each desktop window would be displaying a vim tab. Right?

I think you used Xmonad workspaces as an analogy there, and using tabgrid to solve this issue would make external apps that embed neovim use up a whole grid, which works for big external apps, like embedding a browser in neovim, but it would mess up any less intrusive user widgets, like if I wanted to extend the Shell widget to render image links inside the buffer or add a save button. Those types of widgets wouldn't be worth giving their own tab because you couldn't use splits. Sure a tiling window manager could mitigate that, but then you have to context switch for somewhat minimal gain. It was worth it to integrate terminals so we didn't have to use tmux. This seems similar. Especially since external tiling won't perfectly integrate into vim splits.

[equalsraf]

Hi @AndrewECarlson

(slightly disorganised rambling follows :D)

First concerning the stackoverflow discussion. It seems to rely on Qt embedding foreign windows which is probably very tricky (e.g. https://bugreports.qt.io/browse/QTBUG-40320).
The emacs application framework is also X11 specific and should not be that different.

With regards to EAF there are a couple of interesting things there. It seems to treat these apps as custom buffers where certain actions in the buffer map to some action in the app (say play or pause video). It seems like the input is being forwarded through EAF or emacs, but i did not look at this into detail. On a side note, a lot of the example apps in EAF were implemented from scratch which kind of defeats the purpose.

---

What is usually unclear to me in these discussions is what model do we follow on how to handle input i.e.

1. does all input get forwarded to nvim - allowing for the full power of vim mappings (this is my usual) stance
2. does the app take a stance on this and routes input between widgets based on some special user action - I would probably never enable this by default, but it can be a setting in init.vim (or somewhere else)

Now, in the title you specifically mention buffers as being external apps. In which the app gets redrawn over the space meant for some specific buffer. This means that:

1. you need to keep track of the buffer coordinates
2. your app, or widget gets drawn on top
3. there are corner cases concerning modes, and what happens if the user triggers e.g. cmdline mode in the buffer, do we leave the cmdline visible? if so what about sizes and resizing.
4. vim has lots of special buffers, so it seems this is another special buffer that spawns an app - a plugin figures this out and calls the UI to do the work

On a side note: The buffer model might not match here, since we can have one buffer in multiple windows, which would mean drawing the window in .... multiple locations/sizes. The extgrid approach makes more sense if we think about real application windows. But I dont know if vim/nvim has the notion of a special tab.

but it would mess up any less intrusive user widgets, like if I wanted to extend the Shell widget to render image links inside the buffer or add a save button.

These are a bit different but we can rely on past experience for this:

• Vim had GUI specific buffer content through icons and tooltips in the buffer. Icons require no interaction so they are easier. Vim required those to fit in one or two cells, Buttons can be done with a tooltip (but that probably means you cannot use the keyboard or you need custom mappings)
• images inside the buffer need custom drawing logic, that seems unlikely

What i mean is buttons and little elements are usually easier to do using custom buffers and plugins that interact with the UI.

Anyway i lost my train of thought and have to go :S. Apologies @AndrewECarlson this ended up not being a very productive comment.

PS: elvis (the best Vi clone) actually rendered html

[jgehrig]

@AndrewECarlson Have you tried Oni or Oni2?

• https://github.com/onivim/oni2
• https://github.com/onivim/oni

It sounds like you're looking for a fully-featured/modern Neovim frontend. Your description sounds similar to the goals of OniVim to me :)

Personally, I would like to see Neovim-Qt stay a lightweight and simple frontend. I like that this app is written with simple C++/Qt (ha!), and that it delivers a consistent experience across Windows/Mac/Linux.

A live-update Markdown preview widget might be cool...

[AndrewECarlson]

@jgehrig I have seen oniviim, though I haven't tried it. I don't think thats really the solution. I'm looking for a way to extend the vim gui as needed. Onivim certianly has a lot of features that many would want out of a more full-featured vim gui, but it seems very opinionated and requires a license for commercial use. For me, that's enough to not want to contribute to it, and I suspect others feel the same.
I also appreciate how lightweight neovim-qt is and how true it feels to traditional nvim. It is also the closest to an official neovim gui we have since it ships with the windows install. I think that some form of externalized form of gui extensibility like this (as opposed to the batteries included approach of Onivim) could keep the core of neovim-qt lightweight and improve community cohesiveness regarding guis.

@equalsraf
(Disclaimer: I'm not as familiar with Qt developement as I'd like to be for this conversation)
Hm. If thats a buggy feature, I wouldn't want to use it. Though that was from 2014, maybe its smoothed out?

In EAF, I was mostly excited about qutebrowser. The others being made from scratch mostly seemed like trying to showcase the framework.

I think for handling input sending everything to nvim is a good idea, and it could be like in terminal mode where by default only <Ctrl-\> <Ctrl-N> actually gets intercepted by default.

Currently it looks like Shell handles the window splits, right? Would it be possible to create a QSplitter like with the file tree for each split? That way we could just inherit the position from the parent, or is that more/the same work?

I would think you'd keep the cmdline visible below the statusline and just keep the widget sized to fit above that, like with terminal mode. What do you think?

For getting around the multiple buffers issue, I was thinking if you open a widget buffer thats already in a window in a new window, you could move it the the new window and just display a text that says " open in another window, hit enter to move it here". The buffer workflow is so so nice, I want to use it for everything. I mainly think about this feature when I have dozens and dozens of desktop windows open.

Your comment about elvis is something else I had been thinking about. A very nvim-centric approach would be to use html for the gui and let nvim plugins handle the gui plugins and just have nvim-qt render them. That would be cool in a lot of ways to further discuss. Would you be more on board with something in that direction?

[equalsraf]

In EAF, I was mostly excited about qutebrowser. The others being made from scratch mostly seemed like trying to showcase the framework.

I would not want to bundle a browser with nvim-qt. Probably a separate app that links against the lib in this repo then.

Currently it looks like Shell handles the window splits, right? Would it be possible to create a QSplitter like with the file tree for each split? That way we could just inherit the position from the parent, or is that more/the same work?

I think the native splitter use case is covered by neovim's ui-multigrid extension, but we currently do not support it. https://github.com/neovim/neovim/blob/master/runtime/doc/ui.txt

We actually do have a file tree view. Call :GuiTreeviewToggle. Double click on files to open.

Your comment about elvis is something else I had been thinking about. A very nvim-centric approach would be to use html for the gui and let nvim plugins handle the gui plugins and just have nvim-qt render them. That would be cool in a lot of ways to further discuss. Would you be more on board with something in that direction?

I have no specific preference for html as a format. But yes i would prefer to offload to existing solutions.

Its probably easier to pick one use case and start with that.

[yatli]

@AndrewECarlson I'm very interested in evolving the frontends this way.

If I understand correctly, you're describing something like a declarative ui.vim, where neovim provides command primitives like scroll, find, goto, and the UI implementation provides event feedback, like buttonClicked etc.

If so, here's a dump of my thoughts:

• How to ensure compatibility between different frontends? Would this be a nvim-qt specific framework, or, a general and more declarative one that's usable from even TUI?
• On displaying images in buffers: if neovim provides scroll information and the cell coordinate synchronization, one can anchor widgets to a specific position of a buffer, much like what's done with a floating window, a popup menu etc.
• Control standards would be awesome (what's a button, a drop-down list, etc.)

[AndrewECarlson]

@yatli
I'm glad to see you'd be excited for something like this too!
Yeah, I was thinking that user-widgets could provide neovim functions for whatever was needed to control them, while trying to make as much use out of built in neovim functionality. I like your idea for displaying images in the buffer, and that's part of why I'm pushing for this rather than relying on desktop window level integration between different user-widgets, using them as their own buffer allows more of the composability that we love about nvim. I address your cross GUI point in my section to equalsraf.

@equalsraf

I have no specific preference for html as a format. But yes i would prefer to offload to existing solutions.

What, if any, solution do you prefer for users being able to customize/extend the GUI? I'm sorry. I probably should have led with that, but I got excited about having thought of a potential solution myself.

Some other formats I was considering besides reparenting

• html would have the benefit of being a pretty standard tool for people making extensions, but I don't see how you'd use it without somehow bundling a browser(which makes sense why you wouldn't want to do that). Maybe through a terminal browser?
• I did some looking into it and came across qt .ui files as another potential option.
• Another option could be to create a custom format depending heavily on underlying nvim text buffers. I'm not entirely sure how that would work though. Maybe each line/paragraph could be a different gui element and each gui element could have an underlying buffer for its data? That might require more work translating to what should actually be displayed, but it could get you a lot of nvim's power for free. Also, the format wouldn't be neovim-qt specific and easy to parse, so other GUI's could make use of user-widgets too.

Are there any other formats you were considering/would prefer?

I think the native splitter use case is covered by neovim's ui-multigrid extension, but we currently do not support it. https://github.com/neovim/neovim/blob/master/runtime/doc/ui.txt

We actually do have a file tree view. Call :GuiTreeviewToggle. Double click on files to open.

Are you saying that in order to embed user defined widget-buffers(via external apps or some other format), we'd first need to set up multigrid for the native splits? Sorry if I'm being dense, this is my first foray into neovim development, so all of the implications of multigrid aren't immediately apparent.

Its probably easier to pick one use case and start with that.

What do you think would be a good start? I realize this is kind of a huge feature, and I pitched it all at once. I just wanted to get it all out there so we could make sure development of each use case feeds into the next. One use case might be a widget that displays an image. It'd be pretty minimal on the widget development side but a decent proof of concept and be pretty useful on its own.

Also, to clarify, I'm picturing users having a folder somewhere for all user-widget specific code, so that besides a general interface to widgets, they don't add maintanance burden to nvim-qt.

[jgehrig]

How to ensure compatibility between different frontends? Would this be a nvim-qt specific framework, or, a general and more declarative one that's usable from even TUI?

@yatli Good question! I think Justin touches on this above too.

@AndrewECarlson What are your thoughts on this? Do you have a plan to handle this?

I don't think we should implement a new custom framework specific to neovim-qt. For most features it makes more sense to do the work in neovim instead. Images embedded within buffers are a good example (usefulness aside).

Are you saying that in order to embed user defined widget-buffers(via external apps or some other format), we'd first need to set up multigrid for the native splits? Sorry if I'm being dense, this is my first foray into neovim development, so all of the implications of multigrid aren't immediately apparent.

Not exactly. The UI feature ext_multigrid in neovim gives the front-end more control over how buffers are displayed, sized, and painted. The frontend can set each buffer's size and position independently. This would be useful if you wanted to support something like the camera in your emacs example.

The TreeView works without ext_multigrid; it resizes the single grid, and paints itself on the side.

What do you think would be a good start? I realize this is kind of a huge feature, and I pitched it all at once. I just wanted to get it all out there so we could make sure development of each use case feeds into the next. One use case might be a widget that displays an image. It'd be pretty minimal on the widget development side but a decent proof of concept and be pretty useful on its own.

Adding support for ext_multigrid seems like a good first step to me?

[AndrewECarlson]

@jgehrig
I initially wasn't thinking to make it cross-ui, but I can certianly see the value in it. Using a declarative language, rather than implementational, where different UI's can decide how/if they want to render a certain component would make a lot of sense.
For cross-ui, the only option I listed above would be the buffer oriented approach, but there may be a better option I don't know about.

Multigrid is making more sense. Justin was right, I needed to review it more for this conversation. Should we make a separate issue for multigrid support? I'm happy to work on it, but I'll have some questions and probably need a bit of direction.

[yatli]

adding another potential use case: once I talked to the author of coc.nvim about adding debugger support, and he said the biggest road blocker was that there's no obvious way to draw solid UI for the visual elements (buttons, input boxes, list boxes etc.)

[yatli]

adding some thoughts: perhaps we should design a protocol that runs on the NeoVim jsonrpc channels.
Something like, on the vim side (vimL etc.) one could define view models, and then these models are published to the ui channels, and the UIs can then match the published viewmodels and connect to them.

this paradigm can then be mapped onto UI framework capabilities, for example, qt signal/slot, XAML bindings or html event callbacks.

[yatli]

@AndrewECarlson I've created an RFC in the neovim repo: neovim/neovim#12038

[jgehrig]

As discussed above, this feature would require API support from neovim. Tracking the work above.

Marking as closed.