[RFC] Support for native Wayland plug-ins

[jwolffpsl]

We defined new interfaces which add support for native Wayland plug-in views. See https://github.com/fenderdigital/presonus-plugin-extensions/blob/main/ipslwaylandframe.h
Support for these interfaces is currently being implemented in Studio One.

With Studio One 6.5, we added support for Linux. Studio One is a native Wayland application. The VST3 SDK does not define a way for plug-ins to attach views to the host using Wayland.
These interfaces are meant to close this gap and allow plug-ins to attach views to native Wayland host applications.

See also #1 for additional interfaces which define a way for plug-ins to render simple user interfaces without explicit window system support.

[jwolffpsl]

Q&A from related discussions:

How much work is it to implement a Wayland compositor?

We're still working on it and it's untested in its current state, but the implementation is pretty straightforward.
It's mostly about listening for client (plug-in) requests, translating the server API callbacks to client API calls and forwarding these calls to the session compositor.
When a plug-in binds to a global Wayland interface (e.g. wl_compositor), the host creates a delegate object which listens for plug-in side requests.
When a plug-in creates Wayland objects (e.g. surfaces by calling wl_compositor_create_surface), the host creates a delegate object and creates an actual Wayland object using its Wayland client implementation. The delegate listens for both plug-in side requests and callbacks from the session compositor.

How do you plan to keep it up to date over time when new Wayland protocols are released?

I'm not sure if there is much use for the current staging protocols (https://wayland.app/protocols/) in a plug-in context.
Maybe "XDG decoration", "XDG activation", "Input method", "Fractional scale"?

There are some extensions proposed at https://gitlab.freedesktop.org/groups/wayland/-/merge_requests which could be useful on the host side, but I don't expect new extensions to be added in the near future that could be of use to a plug-in.

What will be the standard set of required Wayland extensions?

We'll obviously need the core Wayland (https://wayland.app/protocols/wayland) and XDG shell protocols (https://wayland.app/protocols/xdg-shell).
This should be all that's needed for most plug-ins, allowing them to open dialogs and context menus, access the clipboard, get pointer/keyboard/touch input (if not implemented through VST interfaces).
These protocols are what I'm currently working at. Once we have a working initial implementation, we can discuss including some of the extension protocols mentioned above.

I think we should follow Wayland's design though. Extensions are extensions. A plug-in must not require a specific extension (other than xdg-shell) to be available.

Would it make sense to release an open source library which all hosts can share and contribute to?

Once we have an initial implementation, we can think about refactoring this into a separate reusable library.
The problem I see with this approach is that a host application still needs to connect its Wayland client implementation to the library's server API implementation. I'm not sure if a separate library would simplify the implementation for a host application.
The host would still need to implement some interfaces which allow this new library to access the hosts client-side implementation objects (wl_display, wl_compositor, xdg_wm_base, ...).
I'm also not sure if all host applications would want to share the same implementation. A host might want to handle specific plug-in requests internally without forwarding to the session compositor.
For example, when implementing the "XDG decoration" protocol, some hosts might want to add session window manager decorations (e.g. KWin decorations) to plug-in dialog windows, other hosts might want to add custom decorations which match the host application design.

[jwolffpsl]

We updated the IWaylandFrame interface in https://github.com/fenderdigital/presonus-plugin-extensions/blob/main/ipslwaylandframe.h

A new BSD-licensed wayland-server-delegate library is available at https://github.com/cclsoftware/wayland-server-delegate. This library may be used by host applications to allow plug-ins to connect to the host using Wayland protocols.

As a proof of concept, we implemented plug-in side Wayland support in https://github.com/fenderdigital/vst3sdk/tree/waylandframe.
This implementation is not complete, but it may serve as a starting point for plug-in developers. Expect bugs.