Add OpenXR Overlay Support

[ArgentumIpsum]

The ability to create an OpenXR session as an overlay would allow Godot users to make games/applications that display over top of another XR experience. This would be useful for creating something like a VR chess game that you could play while in another game, perhaps watching a movie. You could also make a heads-up display with debug information or social information to join friends in games without builtin social features.

This functionality is supported through the XR_EXTX_overlay OpenXR extension. I'm not aware of any way to have similar behavior with WebXR so web export wouldn't be supported.

Some new settings would be needed to determine whether the extension should be enabled (assuming the user's openxr runtime supports it), and how the overlay should be composited relative to other potentially running overlays. I'm not sure what changes, if any, would be required to support the transparent background/skybox that is necessary for overlays, or if setting the default clear color alpha to 0 and enabling per-pixel transparency would do the trick.

This improvement probably wouldn't be used very often, but would be absolutely critical to people who want to make overlays that utilize Godot's features, shouldn't require an awful lot of changes, and would have a negligible effect on binary size. This cannot be worked around with a few lines of script, and I'm not sure if it could even be done as a GDExtension.

EDIT: As a side note, this extension is only provisional as of OpenXR 1.0, and isn't yet supported by SteamVR, Windows Mixed Reality, or Meta so no rush. :)

[Da-Boom]

I agree - i use monado, which does support the extension. I think the more engines that support it the more likely that steamVR and other XR systems will add support for it. Its already quite well used by the likes of wlx-overlay-s, xrdesktop.. its even supported by the LOVR lua "engine" - just add a warning in documentation that its not implemented by all systems.

Honestly, i think its annoying to have to support OpenVR overlays while writing the rest of the app in OpenXR - we shouldn't be splitting between APIs like that.

[RangerMauve]

This would be nice to have. Godot's built in UI engine and easy spatial interaction make it appealing for use as a baseline for overlays.

Is it a matter of adding a flag somewhere when initializing the openxr code?

[RangerMauve]

It seems like a quick enough change. Just need to detect and set the flag. Would PRs be welcome? https://gitlab.freedesktop.org/monado/demos/xrgears/-/commit/c88f0f5f7db06a890cd336037143d1573c9315d8

[Calinou]

cc @BastiaanOlij

[dsnopek]

Looking at the OpenXR extension, I think it should be possible to use it via GDExtension: I think you'd just need to make a sub-class of OpenXRExtensionWrapperExtension which enables the extension and implements _set_session_create_and_get_next_pointer() to create and set the XrSessionCreateInfoOverlayEXTX struct. I haven't looked too deeply, but at first glance it seems like a fairly easy one to implement.

As far as including it upstream Godot: normally, any non-vendor extension in the OpenXR spec is fair game, however, as this is still a "provisional" extension, we'll need to decide if we want to include those in upstream. If we don't, this could perhaps be added to the godot_openxr_vendors GDExtension?

[BastiaanOlij]

I wouldn't worry to much about the provisional status. The real question is how well it's supported and in there lies a problem:
https://github.khronos.org/OpenXR-Inventory/extension_support.html#XR_EXTX_overlay

The most relevant platform seems to be missing from this list, Valve is not listed.

That said, I would be happy to accept a core implementation for this. Have a look at how we implemented this in our OpenVR plugin for any gotchas, I remember Ben had to get creative around handling viewports.

Having an implementation might persuade more vendors to adopt the extension and get it out of its provisional status.

[Da-Boom]

Just to note.. considering OpenXR is the modern XR/VR API that most people are switching to/using, It makes sense to add support for overlays. its my opinion that this should not be a provisional or optional extension. hopefully it gets moved from the provisional extensions into the main spec soon.

Most apps that use overlays and regular systems these days are combining 2 disparate APIs to accomplish the task so they can have both modern OpenXR features and OpenVR overlays, which doesn't always work. This also doesn't make sense from a support POV to have to support an older API just to get a feature that is incredibly common and desirable. Honestly, its the one addition that has been requested to steamVR on the steam community forums time and time again.

I think its a good thing for game and engine developers to be able to say "we support it. its XYZ runtime that doesn't, take it up with them" as it puts more pressure on the runtime developers to support it.

[BastiaanOlij]

@Da-Boom indeed, however we have no control over this. The overlay extension has been around for some time now, if vendors do not adopt it, then it remains in limbo. The only way to break through this is to put pressure on the vendors themselves.

Godot implementing this extension would help with that so I'm all for it, however this is the type of functionality best implemented by someone who wants to build an overlay application and is setup on one of the platforms that already supports it. I'm happy to dive into it myself in due time if nobody else picks it up, but right now I don't have time for it.

[Da-Boom]

while i do not have much experience in VR development. I do have a working linux/monado setup and a HTC VIVE. So if you want, i could have a go at testing things.

I might even try to pick up Godot again and see what i can come up with using the XR plugin. Though I'm an amateur at best when it comes to using Godot or most engines to develop things, we shouldn't need much more than a blank colored 2D plane on screen to overlay onto another app so it shouldn't be too difficult to figure out.