Custom Camera3D projection : a practical approach

[Flarkk]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

There is a longstanding and strong push from the community to get fine control over the Camera3D projection.

The most plebiscited solution is to expose the camera’s Projection matrix to the end user (#2713, #4932).

While it remains a very valid target state, it comes with a number of yet unresolved problems :

• The projection matrix not only depends on the camera but also on the viewport’s aspect ratio to avoid undesirable stretching
• Many pieces of code across physics (picking) and rendering (including shaders) make strong assumptions on the frustum as optimizations, as it avoids full matrix multiplications : rectangular projection surface, parallel and z-orthogonal / z-symmetrical near and far planes, matrix inversibility

An arbitrary matrix may not meet these constraints and have visual and performance side effects. This is the main reason why no attempt has been successful so far (godotengine/godot#85529, godotengine/godot#84454).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Give back full control solely over the parts of the matrix that really matter for an usage with a camera.

Expose it through an interface that guarantees the above constraints cannot be broken.

Benefits

• Almost straightforward to implement under the current status quo, unlike exposing the full raw matrix
• Still doesn’t restrict us in any way to move later towards exposing the full matrix
• Bring some foundations that may actually facilitate it … unless it makes us eventually realize that giving full access to the matrix is not needed after all

Use cases

Thorough reading through #2713, #501 and #3779 gives a pretty clear picture of what 💡 users need, what's ✅ already supported, and of what’s ❌ not needed :

	Symmetrical	XY panning	Shearing	Slanted near and far	Non-rectangular	Perspective division
Perspective	✅ Common use (current PERSPECTIVE mode)	✅ No use (can be achieved anyway by translating the camera)	✅ Doom-like (current FRUSTUM mode)	💡 Planar reflections (mirrors, portals)	❌ No use	✅ Across Z axis only
Orthographic	✅ Common use (current ORTHOGONAL mode)	✅ Already supported (only through rendering server)	💡 Oblique projections (cavalier, cabinet, etc...)	💡 Planar reflections and 2.5D	❌ No use	n/a

💡 Interpolation between perspective and orthographic
💡 Non-rectilinear projections (pannini, fisheye, etc...)

Impact on matrix components

Above user needs are intentionally categorized to match the projection matrix' components.
This gives a clear picture of what really needs to be accessed :

Color coded	Analytical form
	With : $h=2\tan(fovy/2)$ projection plane height $n$ near distance $f$ far distance $\alpha$ aspect ratio $\lambda\in[0; 1]$ interpolation factor between perspective and orthographic $p_x, p_y$ panning values $s_x, s_y$ shear values $t_x, t_y$ slant values

💡 Access needed :

• ⚪ White : symmetrical
• 🟠 Amber : XY panning
• 🟣 Indigo : Shearing
• 🔵 Dark blue : Slanted projection plane
• 🔵 Light blue : Perspective division across Z vs orthographic

Light and dark blue require changes across rendering and picking code.
The rest only requires guardrails at input time regarding aspect ratio and inversibility.

❌ Access not needed :

• 🔴 Red : non rectangular projection plane
• 🔴 Pink : perspective division across XY

Giving access to these ones would have extensive impacts across rendering and physics code as it breaks the above assumptions.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

At server level

• Add function camera_set_projection() that lets the user define the projection in terms of [height, near, far, skew, slant, and persp/ortho mix]. This function enforces inversibility and does not constrain aspect ratio
• For compatibility and convenience, keep the current camera_set_projection_* but reimplement them by wrapping calls to the above
• Keep the current architecture : the server builds the matrix at each frame from the above user inputs. As an optimization, analytically build the inverse too, as well as the 6 frustum planes at each frame as those 2 representations are widely used across rendering code (today generated on the fly multiple times)
• Add additional branches required to handle persp / ortho mixes across rendering and picking. This is the only pervasive change needed
• Do not branch on slanted projection planes. This is not optimal and has side effects, but acceptable for an auxiliary camera used for planar reflections only

At node level

• Camera3D attributes : height, fovy, near, far, persp/ortho mix (hidden by default), plus skew and slant sections :
  • Skew : a toggle bool and a Vector2
  • Slant : a drop-down to pick a mode (disabled, reflect, manual), a NodePath for reflect mode and a Vector2 for manual mode
  • Slant Reflect mode takes the path to a target Node3D to copy forward direction from (as proposed here). This is convenient for setting up reflections without having to adjust near and far manually
• Three presets PERSPECTIVE, ORTHO and CUSTOM enable/disable/override certain attributes to facilitate user input

Non-rectilinear projections

Projections like pannini, fisheye and similar require additional changes to the rendering pipeline.
Typically a specific code path in the scene vertex shader is needed to perform a non linear projection before / after the projection matrix is applied.

This is a bit outside of the scope of this proposal, but for later reference :

• these code paths could be added as #ifdef sections, exactly like dual paraboloid rendering is handled today
• the rendering server could have specific methods to enable/disable these specific projection modes
• Camera3D interface could have additional presets like PANNINI, PARABOLOID, FISHEYE and more

Exposing the current dual paraboloid mode to the end-user as an additional projection mode could be a first good place to start, as the bulk of the rendering code is already there.

XR support

Either left as is (the server bypasses entirely the user settings and retrieve the camera directly from the XR api) or reworked so that XRCamera3D extracts the camera attributes from the matrix provided by the API, then call the regular rendering server methods.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

It cannot be an add-on