[p5.js 2.0 RFC Proposal]: WebGL Outlines

[davepagurek]

Increasing access

Lines in WebGL right now are catered towards line drawings rather than 3D shape outlines. Many older sketches that treat them more as outlines currently run significantly slower than they used to in earlier versions. Ideally, there would be a way to make that use case work again so that earlier teaching materials work as intended without also breaking the sketches than now use lines and rely on the current behaviour.

Which types of changes would be made?

• Breaking change (Add-on libraries or sketches will work differently even if their code stays the same.)
• Systemic change (Many features or contributor workflows will be affected.)
• Overdue change (Modifications will be made that have been desirable for a long time.)
• Unsure (The community can help to determine the type of change.)

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

What's the problem?

WebGL lines currently are too slow to be drawing them in large numbers in immediate mode shapes. Many older teaching examples do this because earlier lines were lighter, and those are now much slower. The use case of 3D shape outlines also does not need the high fidelity caps and joins that are required for smooth curves.

This particular example from The Coding Train keeps coming up, as one that slows down a lot in newer p5 versions https://editor.p5js.org/codingtrain/sketches/OPYPc4ueq but it also does not need complex lines to accomplish its visual goal.

What's the solution?

There are a few options I can think of currently, all with different compromises:

• As suggested by p5.js 2.0 RFC #6678 (comment), implement a line mode that uses GL_LINES to draw outlines. This will be much faster (we can pass start + end points directly to WebGL without any conversion in JS into quads). However, it will be limited to 1px thick lines, which also means that line thickness will look different when you change a sketch's pixel density, as 1px is smaller relative to the canvas size on a high-dpi display.
• Previously, lines were faster, and did not include caps and joins, so we could try to keep the line system mostly the same but add an option that doesn't create caps and joins. This will mean that any retained geometry (e.g. from buildGeometry) created in this simpler lines mode will never be able to gain caps+joins later on, as it will just not include that info. We will also maybe need to revisit the design of the line drawing system to accommodate having less attributes for some shapes, since we now pass more data along with each line vertex in order to be able to handle caps+joins, even if we only add segments, and this extra data is likely partly responsible for the slowdown. Per-vertex stroke color also is partially to blame for the slowdown.
• We could create outlines in a shader using depth information rather than doing it with geometry. Some more background on this technique, including interesting artistic variations, can be found here: https://lettier.github.io/3d-game-shaders-for-beginners/outlining.html I made a quick prototype (https://editor.p5js.org/davepagurek/sketches/aC--DvWmb) but updates will be required if we want to support variable thickness (which we maybe don't need.) This technique needs depth info, which only can be read from framebuffers, so we'd need to refactor the WebGL renderer to always have a main framebuffer (which might have side benefits, like being able to use depth in filters in general.) Being a filter shader, it would also apply to everything on canvas, so you couldn't control it and its color on a per-shape basis.

In addition to the above, we can maybe speed up some of the existing line drawing via the things mentioned in #7237.

Pros (updated based on community comments)

• Lines are currently the slowest part of the WebGL renderer, so if we default to something lighter, the perceived speed of p5 will have a big boost

Cons (updated based on community comments)

• Not sure that it's possible to be backwards compatible to both sketches that used lines as outlines for 3D shapes, and to sketches that use lines for high fidelity line drawings. We'll likely need a slightly different API for one use case or the other.
• Lines are already a pretty complicated system since vanilla WebGL does not offer any help. We'll need to be careful we don't turn this into an even more complex and brittle system
• Some of these solutions involve more complex refactors

Proposal status

Under review

[perminder-17]

Thanks for summarizing the solutions @davepagurek . Each approach solves the issue but leaves us with smaller challenges :"). Personally, I like the first and third solutions the most.

1. The first solution, using GL_LINES, is straightforward, but I feel its limitations could be noticeable. On high-DPI displays, where the pixel density is higher, the line thickness might appear smaller relative to the canvas. So, while it’s a good option, I’m unsure if it’s the one I’d choose.

2. The second solution is easy to implement, but I don’t find it very user-friendly. One point of confusion for me is whether we're giving users the option to add caps and joins, or if we’re removing them entirely. If we offer users these options, it could speed up the line drawing, but it also adds complexity by requiring them to specify it. If we’re just removing these features when drawing outlines, that's not good either. Could you clarify? Or maybe its kind of like for the outline purpose we are removing the joins and caps?

3. The third solution seems like the best fit for me. There are several reasons for this:

   • It will fully utilize the GPU, reducing unnecessary CPU and GPU computations.
   • Refactoring the WebGL renderer to always use a main framebuffer will unlock future possibilities like depth-blur, fog, ambient occlusion (AO), and more filters.
   • We have seen that using a framebuffer is faster than a standard WEBGL canvas, especially with optimizations like imageLight() and filter shaders drawing into the framebuffers. It improved the performance so in future, we are using framebuffer could be faster as well.

The main challenge could be figuring out when to draw the main framebuffer onto the canvas to make it visible—probably at the end of the draw loop. (we can figure that out)

Being a filter shader, it would also apply to everything on canvas, so you couldn't control it and its color on a per-shape basis.

If that’s not a major concern, I’d prefer going with the third solution, even if it means a breaking change (internally in p5 system). It could open up a lot of possibilities for future improvements. Let me know what you think!

[davepagurek]

to clarify that second option: I was imagining the simpler lines mode being an option you can set (maybe at the start of your sketch?) but it may not be easy to have all the features of the current system be optional without introducing bugs, since there are a number of parts we'd have to disable to return to the earlier performance.

[perminder-17]

too clarify that second option: I was imagining the simpler lines mode being an option you can set (maybe at the start of your sketch?) but it may not be easy to have all the features of the current system be optional without introducing bugs, since there are a number of parts we'd have to disable to return to the earlier performance.

Ooh... Got it. Thanks for the clarification 🙂

[davepagurek]

Actually, we might not need to do a huge refactor in order to support the earlier, low fidelity outlines. I did a little test here https://editor.p5js.org/davepagurek/sketches/jfNPJGoqV where setting testNoCapsOrJoins = true at the start comments out the calls to addCap and addJoin in _edgesToVertices without changing any of the other infrastructure. On my mac, it makes the perf of this sketch go from 20fps to 40fps on average. Rather than commenting these out, we could just wrap those sections in an if statement that checks whether the current webgl context uses simple lines or not, e.g. with strokesMode(SIMPLE) (vs a default strokesMode(FULL), for example.)

That might actually be the fastest way to accomplish the original goal of just making strokes faster for use cases that need them. It doesn't go quite as far as to reach the 60fps the original in p5 0.7.2 gets me, but maybe it's enough?

[davepagurek]

One more test case: the Coding Train sketch with noStroke() added, and using a filter shader to do the outlines. This gets me 60fps: https://editor.p5js.org/davepagurek/sketches/ZgKTL4Tmm

[perminder-17]

Fixed by #7390,