Add a "shading" language to Audio nodes, to create sounds effects on top of streams

[vPumpking]

Describe the project you are working on

it's called Godot Engine 😉
just searching how to increase the Godot's power

Describe the problem or limitation you are having in your project

the list of possible effects for sounds is very short

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add a thing like shading language to enable to freely post-effect the streams of audio and video players

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A bit like in a canvas item or in a spatial node, the user can open an Effects section in the inspector when an Audio or video player is selected. In this section, the user can create a new "AudioShader" file, using a new language like GLSL to post-effect the stream output.

If this enhancement will not be used often, can it be worked around with a few lines of script?

no

Is there a reason why this should be core and not an add-on in the asset library?

it's too hard with an addon

[Calinou]

Moving to discussion, as there is no concrete technical implementation provided.

[vPumpking]

ok

[theraot]

ShaderToy does something similar. This is the default example:

vec2 mainSound( int samp, float time )
{
    // A 440 Hz wave that attenuates quickly overt time
    return vec2( sin(6.2831*440.0*time)*exp(-3.0*time) );
}

I could not figure out what samp is.

The result is a vec2 where the x and y components are left and right audio channel for stereo sound.

Yes, it uses the same language that they use for graphics.

I don't know how it works behind the scenes. However, I can imagine that using the GPU to compute an audio buffer this way could use a fragment shader with time passed as a varying uniform and render to a texture, or whatever the equivalent is using compute shaders. From there the buffer would go the audio pipeline.

[vPumpking]

It gets good results! I suggest we continue to explore this way

[theraot]

Fixed: GPU2Audio.zip

It is also a lesson in using the AudioStreamGeneratorPlayback class. Instead of calling _fill_buffer from _process (which is what the official demo does), I'm using call_deferred("_fill_buffer"), which is what ultimately fixed the audio popping. The other changes making sure I don't try to generate more samples than I fit in a frame, and that I make the next frame from the exact next sample after the last one.

[theraot]

Optimization: GPU2Audio.zip

This version is using the four color channels to generate two audio frames per pixel.

I also did some renaming.

I tried different approaches to try to pass the data from the texture to the audio buffer. But the naive approach seems to be the faster way to do it in GDScript.

[sairam4123]

int samp could mean sampling of sound, though it's confusing, it's most likely audio sampling

[somebody1234]

samp seems to be an integer sample number (a count starting from 0), as an alternative to the float time value