Implement environmental audio using OpenAL EFX

[Hiradur]

This is a WIP PR for implementing reverb support using the OpenAL EFX extensions. To make it easier for me, this PR is currently based on #3180 and #3181.
The basic foundation is already there. If the listener (camera) is underwater, an underwater effect becomes active.
The preset can be changed to ones listed here using the console command set audio_listener_efx_preset(see #3182 (comment)). Resetting the preset is currently not possible via console.

There is still some work to do:

• efficiently handle the creation of alEffects
• make changing the reverb scriptable
• avoid harsh transitions when presets change
• automatically activate an in-cockpit preset when appropriate? -> not easy to know whether a truck has a closed cabin or not
  • do not enable it if the listener is very close but outside of the truck (happens because there is a check whether the listener is inside the BB of the driven vehicle, but certain vehicles provide outside camera views)
• allow to define environmental areas in terrn2 files? -> possible using collision boxes
• ensure that sound sources always get updated: Sometimes they don't receive updates to e.g. the obstruction filter. This is a bug.
• Delete alEffects (and potentially alFilters) when returning to main menu.

[ohlidalp]

Looks fantastic so far. I'll test on friday.

[Hiradur]

I'd like to have the reverb presets as well as the air absorption related parameter to be scriptable, manually overridable or to automatically change depending on whether the listener is underwater or not (and perhaps they are in a cockpit or not).
Updating the air absorption parameters of every source every frame even if they haven't changed also seems wasteful.

I haven't decided how to bring all this together yet. Might be necessary to implement a state machine to track where the parameters are allowed to be set from.

[Hiradur]

Since somewhere in the OpenAL docs it was suggested to implement occlusion and obstruction effects alongside reverb effects to put more emphasize on the latter, I added a basic obstruction filter.
It's disabled by default and hidden behind the CVar audio_enable_obstruction. The collision detection definitely needs to be improved before this can be enabled by default (see commit message and comments in code).

[ohlidalp]

I appreciate your focus on scripting, some notes to hopefully make life easier for you:

• we have joint C++/Angelscript Doxygen doc at https://developer.rigsofrods.org/d2/d42/group___script_side_a_p_is.html
• cvars can be modified directly without routing via msg queue https://developer.rigsofrods.org/d3/d4f/class_script2_game_1_1_console_class.html
• instead of laborously adding MSGs, I'd just create thin wrappers around the al functions and export those directly to Angelscript. The game global script object already hosts audio APIs https://developer.rigsofrods.org/dc/d63/class_script2_game_1_1_game_script_class.html
  ... Side note: please ignore the fact the func signatures are C++ and not AngelScript, I'm actually faking the AngelScript classes as C-classes for doxygen, I'm not sure there's other way: https://github.com/RigsOfRods/rigs-of-rods/tree/master/doc/angelscript/Script2Game - I usually fix it manually but this place slipped

Improving sounds including terrain/object interference has been on my mind for years, but there's always too much other stuff, regarding scripting try in console 'loadscript demo_script.as', it showcases the current basic audio bindings. There's plenty more examples in repo under /resources/scripts.

Also come hang around other Devs in Discord #dev https://discord.com/channels/136544456244461568/189904947649708032

[cryham]

Sounds like a lot of awesome changes. I'm just saying, didn't test yet. Will see if that can be merged in SR3, since I based new audio on RoR's.

[ohlidalp]

I'm thrilled, there hasn't been this much focus on audio in last 10 years.

I'd like to know some test scenarios to observe and enjoy these effects. I'll try to explore myself for now.

[Hiradur]

Sorry for the belated response, I was on vacation. Thanks for your kind words. After reading the OpenAL Effects Extensions Guide, I got really motivated to try out many of the features the API provides.

The PR has grown quite large and in terms of features, I think it has almost reached a viable state. I still intend to do some minor refactoring. As far as scripting support goes, I think the most important thing is that reverb presets can be defined for meshes and terrains and that custom presets can be loaded into the audio engine. The first is made possible via the reverb_preset parameter of collision boxes and I intend to make the latter possible via Angelscript.

@cryham I would be very happy to see the code being used in SR3. The OpenAL Soft implementation always fascinated me with the high quality audio it provides through EFX, binaural audio and Ambisonics and I would welcome more games using the advanced features the OpenAL API provides.

Looking at the PR, there is still room for improvements, of course:

• some OpenAL EFX features are not supported yet, e.g. late reverb panning and rendering of multiple reverb environments simultaneously
• in general, the collision checks are functional but not optimized und could cause uneven framepacing
• defining reverb presets currently relies on collision boxes. It is convenient to use the existing infrastructure, but this also means that an odef file containing the definition of a collision box with a reverb preset has to import a mesh, which might not always be desirable, e.g. if you want to define a reverb zone quarry in a terrain that is not bound to a single mesh.
• the audio parameters would benefit from some finetuning, e.g. the parameters for the obstruction filter or the reverb presets for the prefab models. However, I think it would make sense to do this iteratively with growing listening experience.

Reflection panning is still marked as an experimental feature because:

• There is something off with the math for reflection panning. It is not easily audible but you will notice if you print the panning vector's values.
• Reflection panning reveals that the check if a ray collides with a collision mesh is not entirely reliable, resulting in some instabilities, although this has been made less obvious with the smooth interpolation between reverb changes

Here is what you can currently listen for:

@ohlidalp
I created my own very simple map for audio testing:
simple2-audio_terrain.zip

• ensure efx reverb is enabled and drive/walk into the shops and notice how the reverb adjusts. Even the offices have their own reverb presets.
• ensure efx reverb is enabled and drive under the bridge to hear the city_underpass reverb preset that was defined using the mesh of the sign (use the collision debug -> Show event boxes features to see where the reverb zone is)
• ensure the obstruction filter is enabled, start a vehicle, leave the engine running and move the listener (which is the camera) behind walls to hear the effect of the obstruction filter
• ensure efx reverb is enabled, open the top bar menu, go to settings and set any supported reverb preset of your choosing in the field Force Listener EFX Preset to hear its effect anywhere in the world (set to empty string to reset).
• zoom out to hear high-frequency sound drop in volume faster than low-frequency sound
• ensure early reflections panning is enabled, set a default EFX preset of your choosing, and drive close to a wall to hear the direction of early reflections being adjusted and their volume being boosted (experimental)

For the underwater effects, you would have to use another map, though. Simply move the camera between above water and underwater and notice how the audio changes as well as the smooth transitions between the effects.

Some other features that would theoretically possible with OpenAL that I don't see within the scope of this PR:

• rendering of multiple reverb environments at once. Currently, only the environment the listener is in, is rendered.
  • example: consider standing in a quarry and a truck with a running engine is parked in a truckshop inside the quarry. In this scenario, the engine sound would need to have the truckshop reverb preset being applied, and the wet sound would have to be panned to toward the truckshop. The reverb preset of the quarry would have to be applied on top.
• simulating the diffraction path of audio for more realistic positioning and obstruction of obstructed sources
• when the listener is inside a truck, an obstruction filter should be applied to the direct path of outside audio sources and an occlusion filter should be applied to the wet path. However, this would require to know whether the listener is inside a closed cabin or not and I don't think this information is easily accessible in the engine.
• directed sound emissions from exhausts, directed truck horns and sirens using OpenAL cones
• beam-traced early reflections (not natively supported by the AI but it should be possible to do the tracing inside RoR's engine and use OpenAL to place the audio sources. The OpenAL Soft engine could then do the heavy lifting of rendering them with HRTF/Ambisonics and applying late reverb)
• applying unrealistic audio effects, such as flanger or chorus to certain sounds. Compare e.g. with the arcade racer Blur, where being hit by a power-up applies a temporary filter to the music to provide audible feedback by which power-up one was hit.

[ohlidalp]

Wow, thanks for the elaborate write-up and don't worry about responsiveness, I myself react very irregularly.

ODEFs can omit mesh if you set mesh name to "none", see

rigs-of-rods/source/main/terrain/TerrainObjectManager.cpp

Line 609 in 0450b2f

if (odef->header.mesh_name != "none")

- this trick works on many places and is largely undocumented. This means you can pre-define ODEFs with just a reverbzone.

Unfortunately, there's currently no way to scale ODEFs from TOBJ. I was going to add this functionality anyway, for i.e. varied vegetation.

Either way, an usable terrain editor would be a tremendous help here. We have a crude terrain editing mode, but it currently only writes a logfile insted of actually editing terrain files. I have an active WIP branch where I added this missing piece: https://github.com/ohlidalp/rigs-of-rods/tree/terrn_export

[ohlidalp]

I'll create a pull request with the new terrain editor during the coming week, then we can work the reverb and obstruction feats into an editor UI, which will both showcase and document it visually.

[Hiradur]

ODEFs can omit mesh if you set mesh name to "none"

Neat trick, I didn't notice this. Thanks for the info, this makes using collision boxes much more viable.

I'll create a pull request with the new terrain editor during the coming week, then we can work the reverb and obstruction feats into an editor UI, which will both showcase and document it visually.

Sounds great.
As far as the obstruction filter is concerned, it checks against obstruction by terrain, collision boxes, collision meshes, and AABBs of actors. Unlike reverb environments, it doesn't need special entities; hence there wouldn't be a need for UI elements in the terrain editor.
In theory, realism could be improved by making the obstruction filter parameters dependent on acoustic material properties and thickness. I don't see this within the scope of this PR though since it should be done with diffraction path simulation for more realistic behavior. Many older games just use line of sight checks and apply a simple obstruction filter if an obstacle was detected, which is also what is done in this PR at the moment. Currently there is only one filter parameter set for every obstruction scenario. One could create more for different scenarios (e.g. there is probably a difference if a sound is obstructed by a hill, a house or a car). For the most realistic behavior, however, the parameters should be determined dynamically based on the diffraction path properties.

[cryham]

Hi.
I have integrated this code into SR3 now, on branch sound 🎉. It's still WIP though, I got some stuff commented out and I just started restoring SR3 sound using it.

I have a few comments, about this PR's code style. Obviously, I'm not a RoR developer, so it's up to you.

• inconsistencies with if( vs if ( (I'd say with space is better), also for( vs for (
• some variables start with m_ and the rest don't (I'd say m_ isn't needed)
• some code is densely packed, few empty lines would be better, e.g. under // OpenAL EFX stuff
• less readable return statements in same line e.g. bool isDisabled() { return disabled; },
  few methods are very packed (getSoundName, getSoundPitch, getSound).
  I'd either move all to next line (unless for short get/set), or put 2 spaces before { and } (in SR3 code)
• some very long names e.g. m_listener_is_inside_the_player_coupled_actor
• generally some methods are big, I usually put 2 empty lines at start and end (in SR3 code)

I fixed all those and more in this stuntrally/stuntrally3@3588cad
but surely many things there are related to SR3 code only. E.g. I don't have CVars, scripts etc.

[Hiradur]

Great to see it being tried out in SR3 already and I appreciate your feedback. I still intend to do some cleanup and refactoring before gettings this merged.

Onto your suggestions:

inconsistencies with if( vs if ( (I'd say with space is better), also for( vs for (

I should fix that.

some variables start with m_ and the rest don't (I'd say m_ isn't needed)

All code I added should follow the RoR coding style guidelines, which say //STYLE: privates are prefixed m_. Since a lot of the audio code predates my work, there is a mix of styles, unfortunately. This problem also exists elsewhere in the RoR codebase.

some code is densely packed, few empty lines would be better, e.g. under // OpenAL EFX stuff

I'll take a look if it can be improved. I'll probably just copy your improvement.

less readable return statements in same line e.g. bool isDisabled() { return disabled; }

The coding style guidelines have an example of this, which I followed since I like it for simple getters:

int  GetMemberValue() { return m_private_member_variable; }  //STYLE: Getter function -> `Get` in name

few methods are very packed (getSoundName, getSoundPitch, getSound).
I'd either move all to next line (unless for short get/set), or put 2 spaces before { and } (in SR3 code)

The mentioned functions aren't part of this PR and cleaning them up should be left for another PR.

some very long names e.g. m_listener_is_inside_the_player_coupled_actor

I prefer long variable/function names over adding comments to prevent comments getting outdated and make the code read more like natural language (see also clean code principles). I could change it, if desired.
I eliminated this particular variable in the last commits since it was used for enabling the obstruction filter when the listener was inside a truck/actor. However, since RoR also has trucks with open cabins, the obstruction filter shouldn't be enabled unconditionally. I saw you renamed it to listener_inside_player. If you want to keep the variable I recommend renaming it since it's really about whether the listener is inside a truck/actor.

[cryham]

Cool. I am not a developer here, just went through the code BTW. Definitely ohlidalp has the say, and I forgot there is coding style guidelines, so better to keep to it.

Anyways I'm not sure I will use this code, it seems buggy right now, not sure why.
I may keep my previous Sound code fork of RoR, which I modified and it works fine (I added reverb, it has ogg, 2D sounds (for Hud and for all in Split Screen), from config file I also removed triggers etc).
I also don't see in any of them, the 2 features which I'd like most now: dynamic sounds and ambient sound (ogg).
I saw in this video that e.g. hit sounds (for hitting barrels, boxes in SR3) would need to be deleted/removed after playing and there also should be a pool for few at most.
I don't see anything like this in RoR, remove is only done after deleting actor, and on load all needed sources are created, no dynamic sounds on play. This isn't an issue for RoR just for SR3. But yeah I wrote that since you probably know (RoR's) sound code best now.

[Hiradur]

Anyways I'm not sure I will use this code, it seems buggy right now, not sure why.

Can you elaborate? I didn't notice any bugs in RoR.

I also don't see in any of them, the 2 features which I'd like most now: dynamic sounds and ambient sound (ogg). I saw in this video that e.g. hit sounds (for hitting barrels, boxes in SR3) would need to be deleted/removed after playing and there also should be a pool for few at most.

I see buffer management as a different topic. For this PR I only want to focus on environmental audio implemented via OpenAL EFX. I didn't pay too much attention to the already existing buffer management but I think your assessment is correct and that there is room for improvement, especially for ambient sounds.

[cryham]

No I meant it's buggy in SR3, after my somewhat reworked version. And different code in SR3 compared to RoR. I don't have such setup as in RoR with all SoundTriggers, ModulationSources in ActorSpawner.cpp, I don't have that and I just set gain and pitch myself in update. Could be something bad with this in SR3.