Iox #325 deadlock in shared mem mutex

[marthtz]

Introduce a robust, recursive mutex.
If recursive is enabled, the same thread, which has already locked the mutex, can lock the mutex again without getting blocked.
If robust is enabled, a process or thread can exit while having the lock without causing an invalid mutex. In the next lock call to this mutex the system recognizes the mutex is locked by a dead process or thread and allows the mutex to be restored.

[marthtz]

Robust mutex is not supported on Win/Mac. We need to find a solution!

[mossmaurice]

Windows and Mac builds are broken :-( IMHO best solution would be to add empty wrappers for e.g. pthread_mutexattr_setrobust in the iceoryx_utils/platform/*/include/iceoryx_utils/platform/pthread.hpp

[budrus]

This will prevent a deadlock and makes the mutex consistent again.
But how to deal with the problem that the process that died while holding the mutex could leave some shared data objects in an inconsistent state. The locking will work now but depending on the critical section and the involved shared data objects this opens the road to undefined behavior.

What's your feeling on this @marthtz @sculpordwarf ?

[sculpordwarf]

Windows: WAIT_ABANDONED
Mac: EDEADLK -> you need to make an example an step in with the debugger, there isn´t enough docu

[marthtz]

What's your feeling on this @marthtz @sculpordwarf ?

You're right @budrus, we're getting into difficult terrain here. All shared data protected by an invalid mutex should be treated as corrupted. Off the top of my head we could do two things when an invalid mutex has been detected. We could either re-init these data blocks or we throw an error (and may shutdown gracefully).

[elfenpiff]

• Fix Mac OS behavior so that is consistent with linux (if possible)
• Find robust mutex solution for windows (if possible)
• Add additional unit tests for mutex

[elfenpiff]

Please add noexcept

[elfenpiff]

I think this call is not supported for Mac OS and therefore has to be reimplemented and I am not aware if Windows is supporting robust mutexes?!

But maybe you can follow up this link: https://stackoverflow.com/questions/24946886/is-there-pthread-mutex-robust-equivalent-in-mac-os-x or contact me and we can work together on a solution.

[elfenpiff]

Test cases for recursive and robust mutexes are missing.
Like:

• normal creation and usage should work for recursive, robust and recursive and robust mutex
• multiple locks should not lead to deadlock in a recursive mutex in a single thread
• multiple locks should cause a deadlock in a recursive mutex from different threads (start multiple threads, let the first thread deadlock and then unlock the mutex in the thread it was locked before)
• robust mutex cleanup should work ... here we maybe have to think how we can test it. @marthtz do you have an idea?

[marthtz]

Isn't it the case that with the introduction of the building blocks / new API the number of mutexes is reduced?! So, maybe the robust approach is a bit over the top?

[budrus]

You're right @budrus, we're getting into difficult terrain here. All shared data protected by an invalid mutex should be treated as corrupted. Off the top of my head we could do two things when an invalid mutex has been detected. We could either re-init these data blocks or we throw an error (and may shutdown gracefully).

So we would have to provide this information and not silently make the mutex consistent again.

[budrus]

Isn't it the case that with the introduction of the building blocks / new API the number of mutexes is reduced?! So, maybe the robust approach is a bit over the top?

They are reduced but not completely eliminated

[budrus]

My proposal after some discussion with @sculpordwarf :

• Windows and Mac. No support of robust mutex so far. We would fall back to the deadlock situation as we have today
• We need the information whether a mutex was held by a dead application and the lock() call made it consistent again. The user of the mutex needs to be able to decide what the consequence is. I.e. if the data protected by the mutex could be inconsistent or hast to be reset etc. The idea is to introduce a was_inconsistent() method that can be called after lock
• For the old ports we came to the conclusion that a robust mutex is fine for now. For the locking policy in the new building blocks we start with a non-robust mutex and first have to analyze if and how the robust mutex could help
• For tests we think it should be possible to have a global mutex and several threads to force the interesting situations

[elfenpiff]

@marthtz is this PR dead? Can we remove it?

[elfenpiff]

@marthtz I assume that this PR is dead and remove it. If I am mistaken just reopen it.