Fix sending local state to other participants

[danxuliu]

Fixes #3358

Requires #4536

First of all, sorry for the long, long delay and thanks a lot to everyone who provided information on the issue.

Note that the description below treats #4536 as part of this pull request (so the behaviours of the code are described in a pre-4536 state).

The problem started to happen in a0fa841 because, as described in the first part of #3358 (comment), the video is set as not available (or, rather, not known if available) when the connection state changes to NEW or CHECKING. This was done following the implementation of the WebUI; the idea is that until the connection is completly established it is not really known if the video is enabled or not, even if the stream has a video track (it could be a disabled video anyway, so the legacy behaviour to set it available if there is a track should be removed, but that is a different story), so the other client should explicitly provide that information once the connection is established (turns out that, although all that is correct with data channels, with signaling messages it would be possible to provide the state even before the peer connection is established, so this might need to be adjusted in the future). Although the change itself caused the video to no longer be shown (in some cases) the problem is in the sending side of the Android app.

As also described in the second part of #3358 (comment) sendInitialMediaStatus would not do anything when called on a subscriber peer connection, both because it does not have a local stream and because, even if it had, it is not the right connection to send data channel messages when using the HPB, as they must be sent in the publisher connection. But independently of that sendInitialMediaStatus will not be called anyway in most cases; it will not be called when the data channel is open, because for subscribers the observer is registered after the data channel was open already, and it will be randomly called when the connection state changes to CONNECTED, because it depends on hasInitiated, which is meant to be used to initiate the peer connection without HPB if the local participant session ID is "higher" than the remote participant session ID, but is not needed and, moreover, is wrong when using the external signaling server as in that case the comparison is made between the Nextcloud session ID and the signaling session ID.

In the case of publisher connections the data channel message will be also randomly sent when the connection state changes to CONNECTED due to the comparison between Nextcloud session ID and signaling session ID, although it will be always sent when the data channel is open. This means that if participant A is in a call and participant B joins then the state of participant A will not be sent to participant B (because it will be sent on the subscriber connection), but the state of participant B will be sent to participant A (because it will be sent on the publisher connection when the data channel is open). But then... why is the video of participant B visible for participant A only sometimes?

The reason is that when the HPB is used the connection is not established directly between both clients, but between the clients and the HPB. Therefore, when the data channel is open for participant B it is open between participant B and the HPB, but the data channel may not be open yet between participant A and the HPB. If the message is sent at this point it will reach the HPB, but it will not be relayed to participant A. Due to that reason the WebUI does not send just a single state message after the connection is established, but several ones with an exponential backoff to "ensure" that the state is received by the other participants even if their subscriber connection takes a while to be established.

But now the question is, why is video coming from the Android app always shown in the iOS app and the WebUI, despite all of the above?

The iOS app shows the video by default, so if a video track is sent by the Android app it is shown even if the Android app does not send a data channel message to enable it.

In the case of the WebUI received videos are disabled by default, but they are automatically enabled if it is detected that a video is being sent (which is a legacy behaviour and should not work like that, video should be shown only if explicitly enabled, but that is a different story). Therefore, again the video track sent by the Android app is shown even if the Android app does not send a data channel message to enable it.

In order to solve the issue, this pull request introduces the LocalStateBroadcasterXXX helper classes that take care of sending the local state to the other participants as needed. That is:

• Send the current state to another participant in the call when that participant joins (if the local participant is the one that joined all the other participants joined from its point of view, so the state is sent to all of them); if the HPB is used the state is sent several times with an exponential backoff to solve the problem explained above about the connection not being established yet in the other end
• Send the state changes to all the participants in the call; in this case a single message is sent
  • If the state is changed but a connection was not established yet the changed state should be eventually received through the initial exponential backoff (as it sends the current state, not the original state)

Before this pull request the state was sent only through data channels; now the state is sent through signaling messages too, as it is expected by other clients (in the past data channels were found to be problematic, but the signaling messages were more reliable and convenient, so that is why the state was moved to be sent also through signaling messages; the only exception is the speaking state, which, due to its potential frequency and being not so relevant, was left only as data channel messages to avoid hammering the database when the HPB is not used). Note that this should be backwards compatible with older Nextcloud versions, as if any signaling message was not handled it was just ignored, so there is no problem sending them.

The LocalStateBroadcaster (or, rather, its subclasses) also takes into account the differences between the internal or the external signaling server in order to send the messages using the appropriate connection.

All this is almost the same done in the WebUI, with some little improvements:

• In the WebUI, when another participant joins the call, the message is sent to all participants rather than only to the participant that joined.
  • This is still the case in the Android app with Janus and data channels, but there is no way around that.
• In the WebUI the state is sent again whenever the connection changes to the connected state. Therefore the state is sent again without need when a connection is temporarily interrupted (changes to disconnected and then back to connected). However, in the case of an ICE restart that would also send the state again, which would happen without HPB after the connection failed.
• In the WebUI the state is not sent if the connection changes to the completed state; although typically the connection will go through the connected state before reaching the completed state it is possible to go directly from checking to completed, and in that case the state would not be sent.
• In the WebUI, when the HPB is used, the state starts to be sent when a receiver connection is established with the other peer. Here the state is sent as soon as the remote participant is found, as the remote participant might already have a receiver connection to the local participant before the local participant has a receiver connection to the remote participant (although in most cases it should not make any difference, as establishing connections when the HPB is used is usually pretty fast).

Despite the improvements this approach is far from perfect and there is still an excessive amount of initial messages sent with the HPB due to the repeated sending. But this is tracked in nextcloud/spreed#8549 and it will need to be solved across all the clients at the same time.

Follow ups

• Send nick and raised hand state in LocalStateBroadcaster
• Send status also when no peer connection will be established (right now it is not a problem because only media state is sent, but it will be with non-media state, like the nick and raised hand)

How to test

• Setup the HPB
• Start a call with the Android app
• Join the call with the Android app in another device

Result with this pull request

The video of the other participant is visible

Result without this pull request

The video of the other participant may or may not be visible (most likely it will not be)

[danxuliu]

/backport to stable-20.1

[github-actions]

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/4558-talk.apk



To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud Talk app.

[github-actions]

Codacy

Lint

Type	master	PR
Warnings	158	158
Errors	71	71

SpotBugs

Category	Base	New
Bad practice	6	6
Correctness	222	222
Dodgy code	71	71
Internationalization	3	3
Malicious code vulnerability	3	3
Performance	4	4
Security	1	1
Total	310	310