predictoor.ai 5m feed is down often, bc pdr-websocket fails to renew subscription

[KatunaNorbert]

Background / motivation 1 - Oct 2023

Norbert: Websocket fails when tries to renew the subscription and has to be manually restarted

Background / motivation 2 - Feb 2024

From this slack thread on Feb 19, 2024

Trent: predictoor.ai is giving a "?" for BTC 5m prediction. Any ideas why?

Trent: BTW it's not doing it for 1h prediction. (Good.) (edited)

Mustafa: I guess the websocket app has an issue

Mustafa: Can you restart it @norbert

Norbert: Yes, restarted it. It fails sometimes when trying to renew the subscription

Berkay: there's a cronjob to restart it every hour yet we frequently encounter this issue

[trentmc]

We should take steps to make this more reliable, now that more eyes are on predictoor

Perhaps simply re-starting every 10min? @trizin

[KatunaNorbert]

If we are restarting it at the same time each day then the cron is probably trying to restart too early and the subscription is not expired yet. What we could do is keep track of the time it was restarted last time and add +5min to the next time so it gives some extra time for the subscription to expire before restarting.

[KatunaNorbert]

Another solution might be switching the rpc from wss://sapphire.oasis.io/ws to the Chainstack rpc which is more reliable

[trizin]

or pdr-websocket could manage the subscription and buy a new one when it expires.

[KatunaNorbert]

It does but it's not working.. was spending some time with @kdetry on fixing it but didn't find a solution so decided to rely on the cron approach. We could take another look again if the cron approach it's not working.

[trizin]

I updated the cronjob to restart it every 15 minutes. The startup time is around 10 seconds so we will have a downtime of 10 seconds every 15 minutes until we come up with a better solution, let's hope this works for now 🤞

[idiom-bytes]

It does but it's not working.. was spending some time with @kdetry on fixing it but didn't find a solution so decided to rely on the cron approach. We could take another look again if the cron approach it's not working.

Whether the cron job is working or not, having an unstable backend service isn't good. I agree with @trizin that we need to address this at the service-level (code), not system-level (cron).

There is a PR that's "ready to go", that "might fix this issue" but I can't quite validate as there are no logs/errors/trace/context as to the problem explained in the ticket... so, should the PR be getting merged? It's been there since November.

[KatunaNorbert]

I don't think that PR fixes the issue. As far as I know it's not solved that's why it wasn't merged.
The issue I encountered is either the subscription has ended and we try to get the prediction or has not ended and we try to renew it. To sum it up the subscription renewal is well timed.

[trentmc]

I don't think that PR fixes the issue. As far as I know it's not solved that's why it wasn't merged.

Correct, this is what Mustafa said a few days ago (in slack I think?)