Subscriptions stop working after redis reconnect #409
Comments
|
Thank you for submitting this issue! We, the Members of Meteor Community Packages take every issue seriously. However, we contribute to these packages mostly in our free time. If you think this issue is trivial to solve, don't hesitate to submit Please also consider sponsoring the maintainers of the package. |
|
Okay, it seems that the We just experienced the issue again, and no new lines are logged for [RedisSubscriptionManager] Received event: "u" to "production.tables". |
|
It happened again, and more info. So we received the last update at 10:16:55.620 AM So after 20 minutes it noticed that the redis connection had timed out. After which it established a new connection at 10/23/2024, 10:36:12.601 AM, and things work again |
|
FWIW, this sounds like #365. Unfortunately I've never found a fix; I just have to restart my Meteor servers occasionally. It would be great to get more attention on this and finally track down a solution. |
|
Yes, that indeed sounds like the same problem indeed. I have not been able to find more information as of now. It seems that we have two different problems, one where the TCP connection times out, after ~20 minutes(!), after which it reconnects. This is most probably an Azure issue. And the second problem is that when a reconnect happens sometimes it does not "activate" the existing subscriptions again. That sounds like #365. I've spent most of my time today hunting the first problem. But when I get more info on the second problem I'll update here. Thanks for your comment. |
|
Hey @rikvdlooi, I'd like to know if you have any news on this issue. I'm also using Azure with Container Apps, and I'm having a similar problem where the subscription initializes and stops immediately after. Then, it tries to initialize again but never gets ready or returns any data. I have a "testing" system running on Digital Ocean, and it works fine there, using precisely the same image and setup. I appreciate what you've done already on this!
|
|
So an update from our side. I've not been working on this issue for some time, but my colleagues have been testing more. It seems that Azure ContainerApps is doing something funky with it's connections. We're not sure what is happening, we've Azure support but they weren't able to help any further as well. We've implemented ioredis, and configured pings + short timeouts for connections and reconnections, and things seem to be working okay now. It now detects a lot better/faster that the connection is (slightly) broken, and does a reconnect with a new connection. We think that linux does not properly detect that the connection is (slightly) broken, and it takes a long time before it notices that the connection can not be used. But with the workarounds configured (short timeouts + ping) in ioredis this seems to be handled. My colleague (@jdgjsag67251) will make a pull request this week with the changes we made. Be aware that we use an external publisher, so we only use this redis-oplog package to read from redis. |
|
Hi, our changes can be found here: https://github.com/jdgjsag67251/redis-oplog/tree/ioredis |
We are running our meteor app on Azure with Container Apps. We are having a separate issue with Azure where the connection to Redis is closing and reconnecting a lot of times. We are in contact with Azure people to fix that issue.
However after such a reconnect, sometimes (?) the subscriptions are not working anymore.
Description of what we see:
Updating the position is just an example. Any other updates also do not seem to work. But this is just the most easy thing to test for us.
Because this issue does not happen always, it's hard to reproduce it, and figure out the conditions of why this is happening. We're still actively debugging this ourselves. I'm posting here to check if maybe someone has experienced something similar before, or can point us in the right direction. Or maybe help us isolate the issue, because our logs are quite noisy at the moment.
We have enabled debug, and through the logs it seems that the update to tables is actually received. However the update to the specific table id is ignored (once).
Config
Our config
We've set
externalRedisPublisherto true because we are using changestream-to-redis by @radekmie.That seems to be working fine, as through
redis-cliI can confirm that the changes are published.Logs
Extra logging
The
Ignored (no subscribers)is a custom debug we added in theprocessmethod of theRedisSubscriptionManager.jsfileThe text was updated successfully, but these errors were encountered: