Ensuring Scalability: Redis Connection Management in Production

[anwar-al-jahwari]

Hello @geofmureithi,

Thank you for creating such an excellent library! I truly appreciate your hard work, and I'm excited to integrate it into our production setup.

To give you a bit more context, I’m building a WhatsApp service expected to handle significant traffic. The architecture involves a webhook that receives messages from Meta servers, which are then routed through a load balancer and a WAF before reaching multiple /webhook/messages endpoints. Each of these endpoints enqueues messages as jobs into Redis. We have multiple job processors that then handle these jobs and respond to customers through WhatsApp.

Could you clarify whether the Redis connection in your library uses connection pooling? If not, would this impact performance or scalability in such a high-load environment? I'm particularly interested in ensuring that our Redis interactions remain efficient given the high volume and rapid queuing required.

Thanks again for your fantastic work, and I’d appreciate any insights you could share based on this setup.

[geofmureithi]

Hey, good to hear you are checking out apalis and I appreciate your questions they are pretty important.

1. Does apalis support connection pooling in redis? The answer is yes with an asterisk. Basically RedisStorage only cares if you have implemented aio::ConnectionLike. We already have an example that uses deadpool for pooling:
   

   apalis/examples/redis-deadpool/src/main.rs

   Lines 21 to 30 in f6442fb

   	let cfg = Config::from_url("redis://127.0.0.1/");
   	let pool = cfg.create_pool(Some(Runtime::Tokio1)).unwrap();
   	let conn = pool.get().await.unwrap();
   	let mut storage = RedisStorage::new_with_config(conn, config);
   	// This can be in another part of the program
   	produce_jobs(&mut storage).await?;
   	let worker = WorkerBuilder::new("rango-tango")
   	.data(pool)
   	.backend(storage)

   
   If you want more fine grained approach, feel free to find crates based on the redis crate here or implement the trait on your own and let me know how it goes.

2. As long as your processing functions are pure*, you can also scale well horizontally with running several instances of the same binary worker. Eg with heroku/dokku::

heroku ps:scale web=1 worker=4

Let me know if that answers your question.

[anwar-al-jahwari]

Thank you so much for your input—I really appreciate it, and I think things are going really well!

Could you explain a bit more about what you mean by the "pure" aspect?

Also, what’s the difference between setting the worker count to 4 versus running 4 separate instances of the worker? Does each worker create a separate connection?

[geofmureithi]

Could you explain a bit more about what you mean by the "pure" aspect?

By pure, I mean a that it doesn't matter which worker takes the job, the result of the processing should be the same given the same inputs. Like in functional programming.

Also, what’s the difference between setting the worker count to 4 versus running 4 separate instances of the worker?

Currently there is not much difference. The worker count feature will be replaced by concurrency in #428.
This means all you need to set is concurrency of 4 and there will be a max of 4 workers and 4 jobs being processed at any time.

[geofmureithi]

That said if you want to provide a pool for each worker, you will have to setup the storage and worker with Monitor::register several times. This may improve perf, but I dont believe by much.
If you configure your storage well, ie buffer_size, poll_interval then you should have an optimal experience.

[anwar-al-jahwari]

Thank you i will try different approaches and check the metrics and come back with a result if any.