fix: improve celery performance

[Ian2012]

Description

This removes the gossip and mingle features to improve Celery's performance and reduce the number of messages sent to Redis. It also increases the heartbeat interval to 60s from 2s.

• without-gossip: Based on the official celery docs, workers are subscribing to worker-related events like heartbeats, and can detect if any worker goes offline, Celery by default does nothing with this information, but this opens the door for task rerouting or restarting workers when they crash (not implemented). In the context of Open edX, there isn't any implementation that depends on this feature, and the Celery changelog only shows fixes for this flag, also this scales to n^2 increasing dramatically network traffic which isn't needed.
• without-mingle: It syncs the revoked tasks when a worker in the same cluster starts or restarts, avoiding rerunning revoked tasks. I don't have strong arguments against this flag, however, it's highly recommended by different operators to remove it.

For the heartbeat, maybe 20s is a better default in case of network issues or Redis downtimes. However, keep in mind that this heartbeat does nothing. If you disable the heartbeat the workers still will detect if the broker goes down and start the reconnect process, so we should consider removing it completely.

References:

https://www.cloudamqp.com/blog/python-celery-and-rabbitmq.html
https://ankurdhuriya.medium.com/understanding-celery-workers-concurrency-prefetching-and-heartbeats-85707f28c506

[Ian2012]

Other setting we should take into account:

• -O fair: Prevent long-running tasks from blocking neighbour workers (in-process workers) See Common Issues Using Celery, Three quick tips from two years with Celery and The problems with Celery. This is especially true now that we default to only one worker per service which handles all types of Celery tasks. This could be mitigated by adding more workers for different queues, but I'm not sure if that's in the roadmap and this has already been implemented in tutor-contrib-celery.

[regisb]

I agree that we should add --without-mingle and --without-gossip. But it seems that there is a misunderstanding about the heartbeat. If I understand this article correctly (which has more information than the official docs...), the heartbeat is not to detect whether the broker is down, but for the broker to detect when a worker is down. When a worker is down, then its tasks are reassigned to another worker. Surely, we don't want to wait 1 minute until tasks are reassigned. I'm not sure whether we should increase from 2s. If we do, we shouldn't go beyond 5-10s (IMHO).

Concerning -O fair: @igobranco was suggesting here that we add --prefetch-multiplier=1. This would resolve the issue without adding -O fair, right?

[Ian2012]

If there is a concern about the heartbeat, we can use the default for now.

Yes, setting the prefetch-multiplier to 1 should resolve that issue too, this setting is what improved the performance on a large instance and helped to not lose tasks when workers are OOM Killed or pods are deleted forcefully, this with ACKS_LATE enabled, however, this would require celery tasks to be idempotent, such as course indexing, grade calculation or reports, however I think that's a large discussion that's out of scope for this PR. See the docs for more information.

I will update the PR with the new defaults.

[regisb]

Can you please add a changelog entry? https://docs.tutor.edly.io/tutor.html#contributing

[regisb]

Alright, let's merge this!