Skip to content

Commit

Permalink
feat: Celery worker concurrency setting
Browse files Browse the repository at this point in the history
This allows the user to configure how many Celery workers are spawned
independently of how many CPUs there are in the system. The default is
to spawn as many workers as there are CPUs, which in some cases can
consume too many resources.

(The setting should be particularly useful to people running Tutor for
development on Linux machines, where reducing the concurrency to "1" can
reduce RAM usage significantly.)
  • Loading branch information
arbrandes committed Feb 29, 2024
1 parent 13c420c commit 1dd4b5a
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 4 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- [Feature] Allow the user to configure Celery worker concurrency. (by @arbrandes)
5 changes: 5 additions & 0 deletions docs/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,11 @@ This defines the version that will be pulled from just the Open edX platform git

By default, there are 2 `uwsgi worker processes <https://uwsgi-docs.readthedocs.io/en/latest/Options.html#processes>`__ to serve requests for the LMS and the CMS. However, each worker requires upwards of 500 Mb of RAM. You should reduce this value to 1 if your computer/server does not have enough memory.

- ``OPENEDX_LMS_CELERY_WORKERS`` (default: ``"0"``)
- ``OPENEDX_CMS_CELERY_WORKERS`` (default: ``"0"``)

This sets the number of Celery workers at startup. Defaults to the number of CPUs detected on the system (represented by "0"). If your system's RAM is limited, you may need to set this to a low integer value in order to conserve it. (On development deployments, it is suggested to use "1" for both.)

- ``OPENEDX_CELERY_REDIS_DB`` (default: ``0``)
- ``OPENEDX_CACHE_REDIS_DB`` (default: ``1``)

Expand Down
2 changes: 2 additions & 0 deletions tutor/templates/config/defaults.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ OPENEDX_AWS_ACCESS_KEY: ""
OPENEDX_AWS_SECRET_ACCESS_KEY: ""
OPENEDX_CACHE_REDIS_DB: 1
OPENEDX_CELERY_REDIS_DB: 0
OPENEDX_CMS_CELERY_WORKERS: 0
OPENEDX_CMS_UWSGI_WORKERS: 2
OPENEDX_LMS_CELERY_WORKERS: 0
OPENEDX_LMS_UWSGI_WORKERS: 2
OPENEDX_MYSQL_DATABASE: "openedx"
OPENEDX_MYSQL_USERNAME: "openedx"
Expand Down
4 changes: 2 additions & 2 deletions tutor/templates/k8s/deployments.yml
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ spec:
containers:
- name: cms-worker
image: {{ DOCKER_IMAGE_OPENEDX }}
args: ["celery", "--app=cms.celery", "worker", "--loglevel=info", "--hostname=edx.cms.core.default.%%h", "--max-tasks-per-child", "100", "--exclude-queues=edx.lms.core.default"]
args: ["celery", "--app=cms.celery", "worker", "--loglevel=info", "--hostname=edx.cms.core.default.%%h", "--concurrency={{ OPENEDX_CMS_CELERY_WORKERS }}", "--max-tasks-per-child", "100", "--exclude-queues=edx.lms.core.default"]
env:
- name: SERVICE_VARIANT
value: cms
Expand Down Expand Up @@ -250,7 +250,7 @@ spec:
containers:
- name: lms-worker
image: {{ DOCKER_IMAGE_OPENEDX }}
args: ["celery", "--app=lms.celery", "worker", "--loglevel=info", "--hostname=edx.lms.core.default.%%h", "--max-tasks-per-child=100", "--exclude-queues=edx.cms.core.default"]
args: ["celery", "--app=lms.celery", "worker", "--loglevel=info", "--hostname=edx.lms.core.default.%%h", "--concurrency={{ OPENEDX_LMS_CELERY_WORKERS }}", "--max-tasks-per-child=100", "--exclude-queues=edx.cms.core.default"]
env:
- name: SERVICE_VARIANT
value: lms
Expand Down
4 changes: 2 additions & 2 deletions tutor/templates/local/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ services:
environment:
SERVICE_VARIANT: lms
DJANGO_SETTINGS_MODULE: lms.envs.tutor.production
command: celery --app=lms.celery worker --loglevel=info --hostname=edx.lms.core.default.%%h --max-tasks-per-child=100 --exclude-queues=edx.cms.core.default
command: celery --app=lms.celery worker --loglevel=info --hostname=edx.lms.core.default.%%h --concurrency={{ OPENEDX_LMS_CELERY_WORKERS }} --max-tasks-per-child=100 --exclude-queues=edx.cms.core.default
restart: unless-stopped
volumes:
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
Expand All @@ -177,7 +177,7 @@ services:
environment:
SERVICE_VARIANT: cms
DJANGO_SETTINGS_MODULE: cms.envs.tutor.production
command: celery --app=cms.celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --max-tasks-per-child 100 --exclude-queues=edx.lms.core.default
command: celery --app=cms.celery worker --loglevel=info --hostname=edx.cms.core.default.%%h --concurrency={{ OPENEDX_CMS_CELERY_WORKERS }} --max-tasks-per-child 100 --exclude-queues=edx.lms.core.default
restart: unless-stopped
volumes:
- ../apps/openedx/settings/lms:/openedx/edx-platform/lms/envs/tutor:ro
Expand Down

0 comments on commit 1dd4b5a

Please sign in to comment.