From 66df3dc25bb2da7cf26d1c454e232cee0612e32b Mon Sep 17 00:00:00 2001 From: michaelroytman Date: Mon, 2 Oct 2023 14:45:57 -0400 Subject: [PATCH] docs: local event bus development --- README.rst | 101 +++++++++++++++++++++++++++++++++++++++++++-- docker-compose.yml | 4 +- 2 files changed, 100 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 34a9bf03..35707d68 100644 --- a/README.rst +++ b/README.rst @@ -20,8 +20,8 @@ Documentation Development Workflow -------------------- -One Time Setup -~~~~~~~~~~~~~~ +Local Development Set Up +~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: # Clone the repository @@ -43,9 +43,24 @@ One Time Setup # Run edx-exams locally python manage.py runserver localhost:18740 --settings=edx_exams.settings.local +Devstack Set Up +~~~~~~~~~~~~~~~ +.. code-block:: + + # Clone the repository + git clone git@github.com:edx/edx-exams.git + cd edx-exams -Every time you develop something in this repo -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + # Start LMS in devstack from your local devstack directory + make dev.up.lms + + # Return to the edx-exams repo directory and provision the edx-exams containers + bash provision-edx-exams.sh + +You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers. + +Development Workflow +~~~~~~~~~~~~~~~~~~~~ .. code-block:: # Activate the virtualenv @@ -79,6 +94,84 @@ Every time you develop something in this repo # Open a PR and ask for review. +Event Bus Set Up +~~~~~~~~~~~~~~~~ + +The ``edx-exams`` service uses the Open edX event bus to publish events relating to the exam attempt lifecycle and +others important exam events. These Open edX events are emitted by the service and pushed onto the event bus. Downstream +services, like the LMS, receive these events and implement downstream effects of these events. For more details, +please see `Implementation of Event Driven Architecture for Exam Downstream Effects`_. + +Currently, the event bus is only supported in environments running Docker containers, like `devstack`_. This is because +the interactions between services on the event bus is implemented in the devstack networking layer. + +In order to run the event bus locally, follow these steps. These steps assume that you both have `devstack`_ running and +that you are running the ``edx-exams`` Docker container, as described in the Devstack Set Up section. These steps +describe how to install and run the Kafka-based event bus. + +1. In a ``requirements/private.txt`` file, add the following Python package. These requirements are necessary for the + Kafka-based event bus. They are not included as a part of the standard set of requirements because installation of + confluent_kafka poses issues for users of Tutor on M1 Macs, which includes many users in the Open edX community. + For more details, please see `Optional Import of Confluent Kafka`_. + + + .. code-block:: + + confluent_kafka[avro,schema-registry] + +2. Install the application requirements to install ``confluent_kafka``. + + .. code-block:: + + # Shell into the application Docker container + make app-shell + + # Install requirements + make requirements + +3. Follow the `manual testing`_ instructions to set up the Kafka-based Open edX event bus in the service that contains + the event handler(s) for your event(s) - for example, the LMS or Studio. + +Producing Events +################ + +Events will be produced at key stages of the exam attempt lifecycle and other points in the special exam feature. If you +are using the local Kafka cluster, you will be able to see the topics and events there. + +Consuming Events +################ + +In order to consume events off the event bus, you must run a management command that starts an infinite loop to read +from the event bus. + +Shell into the application Docker container and run the following management command to start the loop. See the +`consume_events management command documentation`_ for a description of the arguments. + +.. code-block:: + + python3 manage.py consume_events -t -g + +Here is an example of a command to consume events from the ``learning-exam-attempt-lifecycle`` topic in the LMS. + +.. code-block:: + + python3 manage.py ls consume_events -t learning-exam-attempt-lifecycle -g dev-lms + +When your event is successfully consumed, you should see logs like the following. + +.. code-block:: + + 2023-10-04 15:50:17,508 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:513 - Message received from Kafka: topic=dev-learning-exam-attempt-lifecycle, partition=0, offset=7, message_id=b71c735c-62cd-11ee-9064-0242ac120012, key=b'\x00\x00\x00\x00\x010course-v1:edX+777+2023FW', event_timestamp_ms=1696434617498 + + 2023-10-04 15:50:17,593 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:393 - Message from Kafka processed successfully + + +.. _Implementation of Event Driven Architecture for Exam Downstream Effects: https://github.com/edx/edx-exams/blob/main/docs/decisions/0004-downstream-effect-events.rst +.. _devstack: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/ +.. _Optional Import of Confluent Kafka: https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst. +.. _manual testing: https://github.com/openedx/event-bus-kafka/blob/main/docs/how_tos/manual_testing.rst +.. _consume_events management command documentation: https://github.com/openedx/openedx-events/blob/7e6e92429485133bf16ae4494da71b5a2ac31b9e/openedx_events/management/commands/consume_events.py + Setting up an exam and proctoring tool -------------------------------------- diff --git a/docker-compose.yml b/docker-compose.yml index 42ed54fa..40d181c9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,9 @@ version: "2.1" services: db: - image: mysql:5.6 + # Oracle-packaged version includes a `linux/arm64/v8` version, needed for + # machines with Apple Silicon CPUs (Mac M1, M2) + image: mysql:8.0.33-oracle container_name: edx_exams.db environment: MYSQL_ROOT_PASSWORD: ""