diff --git a/README.rst b/README.rst index fbc6ba8..bfd8cb4 100644 --- a/README.rst +++ b/README.rst @@ -104,11 +104,11 @@ can provide. https://github.com/openedx/openedx-filters/issues -For more information about these options, see the `Getting Help`_ page. +For more information about these options, see the `Open edX Getting Help`_ page. .. _Slack invitation: https://openedx.org/slack .. _community Slack workspace: https://openedx.slack.com/ -.. _Getting Help: https://openedx.org/getting-help +.. _Open edX Getting Help: https://openedx.org/getting-help License ******* diff --git a/docs/concepts/glossary.rst b/docs/concepts/glossary.rst new file mode 100644 index 0000000..05046c8 --- /dev/null +++ b/docs/concepts/glossary.rst @@ -0,0 +1,86 @@ +Open edX filters glossary +########################## + +This glossary provides definitions for some of the concepts needed to use the Open edX Filters library. + + +Pipelines +--------- + +A pipeline is a list of functions that are executed in order. Each function receives the output of the previous function as input. The output of the last function is the output of the pipeline. + +Pipeline steps +-------------- + +A pipeline step is a function that receives data, manipulates it and returns it. It can be used to transform data, to validate it, to filter it, to enrich it, etc. + +Open edX Filter +--------------- + +An Open edX Filter is a Python class that inherits from `OpenEdXPublicFilter`, which is used for executing pipelines or list of functions in specific order. It implements a `run_filter` method that receives the data to be processed and returns the output of the pipeline. + +Open edX Filter signature +------------------------- + +It's the signature of the `run_filter` method of each filter. It defines the input and output of the filter. The input is a dictionary with the data to be processed and the output is a dictionary with the processed data. + +Open edX Filters' pipeline steps +-------------------------------- + +In the context of Open edX Filters, a pipeline step is a class that inherits from ``PipelineStep`` that implements the `run_filter` method which must match the Open edX Filter signature. + +Filter type +----------- + +It's the filter identifier. It's used to identify the filter in the configuration settings. When configuring the pipeline for a filter, the type is as an index for the filter configuration. + +Filter exceptions +----------------- + +Besides acting as a filter, an Open edX Filter can also raise exceptions. These exceptions are used to control the execution of the pipeline. If an exception is raised, the pipeline execution is stopped and the exception is raised again as the output of the pipeline. These exceptions are intentionally raised by the developer during the filter's execution when a condition is met. + +Filter configuration +-------------------- + +The filter configuration is a dictionary with the configuration settings for the filter. It's used to configure the pipeline for a filter. The configuration settings are specific for each filter type. The dictionary looks like this: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "": { + "fail_silently": , + "pipeline": [ + "", + "", + ... + "", + ] + }, + } + +Where: + +- ```` is the filter type. +- ``fail_silently`` is a boolean value. + +If ``fail_silently`` is ``True``: when a pipeline step raises a runtime exception -- like ``ImportError`` or ``AttributeError`` exceptions which are not intentionally raised by the developer during the filter's execution; the exception won't be propagated and the pipeline execution will resume, i.e the next steps will be executed +If ``fail_silently`` is ``False``: the exception will be propagated and the pipeline execution will stop. + +For example, with this configuration: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "": { + "fail_silently": True, + "pipeline": [ + "non_existing_module.non_existing_function", + "existing_module.function_raising_attribute_error", + "existing_module.existing_function", + ] + }, + } + +The pipeline tooling will catch the ``ImportError`` exception raised by the first step and the ``AttributeError`` exception raised by the second step, then continue and execute the third step. Now, if ``fail_silently`` is ``False``, the pipeline tooling will catch the ``ImportError`` exception raised by the first step and propagate it, i.e the pipeline execution will stop. + +- ``pipeline`` is list of paths for each pipeline step. Each path is a string with the following format: ``.``. The module path is the path to the module where the pipeline step class is defined and the class name is the name of the class that implements the ``run_filter`` method to be executed. diff --git a/docs/concepts/hooks-extension-framework.rst b/docs/concepts/hooks-extension-framework.rst new file mode 100644 index 0000000..0e724e8 --- /dev/null +++ b/docs/concepts/hooks-extension-framework.rst @@ -0,0 +1,43 @@ +Extending Open edX with the Hooks Extensions Framework +###################################################### + +To sustain the growth of the Open edX ecosystem, the business rules of the +platform must be open for extension following the open-closed principle. This +framework allows developers to do just that without needing to fork and modify +the main edx-platform repository. + +Context +******* + +Hooks are predefined places in the edx-platform core where externally defined +functions can take place. In some cases, those functions can alter what the user +sees or experiences in the platform. Other cases are informative only. All cases +are meant to be extended using Open edX plugins and configuration. + +Hooks can be of two types, events and filters. Events are in essence Django signals, in +that they are sent in specific application places and whose listeners can extend +functionality. On the other hand Filters are passed data and can act on it +before this data is put back in the original application flow. In order to allow +extension developers to use the Events and Filters definitions on their plugins, +both kinds of hooks are defined in lightweight external libraries. + +* `openedx-filters`_ +* `openedx-events`_ + +Hooks are designed with stability in mind. The main goal is that developers can +use them to change the functionality of the platform as needed and still be able +to migrate to newer open releases with very little to no development effort. In +the case of filters, this is detailed in the `naming and versioning ADR`_. + +A longer description of the framework and it's history can be found in `OEP 50`_. + +.. _OEP 50: https://open-edx-proposals.readthedocs.io/en/latest/oep-0050-hooks-extension-framework.html +.. _naming and versioning ADR: https://github.com/openedx/openedx-filters/blob/main/docs/decisions/0004-filters-naming-and-versioning.rst +.. _openedx-filters: https://github.com/openedx/openedx-filters +.. _openedx-events: https://github.com/openedx/openedx-events + +On the technical side, filters are implemented using a pipeline mechanism, that executes +a list of functions called ``steps`` configured through Django settings. Each +pipeline step receives a dictionary with data, processes it and returns an output. During +this process, they can alter the application execution flow by halting the process +or modifying their input arguments. diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst index 8a2b4bd..e3f391e 100644 --- a/docs/concepts/index.rst +++ b/docs/concepts/index.rst @@ -1,2 +1,9 @@ Concepts -######## +======== + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + hooks-extension-framework + glossary diff --git a/docs/how-tos/create-new-filter.rst b/docs/how-tos/create-new-filter.rst new file mode 100644 index 0000000..3d879bf --- /dev/null +++ b/docs/how-tos/create-new-filter.rst @@ -0,0 +1,203 @@ +How-to Create a new Filter +########################## + +.. How-tos should have a short introduction sentence that captures the user's goal and introduces the steps. + +The mechanisms implemented by the Open edX Filters library are supported and maintained by the Open edX community. The +library is designed to be extensible, and we welcome contributions of new filters. + +Therefore, we've put together this guide that will walk you through the process of adding a new filter to the library, +and will provide you with a template to follow when adding new filters. + +Assumptions +*********** + +.. This section should contain a bulleted list of assumptions you have of the + person who is following the How-to. The assumptions may link to other + how-tos if possible. + +* You have a development environment set up. +* You have a basic understanding of Python and Django. +* You understand the concept of filters or have reviewed the relevant + :doc:`/concepts/index` docs. + +Steps +***** + +.. A task should have 3 - 7 steps. Tasks with more should be broken down into digestible chunks. + +#. Propose a new filter to the Open edX community + + When creating a new filter, you must justify its implementation. For example, you could create a post in Discuss, + send a message through slack or open a new issue in the library repository listing your use cases for it. Or even, + if you have time, you could accompany your proposal with the implementation of the filter to illustrate its behavior. + +#. Place your filter in an architecture subdomain + + As specified in the Architectural Decisions Record (ADR) filter naming and versioning, the filter definition needs an Open edX Architecture + Subdomain for: + + - The type of the filter: ``{Reverse DNS}.{Architecture Subdomain}.{Subject}.{Action}.{Major Version}`` + - The package name where the definition will live, eg. ``learning/``. + + For those reasons, after studying your new filter purpose, you must place it in one of the subdomains already in use, or introduce a new subdomain: + + +-------------------+----------------------------------------------------------------------------------------------------+ + | Subdomain name | Description | + +===================+====================================================================================================+ + | Learning | Allows learners to consume content and perform actions in a learning activity on the platform. | + +-------------------+----------------------------------------------------------------------------------------------------+ + + New subdomains may require some discussion, because there does not yet exist and agreed upon set of subdomains. So we encourage you to start the conversation + as soon as possible through any of the communication channels available. + + Refer to `edX DDD Bounded Contexts `_ confluence page for more documentation on domain-driven design in the Open edX project. + +#. Define the filter's behavior + + Defining the filter's behavior includes: + + - Defining the filter type for identification + - Defining the filter's signature + - Defining the filter's behavior for stopping the process in which it is being used + + The filter type is the name that will be used to identify the filter's and it'd help others identifying its purpose. For example, if you're creating a filter that will be used during the student registration process in the LMS, + according to the documentation, the filter type is defined as follows: + + ``{Reverse DNS}.{Architecture Subdomain}.student.registration.requested.{Major Version}`` + + Where ``student`` is the subject and ``registration.requested`` the action being performed. The major version is the version of the filter, which will be incremented + when a change is made to the filter that is not backwards compatible, as explained in the ADR. + + Now that you have the filter type, you'll need to define the filter's signature and overall behavior. The filter's signature, which is the set of parameters that the filter will manipulate, depends on where the filter is located. For example, + if you're creating a filter that will be used during the student registration process in the LMS, the filter's signature will be the set of parameters available for that time for the user. In this case, the filter's signature will be the set of parameters that the registration form sends to the LMS. + + You can ask yourself the following questions to help you figure out your filter's parameters: + + - What is the filter's purpose? (e.g. to validate the student's email address) + - What parameters will the filter need to to that? (e.g. the email address) + - Where in the registration process will the filter be used? (e.g. after the student submits the registration form but before anything else) + + With that information, you can define the filter's signature: + + - Arguments: ``email``. Since we want this filter to be broadly used, we'll add as much relevant information as possible for the user at that point. As we mentioned above, we can send more information stored in the registration form like ``name`` or ``username``. + - Returns: since filters take in a set of parameters and return a set of parameters, we'll return the same set of parameters that we received. + + Since filters also can act according to the result of the filter's execution, we'll need to define the filter's behavior for when the filter stops the process in which it is being used. For example, if you're using the filter in the LMS, you'll need to define + what happens when the filter stops the registration process. So, for this filter we'll define the following behavior: + + - When stopping the registration process, we'll raise a ``PreventRegistration`` exception. + +#. Implement the new filter + +.. Following the steps, you should add the result and any follow-up tasks needed. + + Up to this point, you should have the following: + +.. code-block:: python + + class StudentRegistrationRequested(OpenEdxPublicFilter): + """ + Custom class used to create registration filters and its custom methods. + """ + + filter_type = "org.openedx.learning.student.registration.requested.v1" + + class PreventRegistration(OpenEdxFilterException): + """ + Custom class used to stop the registration process. + """ + + @classmethod + def run_filter(cls, form_data): + """ + Execute a filter with the signature specified. + + Arguments: + form_data (QueryDict): contains the request.data submitted by the registration + form. + """ + sensitive_data = cls.extract_sensitive_data(form_data) + data = super().run_pipeline(form_data=form_data) + return data.get("form_data") + +.. note:: + This is not exactly what the registration filter looks like, but it's a good starting point. You can find the full implementation of the registration filter in the library's repository. + + Some things to note: + + - The filter's type is defined in the ``filter_type`` class attribute. In this case, the filter type is ``org.openedx.learning.student.registration.requested.v1``. + - The filter's signature is defined in the ``run_filter`` method. In this case, the signature is the ``form_data`` parameter. + - The ``run_filter`` is a class method that returns the same set of parameters that it receives. + - The ``run_filter`` class method calls the ``run_pipeline`` method, which is the method that executes the filter's logic. This method is defined in the ``OpenEdxPublicFilter`` class, which is the base class for all the filters in the library. This method returns a dictionary with the following structure: + + .. code-block:: python + + { + "": , + "": , + ... + "": , + } + + Where in this specific example would be: + + .. code-block:: python + + { + "form_data": form_data, + } + + Where ``form_data`` is the same set of parameters that the filter receives, which is the accumulated output for the filter's pipeline. That is how ``run_filter`` should always look like. + - The filter's behavior for stopping the process is defined in the ``PreventRegistration`` exception which inherits from the ``OpenEdxFilterException`` base exception. In this case, the exception is raised when the filter stops the registration process. This is done in the service where the filter is being used, which in this case is the LMS. + - The class name is the filter's type ``{Subject}.{Action}`` part in a camel case format. In this case, the filter's name is ``StudentRegistrationRequested``. + +#. Add tests for the new filter + + Each filter has its own set of tests. The tests for the filter you're creating should be located in the ``tests`` directory in the library's repository. The tests should be located in the ``test_filters.py`` file, which is where all the tests for the filters are located. Each set of tests is related to a specific type of filter, so you should add your tests to the set of tests that are related to the filter you're creating. + For example, if you're creating a filter that will be used during the student registration process in the LMS, you should add your tests to the ``TestAuthFilters`` set of tests. This is how the tests for the registration filter look like: + + +.. code-block:: python + + def test_student_registration_requested(self): + """ + Test StudentRegistrationRequested filter behavior under normal conditions. + + Expected behavior: + - The filter must have the signature specified. + - The filter should return form data. + """ + expected_form_data = { + "password": "password", + "newpassword": "password", + "username": "username", + } + + form_data = StudentRegistrationRequested.run_filter(expected_form_data) + + self.assertEqual(expected_form_data, form_data) + + @data( + ( + StudentRegistrationRequested.PreventRegistration, {"message": "Can't register in this site."} + ), + ) + @unpack + def test_halt_student_auth_process(self, auth_exception, attributes): + """ + Test for student auth exceptions attributes. + + Expected behavior: + - The exception must have the attributes specified. + """ + exception = auth_exception(**attributes) + + self.assertDictContainsSubset(attributes, exception.__dict__) + +.. note:: + Basically, we're testing the filter's signature and the filter's behavior for stopping the process. The first test is testing the filter's signature, which is the set of parameters that the filter receives and returns. The second test is testing the filter's behavior for stopping the process, which is the exception that is raised when the filter stops the process. + +.. .. seealso:: + + :ref:`title to link to` diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 5147f80..836fad4 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -1,2 +1,9 @@ How-tos -####### +======= + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + create-new-filter + using-filters diff --git a/docs/how-tos/using-filters.rst b/docs/how-tos/using-filters.rst new file mode 100644 index 0000000..e3f71a0 --- /dev/null +++ b/docs/how-tos/using-filters.rst @@ -0,0 +1,157 @@ +How to use Open edX Filters +--------------------------- + +Using openedx-filters in your code is very straight forward. We can consider the +various use cases: implementing pipeline steps, attaching/hooking pipelines to filter, +and triggering a filter. We'll also cover how to test the filters you create in your service. + + +Implement pipeline steps +************************ + +Let's say you want to consult student's information with a third party service +before generating the students certificate. This is a common use case for filters, +where the functions part of the filter's pipeline will perform the consulting tasks and +decide the execution flow for the application. These functions are the pipeline steps, +and can be implemented in an installable Python library: + +.. code-block:: python + + # Step implementation taken from openedx-filters-samples plugin + from openedx_filters import PipelineStep + from openedx_filters.learning.filters import CertificateCreationRequested + + class StopCertificateCreation(PipelineStep): + """ + Stop certificate creation if user is not in third party service. + """ + + def run_filter(self, user, course_id, mode, status): + # Consult third party service and check if continue + # ... + # User not in third party service, denied certificate generation + raise CertificateCreationRequested.PreventCertificateCreation( + "You can't generate a certificate from this site." + ) + +There's two key components to the implementation: + +1. The filter step must be a subclass of ``PipelineStep``. + +2. The ``run_filter`` signature must match the filters', eg., the step signature matches the `run_filter` signature in CertificateCreationRequested: + +.. code-block:: python + + class CertificateCreationRequested(OpenEdxPublicFilter): + """ + Custom class used to create certificate creation filters and its custom methods. + """ + + filter_type = "org.openedx.learning.certificate.creation.requested.v1" + + class PreventCertificateCreation(OpenEdxFilterException): + """ + Custom class used to stop the certificate creation process. + """ + + @classmethod + def run_filter(cls, user, course_key, mode, status, grade, generation_mode): + """ + Execute a filter with the signature specified. + + Arguments: + user (User): is a Django User object. + course_key (CourseKey): course key associated with the certificate. + mode (str): mode of the certificate. + status (str): status of the certificate. + grade (CourseGrade): user's grade in this course run. + generation_mode (str): Options are "self" (implying the user generated the cert themself) and "batch" + for everything else. + """ + data = super().run_pipeline( + user=user, course_key=course_key, mode=mode, status=status, grade=grade, generation_mode=generation_mode, + ) + return ( + data.get("user"), + data.get("course_key"), + data.get("mode"), + data.get("status"), + data.get("grade"), + data.get("generation_mode"), + ) + +Attach/hook pipeline to filter +****************************** + +After implementing the pipeline steps, we have to tell the certificate creation +filter to execute our pipeline. + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "org.openedx.learning.certificate.creation.requested.v1": { + "fail_silently": False, + "pipeline": [ + "openedx_filters_samples.samples.pipeline.StopCertificateCreation" + ] + }, + } + +Triggering a filter +******************* + +In order to execute a filter in edx-platform or your own plugin/library, you must install the +plugin where the steps are implemented and also, ``openedx-filters``. + +.. code-block:: python + + # Code taken from lms/djangoapps/certificates/generation_handler.py + from openedx_filters.learning.filters import CertificateCreationRequested + + try: + user, course_id, mode, status = CertificateCreationRequested.run_filter( + user=user, course_id=course_id, mode=mode, status=status, + ) + except CertificateCreationRequested.PreventCertificateCreation as exc: + raise CertificateGenerationNotAllowed(str(exc)) from exc + +Testing filters' steps +********************** + +It's pretty straightforward to test your pipeline steps, you'll need to include the +``openedx-filters`` library in your testing dependencies and configure them in your test case. + +.. code-block:: python + + from openedx_filters.learning.filters import CertificateCreationRequested + + @override_settings( + OPEN_EDX_FILTERS_CONFIG={ + "org.openedx.learning.certificate.creation.requested.v1": { + "fail_silently": False, + "pipeline": [ + "openedx_filters_samples.samples.pipeline.StopCertificateCreation" + ] + } + } + ) + def test_certificate_creation_requested_filter(self): + """ + Test filter triggered before the certificate creation process starts. + + Expected results: + - The pipeline step configured for the filter raises PreventCertificateCreation + when the conditions are met. + """ + ... + with self.assertRaises(CertificateCreationRequested.PreventCertificateCreation): + CertificateCreationRequested.run_filter( + user=user, course_key=course_key, mode="audit", + ) + + # run your assertions + +Changes in the ``openedx-filters`` library that are not compatible with your code +should break this kind of test in CI and let you know you need to upgrade your code. +The main limitation while testing filters' steps is their arguments, as they are +in-memory objects, but that can be solved in CI using Python mocks. diff --git a/docs/quickstarts/index.rst b/docs/quickstarts/index.rst index a60e916..103ad3d 100644 --- a/docs/quickstarts/index.rst +++ b/docs/quickstarts/index.rst @@ -1,2 +1,9 @@ Quickstarts ########### + + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + use-filters-to-change-enrollment diff --git a/docs/quickstarts/use-filters-to-change-enrollment.rst b/docs/quickstarts/use-filters-to-change-enrollment.rst new file mode 100644 index 0000000..694ddc2 --- /dev/null +++ b/docs/quickstarts/use-filters-to-change-enrollment.rst @@ -0,0 +1,11 @@ +Using Open edX Filters in the LMS service +----------------------------------------- + +Live example +************ + +For filter steps samples you can visit the `openedx-filters-samples`_ plugin, where +you can find minimal steps exemplifying the different ways on how to use +``openedx-filters``. + +.. _openedx-filters-samples: https://github.com/eduNEXT/openedx-filters-samples diff --git a/docs/reference/django-plugins-and-filters.rst b/docs/reference/django-plugins-and-filters.rst new file mode 100644 index 0000000..5965e54 --- /dev/null +++ b/docs/reference/django-plugins-and-filters.rst @@ -0,0 +1,41 @@ +Django Plugins and Filters +########################## + +Django plugins is one of the most valuable extension mechanisms for the Open edX platform. In this section, we will +guide you through the process of using filters inside your own plugin. + + +Use filters inside your plugin +****************************** + +Imagine you have your own registration plugin and you want to add a filter to it. The first thing you need to do is +adding ``openedx-filters`` to your requirements file. Then, you can import the registration filter and use it inside +your registration flow as it's used in the LMS registration flow. You can even add your own filters to your registration, +after implementing their definitions in your plugin. + +Configure filters +***************** + +Filters are configured in the ``OPEN_EDX_FILTERS_CONFIG`` dictionary which can be specified in your plugin's settings +file. The dictionary has the following structure: + +.. code-block:: python + + OPEN_EDX_FILTERS_CONFIG = { + "": { + "fail_silently": , + "pipeline": [ + "", + "", + ... + "", + ] + }, + } + + +Create pipeline steps +********************* + +In your own plugin, you can create your own pipeline steps by inheriting from ``PipelineStep`` and implementing the +``run_filter`` method. You can find examples of pipeline steps in the ``openedx-filters-samples`` repository. See :doc:`/quickstarts/index` for more details. diff --git a/docs/reference/index.rst b/docs/reference/index.rst index ba5ea57..06f61ee 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -1,2 +1,8 @@ References ########## + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + django-plugins-and-filters diff --git a/setup.py b/setup.py index c818396..6dbb361 100755 --- a/setup.py +++ b/setup.py @@ -70,7 +70,7 @@ def is_requirement(line): name="openedx-filters", version=VERSION, description="""Open edX Filters from Hooks Extensions Framework (OEP-50).""", - long_description=README + "\n\n" + CHANGELOG, + long_description=README, long_description_content_type='text/x-rst', author="eduNEXT", author_email="technical@edunext.co", diff --git a/tox.ini b/tox.ini index 843c318..84a6225 100644 --- a/tox.ini +++ b/tox.ini @@ -57,8 +57,7 @@ commands = rm -f docs/modules.rst make -C docs clean make -C docs html - python -m build --wheel - twine check dist/* + python setup.py check --restructuredtext --strict [testenv:quality] whitelist_externals =