Skip to content

Commit

Permalink
docs: center tests around local dev context
Browse files Browse the repository at this point in the history
- added better setup
  • Loading branch information
ilee2u committed Nov 7, 2023
1 parent 788eb68 commit d0f1977
Show file tree
Hide file tree
Showing 2 changed files with 65 additions and 24 deletions.
30 changes: 30 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,36 @@ Devstack Set Up
You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.

LTI Configuration for Local Development
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:

#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.
#. In your terminal, navigate to your
#. In your terminal, run the frontend-app-course-authoring MFE by navigating to the devstack folder and running large-u .
#. Ensure that the exams IDA is enabled for your course. In your LMS' django admin's course waffle flag table (http://localhost:18000/admin/waffle_utils/waffleflagcourseoverridemodel/) add an entry:
#. Enter course_apps.exams_ida in the Waffle Flag field
#. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023
#. Override Choice should be Force On
#. Select the Enabled checkbox.
#. At http://localhost:18740/admin/core/proctoringprovider/add/, add an entry for a proctoring provider:
#. Make the "name" something with no spaces, like "custom-proctor"
#. Set the "verbose name" to whatever you want, maybe something like "Custom Proctoring Service"
#. The LTI configuration id and Tech support phone & email are not relevant for local testing, so just set them all to "1”.
#. From whatever root folder you keep your devstack folders in, navigate to edx-platform/cms/envs/
#. If you have not already, create a private.py file in this /envs/ directory
#. In this private.py file, add the line FEATURES['ENABLE_EXAM_SETTINGS_HTML_VIEW'] = True
#. Again from whatever root folder you keep your devstack folders in, navigate to frontend-app-course-authoring/.env.development
#. Set the environment variable EXAMS_BASE_URL="http://localhost:18740"
#. You will need to reset the frontend-app-course-authoring MFE if it is running for this setting to work
#. Setup the proctoring provider in your course in studio (settings dropdown > proctored exam settings)
#. Set proctored exams to enabled
#. Set the proctoring provider to the name you entered in step #3
#. Create a new proctored exam: 13.3. Creating Proctored Exams — Building and Running an Open edX Course documentation
#. The exam registration task that registers exams with the exams service requires that the subsection due date (i.e. exam due date) is set and/or the course end date is set. Be sure to do do this; otherwise, your exam will not be registered with the exams service.
#. Publish the course
#. Ensure your exam in successfully created in edx-exams at http://localhost:18740/admin/core/exam/

Development Workflow
~~~~~~~~~~~~~~~~~~~~
.. code-block::
Expand Down
59 changes: 35 additions & 24 deletions docs/test_plan/proctored_exam.rst
Original file line number Diff line number Diff line change
@@ -1,27 +1,36 @@
Proctored Exams
===============

Helpful ReadTheDocs Pages:

Helpful ReadTheDocs Pages
-------------------------
- `Configuring a Proctoring Provider <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/proctored_enabling.html#configuring-proctoring-provider>`_
- `Creating a Proctored Exam <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/pt_create.html#creating-a-proctored-exam>`_

Prerequisite steps:
#. Use Google Chrome
#. Have a staff account
#. Have a non-staff learner account enrolled in the verified track for this course
#. As your non-staff user in your current window, and enroll in the course where you have set up your proctored exam
#. In an incognito window, login to your staff account and visit your Support Tools page and search in the email for your non-staff user. You should now see the option to change the enrollment track of that user. Change the enrollment track of the user to ""verified"" (choose any reason when prompted, it doesn't matter)
#. In that same incognito, still signed in as staff, navigate to the Instructor dashboard
#. Find or create a test course within Stage with proctoring enabled and an LTI proctoring provider chosen, certificates are enabled, and where the exams IDA waffle flag is enabled
#. The exams IDA waffle flag can be found/created in the LMS' Django Admin under "Waffle Flag Course Overrides"
#. If needed, you can make a new entry using "course_apps.exams_ida" as the waffle flag, the course ID, and by setting it to "Enabled"
#. Find or create two exam subsections with a type of 'Proctored exam' in your test course
#. This can be done by creating a subsection and clicking the 'Configure' button (should look like a gear), going to the "Advanced Tab", and selecting "Proctored"
#. You MUST also set this subsection to be Graded as a Midterm or Final exam
#. This subsection should contain at least one unit with one gradeable block, e.g. a generic multiple choice question in order to test downstream effects to the gradebook
#. You should also have a tab open with your event bus logs

Proctoring Set Up Steps
-----------------------
- As of November 2023, there is a course available for testing in edX's staging environment at https://learning.stage.edx.org/course/course-v1:edx+PROC100+1T2023/home. You should be able to test using this course so long as you have access to both an instructor and learner account.
- Otherwise, if you are developing on Open edX locally, then you should follow the instructions below
#. Find or create a test course that has proctoring enabled and an LTI proctoring provider chosen, where a verified track exists, and where the exams IDA waffle flag is enabled
- The exams IDA waffle flag can be found/created in the LMS' Django Admin under "Waffle Flag Course Overrides"
- If needed, you can make a new entry using "course_apps.exams_ida" as the waffle flag, the course ID, and by setting it to "Enabled"
#. Find or create two exam subsections with a type of 'Proctored exam' in your test course
- This can be done by creating a subsection and clicking the 'Configure' button (should look like a gear), going to the "Advanced Tab", and selecting "Proctored"
- You MUST also set this subsection to be Graded as a Midterm or Final exam
- This subsection should contain at least one unit with one gradeable block, e.g. a generic multiple choice question in order to test downstream effects to the gradebook

User Set Up Steps
-----------------
#. Use Google Chrome as your browser for these tests
#. In your platform, log in with a non-staff learner account enrolled in the verified track for this course
- The default credentials for local development are ([email protected] Password=edx)
#. As your non-staff user in your current window, and enroll in the course where you have set up your proctored exam
#. In an incognito window, login into an account with staff or instructor level permissions
- The default credentials for local development are ([email protected] Password=edx)
#. Visit your Support Tools page for managing enrollments. Enter the email for your non-staff user in the search bar. You should now see the option to change the enrollment track of that user. Change the enrollment track of the user to "verified" (choose any reason if prompted, it doesn't matter)
- Default local URL for changing enrollments is http://localhost:18000/support/enrollment
#. In that same incognito window, while still signed into your staff or instructor level account, navigate to the exams dashboard
- This can be found by going to the course view for the course you are testing (the one with all the checkboxes and collapsable course sections), then navigating to the "Instructor" tab to get to the Instructor Dashboard, then navigating to the "Special Exams" tab within the Instructor dashboard.

Proctored Exam Flow
===================
Expand All @@ -46,10 +55,12 @@ Can't Start Exam Without Proctoring Software Running

Exam Flow Works With Proctoring Software Installed & Running
------------------------------------------------------------
#. Go through the setup again, this time without closing your proctoring software. Clicking "Start Exam" should start the exam
#. Go through the steps 1-4 in the "Can't Start Exam Without Proctoring Software Running" section again
#. This time without closing your proctoring software, clicking "Start Exam". You should be brought to your exam, with a timer running above the exam questions.
#. Complete whatever question you put inside your exam w/ the correct answer. Clicking "End My Exam" should direct you to a "Are you sure you want to end your proctored exam?" interstitial
#. Clicking "No I'd like to continue working" should send you back to the exam
#. Clicking "Yes end my proctored exam" should end the exam and directs you to a "You have submitted this proctored exam for review" page
#. Clicking "Yes end my proctored exam" should also close your proctoring software automatically.
#. In the course view, the exam you took should be marked as completed (with a green checkmark)


Expand All @@ -73,6 +84,8 @@ NOTE: These instructions currently do not account for downstream effects involvi

Dashboard UI Loads Exam Attempts
--------------------------------
INSTRUCTIONS: In an incognito window, log in to a staff or instructor account Go to the course view for the course you are testing (the one with all the checkboxes and collapsable course sections), then navigating to the "Instructor" tab to get to the Instructor Dashboard, then navigating to the "Special Exams" tab within the Instructor dashboard.

#. Clicking on the "Special Exams" tab should load the *new* version of the exams dashboard
#. Clicking on the "Review Dashboard" tab within the exams dashboard should open a dashboard owned by your selected procotoring software (e.g. a Proctortrack or Proctorio dashboard for reviewing exam attempt details)
#. Clicking the "Select An Exam" dropdown should show the list of exams for the course
Expand All @@ -83,16 +96,16 @@ Dashboard UI Loads Exam Attempts
#. Clicking the "Review Dashboard" link in the modal should load the review dashboard for your chosen Proctoring Software (e.g. Proctortrack or Proctorio)

Can Reject/Verify "Review Required" attempts
---------------------------------------------------
--------------------------------------------
INSTRUCTIONS: Go into another exam and be suspiscious. Block your webcam, leave for a few seconds, make strange noises, open a bunch of tabs, watch some youtube, open wikipedia, ask chatgpt for the answer to life, etc. Note you will need to do this a couple of times

#. Submit your suspiscious exam attempt, and check the exams dashboard. Assuming you were suspiscious enough, you should see a row that has a status of "Review Required" AND has a "Review Required" button for your attempt
#. NOTE: This exam attempt may appear with the status "Satisfied" if you are using Proctorio. This means Proctorio's system is still analyzing your attempt. If this happens, just wait a bit and refresh the page until the status changes to "Verified"
#. Clicking the "Review Required" button should spawn a modal that says "Update review status"
#. Clicking "Cancel" should close the modal
#. Clicking "Verify" should mark attempt as verified both in the dashboard
#. Clicking "Verify" should mark attempt as verified the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100
#. Clicking "Reject" should mark attempt as reject both in the dashboard
#. Clicking "Reject" should mark attempt as reject the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0

Can Review Verified Attempts
Expand All @@ -104,17 +117,15 @@ INSTRUCTIONS: Go into another exam and don't be suspiscious. Keep your webcam on
#. A modal should pop up upon clicking "Manual Review" that contains "Cancel" and "Reject" Buttons
#. Clicking "Cancel" should close the modal
#. Clicking "Reject" should change the attempt status to "rejected" in the UI
#. Upon refreshing a filtering for this exam attempt, it is still marked as "rejected"
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0
#. Check the Generated Certificates table in the LMS admin at "{your platform url}/admin/certificates/generatedcertificate/" to see if the user's certificate for the course has been marked as "invalidated"

Can Review Rejected Attempts
----------------------------
#. The exam attempt that you just marked as "Rejected" should have a button at the end of their rows saying "Manual Review"
#. A modal pops up upon clickin "Manual Review" that only should ha the "Cancel" and "Verify" Buttons
#. A modal pops up upon clicking "Manual Review" that should only have the "Cancel" and "Verify" Buttons
#. Clicking "Cancel" should close the modal
#. Clicking "Verify" should change the attempt status to "verified" in the UI
#. Upon refreshing the page, the same attempt should still be marked as "verified"
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100


Expand Down

0 comments on commit d0f1977

Please sign in to comment.