Skip to content

Commit

Permalink
docs: improved test names + organization
Browse files Browse the repository at this point in the history
  • Loading branch information
ilee2u committed Nov 3, 2023
1 parent 77151c1 commit d1455d8
Showing 1 changed file with 68 additions and 60 deletions.
128 changes: 68 additions & 60 deletions docs/test_plan/proctored_exam.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,100 +27,108 @@ Proctored Exam Flow
-------------------
Expected Behavior: "A paid track learner is able to start, complete, and submit a proctored exam using Proctoring Software that is launched via LTI."

Without Proctoring Software Installed
=====================================
Can't Start Exam Without Proctoring Software Installed
======================================================
#. Enter a Proctored Exam without proctoring software installed. You should see an interstitial stating that "This exam is proctored" with a button that says "Continue to Proctored Exam"
#. Clicking "Continue to Proctored Exam" should bring you to an interstitial titled "Set up and start your proctored exam." with "Start System Check" and "Start Exam" buttons
#. Clicking the "Start Exam" button should pop up a modal that says "Cannot Start Exam"
#. Clicking "OK" closes that modal
#. Clicking "Start System Check" redirects you to a (likely broken) page
#. Navigating back one page and clicking "Start Exam" still blocks you from starting the exam
#. Clicking "OK" should close that modal
#. Clicking "Start System Check" should redirect you to your proctoring setup page
#. Navigating back one page and clicking "Start Exam" should still block you from starting the exam

With Proctoring Software Installed
==================================
Can't Start Exam Without Proctoring Software Running
====================================================
#. Install the proctoring software that you've configured with your platform. Go through the opening interstitials and click "Start System Check". This should load your respective Proctoring Software Setup page
#. Immediately navigating back and trying to click "Start Exam" still doesn't allow you to start the exam
#. Navigating back to your proctoring software setup page and refreshing on that page causes no issues
#. Immediately navigating back and trying to click "Start Exam" should't allow you to start the exam
#. Navigating back to your proctoring software setup page and refreshing on that page should cause no issues
#. Go through your proctoring software setup for your Webcam, Desktop, etc. Upon setup completion, you should be redirected back to an exam interstitial with just the "Start Exam" button
#. Clicking "Start Exam" starts the exam. Make sure to complete whatever question you put inside your exam w/ the correct answer.
#. Clicking "End My Exam" directs you to a "Are you sure you want to end your proctored exam?" interstitial
#. Clicking "No I'd like to continue working" sends you back to the exam
#. Clicking "Yes end my proctored exam" ends the exam and directs you to a "You have submitted this proctored exam for review" page
#. In the course view, the exam you took is marked as completed (with a green checkmark)
#. Close your proctoring software and click "Start Exam". The exam should not start

Proctorio Specific
==================
Exam Flow Works With Proctoring Software Installed & Running
============================================================
#. Go through the setup again, this time without closing your proctoring software. Clicking "Start Exam" should start the exam
#. Complete whatever question you put inside your exam w/ the correct answer. Clicking "End My Exam" should direct you to a "Are you sure you want to end your proctored exam?" interstitial
#. Clicking "No I'd like to continue working" should send you back to the exam
#. Clicking "Yes end my proctored exam" should end the exam and directs you to a "You have submitted this proctored exam for review" page
#. In the course view, the exam you took should be marked as completed (with a green checkmark)


Proctorio-Specific Tests
------------------------
These tests are for those using Proctorio as their proctoring provider

Exam Flow Works as Expected
===========================
#. With Proctorio not installed, clicking "Start System Check" redirects you to https://getproctorio.com/
#. Install Proctorio and go through the opening interstitials and click "Start System Check". This should load the Proctorio Setup page
#. Immediately navigating back and trying to click "Start Exam" still doesn't allow you to start the exam
#. Navigating back to the Proctorio Setup and refreshing on that page causes no issues
#. Go through the Proctorio Setup for your Webcam, Desktop, etc. Upon setup completion, your webcam should appear to the right of the screen along with some controls. You should land on a page that says "Sending you back to your exam." with a link to "Return to Exam". Clicking this link should send you to another interstitial. (NOTE: Clicking this link is time-bound so make sure to click it fast)
#. Clicking "Return to Exam" directs you to a page titled "Important" w/ another header saying "Proctored Exam Rules", and a "Start Exam" Button
#. In a separate exam, go through the setup again and, but instead of clicking on "Return to exam" on the "Sending you back to your exam" page, simply wait, You should be sent to the "Important"/"Start Exam" interstitial automatically
#. In this separate exam, start the exam, then open devtools using Cmd+Option+I on Mac (F12 on Windows). This should error out the exam and load a "Error with proctored exam" page AND open the "Proctorio Support" page in another tab
#. Go through the Proctorio Setup for your Webcam, Desktop, etc. Upon setup completion, your webcam should appear to the right of the screen along with some controls.
#. On the "Return to exam" page, DO NOT click on the "Sending you back to your exam" page. Wait instead, and you should be sent to the "Important"/"Start Exam" interstitial automatically
#. In a separate exam, go through the setup again and on the "Sending you back to your exam." page, click "Return to Exam". Clicking this link should send you to another interstitial. (NOTE: Clicking this link is time-bound so make sure to click it fast)
#. Clicking the "Return to Exam" button should direct you to a page titled "Important" w/ another header saying "Proctored Exam Rules", and a "Start Exam" Button


Exams Dashboard
---------------
Expected Behavior: Calls to the ACS Endpoint work, instructors can review/reset exam attempts, and expected downstream effects (e.g. w/ completion, credits, certificates, etc) of modifying an exam attempt's status occur
NOTE: These instructions currently do not account for downstream effects involving a learner's credit requirement status. We currently have no plan to include these.
NOTE: These instructions currently do not account for downstream effects involving a learner's credit requirement status. We currently have no plan to include these

Dashboard UI
============
#. Clicking on the "Special Exams" tab loads the *new* version of the exams dashboard
#. Clicking on the "Special Exams" tab should load the *new* version of the exams dashboard
#. Clicking on the "Review Dashboard" tab within the exams dashboard should open a dashboard owned by your selected procotoring software (e.g. a Proctortrack or Proctorio dashboard for reviewing exam attempt details)
#. Clicking the "Select An Exam" dropdown shows the list of exams for the course
#. Filtering by exam name in the exam selection dropdown brings up the expected exams (e.g. querying "M4-M6" shows all the exams in that section)
#. Selecting a specific exam (i.e. the ones you just took) loads the respective exam attempts for those exams
#. The exam attempts are in the expected status, e.g. successful attempts are submitted or verified, and the errored attempts are errored
#. Filtering by username only shows exam attempts for that username
#. The status changes for these exam attempts look the same in the edx-exams admin
#. Clicking the "Select An Exam" dropdown should show the list of exams for the course
#. Filtering by exam name in the exam selection dropdown should bring up the expected exams (e.g. querying "M4-M6" shows all the exams in that section)
#. Selecting a specific exam (i.e. the ones you just took) should load the respective exam attempts for those exams
#. The exam attempts should be in the expected status, e.g. successful attempts are submitted or verified, and the errored attempts are errored
#. Filtering by username only should show exam attempts for that username
#. Clicking the "Review Dashboard" link in the modal should load the review dashboard for your chosen Proctoring Software (e.g. Proctortrack or Proctorio)


Reviewing "Second Review Required" attempts
===========================================
Instructions: Go into another exam and be suspiscious. Block your webcam, leave for a few seconds, make strange noises, open a bunch of tabs, watch some youtube, open wikipedia, ask chatgpt for the answer to life, etc. Note you will need to do this a couple of times
INSTRUCTIONS: Go into another exam and be suspiscious. Block your webcam, leave for a few seconds, make strange noises, open a bunch of tabs, watch some youtube, open wikipedia, ask chatgpt for the answer to life, etc. Note you will need to do this a couple of times
#. Submit your suspiscious exam attempt, and check the exams dashboard. Assuming you were suspiscious enough, you should see a row that has a status of "Second Review Required" AND has a "Review Required" button for your attempt
#. Clicking the "Review Required" button spawns a modal that says "Update review status"
#. Clicking "Cancel" closes modal
#. Clicking "Verify" marks attempt as verified both in the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100.
#. Clicking "Reject" marks attempt as reject both in the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0.
#. Clicking the "Review Dashboard" link in the modal "loads" the review dashboard (NOTE: This shouldn't work yet, and will load a broken page instead)
#. NOTE: This exam attempt may appear with the status "Satisfied" if you are using Proctorio. This means Proctorio's system is still analyzing your attempt. If this happens, just wait a bit and refresh the page until the status changes to "Verified"
#. Clicking the "Review Required" button should spawn a modal that says "Update review status"
#. Clicking "Cancel" should close the modal
#. Clicking "Verify" should mark attempt as verified both in the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100
#. Clicking "Reject" should mark attempt as reject both in the dashboard
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0

Reviewing Verified Attempts
===========================
#. Exam attempts with a status of "Verified" have a button/link at the end of their rows titled "Manual Review"
#. A modal pops up upon clicking "Manual Review" that contains "Cancel" and "Reject" Buttons
#. Clicking "Cancel" closes the modal
#. Clicking "Reject" changes the attempt status to "rejected" in the UI
INSTRUCTIONS: Go into another exam and don't be suspiscious. Keep your webcam on with your face in frame. Complete the exam
#. An exam attempt with a status of "Verified" should appear and have a button/link at the end of their rows titled "Manual Review"
#. NOTE: This exam attempt may appear with the status "Satisfied" if you are using Proctorio. This means Proctorio's system is still analyzing your attempt. If this happens, just wait a bit and refresh the page until the status changes to "Verified"
#. A modal should pop up upon clicking "Manual Review" that contains "Cancel" and "Reject" Buttons
#. Clicking "Cancel" should close the modal
#. Clicking "Reject" should change the attempt status to "rejected" in the UI
#. Upon refreshing a filtering for this exam attempt, it is still marked as "rejected"
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0.
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 0
#. Check the Generated Certificates table in the LMS admin at "{your platform url}/admin/certificates/generatedcertificate/" to see if the user's certificate for the course has been marked as "invalidated"

Reviewing Rejected Attempts
===========================
#. The same exam attempts with a status of "Rejected" have a button at the end of their rows saying "Manual Review"
#. A modal pops up upon clickin "Manual Review" that only has the "Cancel" and "Verify" Buttons
#. Clicking "Cancel" closes the modal
#. Clicking "Verify" changes the attempt status to "verified" in the UI
#. Upon refreshing a filtering for this exam attempt, it is still marked as "verified"
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100.
#. The exam attempt that you just marked as "Rejected" should have a button at the end of their rows saying "Manual Review"
#. A modal pops up upon clickin "Manual Review" that only should ha the "Cancel" and "Verify" Buttons
#. Clicking "Cancel" should close the modal
#. Clicking "Verify" should change the attempt status to "verified" in the UI
#. Upon refreshing the page, the same attempt should still be marked as "verified"
#. Check the Gradebook (Instructor Dashboard -> Student Admin -> Gradebook) and check that the grade for the exam is 100

Reviewing "Errored" attempts
============================
Instructions: Create a couple of errored attempts by opening Chrome DevTools during an exam
#. Clicking the "Review Required" button pops up a modal that says "Update review status"
#. Clicking "Cancel" closes modal
#. Clicking "Verify" marks attempt as verified both in the dashboard and in the edx-exams admin
#. Clicking "Reject" marks attempt as reject both in the dashboard and in the edx-exams admin
INSTRUCTIONS: In a separate exam, start the exam, then open devtools using Cmd+Option+I on Mac (F12 on Windows). This should error out the exam and load a "Error with proctored exam" page
#. If you are using Proctorio, this should open the "Proctorio Support" page in another tab
#. Clicking the "Review Required" button should pop up a modal that says "Update review status"
#. Clicking "Cancel" should close the modal
#. Clicking "Verify" should mark the attempt as verified both in the dashboard and in the edx-exams admin
#. Clicking "Reject" should mark the attempt as reject both in the dashboard and in the edx-exams admin

Resetting an exam attempt
=========================
#. Clicking "Reset" on an exam attempt brings up a confirmation modal
#. Clicking "No (Cancel)" closes the modal and nothing changes
#. Clicking "Yes I'm Sure" removes the exam attempt from the table (effectively confirming that it's been reset)
#. The exam attempt no longer appears in the edx-exams admin
#. The learner's completion status for the exam is reset (green checkmark disappears)
#. Clicking "Reset" on an exam attempt should bring up a confirmation modal
#. Clicking "No (Cancel)" should close the modal
#. Clicking "Yes I'm Sure" should remove the exam attempt from the table (effectively confirming that it's been reset)
#. The learner's completion status for the exam should be reset (i.e. the green checkmark is now gone)
#. As a learner, try taking the exam again. You should be brought to the initial interstitials, as though this were your first time taking the exam.

0 comments on commit d1455d8

Please sign in to comment.