docs: proctored exam testing instructions #205

ilee2u · 2023-10-26T15:18:53Z

JIRA: Link to JIRA ticket

Description: Describe in a couple of sentences what this PR adds

Author concerns: List any concerns about this PR - inelegant
solutions, hacks, quick-and-dirty implementations, concerns about
migrations, etc.

Dependencies: dependencies on other outstanding PRs, issues, etc.

Installation instructions: List any non-trivial installation
instructions.

Testing instructions:

Open page A
Do thing B
Expect C to happen
If D happened instead - check failed.

Merge checklist:

Post merge:

Delete working branch (if not needed anymore)

zacharis278

few comments to get started.

High level, it would be nice to make this less Proctorio specific. If we do have a few Proctorio specific cases we want to cover we keep that as a separate test. The user flow should mostly be the same regardless of the tool used.

docs/test_plan/proctored_exam.rst

zacharis278 · 2023-11-02T20:01:24Z

docs/test_plan/proctored_exam.rst

+    #. You should also have a tab open with your event bus logs
+
+
+Proctored Exam Flow


I think these another set of cases here for submitting an attempt that is auto verified and one needs a second review.

I still think we're missing these test cases ensuring a review by the proctoring provider is received and the attempt is updated. We kind of do that as a side effect of setup for testing the instructor dashboard but I think it's best to be explicit. If someone is just testing exam flow they should be validating this case.

I added a test to the Proctorio section about checking that a review of the exam exists. Regardless of whether or not we're using Proctorio, wouldn't the test that we received a review by a proctoring provider be checking the exams dashboard itself?

I think the test I'm looking for is something that validates the ACS works as intended. So a test along the lines of "if a user submits an attempt without violation the proctoring software can update the attempt to verified" / "if a user submits with violations the proctoring software can update the attempt to review required".

I think that's a distinctly different test from testing that a verified/rejected/second review attempt can be reset in the UI since that testing instructor experience but the ACS is part of the learner / exam flow.

docs/test_plan/proctored_exam.rst

MichaelRoytman

I know I've left a lot of feedback. I read these instructions from the perspective of a team member with limited familiarity with LTI for proctoring, Proctorio, and how to set up proctoring. I think there there is a good amount of assumed knowledge in this document that not everyone will have, and my comments reflect that. I'm not marking my feedback as blocking. I know it's a pain, but I do think this document could really benefit from being a lot more explicit.

MichaelRoytman · 2023-11-07T15:16:27Z

docs/test_plan/proctored_exam.rst

+    #. As your non-staff user in your current window, and enroll in the course where you have set up your proctored exam
+    #. In an incognito window, login to your staff account and visit your Support Tools page and search in the email for your non-staff user. You should now see the option to change the enrollment track of that user. Change the enrollment track of the user to ""verified"" (choose any reason when prompted, it doesn't matter)
+    #. In that same incognito, still signed in as staff, navigate to the Instructor dashboard
+    #. Find or create a test course within Stage with proctoring enabled and an LTI proctoring provider chosen, certificates are enabled, and where the exams IDA waffle flag is enabled


I think that the order of these prerequisite steps is a little confusing. For example, you're enrolling a non-staff learner account in "this course" before you have set up a course with proctoring. Based on your instructions, you may set up an entirely new course in which the learner is not enrolled.

I'd recommend breaking this up into "Proctoring Set Up Steps" and "User Set Up Steps" to make this clearer.

Do you have a particular environment in mind here (e.g. stage)? Stage will have proctoring set up in a way that someone just starting in their devstack will not, so I think we should either not make references to a particular environment and link out to documentation for each environment (e.g. stage/production and development), or specify that this is just for stage/production.

"certificates are enabled" is a little vague and could mean a variety of things. I think we should be more specific here. Do you mean "where there is a verified track" or something else?

I know we don't have official documentation for this yet, but it would be good to clarify what "an LTI proctoring provider chosen" means - how does one do this or verify this? Depending on which environment this documentation applies to, you could just refer to the Proctoring Settings Page and/or reference our Local Development and LTI Configuration document for the development environment.

Changed the order of the steps as you recommended (still need to push the change)

I have local in mind, but based on this thread it seems like it's also fair to mention that we do have a test course in stage (at least as of now)

I'm honestly not too sure how certificates work so if "verified track" makes more sense I can just put that.

Assuming edx-exams ends up in Open edX, would it make sense to just move the instructions for Local Development and LTI Configuration to the README and just make that the source of truth?

for the local instructions you are correct and those do need to move as part of the release to openedx. We've got that noted as a requirement here: https://2u-internal.atlassian.net/wiki/spaces/PT/pages/626753683/Beta+Release+and+Beyond.

I do think these tests need to be done in a deployed environment and not Tutor/Devstack in order to say we're confident in releasing these features.

@MichaelRoytman I do think there's also going to be a gap until we get instructions in readthedocs. I don't think we should duplicate how to setup a proctored exam, a user, a proctoring provider, etc here and should just reference readthedocs where needed.

Yeah, totally agree with linking out. I didn't mean to suggest we should duplicate - just that I felt like some information was missing, which can definitely be added by referencing other documentation, like "Set up a proctored exam: ."

docs/test_plan/proctored_exam.rst

MichaelRoytman · 2023-11-07T15:26:38Z

docs/test_plan/proctored_exam.rst

+- `Configuring a Proctoring Provider <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/proctored_enabling.html#configuring-proctoring-provider>`_
+- `Creating a Proctored Exam <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/pt_create.html#creating-a-proctored-exam>`_


Neither of these links is used in this document.

Ah these are copied over from timed_exam.rst, where it seems like they aren't used either. I see what you're saying though, I can have these links referenced in the setup instructions where it makes sense.

I think this might be how they are referenced the other docs has them right next to the step to create a proctored exam. Maybe we just want to point back to this if these instructions are needed anywhere later in the doc. If you use headings you can add internal links within the doc like this https://github.com/openedx/edx-proctoring/blob/master/docs/testing/test_plan.md?plain=1#L220. That can also be good to avoid repeating yourself if one test depends on the setup of another

Ah, I see. So it's like reference materials? My bad - got confused!

docs/test_plan/proctored_exam.rst

- added better setup

- Also added instructions for creating exam attempt data

ilee2u · 2023-11-07T21:17:24Z

Made some changes based off @MichaelRoytman and @zacharis278's comments, still have some work to do though.

MichaelRoytman

Thanks for addressing my first set of feedback.

I added some non-blocking feedback, which you can take or leave.

I agree with Zach regarding the ACS tests.

MichaelRoytman · 2023-11-08T18:46:46Z

README.rst

@@ -59,6 +59,36 @@ Devstack Set Up

 You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.

+LTI Configuration for Local Development
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:


Suggested change

In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:

In order to use edx-exams as the exams backend for your courses (and to get it to work in general), you will also need to complete the following configuration steps:

MichaelRoytman · 2023-11-08T18:47:10Z

README.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:
+
+#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.


Nit.

Suggested change

#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.

#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment settings file.

MichaelRoytman · 2023-11-08T18:48:24Z

README.rst

+In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:
+
+#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.
+#. In your terminal, navigate to your 


It looks like you forgot to complete the sentence or finished it below and forgot to remove this fragment.

MichaelRoytman · 2023-11-08T18:49:20Z

README.rst

+
+#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.
+#. In your terminal, navigate to your 
+#. In your terminal, run the frontend-app-course-authoring MFE by navigating to the devstack folder and running large-u .


Do you mean make dev.up.large-and-slow?

MichaelRoytman · 2023-11-08T18:49:45Z

README.rst

+#. In your terminal, run the frontend-app-course-authoring MFE by navigating to the devstack folder and running large-u .
+#. Ensure that the exams IDA is enabled for your course. In your LMS' django admin's course waffle flag table (http://localhost:18000/admin/waffle_utils/waffleflagcourseoverridemodel/) add an entry:
+    #. Enter course_apps.exams_ida in the Waffle Flag field
+    #. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023


Nit.

Suggested change

#. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023

#. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023)

MichaelRoytman · 2023-11-08T18:56:42Z

README.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:
+
+#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.


It can be helpful to add a little code formatting to isolate settings and code from the instruction.

Suggested change

#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.

#. Add ``EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1'`` to your cms and lms private.py environment.

MichaelRoytman · 2023-11-08T18:59:39Z

README.rst

@@ -59,6 +59,36 @@ Devstack Set Up

 You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.

+LTI Configuration for Local Development


Very nice to have these instructions here now!

Are these meant to replace the Local Development and LTI Configuration Confluence page? If so, it would be good to leave a note on that Confluence page referring to this page and to remove the Setting up an exam and proctoring tool section at the bottom of this page, which refers to that Confluence document.

If not, disregard!

I'd say so. Since this is meant to be seen by Open Edx's community, I believe it makes sense for this public README to be the source of truth instead of our internal doc. I'll make sure to do that link once this is merged.

MichaelRoytman · 2023-11-08T19:00:23Z

docs/test_plan/proctored_exam.rst

+- `Configuring a Proctoring Provider <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/proctored_enabling.html#configuring-proctoring-provider>`_
+- `Creating a Proctored Exam <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/pt_create.html#creating-a-proctored-exam>`_


Ah, I see. So it's like reference materials? My bad - got confused!

MichaelRoytman · 2023-11-08T19:03:06Z

docs/test_plan/proctored_exam.rst

+    #. As your non-staff user in your current window, and enroll in the course where you have set up your proctored exam
+    #. In an incognito window, login to your staff account and visit your Support Tools page and search in the email for your non-staff user. You should now see the option to change the enrollment track of that user. Change the enrollment track of the user to ""verified"" (choose any reason when prompted, it doesn't matter)
+    #. In that same incognito, still signed in as staff, navigate to the Instructor dashboard
+    #. Find or create a test course within Stage with proctoring enabled and an LTI proctoring provider chosen, certificates are enabled, and where the exams IDA waffle flag is enabled


Yeah, totally agree with linking out. I didn't mean to suggest we should duplicate - just that I felt like some information was missing, which can definitely be added by referencing other documentation, like "Set up a proctored exam: ."

docs/test_plan/proctored_exam.rst

zacharis278

I'd say we're looking good. One quick thing I noticed, there's a parent readme for this folder that has a link to the other sets of tests. Can you add a link there? https://github.com/edx/edx-exams/tree/main/docs/test_plan

ilee2u force-pushed the ilee2u/proctored_test_docs branch from 432ac5d to 463f75b Compare October 26, 2023 15:19

ilee2u changed the title ~~docs: proctored exam testing draft~~ docs: proctored exam testing instructions Oct 26, 2023

zacharis278 reviewed Oct 30, 2023

View reviewed changes

ilee2u force-pushed the ilee2u/proctored_test_docs branch from f48b9cc to 6f69996 Compare November 1, 2023 15:10

zacharis278 reviewed Nov 2, 2023

View reviewed changes

alangsto reviewed Nov 3, 2023

View reviewed changes

docs/test_plan/proctored_exam.rst Show resolved Hide resolved

alangsto reviewed Nov 3, 2023

View reviewed changes

docs/test_plan/proctored_exam.rst Outdated Show resolved Hide resolved

ilee2u force-pushed the ilee2u/proctored_test_docs branch from 3698f55 to 9bfa421 Compare November 6, 2023 20:08

MichaelRoytman reviewed Nov 7, 2023

View reviewed changes

ilee2u added 7 commits November 7, 2023 16:02

docs: proctored exam testing draft

973cfc2

docs: improved testing instructions

2f47542

docs: less Proctorio specific, more UI focused

2f2371f

docs: improved test names + organization

b687b0b

docs: fixed headers + error/reset info

d4d5642

chore: typo

788eb68

docs: center tests around local dev context

d0f1977

- added better setup

ilee2u force-pushed the ilee2u/proctored_test_docs branch from 72ee12c to d0f1977 Compare November 7, 2023 21:02

ilee2u added 2 commits November 7, 2023 16:13

docs: distinguishing new exams dashboard

c68f7a9

- Also added instructions for creating exam attempt data

docs: added better dashboard setup instructions

441e851

ilee2u added 2 commits November 7, 2023 16:53

docs: better exam flow/dashboard test split

6ff3a30

docs: removed test for vendor suspicion level

acb7e9a

MichaelRoytman reviewed Nov 8, 2023

View reviewed changes

ilee2u added 3 commits November 9, 2023 13:56

docs: added tests for proctorio ACS

75107a9

docs: fixed README local dev setup info

d18553d

chore: removing development readme bit

8b48282

zacharis278 approved these changes Nov 9, 2023

View reviewed changes

ilee2u added 2 commits November 9, 2023 15:20

docs: parent link

29d15ff

chore: small nit

394568a

ilee2u merged commit c54428a into main Nov 9, 2023
7 checks passed

ilee2u deleted the ilee2u/proctored_test_docs branch November 9, 2023 20:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: proctored exam testing instructions #205

docs: proctored exam testing instructions #205

ilee2u commented Oct 26, 2023

zacharis278 left a comment

zacharis278 Nov 2, 2023

zacharis278 Nov 7, 2023

ilee2u Nov 7, 2023

zacharis278 Nov 8, 2023

MichaelRoytman left a comment

MichaelRoytman Nov 7, 2023

MichaelRoytman Nov 7, 2023

MichaelRoytman Nov 7, 2023

MichaelRoytman Nov 7, 2023

ilee2u Nov 7, 2023

zacharis278 Nov 8, 2023

zacharis278 Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 7, 2023

ilee2u Nov 7, 2023

zacharis278 Nov 8, 2023

MichaelRoytman Nov 8, 2023

ilee2u commented Nov 7, 2023

MichaelRoytman left a comment

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

ilee2u Nov 9, 2023

MichaelRoytman Nov 8, 2023

MichaelRoytman Nov 8, 2023

zacharis278 left a comment

		#. You should also have a tab open with your event bus logs


		Proctored Exam Flow

		- `Configuring a Proctoring Provider <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/proctored_enabling.html#configuring-proctoring-provider>`_
		- `Creating a Proctored Exam <https://edx.readthedocs.io/projects/edx-partner-course-staff/en/latest/proctored_exams/pt_create.html#creating-a-proctored-exam>`_

	In order to use edx-exams as the exams backend for your courses (and to get it to wor in general), you will also need to complete the following configuration steps:
	In order to use edx-exams as the exams backend for your courses (and to get it to work in general), you will also need to complete the following configuration steps:

	#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.
	#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment settings file.

	#. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023
	#. Enter the course ID for the course ID field (e.g. something like course-v1:edX+COURSENAME+T2023)

	#. Add EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1' to your cms and lms private.py environment.
	#. Add ``EXAMS_SERVICE_URL = 'http://host.docker.internal:18740/api/v1'`` to your cms and lms private.py environment.

		@@ -59,6 +59,36 @@ Devstack Set Up

		You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.

		LTI Configuration for Local Development

docs: proctored exam testing instructions #205

docs: proctored exam testing instructions #205

Conversation

ilee2u commented Oct 26, 2023

zacharis278 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelRoytman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilee2u commented Nov 7, 2023

MichaelRoytman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zacharis278 left a comment

Choose a reason for hiding this comment