add controls for when grades are sent to the LMS

[Alex-Jordan]

This is a reworking of #2614. Some discussion there may be helpful but imo this can be considered a new PR. As with that PR though, I can not test this with D2L until it's less risky, because for now I can only test in production. If anyone else can test in Canvas or Moodle, and things get to the point they are working, I will test with D2L. I apologize in advance for my mistakes! I can at least say I tested that webwork2 starts, and nothing goes bad when I visit pages.

A few changes are packaged here. The big things are:

• $LTISendScoresAfterDate can be set, for example, to "close_date". It's described in the new conf/authen_LTI.conf.dist.
• $LTISendGradesEarlyThreshold can be set to a number from 0 to 1, or 'attempted'. It's described in conf/authen_LTI.conf.dist.

There are 2x2 pathways to consider when implementing these. First, is it LTI 1.1 or 1.3? And then, is grade passback in "course" or "homework" mode? For "homework" passback, the work is almost entirely in the two SubmitGrade.pm files. Specifically in the submit_set_grade subroutines. There is a small reliance on three new utilities in lib/WeBWorK/Utils/Sets.pm.

For "course" grade passback, the work happens in lib/WeBWorK/Utils/Sets.pm with its grade_all_sets subroutine, where it always has. But now with parameters sent so that grade_all_sets can understand the two new config settings and act accordingly.

Additional changes concerning $LTICheckPrior:

• Add $LTICheckPrior to the list of LTI config options that could be enabled on the Config page
• Before this, if $LTICheckPrior is set and somehow an LMS score is like 0.9999, it would not update to 1 even when the WeBWorK score is now 1. So I changed it to ignore a small nonzero difference if the WW score is 1.
• Now if using $LTICheckPrior, suppose an LMS score is null and the corresponding WW score is 0. Previously, the LMS would not be updated to 0. Now it will be updated to 0 if we are past the new $LTISendScoresAfterDate.

[somiaj]

I suggest that the defaults should be $LTISendScoresAfterDate = 'close_date' and $LTISendGradesEarlyThreshold = 'attempted'. I don't think we need to keep behavior the same as it is here, and this seems reasonable, any attempt will be sent back, but no need to send non attempted zeros back early.

[drgrice1]

I just looked at your changes with $LTIGradeOnSubmit now incorporated, and if I understand things correctly I don't think I like the change. What I want is when grades are submitted because $LTIGradeOnSubmit is true is that those are always sent (except the $LTICheckPrior case when the grades are the same). However, when a mass grade pass back occurs, then I want that to not send zero scores. Is that possible with what you have set up?

[Alex-Jordan]

I don't think I intentionally changed anything with how/when $LTIGradeOnSubmit has effect. But I can think through what you described and try it.

[somiaj]

@drgrice1 Wouldn't my suggested defaults above do just that? Grades will be sent once attempted (from the first time they hit submit), or after the closed date?

[Alex-Jordan]

Do I understand this right? If there is a set of 10 questions, and $LTIGradeOnSubmit is true, and a student answers the first question (correctly or incorrectly) then you want the score (0% or 10%) passed back? That's what I am trying to avoid, although I do not have $LTIGradeOnSubmit set.

Just seeing Jaimos's post. Is the behavior you want achieved by using $LTISendGradesEarlyThreshold = 'attempted'?

This is reminding me that I will have to change the message students get about whether or not (and when) their score will be sent to the LMS. I hadn't considered that.

[somiaj]

But I too found it confusing that $LTIGradeOnSubmit was linked into this, as I originally thought this was only about LTI mass update (from either the time interval or manually triggering it). But from my understanding the defaults I suggested would probably meet my use case.

[Alex-Jordan]

Oh I see, you do not want the score passed back during mass grade passback, and using attempted would do that. I'll think about it more. Signing off for a while now though.

[Alex-Jordan]

Did I change something with LTIGradeOnSubmit? I know I added to its documentation but I didn't think I changed its behavior. Just tried to improve the description.

[Alex-Jordan]

OK, I read my own words and I see it. I was acting on the idea that whatever calls the routines in SubmitGrade.pm is what it is (on submit or a mass passback). And then its those routines that were modified.

So we will want to know if these routines were called my mass passback or by a passback upon submit.

[somiaj]

@Alex-Jordan I almost commented on this, but then thought the defaults I suggested would work for me, so left it alone. I did find the wording a little confusing, "This is subject to the above settings." makes it sound like you meant that now the grade will only pass back if it meets the conditions set in $LTISendScoresAfterDate and $LTISendGradesEarlyThreshold. But I can't tell in the code if this is actually the case. Maybe you should put your new options after $LTIGradeOnSubmit as I'm still confused on if those settings affect that or not.

[somiaj]

Okay, I just double checked, and yes these settings do affect $LTIGradeOnSubmit because the logic calls the submit_course_grade or submit_set_grade methods appropriately, which are now subject to these new constraints. So this can change its behavior (though the defaults open_date and 0 won't chance current behavior).

[somiaj]

@Alex-Jordan ahh I see you answered your own question while I was typing and I missed it. One thing this already appears to do is make the configuration a bit more complicated.

[Alex-Jordan]

Right, I set the defaults so that nothing should change. [Technically, there are currently some unrealistic situations where a grade would be submitted before the set opens, and this PR would change things in those unrealistic situations and no longer send grades.]

Three things can trigger grade passback:

• an exercise submission
• mass passback because the mass passback time interval has passed
• mass passback because someone used the LTI Grades update tool

They all call the subroutines in SubmitGrades.pm which is where the code modification for this PR starts. So there is no distinguishing those three triggers as of this time.

But I could make the three things behave differently and that won't be hard. The question is though, that what @drgrice1 is requesting is not what I would want, even if I were using $LTIGradeOnSubmit. So I'm not sure about hard coding anything to work as Glenn describes.

How about if $LTIGradeOnSubmit can be

• 0 for off
• 1 (or most other truthy things) for behaving like it does at the moment with this PR
• some third setting to behave as Glenn describes: always pass the grade when the trigger was a submission, even if the threshold has not been met.

I don't have a preference about defaults. For my purposes I'll just set the settings to what I would like them to be. So whether defaults preserve current behavior or move to a new better behavior, whatever people want is good.

[drgrice1]

Do I understand this right? If there is a set of 10 questions, and $LTIGradeOnSubmit is true, and a student answers the first question (correctly or incorrectly) then you want the score (0% or 10%) passed back? That's what I am trying to avoid, although I do not have $LTIGradeOnSubmit set.

Yes, that is what I want. In fact, I realized that all I really want is when mass grade pushback occurs is a grade for a test that has no versions is not sent until after the due/course date. The biggest issue that keeps happening is a student opens a test from the LMS but doesn't start it. So now Webwork has what it needs to pass back grades, but no version exists. Then mass pass back submits a zero, and I get an email from a student asking me to re open the test for them (they don't bother to check to see that they can still go into the test and take it).

Note that an empty score from the LMS is not zero, and for LTI 1.1, even $LTICheckPrior doesn't prevent grade pass back.

[drgrice1]

So we will want to know if these routines were called my mass passback or by a passback upon submit.

As I mentioned in the previous pull request, this is already known in all of the SubmitGrade.pm methods (for both LTI 1.1 and 1.3). If $self->{post_processing_mode} is set, then it is a mass pass back, and otherwise it is from a $LTIGradeOnSubmit submission.

[Alex-Jordan]

I was somewhat surprised to find that:

$a;
$b = $a / 10;

does not cause an error. It results in $b being 0. The same happens with $a = '';.

With LTI 1.3 at least, I confirmed that '' is passed back during a "CheckPrior" check when the LMS grade is null. It then becomes 0 like the above when it gets divided by the total weight of the LMS assignment. So it ultimately compares as 0 against a WW score of 0 with no issue. But we can still detect when $priorData->[0]{resultScore} is empty and distinguish that from when it is 0.

Yeah I don't know about LTI 1.1. I also wonder if the LMS might pass back different things for different LMSs when the grade is null in the LMS. I can't test that with our D2L instance anymore.

[drgrice1]

For LTI 1.1, the LMS returns XML containing <textString/> when it has no grade. If it has a grade instead it returns <textString>numeric score</textString>. The $LTICheckPrior code in WeBWorK/Authen/LTIAdvanced/SubmitGrade.pm has this analysis, and it translates <textString/> into the empty string. Furthermore, in that case it doesn't even look at the score that webwork has, and it proceeds to submit the grade. So it is not the same as what happens with LTI 1.3 at all.

[drgrice1]

So there is a bug in the LTI 1.3 submit_grade method in the case that a user has opened a test, but not started a test. When mass grade passback occurs and $LTICheckPrior is set and $LTIGradeMode = 'homework'. In this case there are no set versions (of course), and the grade_gateway method just returns (0, 0). Then the submit_grade method fails with a divide by zero error. This is caused on line 277 of WeBWorK/Authen/LTIAdvantage/SubmitGrade.pm added in #2422.

[Alex-Jordan]

Sorry about that. I just added a commit that I think addresses it.

I'll work on the $LTIGradeOnSubmit change now.

[somiaj]

I don't have a setup where I can test this, but I'm okay with additional configuration options (even if it is a bit confusing at first). As for $LTIGradeOnSubmit, it is possible to get the behavior I would use with the current setup, and maybe some users who want to not submit grades until most of the assignment is done may want to pair your changes with $LTIGradeOnSubmit. You could make it more configurable and have a $LTIGradeOnSubmit option that controls if it follow your new rules or not.

[Alex-Jordan]

I have a question about this. Suppose a person is using "course" grade passback mode. What I've done so far means that when an overall course grade is calculated to send to the LMS, it will only factor in those sets where the threshold ('attempted' or some score) has been reached, or for which it is past the $LTISendScoresAfterDate.

Note that during any mass passback, this is how the grade calculation will be made.

@drgrice1 has requested that possibly with the right configuration, that if a student submits an answer to a set or grades a test, then grades should be passed back regardless of that set's date or threshold being met. What does this mean for a set

• that has not met the threshold
• that has not reached its $LTISendScoresAfterDate
• and we are using "course" grade mode?

Either:

1. In that moment, the overall grade that is calculated still omits sets that have not met their threshold or reached the $LTISendScoresAfterDate.
2. In that moment, the overall grade that is calculated still omits most sets that have not met their threshold or reached the $LTISendScoresAfterDate, but includes the set where a submission triggered this passback.
3. In that moment, the overall grade that is calculated using all sets, regardless of meeting thresholds or reaching dates.

(1) is pointless because the score will omit the set that triggered the passback.

(2) and (3) will change the score in the LMS. But then later, suppose there is a mass update and this set has still not met its threshold or reached its $LTISendScoresAfterDate. The mass update may calculate a different score than was calculated from the problem/test submission. And then change the LMS score, which will be confusing.

So should this feature that @drgrice1 would like to have be unavailable for "course" passback mode?

[somiaj]

I agree with your analysis, If you want to add a feature that $LTIGradeOnSubmit ignores these settings, it only makes sense to do so in homework mode.

Also, would it be worth while extending this beyond LMS passback. What about in the WeBWorK gradebook itself, these settings could control what sets are included in the total grade for users who show that to students (just a thought).

[somiaj]

I'm just going to restate my above suggestion, I think since we are adding these features we should update the defaults to be close_date and attempted, I don't think we should keep the old default behavior (and I think this is consistent with what I see most LTIs do, they keep grades empty until something has actually been done or the set is past due)

[Alex-Jordan]

Here is my new write-up to document the intended behavior of all these variables. I'm looking for feedback before I return to checking the behavior matches this description.

# There are several controls for when to report grades to the LMS. Sometimes these controls
# interplay with each other, and the details of how they work may depend on whether
# $LTIGradeMode is set to 'course' or 'homework'. So it is recommended to understand all of
# them and then decide how to set them.

# If $LTICheckPrior is 1, then any time WeBWorK is about to send a grade to the LMS, it will
# first request from the LMS what that grade currently is. Then if there is no significant
# difference between the LMS grade and the WeBWorK grade, WeBWorK will not follow through with
# updating the LMS grade. This is to avoid frequent insignificant updates to a student's grades
# in the LMS. With some LMSs, some students receive notifications each time a grade is updated,
# and setting this variable will prevent too many notifications for them. This does create a
# two-phase process, first querying the current grade from the LMS and then actually updating
# the grade (if there is a significant difference).

# Additional details:
# - If the LMS score is not 100%, but the WeBWorK score is, then even if the LMS score is only
#   insignificantly less than 100%, it will be updated anyway.
# - If the LMS score is null and the WeBWorK score is 0, this is usually considered an
#   insignificant difference and the LMS score will not be updated to 0. However if it is after
#   the $LTISendScoresAfterDate (described below), then the null score will be updated to 0
#   anyway.
# - "Significant" means an absolute difference of 0.001, or 0.1%. At this time this is not
#   configrable.
$LTICheckPrior = 0;

# If $LTIGradeOnSubmit is set to 1, then  each time a user submits an answer or grades a test,
# that will trigger WeBWorK possibly reporting a score to the LMS. See $LTICheckPrior for one
# reason that WeBWorK might not ultimately send a grade. But there are other reasons too.
# - If $LTIGradeMode is 'homework', then WeBWorK will send that assignment's score to the LMS
#   only if either the assignment's $LTISendGradesEarlyThreshold (described below) has been met
#   or if it is past that assignment's $LTISendScoresAfterDate (also described below).
# - If $LTIGradeMode is 'course', then WeBWorK will send the overall course grade to the LMS
#   only if either the assignment's $LTISendGradesEarlyThreshold (described below) has been met
#   or if it is past that assignment's $LTISendScoresAfterDate (also described below).

# If $LTIGradeOnSubmit is set to 'homework_always', then
# - If $LTIGradeMode is 'homework', then WeBWorK will send that assignment's score to the LMS
#   regardless of having met the $LTISendGradesEarlyThreshold or being past the
#   $LTISendScoresAfterDate
# - If $LTIGradeMode is 'course', the behavior will be no different as if $LTIGradeOnSubmit were
#   set to 1.

# Sending grades frequently may put a strain on the server, so you might want to turn this off
# by setting $LTIGradeOnSubmit = 0;

$LTIGradeOnSubmit = 1;
#$LTIGradeOnSubmit = 'homework_always';
#$LTIGradeOnSubmit = 0;

# In addition to scores possibly being sent to the LMS upon submission, they can be sent by an
# instructor or admin user using the LTI Grades Update Tool. And thirdly, the system can
# periodically update student grades on the LMS on its own. For all three possible triggers for
# grades to be passed to the LMS, $LTISendScoresAfterDate and $LTISendGradesEarlyThreshold can
# affect what is sent. $LTISendScoresAfterDate can be set to 'open_date', 'reduced_scoring_date',
# 'close_date', 'answer_date', or 'never'. For a given assignment, if it is *after* the
# $LTISendScoresAfterDate, then WeBWorK will send grades.
# - For 'course' grade passback mode, the assignment will be included in the overall course
#   grade calculation.
# - For 'homework' grade passback mode, the assignemnt's score will be sent.

# If $LTISendScoresAfterDate = 'reduced_scoring_date' and an assignent has no reduced_scoring_date
# the fallback is to use the close_date.

# For a given assignment, if it is *before* the $LTISendScoresAfterDate, WeBWorK *may* send a
# score to the LMS depending on $LTISendGradesEarlyThreshold. This variable can either be the
# string 'attempted' or a number from 0 to 1. If this variable is 'attempted', a given set must
# have been attempted for the threshold to have been met, and then the score can be used even if
# it is before the $LTISendScoresAfterDate. For a regular or jitar set, 'attempted' just  means
# that some exercise in the set was attempted. For a test, 'attempted means that either there
# is one version with a graded submission, or there are at least two versions.

# If $LTISendGradesEarlyThreshold is a number from 0 to 1, the score for an assignment needs to
# have reached that number for the threshold to be met, and then the score can be used even if
# it is before the $LTISendScoresAfterDate.

#$LTISendScoresAfterDate = 'open_date';
$LTISendScoresAfterDate = 'reduced_scoring_date';
#$LTISendScoresAfterDate = 'close_date';
#$LTISendScoresAfterDate = 'answer_date';
#$LTISendScoresAfterDate = 'never';

$LTISendGradesEarlyThreshold = 'attempted';
#$LTISendGradesEarlyThreshold = 0;
#$LTISendGradesEarlyThreshold = 0.7;
#$LTISendGradesEarlyThreshold = 1;

# The system periodically updates student grades on the LMS. If it has been at least this many
# seconds since the last mass passback event and someone in the course does anything to load a
# page, then a new mass passback job will begin. Set this to -1 to disable mass passback.
$LTIMassUpdateInterval = 86400;

[somiaj]

Overall sounds good, a few things I notice over my first quick read through.

• With $LTIGradeOnSubmit=1 description, your course and homework descriptions are the same, you can probably combine them.
• "If $LTISendScoresAfterDate = 'reduced_scoring_date' and an assignent has no reduced_scoring_date the fallback is to use the close_date.", I think this might be incorrect behavior (and also means the code needs updated). I think it is possible for an assignment to have a reduced_scoring_date, but reduced scoring be disabled for that set or for the course. I think you need also check if reduced scoring is indeed enabled here.
• This document seems a bit overly detailed for being aimed at webwork admins (in the server configuration file file). Many users may only encounter these settings in the course configuration settings. So those should probably be updated to match.

[Alex-Jordan]

I tried to use GH tools to address a small merge conflict that arose. Maybe a mistake, maybe not.

Later (this weekend?) I will do testing with the functionality of all this again. If you can confirm you tested with LTI 1.3 with Canvas and Moodle, I may be OK to try this all out with D2L and LTI 1.3. I just have to try it out in production.

[drgrice1]

It looks like you fixed the merge conflict correctly.

I have tested this with Moodle, but I haven't gotten around to testing with Canvas yet. I will do that as soon as I can.

[drgrice1]

I have now tested this with Canvas. Everything seems to be working well. If you feel like it you could add debugging messages as suggested in my comments.

[drgrice1]

We might want to add a debug message here. I was just testing this with Canvas, and found it useful to determine that a grade was not sent due to it not meeting the requirements. Something like

Suggested change

	return -1 unless $score;
	if ($submittedSet && !getSetPassbackScore($db, $ce, $userID, $submittedSet, 1)) {
	$self->warning('It is not after the $LTISendScoresAfterDate and the $LTISendGradesEarlyThreshold was not met. '
	. 'Not updating grade.');
	return -1;
	}

I am not sold on using the variable names in the message, although this is just a debugging message. Perhaps just use words?

Also a similar message above for the submit_course_grade case would be good.

[Alex-Jordan]

I'm just checking if there was importance behind the condition in the if in your comment here. I have something like this now but it's just if (!$score).

[drgrice1]

I think this was inside the submit_course_grade method. Currently line 136 in LTIAdvanced/SubmitGrade.pm and line 218 of LTIAdvantage/SubmitGrade.pm which is return -1 if $submittedSet && !getSetPassbackScore($db, $ce, $userID, $submittedSet, 1); there is no message.

[somiaj]

This looks like it is about ready to go to me. I can't fully test this as I don't have a development system using LTI, but the defaults and ideas look good to me. @Alex-Jordan do you want to update the final debugging suggestions?

[Alex-Jordan]

I'm out of today, but I wanted to do a last round of D2L testing. If we merge now I'll get lazy and not do that :)

…

On Sun, Nov 24, 2024, 10:51 AM Jaimos Skriletz ***@***.***> wrote: This looks like it is about ready to go to me. I can't fully test this as I don't have a development system using LTI, but the defaults and ideas look good to me. @Alex-Jordan <https://urldefense.com/v3/__https://github.com/Alex-Jordan__;!!Ka_JY85zDv0FFw!mMRp1vb9rNQF-zxde64NPoApql2YB0lTkwuqRFvXYRQXZukpcCRmglkPd1FxWDgWuQeHJ8z5hEzZL9_IuN9VvxbKuxA$> do you want to update the final debugging suggestions? — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/openwebwork/webwork2/pull/2617*issuecomment-2496168904__;Iw!!Ka_JY85zDv0FFw!mMRp1vb9rNQF-zxde64NPoApql2YB0lTkwuqRFvXYRQXZukpcCRmglkPd1FxWDgWuQeHJ8z5hEzZL9_IuN9V_PDrA3g$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABEDOAATM2V475MLD6NHIA32CIN4TAVCNFSM6AAAAABROOJ5SCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJWGE3DQOJQGQ__;!!Ka_JY85zDv0FFw!mMRp1vb9rNQF-zxde64NPoApql2YB0lTkwuqRFvXYRQXZukpcCRmglkPd1FxWDgWuQeHJ8z5hEzZL9_IuN9VZE8YkBY$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[drgrice1]

I am wandering if the submit_course_grade method (with the setting $LTIGradeMode = 'course';) is working the way you really want with this pull request. The case in question is the case that the $LTISendGradesEarlyThreshold setting is numeric, say it is set to 0.7.

For the following, I assume that LTI 1.3 is in use.

Assume that it is the beginning of the semester, and a student enters webwork via a link in the LMS, but doesn't submit any answers for anything. So now webwork has everything it needs to pass grades back for this student. Now assume that a mass grade pass back occurs. The grade_all_sets method will return (0, 0) for this student when called in list context because no sets meet the 0.7 threshold. This is not a submission, so there isn't a user set that is initially checked in the submit_course_grade method, and so the submit_grade method is called with the (0, 0) score for this student. The LMS returns no grade, and so the check for a prior score doesn't stop a grade submission, and that (0, 0) score is submitted. I tested this with Moodle, and Moodle just takes that as a grade of 0. Is that what is desired? Or should the course grade be checked to ensure it satisfies the threshold also? If that is the case, then in the submit_course_grade method the grade_all_sets method should be called outside of the parameters for the submit_grade method, and its return value checked to ensure it meets the pass back criteria, and if it does then that return value should be passed on to the submit_grade method. Otherwise, no grade should be submitted.

[Alex-Jordan]

I would not want to check the course grade against the numerical threshold. The score could be lower than the threshold and you may still want that course score to go to the LMS simply because various critical dates have passed.

Here is an idea I have not thought through yet (and I have to leave for the day soon for PT conferences). What if grade_all_sets returns three things (when wantarray is true). The two things it currently returns, plus a count of how many sets were included.

Getting (0,0) can happen because no sets were included in the calculation. Or because a nonzero number of sets were included but each had a "0/0" score. The point is to distinguish these. And in the scenario you described, we would have (0,0,0) and we could just not send anything to the LMS in that case and I think that would be fine. Getting something like (0,0,1) is something I have not thought through yet. Just like all the situations in "homework" grade mode where there could be a "0/0" score.

[drgrice1]

That sounds like a reasonable solution.

[Alex-Jordan]

As long as I'm changing that, is there any reason to make it more than a simple count? An alternative is that it could be an array reference with references to each of the included user set objects.

[Alex-Jordan]

One reason I see where that may be helpful is that I can add a log message listing the sets that were included in the calculation when the overall course score is sent to the LMS. It could be a bit of a long log message. What do you think?

[drgrice1]

Sounds fine to me. Certainly, the list of included user set objects would generally be more useful, and provide much more information.

[Alex-Jordan]

OK, I added a commit that does that. Hopefully it is easy to just review that commit. Also I caught an error during testing that is fixed in a separate commit. Thirdly, I notices that in "homework" mode, the log would have entries like Submitting grade for user alex.jordan and set Demo. even under conditions when WeBWorK was not going to send the LMS a score. I moved those log entries in the code so they only are used if we are indeed going to try to send the LMS a score.

I did a fairly thorough round of testing with LTI 1.1. Things with LTI 1.3 are not really different now, as far as how this PR goes now. I haven't tested with LTI 1.3, but I feel OK that if it works for LTI 1.1, it will work for LTI 1.3. The necessary differences in the two SubmitGrade.pm look right to me.

[drgrice1]

I am not sure that I like that the messages Submitting score for $userID... were moved down. For debugging purposes it is good to have a message in any case. Furthermore, there are now messages that occur earlier then this message that refer to the user. For example, the message "LMS user id is not available for this user" in the submit_set_grade method for LTI 1.3 now occurs with no reference to which user the message regards. So if this message is moved down, then the messages before that need to now include the $userID.

I think moving the message down also makes the messages I previously suggested for when a grade is not submitted necessary.

[Alex-Jordan]

I could move the messages back but then I think they need to change what they say. The log was looking like:

Submitting grade for user XXX and set YYY.

And this was happening in situations where no grade was in fact submitted. Then there was no other message related to an LTI grade passback attempt, and so the implication was something was indeed sent to the LMS. I think that would be pretty misleading to someone reading the logs while debugging/configuring.

[drgrice1]

I agree, but that is why I had previously suggested adding messages in the case that grades were not sent. You don't need to move the messages back, but the other messages need to be updated to accommodate the change. The messages that say there is missing pass back data for the student need to include the $userID in the message to indicate which user the pass back data is missing for.

[drgrice1]

You could also move the message back, but change it to say something like "Considering submitting grade for $userID", although perhaps better wording. Hopefully you get the idea from that poorly chosen wording though.

[Alex-Jordan]

OK, take a look now at the debug messages. Sorry, before I had not seen your comments about those from Nov 21.

Also I reverted to letting grade_all_sets have a scalar context return. But nothing is using it at this time.

[drgrice1]

Yeah, since nothing is using the grade_all_sets method except LTI grade pass back, I almost moved it into the WeBWorK::Authen::LTI::GradePassback package. But then I came up with the callback approach that made it more versatile (and was kind of fun to implement with the default signature parameter that is an anonymous sub).

Now all of the sub grade... methods in the file are consistent in their return behavior though.

[drgrice1]

This should be return -1. A return value of 0 generally means that something went wrong with the grade pass back process. A return value of 1 means success (not sending a grade because it was unchanged with $LTICheckPrior is considered success). A return value of -1 means that the grade was not passed back because the criteria were not met which is what is happening in this case.

The different return values give a different type of messaged in the passbackGradeOnSubmit method. For a return value of -1, the message is just "'Your score will be sent to the LMS at a later time." It isn't appropriate to say that the grade will be passed back in some specified amount of time, because at that time the criteria may not be met still, and so the grade may still not be passed back.

The same change of course is needed at this place in the LTI 1.3 method.

[drgrice1]

Thanks for fixing that. I am going to merge this.