Add unit test cases for DB scheduling (#523)

docs/Dispatcher Scheduling DB Design.md

@@ -17,7 +17,7 @@ The database to be used will be SQLite3.  The database will be used to keep trac
 
 The goal of this design is to have a database design that adheres to the principles of Normal Form.
 
-NOTE:  The introduction of a task ID may be confusing.  It exists because there will be repeated schedules added to the database which will have the same JobID and manifest.   By adding an auto-incremented taskID, we are able to schedule and track subsequent runs of that same job/manifest and maintain our database integrity.
+NOTE: If the Dispatcher receives a new scheduled manifest, the contents of the APSheduler and Database will be removed and replaced with the entire contents of the received manifest.  Any schedules that were in the previous manifest and not in the new one will no longer be scheduled.
 
 ## ER Model
 
@@ -99,10 +99,10 @@ NOTE: These values may not make sense in the real world.  Just for demonstration
 
 | id | request_id | cron_duration | cron_minutes | cron_hours | cron_day_month | cron_month |cron_day_week |
 | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
-| 1 | 123 | P1D | 0 | */3 | * | * | * |
-| 2 | 123 | P7D | 0 | */6 | * | * | * |
-| 3 | 234 | P2D | 0 | */8 | * | * | * |
-| 4 | 234 | P14D | 0 | * | * | * | * |
+| 1 | 6bf587ac-1d70-4e21-9a15-097f6292b9c4 | P1D | 0 | */3 | * | * | * |
+| 2 | 6bf587ac-1d70-4e21-9a15-097f6292b9c4 | P7D | 0 | */6 | * | * | * |
+| 3 | c9b74125-f3bb-440a-ad80-8d02090bd337 | P2D | 0 | */8 | * | * | * |
+| 4 | c9b74125-f3bb-440a-ad80-8d02090bd337 | P14D | 0 | * | * | * | * |
 
 ### SINGLE_SCHEDULE_JOB Table
 
@@ -135,9 +135,9 @@ Example: To do a download, install, and reboot of SOTA at the repeated time in s
 | cron_day_week | field in a cron job schedule that specifies the day of the week on which the task should run, ranging from 0 (Sunday) to 6 (Saturday).     | Yes |
 | end_time | ending date/time for a single scheduled request. | No |
 | id    | Auto-generated by DB.  The request Ids will not be unique in the tables to use as a PK. | Yes |
-| job_id | ID for each job assigned by MJunct.  It will have an abbreviation of the job type in front of a UUID | Yes |
+| job_id | ID for each job assigned by MJunct.  It will have an abbreviation of the job type in front of a UUID.  All manifests in the same schedule will have the same jobId. | Yes |
 | manifest | valid Inband Manageability manifest | Yes |
-| request_id | Request ID generated by MJunct that is used to trace the request.| Yes |
+| request_id | Request ID generated by MJunct that is used to trace the request.  Every schedule and job in the manifest will have the same requestId. | Yes |
 | start_time | starting date/time for a single scheduled request.| Yes |
 | status | indicates if the request has been scheduled.  Not scheduled unless 'scheduled' is indicated in the field.  This is used to ensure the same manifest is not ran after a reboot | No |
-| task_id | Autoincremented number in the JOB table to store jobs and their manifests.   | Yes |
+| task_id | Autoincremented number in the JOB table to store jobs and their manifests.  Used to differentiate when a schedule has multiple manifests.  They would all have the same requestId and JobId, so that taskId ensures the DB conforms to Normal Form.   | Yes |

docs/Scheduled Manifest Parameters.md

@@ -4,6 +4,8 @@
 
 This XML schema defines the structure for a scheduled request. It includes types for cron values, manifests, schedules, and the overall schedule request.
 
+NOTE: If the Dispatcher receives a new scheduled manifest, the contents of the APSheduler and Database will be removed and replaced with the entire contents of the received manifest.  Any schedules that were in the previous manifest and not in the new one will no longer be scheduled.
+
 ## XML Structure
 
 ```xml
@@ -32,29 +34,32 @@ This XML schema defines the structure for a scheduled request. It includes types
 
 ```xml
 <single_schedule>
-    <start_time>2023-04-01T08:00:00</start_time>
-    <end_time>2023-04-01T17:00:00</end_time>
+  <job_id>swupd-f02131f9-b7d9-4e3f-9ee2-615e0fe005a5</job_id>
+  <start_time>2023-04-01T08:00:00</start_time>
+  <end_time>2023-04-01T17:00:00</end_time>
 </single_schedule>
 ```
 
 ## Single Schedule with No End Time
 
 ```xml
 <single_schedule>
-    <start_time>2023-04-01T08:00:00</start_time>
+  <job_id>swupd-f02131f9-b7d9-4e3f-9ee2-615e0fe005a5</job_id>
+  <start_time>2023-04-01T08:00:00</start_time>
 </single_schedule>
 ```
 
 ## Repeated Schedule
 
 ```xml
 <repeated_schedule>
-    <duration>P1D</duration> <!-- P1D means a period of one day -->
-    <cron_minutes>0</cron_minutes> <!-- At minute 0 -->
-    <cron_hours>*/3</cron_hours> <!-- Every 3 hours -->
-    <cron_day_month>*</cron_day_month> <!-- Every day of the month -->
-    <cron_month>*</cron_month> <!-- Every month -->
-    <cron_day_week>*</cron_day_week> <!-- Every day of the week -->
+  <job_id>swupd-f02131f9-b7d9-4e3f-9ee2-615e0fe005a5</job_id>
+  <duration>P1D</duration> <!-- P1D means a period of one day -->
+  <cron_minutes>0</cron_minutes> <!-- At minute 0 -->
+  <cron_hours>*/3</cron_hours> <!-- Every 3 hours -->
+  <cron_day_month>*</cron_day_month> <!-- Every day of the month -->
+  <cron_month>*</cron_month> <!-- Every month -->
+  <cron_day_week>*</cron_day_week> <!-- Every day of the week -->
 </repeated_schedule>    
 ```
 
@@ -73,11 +78,13 @@ This XML schema defines the structure for a scheduled request. It includes types
 
 ### SingleSchedule
 
+- **job_id**: Assigned by MJunct to track each individual schedule request.  In the format of an abbreviated job type descriptor (4-9 characters) followed by a UUID.
 - **start_time** (optional): The start time of the schedule, represented as a dateTime.
 - **end_time** (optional): The end time of the schedule, represented as a dateTime.
 
 ### RepeatedSchedule
 
+- **job_id**: Assigned by MJunct to track each individual schedule request.  In the format of an abbreviated job type descriptor (4-9 characaters) followed by a UUID.
 - **duration**: The duration of the schedule, represented as an XML duration.
 - **cron_minutes**: The minute component of the cron schedule, using `MinutesWithinHourCronValue`.
 - **cron_hours**: The hour component of the cron schedule, using `HourWithinDayCronValue`.

inbm/dispatcher-agent/dispatcher/schedule/apscheduler.py

@@ -15,6 +15,7 @@
 from apscheduler.schedulers.background import BackgroundScheduler
 from .sqlite_manager import SqliteManager
 from ..constants import SCHEDULED
+from ..dispatcher_exception import DispatcherException
 
 logger = logging.getLogger(__name__)
 
@@ -38,7 +39,8 @@ def remove_all_jobs(self) -> None:
         logger.debug("Remove all jobs in APScheduler")
         self._scheduler.remove_all_jobs()
 
-    def add_single_schedule_job(self, callback: Callable, single_schedule: SingleSchedule) -> None:
+    def add_single_schedule_job(self, callback: Callable, 
+                                single_schedule: SingleSchedule) -> None:
         """Add the job for single schedule.
 
         @param callback: The function to be called.
@@ -47,9 +49,13 @@ def add_single_schedule_job(self, callback: Callable, single_schedule: SingleSch
         logger.debug("")
         if self.is_schedulable(single_schedule):
             self._sqlite_mgr.update_status(single_schedule, SCHEDULED)
-            for manifest in single_schedule.manifests:
-                self._scheduler.add_job(
-                    callback, 'date', run_date=single_schedule.start_time, args=[manifest])
+            try:
+                for manifest in single_schedule.manifests:
+                    self._scheduler.add_job(
+                        func=callback, trigger='date', run_date=single_schedule.start_time, args=[manifest])
+            except (ValueError, TypeError) as err:
+                raise DispatcherException(f"Please correct and resubmit scheduled request. Invalid parameter used in date expresssion to APScheduler: {err}")
+
 
     def add_repeated_schedule_job(self, callback: Callable, repeated_schedule: RepeatedSchedule) -> None:
         """Add the job for repeated schedule.
@@ -60,16 +66,19 @@ def add_repeated_schedule_job(self, callback: Callable, repeated_schedule: Repea
         logger.debug("")
         if self.is_schedulable(repeated_schedule):
             self._sqlite_mgr.update_status(repeated_schedule, SCHEDULED)
-            for manifest in repeated_schedule.manifests:
-                self._scheduler.add_job(callback, 'cron', args=[manifest],
-                                        start_date=datetime.now(),
-                                        end_date=self._convert_duration_to_end_time(
-                                            repeated_schedule.cron_duration),
-                                        minute=repeated_schedule.cron_minutes,
-                                        hour=repeated_schedule.cron_hours,
-                                        day=repeated_schedule.cron_day_month,
-                                        month=repeated_schedule.cron_month,
-                                        day_of_week=repeated_schedule.cron_day_week)
+            try:
+                for manifest in repeated_schedule.manifests:
+                    self._scheduler.add_job(func=callback, trigger='cron', args=[manifest],
+                                            start_date=datetime.now(),
+                                            end_date=self._convert_duration_to_end_time(
+                                                repeated_schedule.cron_duration),
+                                            minute=repeated_schedule.cron_minutes,
+                                            hour=repeated_schedule.cron_hours,
+                                            day=repeated_schedule.cron_day_month,
+                                            month=repeated_schedule.cron_month,
+                                            day_of_week=repeated_schedule.cron_day_week)
+            except (ValueError, TypeError) as err:
+                raise DispatcherException(f"Please correct and resubmit scheduled request. Invalid parameter used in cron expresssion to APScheduler: {err}")
 
     def is_schedulable(self, schedule: Schedule) -> bool:
         """Check if the schedule can be scheduled.
@@ -116,7 +125,7 @@ def _check_single_schedule(self, schedule: SingleSchedule) -> bool:
             schedule.start_time = datetime.now() + timedelta(seconds=2)
             return True
 
-        logger.error("The start time or current time is greather than end time. Will not schedule")
+        logger.error("The start time or current time is greater than the end time. Not scheduled.")
         return False
 
     def _check_repeated_schedule(self, schedule: RepeatedSchedule) -> bool:
@@ -131,7 +140,11 @@ def _check_repeated_schedule(self, schedule: RepeatedSchedule) -> bool:
                      f"cron_day_month={schedule.cron_day_month},"
                      f"cron_month={schedule.cron_month},"
                      f"cron_day_week={schedule.cron_day_week}")
-        # TODO: Add any checking
+
+        # No negative duration
+        if schedule.cron_duration[0] == "-":
+            raise DispatcherException("Negative durations are not supported")
+
         return True
 
     def _convert_duration_to_end_time(self, duration: str) -> Union[str, datetime]:

inbm/dispatcher-agent/dispatcher/schedule/schedules.py

@@ -6,7 +6,7 @@
 """
 from datetime import datetime
 from dataclasses import dataclass, field
-from typing import Optional, List, Tuple
+from typing import Optional, List
 
 
 @dataclass

inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/schedule_manifest_schema.xsd

@@ -44,7 +44,7 @@
         </xs:documentation>
     </xs:annotation>
     <xs:restriction base="xs:string">
-      <xs:pattern value="[A-Za-z]{4,6}-[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"/>
+      <xs:pattern value="[A-Za-z]{4,9}-[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}"/>
     </xs:restriction>
   </xs:simpleType>
 

inbm/dispatcher-agent/tests/unit/schedule/test_apscheduler.py

@@ -1,8 +1,9 @@
 from unittest.mock import Mock, patch
 from unittest import TestCase
 
-from dispatcher.schedule.schedules import SingleSchedule
+from dispatcher.schedule.schedules import SingleSchedule, RepeatedSchedule
 from dispatcher.schedule.apscheduler import APScheduler
+from dispatcher.dispatcher_exception import DispatcherException
 from datetime import datetime, timedelta
 
 
@@ -18,7 +19,7 @@ def test_return_true_schedulable_single_schedule(self):
                              end_time=datetime.now() + timedelta(minutes=5),
                              manifests=["MANIFEST1", "MANIFEST2"])
         self.assertTrue(self.scheduler.is_schedulable(ss1))
-
+    
     def test_return_false_too_late_to_schedule(self):
         ss1 = SingleSchedule(request_id="REQ123",
                              start_time=datetime.now() - timedelta(hours=2),
@@ -44,14 +45,32 @@ def test_is_schedulable_with_other_object(self):
         ss1 = "Neither a SingleSchedule nor a RepeatedSchedule object"
         self.assertFalse(self.scheduler.is_schedulable(schedule=ss1))
 
-    def test_add_single_schedule_job(self):
+    def test_successfully_add_single_schedule_job(self):
         ss1 = SingleSchedule(request_id="REQ123",
                              start_time=datetime.now(),
                              end_time=datetime.now() + timedelta(minutes=5),
                              manifests=["MANIFEST1", "MANIFEST2", "MANIFEST3"])
         self.scheduler.add_single_schedule_job(callback=Mock(), single_schedule=ss1)
         self.assertEqual(len(self.scheduler._scheduler.get_jobs()), 3)
 
+    def test_successfully_add_repeated_schedule_job(self):
+        rs1 = RepeatedSchedule(request_id="REQ123",
+                                cron_duration="P2Y", cron_minutes="*", 
+                                cron_hours="*", cron_day_month="*",
+                                cron_month="*", cron_day_week="1",
+                                manifests=["MANIFEST1", "MANIFEST2"])
+        self.scheduler.add_repeated_schedule_job(callback=Mock(), repeated_schedule=rs1)
+        self.assertEqual(len(self.scheduler._scheduler.get_jobs()), 2)
+
+    def test_raise_negative_duration_repeated_schedule_job(self):
+        rs1 = RepeatedSchedule(request_id="REQ123",
+                                cron_duration="-P2Y", cron_minutes="*", 
+                                cron_hours="*", cron_day_month="*",
+                                cron_month="*", cron_day_week="1",
+                                manifests=["MANIFEST1", "MANIFEST2"])
+        with self.assertRaisesRegex(DispatcherException, 'Negative durations are not supported'):
+            self.scheduler.add_repeated_schedule_job(callback=Mock(), repeated_schedule=rs1)
+
     def test_convert_duration_to_end_time_return_default(self):
         self.assertEqual(self.scheduler._convert_duration_to_end_time(duration="*"), "*")
 

inbm/dockerfiles/Dockerfile-check.m4

@@ -30,7 +30,6 @@ RUN source /venv-py3/bin/activate && \
     	pytest-cov==4.1.0 \
         pytest-mock==3.12.0 \
         pytest-xdist==3.3.1 \
-	pytest-mock==3.12.0 \
         -U
 COPY inbm-lib /src/inbm-lib
 ENV PYTHONPATH=/src/inbm-lib
@@ -149,7 +148,7 @@ RUN source /venv-py3/bin/activate && \
     mkdir -p /output/coverage && \
     set -o pipefail && \
     export PYTHONPATH=$PYTHONPATH:$(pwd) && \
-    pytest -n 3 --cov=dispatcher --cov-report=term-missing --cov-fail-under=79.6 tests/unit 2>&1 | tee /output/coverage/dispatcher-coverage.txt
+    pytest -n 3 --cov=dispatcher --cov-report=term-missing --cov-fail-under=81 tests/unit 2>&1 | tee /output/coverage/dispatcher-coverage.txt
 
 # ---cloudadapter agent---
 

retain-3rd-party-notices/gogo-protobuf-LICENSE

@@ -1,16 +1,23 @@
-Copyright (c) 2018 The Go Authors. All rights reserved.
+Copyright (c) 2013, The GoGo Authors. All rights reserved.
+
+Protocol Buffers for Go with Gadgets
+
+Go support for Protocol Buffers - Google's data interchange format
+
+Copyright 2010 The Go Authors.  All rights reserved.
+https://github.com/golang/protobuf
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are
 met:
 
-   * Redistributions of source code must retain the above copyright
+    * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
-   * Redistributions in binary form must reproduce the above
+    * Redistributions in binary form must reproduce the above
 copyright notice, this list of conditions and the following disclaimer
 in the documentation and/or other materials provided with the
 distribution.
-   * Neither the name of Google Inc. nor the names of its
+    * Neither the name of Google Inc. nor the names of its
 contributors may be used to endorse or promote products derived from
 this software without specific prior written permission.
 
@@ -25,3 +32,4 @@ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+