diff --git a/docs/Manifest Parameters.md b/docs/Manifest Parameters.md index 63428f220..266a689f1 100644 --- a/docs/Manifest Parameters.md +++ b/docs/Manifest Parameters.md @@ -14,6 +14,8 @@ The following outlines the manifest parameters used to perform the supported OTA 9. [Configuration LOAD](#Load) 10. [Configuration APPEND](#Append) 11. [Configuration REMOVE](#Remove) +12. [Source Application](#Application) +13. [Source OS](#OS) ## Manifest Rules @@ -41,18 +43,18 @@ The following outlines the manifest parameters used to perform the supported OTA | Tag | Example | Required /Optional | Notes | |:-----------------------------------------|:----------------------------------------------------|:------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `` | `` | R || -| `` | `` | R || +| `` | `` | R | | +| `` | `` | R | | | `` | `ota` | R | Always OTA | -| `` | `` | R || -| `
` | `
` | R || -| `` | `yourID` | O || +| `` | `` | R | | +| `
` | `
` | R | | +| `` | `yourID` | O | | | `` | `YourName` | O | Endpoint Manufacturer Name | -| `` | `YourDescription | O || -| `` | `fota` | R || +| `` | `YourDescription | O | | +| `` | `fota` | R | | | `` | `remote` | O | [local or remote]. If file is already downloaded on the system, then use _**local**_. If it needs to be fetched from remote repository, use **_remote_**. | -| `
` | `
` | R || -| `` | `` | R || +| `
` | `
` | R | | +| `` | `` | R | | | `` | `` | R | Text must be compliant with XML Standards | | `` | `http://yoururl:80/BIOSUPDATE.tar` | R | FOTA path created in repository | | `` | `ABC123` | O | Digital signature of *.tar file. | @@ -65,10 +67,10 @@ The following outlines the manifest parameters used to perform the supported OTA | `` | `7acbd1a5a-33a4-48c3ab11-a4c33b3d0e56` | O | Check for ‘System Firmware Type’ on running cmd:fwupdate -l | | `` | `user` | O | Username used during fetch from remote repository | | `` | `pwd` | O | Password used during fetch from remote repository | -| `` | `` | R || -| `` | `` | R || -| `` | `` | R || -| `` | `` | R || +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | The following table references each XML tag within a manifest that triggers the FOTA update. Using the following XML tags in the order of description will trigger a FOTA update via Manifest. @@ -103,28 +105,28 @@ description will trigger a FOTA update via Manifest. | Tag | Example | Required/Optional | Notes | |:-----------------------------------------|:---------------------------------------------|:-----------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `` | `` | R || -| `` | ` | R || +| `` | `` | R | | +| `` | ` | R | | | `` | `ota` | R | Always OTA | -| `
` | `
` | R || -| `` | `Example` | O || -| `` | `Example` | O || -| `` | `Example` | O || -| `` | `sota` | R || +| `
` | `
` | R | | +| `` | `Example` | O | | +| `` | `Example` | O | | +| `` | `Example` | O | | +| `` | `sota` | R | | | `` | `remote` | R | [local or remote]. If file is already downloaded on the system, then use _**local**_. If it needs to be fetched from remote repository, use **_remote_**. | -| `
` | `
` | R || -| `` | `` | R || -| `` | `` | R || -| `` | `update` | R || -| `` | `full` | O | Valid values: [full, no-download, download-only] +| `
` | `
` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `update` | R | | +| `` | `full` | O | Valid values: [full, no-download, download-only] | | `` | `https://yoururl/file.mender` | O | Used to download mender file from remote repository. (use repo=remote) | | `` | `xx` | O | Username for remote repository | | | `` | `xxx` | O | Password for remote repository | | | `` | `2020-01-01` | O | The release date provided should be in ‘YYYY-MM-DD’ format. | -| `` | `` | R || -| `` | `` | R || -| `` | `` | R || -| `` | ` | R || +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | ` | R | | ### Sample SOTA Manifest - Ubuntu: ```xml @@ -204,7 +206,7 @@ The POTA manifest is used to perform both a FOTA and SOTA update at the same tim | `` | `` | R | | | `` | `` | R | | | `` | `update` | R | -| `` | `full` | O | Valid values: [full, no-download, download-only] | +| `` | `full` | O | Valid values: [full, no-download, download-only] | | `` | `https://yoururl/file.mender` | O | Used to download mender file from remote repository. (use repo=remote) | | `` | `/var/cache/file.mender` | O | Used to update using a local mender file . (use repo=local) | | `` | `xx` | O | Username for remote repository | | @@ -298,7 +300,7 @@ The POTA manifest is used to perform both a FOTA and SOTA update at the same tim update application - yoururl/package.deb/fetch> + yoururl/package.deb yes @@ -905,3 +907,284 @@ The query command can be used to gather information about the system and the Vis ``` + +## Application + +#### Source Application Add Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:-----------------------------------------------------------------------------------------------|:-----------------:|:----------------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `https://dl-ssl.google.com/linux/linux_signing_key.pub` | R | | +| `` | `google-chrome.gpg` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main` | R | | +| `` | `` | R | | +| `` | `google-chrome.list` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source application add Manifest Example +```xml + + + source + + + + https://dl-ssl.google.com/linux/linux_signing_key.pub + google-chrome.gpg + + + + deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main + deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable second + + google-chrome.list + + + + +``` + +#### Source Application Update Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:-----------------------------------------------------------------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main` | R | | +| `` | `` | R | | +| `` | `google-chrome.list` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source Application Update Manifest Example +```xml + + + source + + + + + deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main + + google-chrome.list + + + + +``` + +#### Source Application Remove Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:-----------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `google-chrome.gpg` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `google-chrom.list` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source Application Remove Manifest Example +```xml + + + source + + + + google-chrome.gpg + + + google-chrome.list + + + + +``` + +#### Source Application List Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:-----------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source application list Manifest Example +```xml + + + source + + + + +``` + +## OS + +#### Source OS Add Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:--------------------------------------------------------------------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted ` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source OS Add Manifest Example +```xml + + + source + + + + deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted + deb http://archive.ubuntu.com/ubuntu/ jammy-security universe + + + + +``` + +#### Source os Update Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:--------------------------------------------------------------------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted ` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source OS Update Manifest Example +```xml + + + source + + + + deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted + deb http://archive.ubuntu.com/ubuntu/ jammy-security universe + + + + +``` + +#### Source OS Remove Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:------------------------------------------------------------------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + +#### Source OS Remove Manifest Example +```xml + + + source + + + + deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted + deb http://archive.ubuntu.com/ubuntu/ jammy-security universe + + + + +``` + +#### Source os List Manifest Parameters +| Tag | Example | Required/Optional | Notes | +|:-----------------------------------------|:-----------------------------------------|:-----------------:|:------| +| `` | `` | R | | +| `` | `` | R | | +| `source` | `source` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | +| `` | `` | R | | + + + + +#### Source OS List Manifest Example +```xml + + + source + + + + +``` \ No newline at end of file diff --git a/autopep8.sh b/format-python.sh similarity index 82% rename from autopep8.sh rename to format-python.sh index a2ee3fff9..e9ef58327 100755 --- a/autopep8.sh +++ b/format-python.sh @@ -4,6 +4,6 @@ set -euxo pipefail for dir in inbc-program inbm inbm-lib ; do ( cd "$dir" - ./autopep8.sh + ./format-python.sh ) done diff --git a/inbc-program/.gitignore b/inbc-program/.gitignore index f98570b2c..8cc02e24f 100644 --- a/inbc-program/.gitignore +++ b/inbc-program/.gitignore @@ -4,3 +4,5 @@ coverage.xml *.cover **/cover/ +inbm_lib +inbm_common_lib diff --git a/inbc-program/README.md b/inbc-program/README.md index 6da6deff0..4737645c6 100644 --- a/inbc-program/README.md +++ b/inbc-program/README.md @@ -6,7 +6,7 @@ 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Notes](#-notes) -4. [MQTT Communication](#mqtt-communication) +4. [MQTT Communication](#mqtt-communication-) 1. [Publish Channels](#publish-channels) 2. [Subscribe Channels](#subscribe-channels) 5. [Commands](#commands) @@ -19,6 +19,14 @@ 7. [Configuration Set](#set) 8. [Restart](#restart) 9. [Query](#query) + 10. [Source Application Add](#source-application-add) + 11. [Source Application Remove](#source-application-remove) + 12. [Source Application Update](#source-application-update) + 13. [Source Application List](#source-application-list) + 14. [Source OS Add](#source-os-add) + 15. [Source OS Remove](#source-os-remove) + 16. [Source OS Update](#source-os-update) + 15. [Source OS List](#source-os-list) 6. [Status Codes](#status-codes) 7. [Return and Exit Codes](#return-and-exit-codes) @@ -352,7 +360,7 @@ Remove is only applicable to two config tags, which are trustedRepositories and ### Usage ``` -inbc remove +inbc remove {--path, -p KEY_PATH;...} ``` @@ -404,6 +412,152 @@ inbc query --option hw inbc query --option sw ``` +## SOURCE APPLICATION ADD +### Description +Downloads and encrypts GPG key and stores it on the system under /usr/share/keyrings. Creates a file under /etc/apt/sources.list.d to store the update source information. +This list file is used during 'sudo apt update' to update the application + +### Usage +``` +inbc source application add + {--gpgKeyUri, -gku=GPG_KEY_URI} + {--gpgKeyName, -gkn=GPG_KEY_NAME} + {--sources, -s=SOURCES} + {--filename, -f=FILENAME} +``` + +### Example +#### Add an application source +``` +inbc source application add + --gpgKeyUri https://dl-ssl.google.com/linux/linux_signing_key.pub + --gpgKeyName google-chrome.gpg + --sources "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" + --filename google-chrome.list +``` + +## SOURCE APPLICATION REMOVE +### Description +Removes the GPG key file from under /usr/share/keyrings. Removes the source file from under /etc/apt/sources.list.d/. + +### Usage +``` +inbc source application remove + {--gpgKeyName, -gkn=GPG_KEY_NAME} + {--filename, -f=FILE_NAME} +``` + +### Example +#### Remove an application source +```commandline +inbc source application remove + --gpgKeyName google-chrome.gpg + --filename google-chrome.list +``` + +## SOURCE APPLICATION UPDATE +### Description +Updates Application sources that are used to update the system +NOTE: Currently this only works on Ubuntu + +### Usage +``` +inbc source application update + {--filename, -f=FILEPATH} + {--sources, -s=SOURCES} +``` + +### Examples +#### Update an application source file +``` +inbc source application update + --filename google-chrome.list + --sources "deb [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable test" "debsrc [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable test2" +``` + +## SOURCE APPLICATION LIST +### Description +Lists Application sources +NOTE: Currently this only works on Ubuntu + +### Usage +``` +inbc source application list +``` + +### Examples +#### Lists all application source files +``` +inbc source application list +``` + +## SOURCE OS ADD +### Description +Appends new source(s) to the /etc/apt/sources.list file + +### Usage +``` +inbc source os add + {--sources, -s=SOURCES} +``` + +### Example +#### Adds two sources +``` +inbc source os add + --sources="deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted" "deb http://archive.ubuntu.com/ubuntu/ jammy-security universe" +``` + +## SOURCE OS REMOVE +### Description +Removes the provided source(s) from the /etc/apt/sources.list file, if they are present. + +### Usage +``` +inbc source os remove + {--sources, -s=SOURCES} +``` + +### Example +#### Removes the two provided source(s) from the /etc/apt/sources.list file +``` +inbc source os remove + --sources="deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted" "deb http://archive.ubuntu.com/ubuntu/ jammy-security universe" +``` + +## SOURCE OS UPDATE +### Description +Creates a new /etc/apt/sources.list file with only the sources provided + +### Usage +``` +inbc source os update + {--sources, -s=SOURCES} +``` + +### Example +#### Creates a new /etc/apt/sources.list file with only the two provided sources +``` +inbc source os update + --sources="deb http://archive.ubuntu.com/ubuntu/ jammy-security main restricted" "deb http://archive.ubuntu.com/ubuntu/ jammy-security universe" +``` + +## SOURCE OS LIST +### Description +Lists OS sources +NOTE: Currently this only works on Ubuntu + +### Usage +```commandline +inbc source os list +``` + +### Examples +#### Lists all OS source files +``` +inbc source os list +``` + # Status Codes | Message | Description | Result | diff --git a/inbc-program/autopep8.sh b/inbc-program/format-python.sh similarity index 100% rename from inbc-program/autopep8.sh rename to inbc-program/format-python.sh diff --git a/inbc-program/inbc/broker.py b/inbc-program/inbc/broker.py index 5bdb85ed1..0bb4dfc68 100644 --- a/inbc-program/inbc/broker.py +++ b/inbc-program/inbc/broker.py @@ -43,15 +43,20 @@ def __init__(self, cmd_type: str, parsed_args: Any, tls: bool = True) -> None: except ValueError: mqtt_port = DEFAULT_MQTT_PORT - self.mqttc = MQTT(PROG, - MQTT_HOST, - mqtt_port, - MQTT_KEEPALIVE_INTERVAL, - ca_certs=CA_CERTS, - env_config=True, - tls=tls, - client_certs=CLIENT_CERTS, - client_keys=CLIENT_KEYS) + try: + self.mqttc = MQTT(PROG, + MQTT_HOST, + mqtt_port, + MQTT_KEEPALIVE_INTERVAL, + ca_certs=CA_CERTS, + env_config=True, + tls=tls, + client_certs=CLIENT_CERTS, + client_keys=CLIENT_KEYS) + except AttributeError as e: + logger.exception("MQTT error: Check that MQTT Service is running") + raise + self.mqttc.start() self._subscribe() self._command = create_command_factory(cmd_type, self) diff --git a/inbc-program/inbc/command/command.py b/inbc-program/inbc/command/command.py index 6ad76b43f..b03cbcd30 100644 --- a/inbc-program/inbc/command/command.py +++ b/inbc-program/inbc/command/command.py @@ -6,7 +6,6 @@ """ import logging -import time from typing import Any, Optional from abc import ABC, abstractmethod from inbc import shared @@ -48,7 +47,7 @@ def stop_timer(self) -> None: def invoke_update(self, args: Any) -> None: """Trigger the command-line utility tool to invoke update. - Sub-classes will override this method to provide specific implementations. + Subclasses will override this method to provide specific implementations. @param args: arguments passed to command-line tool. """ @@ -58,7 +57,7 @@ def _send_manifest(self, args: Any, topic: str) -> None: """Send a manifest. This is a default concrete implementation that takes in a topic. It is intended - to be called from a sub-class. + to be called from a subclass. @param args: arguments passed to command-line tool. @param topic: topic on which to send the manifest @@ -74,7 +73,7 @@ def search_response(self, payload: Any) -> None: @param payload: payload received in which to search """ if self._cmd_type != "query": - if search_keyword(payload, ["SUCCESSFUL"]): + if search_keyword(payload, ["SUCCESSFUL", '"status": 200']): self.terminate_operation(COMMAND_SUCCESS, InbcCode.SUCCESS.value) else: self.terminate_operation(COMMAND_FAIL, InbcCode.FAIL.value) diff --git a/inbc-program/inbc/command/source_command.py b/inbc-program/inbc/command/source_command.py index 0dd8d9364..8ab3777ca 100644 --- a/inbc-program/inbc/command/source_command.py +++ b/inbc-program/inbc/command/source_command.py @@ -1,7 +1,7 @@ """ Source Command classes to represent command entered by user. - # Copyright (C) 2020-2023 Intel Corporation + # Copyright (C) 2020-2024 Intel Corporation # SPDX-License-Identifier: Apache-2.0 """ from typing import Any @@ -33,7 +33,6 @@ def search_response(self, payload: Any) -> None: @param payload: payload received in which to search """ - # TODO: Add responses to wait for super().search_response(payload) def search_event(self, payload: Any, topic: str) -> None: diff --git a/inbc-program/inbc/parser/parser.py b/inbc-program/inbc/parser/parser.py index 599c949d1..97201444b 100644 --- a/inbc-program/inbc/parser/parser.py +++ b/inbc-program/inbc/parser/parser.py @@ -65,19 +65,19 @@ def parse_source_args(self) -> None: # Application Add Command app_add_parser = app_subparsers.add_parser('add') - app_add_parser.add_argument('-gpgKeyPath', '--gkp', required=True, + app_add_parser.add_argument('--gpgKeyUri', '-gku', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 1500), - help='Path to GPG key') - app_add_parser.add_argument('-gpgKeyName', '--gkn', required=True, + help='Uri from which to download GPG key') + app_add_parser.add_argument('--gpgKeyName', '-gkn', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 200), help='Name to store the GPG key information') - app_add_parser.add_argument('-source', '--s', required=True, + app_add_parser.add_argument('--sources', '-s', required=True, nargs="*", default=[], type=lambda x: validate_string_less_than_n_characters( - x, 'str', 1000), - help='Source information to store in the file') - app_add_parser.add_argument('-fileName', '--f', required=True, + x, 'List[str]', 3500), + help='List of source information to store in the file') + app_add_parser.add_argument('--filename', '-f', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 200), help='file name to use when storing the source information') @@ -85,11 +85,11 @@ def parse_source_args(self) -> None: # Application Remove Command app_remove_parser = app_subparsers.add_parser('remove') - app_remove_parser.add_argument('-gpgKeyId', '--gki', required=True, + app_remove_parser.add_argument('--gpgKeyName', '-gkn', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 50), - help='GPG Key ID of the source to remove.') - app_remove_parser.add_argument('-fileName', '--f', required=True, + help='GPG key name of the source to remove.') + app_remove_parser.add_argument('--filename', '-f', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 200), help='file name to use when storing the source information') @@ -97,14 +97,14 @@ def parse_source_args(self) -> None: # Application Update Command app_update_parser = app_subparsers.add_parser('update') - app_update_parser.add_argument('-fileName', '--f', required=True, + app_update_parser.add_argument('--filename', '-f', required=True, type=lambda x: validate_string_less_than_n_characters( x, 'str', 200), help='file name to use when storing the source information') - app_update_parser.add_argument('-source', '--s', required=True, + app_update_parser.add_argument('--sources', '-s', required=True, nargs="*", default=[], type=lambda x: validate_string_less_than_n_characters( - x, 'str', 1000), - help='Source information to store in the file') + x, 'List[str]', 3500), + help='List of source information to store in the file') app_update_parser.set_defaults(func=application_update) # Application List Command @@ -119,7 +119,7 @@ def parse_source_args(self) -> None: # OS Add Command os_add_parser = os_subparsers.add_parser('add') - os_add_parser.add_argument('-sources', '--s', required=True, nargs="*", default=[], + os_add_parser.add_argument('--sources', '-s', required=True, nargs="*", default=[], type=lambda x: validate_string_less_than_n_characters( x, 'List[str]', 3500), help='List of source information to store in the file') @@ -127,7 +127,7 @@ def parse_source_args(self) -> None: # OS Remove Command os_remove_parser = os_subparsers.add_parser('remove') - os_remove_parser.add_argument('-sources', '--s', required=True, nargs="*", default=[], + os_remove_parser.add_argument('--sources', '-s', required=True, nargs="*", default=[], type=lambda x: validate_string_less_than_n_characters( x, 'List[str]', 3500), help='Source information to remove from the file') @@ -135,7 +135,7 @@ def parse_source_args(self) -> None: # OS Update Command os_update_parser = os_subparsers.add_parser('update') - os_update_parser.add_argument('-sources', '--s', required=True, nargs="*", default=[], + os_update_parser.add_argument('--sources', '-s', required=True, nargs="*", default=[], type=lambda x: validate_string_less_than_n_characters( x, 'List[str]', 3500), help='Source information to replace in the file') diff --git a/inbc-program/inbc/parser/source_app_parser.py b/inbc-program/inbc/parser/source_app_parser.py index 1d4c58ed6..680dd4615 100644 --- a/inbc-program/inbc/parser/source_app_parser.py +++ b/inbc-program/inbc/parser/source_app_parser.py @@ -10,10 +10,10 @@ def application_add(args: argparse.Namespace) -> str: arguments = { - 'path': args.gkp, - 'keyname': args.gkn, - 'source': args.s, - 'filename': args.f, + 'uri': args.gpgKeyUri, + 'keyname': args.gpgKeyName, + 'sources': args.sources, + 'filename': args.filename, } manifest = ('' + @@ -23,14 +23,16 @@ def application_add(args: argparse.Namespace) -> str: '' '{0}' + '{1}' - '' + - '{2}' - '{3}' - '' + - '').format(create_xml_tag(arguments, "path"), - create_xml_tag(arguments, "keyname"), - create_xml_tag(arguments, "source"), - create_xml_tag(arguments, "filename")) + '').format(create_xml_tag(arguments, "uri"), + create_xml_tag(arguments, "keyname")) + + for source in args.sources: + manifest += '' + source.strip() + '' + + manifest += ('' + f'{create_xml_tag(arguments, "filename")}' + '' + + '') print("manifest {0}".format(manifest)) return manifest @@ -38,41 +40,43 @@ def application_add(args: argparse.Namespace) -> str: def application_remove(args: argparse.Namespace) -> str: arguments = { - 'keyid': args.gki, - 'filename': args.f + 'keyname': args.gpgKeyName, + 'filename': args.filename } manifest = ('' + 'source' + '' + '' - '{0}' + + f'{create_xml_tag(arguments, "keyname")}' + '' + - '{1}' + f'{create_xml_tag(arguments, "filename")}' '' '' + - '').format(create_xml_tag(arguments, "keyid"), - create_xml_tag(arguments, "filename")) + '') - print("manifest {0}".format(manifest)) + print(f"manifest {manifest}") return manifest def application_update(args: argparse.Namespace) -> str: arguments = { - 'source_pkg': args.s, - 'filename': args.f + 'sources': args.sources, + 'filename': args.filename } manifest = ('' + 'source' + '' + - '' - '{0}{1}' + - '' + - '').format(create_xml_tag(arguments, "source_pkg"), - create_xml_tag(arguments, "filename")) + '') - print("manifest {0}".format(manifest)) + for source in args.sources: + manifest += '' + source.strip() + '' + + manifest += (f'{create_xml_tag(arguments, "filename")}' + + '' + + '') + + print(f"manifest {manifest}") return manifest diff --git a/inbc-program/inbc/parser/source_os_parser.py b/inbc-program/inbc/parser/source_os_parser.py index e7178d95e..0570d0fdc 100644 --- a/inbc-program/inbc/parser/source_os_parser.py +++ b/inbc-program/inbc/parser/source_os_parser.py @@ -9,7 +9,7 @@ def os_add(args: argparse.Namespace) -> str: manifest = 'source' \ '' - for source in args.s: + for source in args.sources: manifest += '' + source.strip() + '' manifest += '' @@ -20,7 +20,7 @@ def os_add(args: argparse.Namespace) -> str: def os_remove(args: argparse.Namespace) -> str: manifest = 'source' \ '' - for source in args.s: + for source in args.sources: manifest += '' + source.strip() + '' manifest += '' @@ -31,7 +31,7 @@ def os_remove(args: argparse.Namespace) -> str: def os_update(args: argparse.Namespace) -> str: manifest = 'source' \ '' - for source in args.s: + for source in args.sources: manifest += '' + source.strip() + '' manifest += '' diff --git a/inbc-program/tests/unit/test_ota_parser.py b/inbc-program/tests/unit/test_ota_parser.py index 06d69f7f6..7d9f1ab25 100644 --- a/inbc-program/tests/unit/test_ota_parser.py +++ b/inbc-program/tests/unit/test_ota_parser.py @@ -1,3 +1,4 @@ +import pytest from datetime import datetime from unittest import TestCase from inbc.parser.parser import ArgsParser @@ -138,22 +139,21 @@ def test_aota_docker_compose_down_manifest_pass(self) -> None: @patch('inbc.command.ota_command.FotaCommand.invoke_update') @patch('sys.stderr', new_callable=StringIO) def test_raise_invalid_fota_date_format(self, mock_stderr, mock_trigger) -> None: - with self.assertRaises(SystemExit): + with pytest.raises(SystemExit): self.arg_parser.parse_args( ['fota', '-u', 'https://abc.com/test.tar', '-r', '12-31-2024', '-m', 'Intel', '--target', '123ABC', '456DEF']) - self.assertRegexpMatches(mock_stderr.getvalue(), r"Not a valid date - format YYYY-MM-DD:") + assert "Not a valid date - format YYYY-MM-DD:" in str(mock_stderr.getvalue()) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.reconnect') @patch('sys.stderr', new_callable=StringIO) def test_raise_too_long_fota_signature(self, mock_stderr, mock_reconnect) -> None: - with self.assertRaises(SystemExit): + with pytest.raises(SystemExit): self.arg_parser.parse_args(['fota', '-u', 'https://abc.com/test.tar', '-r', '2024-12-31', '-m', 'Intel', '-s', OVER_ONE_THOUSAND_CHARACTER_STRING]) - self.assertRegexpMatches(mock_stderr.getvalue( - ), r"Signature is greater than allowed string size") + assert "Signature is greater than allowed string size" in str(mock_stderr.getvalue()) @patch('threading.Thread.start') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.reconnect') @@ -179,18 +179,18 @@ def test_create_fota_manifest_clean_input(self, mock_start, m_broker, m_pass, m_ @patch('sys.stderr', new_callable=StringIO) def test_raise_invalid_pota_sota_release_date_format(self, mock_stderr) -> None: - with self.assertRaises(SystemExit): + with pytest.raises(SystemExit): self.arg_parser.parse_args( ['pota', '-fp', './fip.bin', '-sp', './temp/test.mender', '-r', '2024-12-31', '-sr', '12-31-2024']) - self.assertRegexpMatches(mock_stderr.getvalue(), r"Not a valid date - format YYYY-MM-DD:") + assert "Not a valid date - format YYYY-MM-DD:" in str(mock_stderr.getvalue()) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.reconnect') @patch('sys.stderr', new_callable=StringIO) def test_raise_invalid_pota_fota_release_date_format(self, mock_stderr, mock_reconnect) -> None: - with self.assertRaises(SystemExit): + with pytest.raises(SystemExit): self.arg_parser.parse_args( ['pota', '-fp', './fip.bin', '-sp', './temp/test.mender', '-r', '12-25-2021']) - self.assertRegexpMatches(mock_stderr.getvalue(), r"Not a valid date - format YYYY-MM-DD:") + assert "Not a valid date - format YYYY-MM-DD:" in str(mock_stderr.getvalue()) def test_create_ubuntu_update_manifest(self) -> None: s = self.arg_parser.parse_args(['sota']) @@ -252,10 +252,10 @@ def test_create_expected_manifest_from_fota(self, mock_dmi) -> None: @patch('sys.stderr', new_callable=StringIO) def test_raise_invalid_sota_release_date_format(self, mock_stderr) -> None: - with self.assertRaises(SystemExit): + with pytest.raises(SystemExit) : self.arg_parser.parse_args( ['sota', '-u', 'https://abc.com/test.mender', '-r', '12-31-2024']) - self.assertRegexpMatches(mock_stderr.getvalue(), r"Not a valid date - format YYYY-MM-DD:") + assert "Not a valid date - format YYYY-MM-DD:" in str(mock_stderr.getvalue()) @patch('inbc.parser.ota_parser._gather_system_details', return_value=PlatformInformation(datetime(2011, 10, 13), 'Intel Corporation', 'ADLSFWI1.R00', diff --git a/inbc-program/tests/unit/test_source_app_parser.py b/inbc-program/tests/unit/test_source_app_parser.py index 605172092..43cfbaeb8 100644 --- a/inbc-program/tests/unit/test_source_app_parser.py +++ b/inbc-program/tests/unit/test_source_app_parser.py @@ -13,72 +13,77 @@ def setUp(self): def test_parse_add_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'application', 'add', - '-gpgKeyPath', 'https://repositories.intel.com/gpu/intel-graphics.key', - '-gpgKeyName', 'intel-graphics.gpg', - '-source', 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"', - '-fileName', 'intel-gpu-jammy.list']) - self.assertEqual(f.gkp, 'https://repositories.intel.com/gpu/intel-graphics.key') - self.assertEqual(f.gkn, 'intel-graphics.gpg') - self.assertEqual(f.s, 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"') - self.assertEqual(f.f, 'intel-gpu-jammy.list') + '--gpgKeyUri', 'https://repositories.intel.com/gpu/intel-graphics.key', + '--gpgKeyName', 'intel-graphics.gpg', + '--sources', 'deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main', + '--filename', 'intel-gpu-jammy.list']) + self.assertEqual(f.gpgKeyUri, 'https://repositories.intel.com/gpu/intel-graphics.key') + self.assertEqual(f.gpgKeyName, 'intel-graphics.gpg') + self.assertEqual(f.sources, ['deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main']) + self.assertEqual(f.filename, 'intel-gpu-jammy.list') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_add_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'application', 'add', - '-gpgKeyPath', 'https://repositories.intel.com/gpu/intel-graphics.key', - '-gpgKeyName', 'intel-graphics.gpg', - '-source', 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"', - '-fileName', 'intel-gpu-jammy.list']) + '--gpgKeyUri', 'https://repositories.intel.com/gpu/intel-graphics.key', + '--gpgKeyName', 'intel-graphics.gpg', + '--sources', 'deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main', + '--filename', 'intel-gpu-jammy.list']) Inbc(p, 'source', False) expected = 'source' \ - 'https://repositories.intel.com/gpu/intel-graphics.key' \ - 'intel-graphics.gpg' \ - 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"' \ - 'intel-gpu-jammy.list' + 'https://repositories.intel.com/gpu/intel-graphics.key' \ + 'intel-graphics.gpg' \ + 'deb http://example.com/ focal main restricted universe' \ + 'deb-src http://example.com/ focal-security main' \ + 'intel-gpu-jammy.list' self.assertEqual(p.func(p), expected) def test_parse_remove_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'application', 'remove', - '-gpgKeyId', '46C1680FC119E61A501811823A319F932D945953', - '-fileName', 'intel-gpu-jammy.list']) - self.assertEqual(f.gki, '46C1680FC119E61A501811823A319F932D945953') - self.assertEqual(f.f, 'intel-gpu-jammy.list') + '--gpgKeyName', 'intel-gpu-jammy.gpg', + '--filename', 'intel-gpu-jammy.list']) + self.assertEqual(f.gpgKeyName, 'intel-gpu-jammy.gpg') + self.assertEqual(f.filename, 'intel-gpu-jammy.list') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_remove_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'application', 'remove', - '-gpgKeyId', '46C1680FC119E61A501811823A319F932D945953', - '-fileName', 'intel-gpu-jammy.list']) + '--gpgKeyName', 'intel-gpu-jammy.gpg', + '--filename', 'intel-gpu-jammy.list']) Inbc(p, 'source', False) expected = 'source' \ - '46C1680FC119E61A501811823A319F932D945953' \ + 'intel-gpu-jammy.gpg' \ 'intel-gpu-jammy.list' self.assertEqual(p.func(p), expected) def test_parse_update_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'application', 'update', - '-source', - 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"', - '-fileName', 'intel-gpu-jammy.list']) - self.assertEqual(f.s, - 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"') - self.assertEqual(f.f, 'intel-gpu-jammy.list') + '--sources', 'deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main', + '--filename', 'intel-gpu-jammy.list']) + self.assertEqual(f.sources, ['deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main']) + self.assertEqual(f.filename, 'intel-gpu-jammy.list') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_update_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'application', 'update', - '-source', 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"', - '-fileName', 'intel-gpu-jammy.list']) + '--sources', 'deb http://example.com/ focal main restricted universe', + 'deb-src http://example.com/ focal-security main', + '--filename', 'intel-gpu-jammy.list']) Inbc(p, 'source', False) expected = 'source' \ - '' \ - 'echo "deb https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified"' \ - 'intel-gpu-jammy.list' + 'deb http://example.com/ focal main restricted universe' \ + 'deb-src http://example.com/ focal-security main' \ + 'intel-gpu-jammy.list' self.assertEqual(p.func(p), expected) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') diff --git a/inbc-program/tests/unit/test_source_os_parser.py b/inbc-program/tests/unit/test_source_os_parser.py index 6389cc62e..07f6f0d04 100644 --- a/inbc-program/tests/unit/test_source_os_parser.py +++ b/inbc-program/tests/unit/test_source_os_parser.py @@ -17,16 +17,16 @@ def setUp(self): def test_parse_add_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'os', 'add', - '-sources', 'deb http://example.com/ focal main restricted universe', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) - self.assertEqual(f.s, ['deb http://example.com/ focal main restricted universe', + self.assertEqual(f.sources, ['deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_add_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'os', 'add', - '-sources', 'deb http://example.com/ focal main restricted universe', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) Inbc(p, 'source', False) expected = 'source' \ @@ -39,16 +39,16 @@ def test_create_add_manifest_successfully(self, m_connect): def test_parse_remove_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'os', 'remove', - '-sources', 'deb http://example.com/ focal main restricted universe', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) - self.assertEqual(f.s, ['deb http://example.com/ focal main restricted universe', + self.assertEqual(f.sources, ['deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_remove_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'os', 'remove', - '-sources', 'deb http://example.com/ focal main restricted universe', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) Inbc(p, 'source', False) expected = 'source' \ @@ -61,17 +61,17 @@ def test_create_remove_manifest_successfully(self, m_connect): def test_parse_update_arguments_successfully(self): f = self.arg_parser.parse_args( ['source', 'os', 'update', - '-sources', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) - self.assertEqual(f.s, ['deb http://example.com/ focal main restricted universe', + self.assertEqual(f.sources, ['deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') def test_create_update_manifest_successfully(self, m_connect): p = self.arg_parser.parse_args( ['source', 'os', 'update', - '-sources', 'deb http://example.com/ focal main restricted universe', + '--sources', 'deb http://example.com/ focal main restricted universe', 'deb-src http://example.com/ focal-security main']) Inbc(p, 'source', False) expected = 'source' \ diff --git a/inbm-lib/.flakehell_baseline b/inbm-lib/.flakehell_baseline index 75a5a32a7..32b63ace7 100644 --- a/inbm-lib/.flakehell_baseline +++ b/inbm-lib/.flakehell_baseline @@ -20,4 +20,4 @@ f13954c7604ce46048287bab5ace9189 3b9b31bef09601073051011de3d36394 69d09238e7b96e67f5e4128d10b408f4 4a6598d3ba8e439b240626375d07b6cf -436f2af94a646b36baa3bf9d3a32a1d8 +436f2af94a646b36baa3bf9d3a32a1d8 \ No newline at end of file diff --git a/inbm-lib/autopep8.sh b/inbm-lib/format-python.sh similarity index 100% rename from inbm-lib/autopep8.sh rename to inbm-lib/format-python.sh diff --git a/inbm-lib/inbm_common_lib/utility.py b/inbm-lib/inbm_common_lib/utility.py index 50b7f15af..c2abac78f 100644 --- a/inbm-lib/inbm_common_lib/utility.py +++ b/inbm-lib/inbm_common_lib/utility.py @@ -106,20 +106,36 @@ def _check_paths(src: str, destination: str) -> None: raise IOError("Security error: Destination is a symlink") -def remove_file(path: Union[str, Path]) -> None: +def remove_file(path: Union[str, Path]) -> bool: """ Remove file from the given path @param path: location of file to be removed """ canonical_path = get_canonical_representation_of_path(str(path)) if not os.path.exists(canonical_path): - return + return False if os.path.isfile(canonical_path): logger.debug(f"Removing file at {canonical_path}.") os.remove(canonical_path) + return True else: logger.warning("Failed to remove file. Path is a directory.") + return False + + +def create_file_with_contents(path: Union[str, Path], contents: List[str]) -> None: + """ Create a file and add the contents by line + + @param path: location of file to create + @param contents: each item in the list is a line to add to the file + """ + try: + canonical_path = get_canonical_representation_of_path(str(path)) + with open(canonical_path, 'w') as f: + f.writelines([string + '\n' for string in contents]) + except (PermissionError, IsADirectoryError, OSError) as e: + raise IOError(f"Error while writing file: {str(e)}") def remove_file_list(path: List[str]) -> None: @@ -188,6 +204,7 @@ def is_within_directory(directory: str, target: str) -> bool: return prefix == abs_directory + def safe_extract(tarball: tarfile.TarFile, path: str = ".", members: Optional[Iterable[tarfile.TarInfo]] = None, *, numeric_owner: bool = False) -> None: """Avoid path traversal when extracting tarball diff --git a/inbm-lib/inbm_lib/constants.py b/inbm-lib/inbm_lib/constants.py index bfc9cc9d0..00939634c 100644 --- a/inbm-lib/inbm_lib/constants.py +++ b/inbm-lib/inbm_lib/constants.py @@ -22,7 +22,7 @@ CHROOT_PREFIX = "/usr/sbin/chroot /host " # XML parse time limit -PARSE_TIME_SECS = 5 +PARSE_TIME_SECS = 10 # TRTL install location TRTL_PATH = get_canonical_representation_of_path('/usr/bin/trtl') @@ -82,7 +82,7 @@ # OTA STATUS OTA_SUCCESS = "SUCCESS" -OTA_FAIL = "FAIL" +FAIL = "FAIL" OTA_PENDING = "PENDING" FORMAT_VERSION = "v1" diff --git a/inbm-lib/inbm_lib/trtl.py b/inbm-lib/inbm_lib/trtl.py index ed5b296a2..e0f62abe0 100644 --- a/inbm-lib/inbm_lib/trtl.py +++ b/inbm-lib/inbm_lib/trtl.py @@ -13,7 +13,6 @@ from typing import Optional, Tuple import logging -import pipes from .constants import DOCKER, COMPOSE, TRTL_PATH from subprocess import Popen, PIPE @@ -150,7 +149,7 @@ def execute(self, image: str, version: int, opt: bool = False) -> Tuple[str, Opt ", " + str(version) + ", [" + str(command) + "])") (out, err, code) = self.runner.run(self._boilerplate("exec") + " -in=" + image + " -iv=" + str(version) + - " -ec=" + pipes.quote(command)) + " -ec=" + shlex.quote(command)) if err is None: err = "" logger.debug("(2/2) Stdout: [" + out + "]" + "; stderr: [" + err + diff --git a/inbm-lib/pyproject.toml b/inbm-lib/pyproject.toml index d593e6927..a2aeb4903 100644 --- a/inbm-lib/pyproject.toml +++ b/inbm-lib/pyproject.toml @@ -1,5 +1,4 @@ [tool.flakeheaven] -baseline = ".flakehell_baseline" format = "default" show_source = true statistics = false diff --git a/inbm-lib/tests/unit/inbm_common_lib/test_utility.py b/inbm-lib/tests/unit/inbm_common_lib/test_utility.py index d64d1552d..ec1c8a661 100644 --- a/inbm-lib/tests/unit/inbm_common_lib/test_utility.py +++ b/inbm-lib/tests/unit/inbm_common_lib/test_utility.py @@ -1,10 +1,10 @@ import shutil -from unittest.mock import patch, Mock +from unittest.mock import patch, Mock, mock_open from unittest import TestCase from inbm_common_lib.exceptions import UrlSecurityException from inbm_common_lib.utility import clean_input, get_canonical_representation_of_path, canonicalize_uri, \ - validate_file_type, remove_file, copy_file, move_file + validate_file_type, remove_file, copy_file, move_file, create_file_with_contents class TestUtility(TestCase): @@ -94,3 +94,18 @@ def test_move_file_throw_exception(self, os_path: Mock) -> None: def test_raises_when_move_src_is_symlink(self, mock_is_symlink: Mock) -> None: with self.assertRaises(IOError): move_file('/home/usr', '/etc') + + def test_create_file_with_contents_successfully(self) -> None: + try: + m = mock_open() + lines = ['line1', 'line2'] + with patch('builtins.open', m) as m_open: + create_file_with_contents('/etc/apt/sources.list.d/docker.list', lines) + + m_open.assert_called_once_with('/etc/apt/sources.list.d/docker.list', 'w') + + handle = m() + handle.writelines.assert_called_once_with([line + "\n" for line in lines]) + + except IOError as e: + self.fail(f"Unexpected exception raised during test: {e}") diff --git a/inbm-lib/tests/unit/inbm_lib/manifest_schema.xsd b/inbm-lib/tests/unit/inbm_lib/manifest_schema.xsd deleted file mode 100644 index b30422c24..000000000 --- a/inbm-lib/tests/unit/inbm_lib/manifest_schema.xsd +++ /dev/null @@ -1,467 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/inbm-lib/tests/unit/inbm_lib/test_xmlparser.py b/inbm-lib/tests/unit/inbm_lib/test_xmlparser.py index b783ab35e..c95c95f14 100644 --- a/inbm-lib/tests/unit/inbm_lib/test_xmlparser.py +++ b/inbm-lib/tests/unit/inbm_lib/test_xmlparser.py @@ -4,8 +4,20 @@ from inbm_lib.xmlhandler import XmlHandler, XmlException import os -TEST_SCHEMA_LOCATION = os.path.join(os.path.dirname(__file__), - 'manifest_schema.xsd') +TEST_SCHEMA_LOCATION = os.path.join( + os.path.dirname(__file__), + '..', + '..', + '..', + '..', + 'inbm', + 'dispatcher-agent', + 'fpm-template', + 'usr', + 'share', + 'dispatcher-agent', + 'manifest_schema.xsd', + ) GOOD_XML = '' \ 'ota
sampleIdSample FOTA' \ diff --git a/inbm/Changelog.md b/inbm/Changelog.md index 4a75c6935..8047b01b4 100644 --- a/inbm/Changelog.md +++ b/inbm/Changelog.md @@ -8,8 +8,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). ### Changed - RTC 536078 - Added package list option to inbc, cloud, and internal manifest. This allows SOTA to run an install/upgrade command on a set of individual packages rather than all installed packages. -#Added -- RTC 536603, 536604 -[source] Create os and application based manifest +### Added + - RTC 536601 - Added 'source' command to INBM. This command manages `/etc/apt/sources.list` and `/etc/apt/sources.list.d/*` and associated gpg keys on Ubuntu. ### Fixed - RTC 534426 - Could not write to /var/log/inbm-update-status.log on Yocto due to /var/log being a symlink to /var/volatile/log. diff --git a/inbm/dispatcher-agent/.gitignore b/inbm/dispatcher-agent/.gitignore index ecdaa2e20..516d08c19 100644 --- a/inbm/dispatcher-agent/.gitignore +++ b/inbm/dispatcher-agent/.gitignore @@ -100,3 +100,6 @@ fpm-files/ # test coverage report cover/ + +inbm_lib +inbm_common_lib diff --git a/inbm/dispatcher-agent/dispatcher/dispatcher_class.py b/inbm/dispatcher-agent/dispatcher/dispatcher_class.py index 78ae65b88..a88cc1009 100644 --- a/inbm/dispatcher-agent/dispatcher/dispatcher_class.py +++ b/inbm/dispatcher-agent/dispatcher/dispatcher_class.py @@ -30,12 +30,13 @@ from inbm_common_lib.dmi import is_dmi_path_exists, get_dmi_system_info from inbm_common_lib.device_tree import get_device_tree_system_info from inbm_common_lib.platform_info import PlatformInformation -from inbm_lib.constants import QUERY_CMD_CHANNEL, OTA_SUCCESS, OTA_FAIL +from inbm_lib.constants import QUERY_CMD_CHANNEL, OTA_SUCCESS, FAIL from .aota.aota_error import AotaError +from .source.source_exception import SourceError from .common import dispatcher_state from .common.result_constants import CODE_OK, CODE_BAD_REQUEST, CODE_MULTIPLE, \ - CONFIG_LOAD_FAIL_WRONG_PATH, CODE_FOUND + CODE_FOUND from .config_dbs import ConfigDbs from .constants import * from .device_manager.device_manager import get_device_manager @@ -337,21 +338,26 @@ def do_install(self, xml: str, schema_location: Optional[str] = None) -> Result: except (DispatcherException, UrlSecurityException) as error: logger.error(error) result = Result(CODE_BAD_REQUEST, f'Error during install: {error}') - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL + self._update_logger.error = str(error) + except SourceError as error: + logger.error(error) + result = Result(CODE_BAD_REQUEST, f'Error changing sources files: {error}') + self._update_logger.status = FAIL self._update_logger.error = str(error) except XmlException as error: result = Result(CODE_MULTIPLE, f'Error parsing/validating manifest: {error}') - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL self._update_logger.error = str(error) except (AotaError, FotaError, SotaError) as e: result = Result(CODE_BAD_REQUEST, str(e)) - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL self._update_logger.error = str(e) finally: logger.info('Install result: %s', str(result)) self._send_result(str(result)) if result.status != CODE_OK and parsed_head: - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL self._update_logger.error = str(result) self.invoke_workload_orchestration_check(True, type_of_manifest, parsed_head) @@ -650,8 +656,8 @@ def check_sota_state(self) -> None: self._telemetry( 'On Boot, Diagnostics reports some services not up after previous SOTA') self.invoke_sota(action='diagnostic_system_unhealthy', snapshot=None) - self._update_logger.update_log(OTA_FAIL) - logger.info(OTA_FAIL) + self._update_logger.update_log(FAIL) + logger.info(FAIL) def check_fota_state(self, fota_state: dispatcher_state.DispatcherState) -> None: """This method checks the FOTA info in dispatcher state file and validates the release date @@ -661,7 +667,7 @@ def check_fota_state(self, fota_state: dispatcher_state.DispatcherState) -> None @params fota_state: The consumed information from the dispatcher state file. """ # If all the checks pass, the OTA status changes to SUCCESS at the end. - self._update_logger.update_log(OTA_FAIL) + self._update_logger.update_log(FAIL) os_type = platform.system() platform_info = PlatformInformation() diff --git a/inbm/dispatcher-agent/dispatcher/sota/sota.py b/inbm/dispatcher-agent/dispatcher/sota/sota.py index 1f56ba417..128cd6758 100644 --- a/inbm/dispatcher-agent/dispatcher/sota/sota.py +++ b/inbm/dispatcher-agent/dispatcher/sota/sota.py @@ -7,7 +7,6 @@ import logging import os -import re import time from typing import Any, List, Optional, Union, Mapping @@ -17,7 +16,7 @@ from inbm_common_lib.constants import REMOTE_SOURCE, LOCAL_SOURCE from inbm_lib.validate_package_list import parse_and_validate_package_list from inbm_lib.detect_os import detect_os -from inbm_lib.constants import OTA_PENDING, OTA_FAIL, OTA_SUCCESS +from inbm_lib.constants import OTA_PENDING, FAIL, OTA_SUCCESS from dispatcher.dispatcher_exception import DispatcherException from .command_handler import run_commands, print_execution_summary, get_command_status @@ -244,7 +243,7 @@ def execute(self, proceed_without_rollback: bool, skip_sleeps: bool = False) -> rebooter = self.factory.create_rebooter() if self.sota_state == 'diagnostic_system_unhealthy': - self._update_logger.update_log(OTA_FAIL) + self._update_logger.update_log(FAIL) snapshot.revert(rebooter, time_to_wait_before_reboot) elif self.sota_state == 'diagnostic_system_healthy': try: @@ -257,7 +256,7 @@ def execute(self, proceed_without_rollback: bool, skip_sleeps: bool = False) -> msg = "FAILED INSTALL: System has not been properly updated; reverting." logger.debug(str(e)) self._dispatcher_broker.send_result(msg) - self._update_logger.update_log(OTA_FAIL) + self._update_logger.update_log(FAIL) snapshot.revert(rebooter, time_to_wait_before_reboot) else: self.execute_from_manifest(setup_helper=setup_helper, @@ -340,7 +339,7 @@ def execute_from_manifest(self, '{"status": 400, "message": "SOTA command status: FAILURE"}') if download_success and self.sota_mode != 'download-only': snapshotter.recover(rebooter, time_to_wait_before_reboot) - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL self._update_logger.error = "" self._update_logger.save_log() raise SotaError(str(msg)) @@ -366,7 +365,7 @@ def execute_from_manifest(self, else: # Save the log before reboot - self._update_logger.status = OTA_FAIL + self._update_logger.status = FAIL self._update_logger.error = "" self._update_logger.save_log() self._dispatcher_broker.telemetry(SOTA_FAILURE) diff --git a/inbm/dispatcher-agent/dispatcher/source/constants.py b/inbm/dispatcher-agent/dispatcher/source/constants.py index 5680472fa..3d7d3f94c 100644 --- a/inbm/dispatcher-agent/dispatcher/source/constants.py +++ b/inbm/dispatcher-agent/dispatcher/source/constants.py @@ -1,15 +1,15 @@ """ - Copyright (C) 2023 Intel Corporation + Copyright (C) 2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 """ from enum import Enum, unique from dataclasses import dataclass, field -from typing import List UBUNTU_APT_SOURCES_LIST = "/etc/apt/sources.list" UBUNTU_APT_SOURCES_LIST_D = "/etc/apt/sources.list.d" +LINUX_GPG_KEY_PATH = "/usr/share/keyrings" @dataclass(kw_only=True, frozen=True) @@ -18,27 +18,29 @@ class ApplicationSourceList: sources: list[str] -@dataclass(kw_only=True) +@dataclass(kw_only=True, frozen=True) class SourceParameters: - sources: List[str] = field(default_factory=lambda: []) + sources: list[str] = field(default_factory=lambda: []) -@dataclass(kw_only=True) -class ApplicationAddSourceParameters(SourceParameters): - gpg_key_path: str +@dataclass(kw_only=True, frozen=True) +class ApplicationAddSourceParameters: + gpg_key_uri: str gpg_key_name: str file_name: str + sources: list[str] = field(default_factory=lambda: []) -@dataclass(kw_only=True) +@dataclass(kw_only=True, frozen=True) class ApplicationRemoveSourceParameters: - gpg_key_id: str + gpg_key_name: str file_name: str -@dataclass(kw_only=True) -class ApplicationUpdateSourceParameters(SourceParameters): +@dataclass(kw_only=True, frozen=True) +class ApplicationUpdateSourceParameters: file_name: str + sources: list[str] = field(default_factory=lambda: []) @unique diff --git a/inbm/dispatcher-agent/dispatcher/source/linux_gpg_key.py b/inbm/dispatcher-agent/dispatcher/source/linux_gpg_key.py new file mode 100644 index 000000000..ba1fe4be1 --- /dev/null +++ b/inbm/dispatcher-agent/dispatcher/source/linux_gpg_key.py @@ -0,0 +1,64 @@ +""" + Copyright (C) 2024 Intel Corporation + SPDX-License-Identifier: Apache-2.0 +""" +import logging +import subprocess +import os +import requests + +from .source_exception import SourceError +from .constants import LINUX_GPG_KEY_PATH + +logger = logging.getLogger(__name__) + + +def remove_gpg_key_if_exists(gpg_key_name: str) -> None: + """Linux - Removes a GPG key file if it exists + + @param gpg_key_name: name of GPG key file to remove (file under LINUX_GPG_KEY_PATH) + """ + try: + key_path = os.path.join(LINUX_GPG_KEY_PATH, gpg_key_name) + if os.path.exists(key_path): + os.remove(key_path) + # it's OK if the key is not there + + except OSError as e: + raise SourceError(f"Error checking or deleting GPG key: {gpg_key_name}") from e + + +def add_gpg_key(remote_key_path: str, key_store_name: str) -> None: + """Linux - Adds a GPG key from a remote source + + Raises SourceError if there are any problems. + + @param remote_key_path: Remote location of the GPG key to download + @param key_store_name: Name to use to store the GPG under LINUX_GPG_KEY_PATH + """ + + try: + # Download the GPG key from the remote source + response = requests.get(remote_key_path) + response.raise_for_status() + + decoded_ascii = response.content.decode("utf-8", errors="strict") + + key_path = os.path.join(LINUX_GPG_KEY_PATH, key_store_name) + + # Use gpg to dearmor the key and save it to the key store path + subprocess.run( + ["/usr/bin/gpg", "--dearmor", "--output", key_path], + input=decoded_ascii, + check=True, + text=True, + shell=False, + ) + + logger.info(f"GPG key added to {key_store_name}") + + except requests.exceptions.RequestException as e: + raise SourceError(f"Error getting GPG key from remote source: {e}") + + except subprocess.CalledProcessError as e: + raise SourceError(f"Error running gpg command to dearmor key: {e}") diff --git a/inbm/dispatcher-agent/dispatcher/source/source_command.py b/inbm/dispatcher-agent/dispatcher/source/source_command.py index 51c6b7aa6..2f90e0e7e 100644 --- a/inbm/dispatcher-agent/dispatcher/source/source_command.py +++ b/inbm/dispatcher-agent/dispatcher/source/source_command.py @@ -1,12 +1,20 @@ """ - Copyright (C) 2023 Intel Corporation + Copyright (C) 2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 """ +from dataclasses import asdict import logging import json from dispatcher.common.result_constants import Result -from dispatcher.source.constants import ApplicationRemoveSourceParameters, OsType, SourceParameters +from dispatcher.source.constants import ( + ApplicationAddSourceParameters, + ApplicationRemoveSourceParameters, + ApplicationUpdateSourceParameters, + OsType, + SourceParameters, +) + from dispatcher.source.source_manager_factory import create_os_source_manager from dispatcher.source.source_manager_factory import create_application_source_manager from inbm_lib.xmlhandler import XmlException, XmlHandler @@ -25,14 +33,14 @@ def do_source_command(parsed_head: XmlHandler, os_type: OsType) -> Result: logger.debug(f"do_source_command: {parsed_head}") try: - os_action = parsed_head.get_children('osSource') + os_action = parsed_head.get_children("osSource") if os_action: return _handle_os_source_command(parsed_head, os_type, os_action) except XmlException: pass # If we get an XmlException here we still want to try applicationSource try: - app_action = parsed_head.get_children('applicationSource') + app_action = parsed_head.get_children("applicationSource") if app_action: return _handle_app_source_command(parsed_head, os_type, app_action) except XmlException as e: @@ -52,19 +60,42 @@ def _handle_os_source_command(parsed_head: XmlHandler, os_type: OsType, os_actio """ os_source_manager = create_os_source_manager(os_type) - if os_action == {'list': ''}: + if "list" in os_action: return Result(status=200, message=json.dumps(os_source_manager.list())) - if os_action == {'remove': 'remove'}: - source_pkgs = _get_source_packages(parsed_head) - remove_parameters = SourceParameters(sources=source_pkgs) + if "remove" in os_action: + remove_source_pkgs: list[str] = [] + for key, value in parsed_head.get_children_tuples("osSource/remove/repos"): + if key == "source_pkg": + remove_source_pkgs.append(value) + remove_parameters = SourceParameters(sources=remove_source_pkgs) os_source_manager.remove(remove_parameters) return Result(status=200, message="SUCCESS") + if "add" in os_action: + add_source_pkgs: list[str] = [] + for key, value in parsed_head.get_children_tuples("osSource/add/repos"): + if key == "source_pkg": + add_source_pkgs.append(value) + add_parameters = SourceParameters(sources=add_source_pkgs) + os_source_manager.add(add_parameters) + return Result(status=200, message="SUCCESS") + + if "update" in os_action: + update_source_pkgs: list[str] = [] + for key, value in parsed_head.get_children_tuples("osSource/update/repos"): + if key == "source_pkg": + update_source_pkgs.append(value) + update_parameters = SourceParameters(sources=update_source_pkgs) + os_source_manager.update(update_parameters) + return Result(status=200, message="SUCCESS") + return Result(status=400, message="unknown os source command") -def _handle_app_source_command(parsed_head: XmlHandler, os_type: OsType, app_action: dict) -> Result: +def _handle_app_source_command( + parsed_head: XmlHandler, os_type: OsType, app_action: dict +) -> Result: """ Handle the application source commands. @@ -75,28 +106,53 @@ def _handle_app_source_command(parsed_head: XmlHandler, os_type: OsType, app_act """ application_source_manager = create_application_source_manager(os_type) - if app_action == {'list': ''}: - return Result(status=200, message=json.dumps(application_source_manager.list())) + if "list" in app_action: + serialized_list = json.dumps( + [asdict(app_source) for app_source in application_source_manager.list()] + ) + return Result(status=200, message=serialized_list) - if app_action == {'remove': 'remove'}: - keyid = parsed_head.get_children('applicationSource/remove/gpg')['keyid'] - filename = parsed_head.get_children('applicationSource/remove/repo')['filename'] + if "remove" in app_action: + keyname = parsed_head.get_children("applicationSource/remove/gpg")["keyname"] + filename = parsed_head.get_children("applicationSource/remove/repo")["filename"] application_source_manager.remove( - ApplicationRemoveSourceParameters(file_name=filename, gpg_key_id=keyid)) + ApplicationRemoveSourceParameters(file_name=filename, gpg_key_name=keyname) + ) return Result(status=200, message="SUCCESS") - return Result(status=400, message="unknown application source command") - + if "add" in app_action: + gpg_key_uri = parsed_head.get_children("applicationSource/add/gpg")["uri"] + gpg_key_name = parsed_head.get_children("applicationSource/add/gpg")["keyname"] + repo_filename = parsed_head.get_children("applicationSource/add/repo")["filename"] + + add_source_pkgs: list[str] = [] + for key, value in parsed_head.get_children_tuples("applicationSource/add/repo/repos"): + if key == "source_pkg": + add_source_pkgs.append(value) + + application_source_manager.add( + ApplicationAddSourceParameters( + file_name=repo_filename, + gpg_key_name=gpg_key_name, + gpg_key_uri=gpg_key_uri, + sources=add_source_pkgs, + ) + ) + return Result(status=200, message="SUCCESS") -def _get_source_packages(parsed_head: XmlHandler) -> list[str]: - """ - Extract source packages from XML command. + if "update" in app_action: + repo_filename = parsed_head.get_children("applicationSource/update/repo")["filename"] + update_source_pkgs: list[str] = [] + for key, value in parsed_head.get_children_tuples("applicationSource/update/repo/repos"): + if key == "source_pkg": + update_source_pkgs.append(value) + + application_source_manager.update( + ApplicationUpdateSourceParameters( + file_name=repo_filename, + sources=update_source_pkgs, + ) + ) + return Result(status=200, message="SUCCESS") - @param parsed_head: XmlHandler containing the remove source commands - @return List of packages to remove - """ - source_pkgs: list[str] = [] - for key, value in parsed_head.get_children_tuples('osSource/remove/repos'): - if key == 'source_pkg': - source_pkgs.append(value) - return source_pkgs + return Result(status=400, message="unknown application source command") diff --git a/inbm/dispatcher-agent/dispatcher/source/source_exception.py b/inbm/dispatcher-agent/dispatcher/source/source_exception.py new file mode 100644 index 000000000..66279e786 --- /dev/null +++ b/inbm/dispatcher-agent/dispatcher/source/source_exception.py @@ -0,0 +1,13 @@ +""" + Central communication agent in the manageability framework responsible + for issuing commands and signals to other tools/agents + + Copyright (C) 2024 Intel Corporation + SPDX-License-Identifier: Apache-2.0 +""" + + +class SourceError(Exception): + """Class exception Module""" + + pass diff --git a/inbm/dispatcher-agent/dispatcher/source/source_manager.py b/inbm/dispatcher-agent/dispatcher/source/source_manager.py index 8d825d0d0..58025fda4 100644 --- a/inbm/dispatcher-agent/dispatcher/source/source_manager.py +++ b/inbm/dispatcher-agent/dispatcher/source/source_manager.py @@ -1,5 +1,5 @@ """ - Copyright (C) 2023 Intel Corporation + Copyright (C) 2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 """ diff --git a/inbm/dispatcher-agent/dispatcher/source/source_manager_factory.py b/inbm/dispatcher-agent/dispatcher/source/source_manager_factory.py index 54d1f1db2..0f11c4b55 100644 --- a/inbm/dispatcher-agent/dispatcher/source/source_manager_factory.py +++ b/inbm/dispatcher-agent/dispatcher/source/source_manager_factory.py @@ -1,7 +1,7 @@ """ Creates concrete classes based on OS Type and type of source file being manipulated. - Copyright (C) 2023 Intel Corporation + Copyright (C) 2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 """ import logging diff --git a/inbm/dispatcher-agent/dispatcher/source/ubuntu_source_manager.py b/inbm/dispatcher-agent/dispatcher/source/ubuntu_source_manager.py index fbece955d..220b8be3f 100644 --- a/inbm/dispatcher-agent/dispatcher/source/ubuntu_source_manager.py +++ b/inbm/dispatcher-agent/dispatcher/source/ubuntu_source_manager.py @@ -1,12 +1,13 @@ """ - Copyright (C) 2023 Intel Corporation + Copyright (C) 2024 Intel Corporation SPDX-License-Identifier: Apache-2.0 """ import glob import logging import os -from dispatcher.dispatcher_exception import DispatcherException + +from dispatcher.source.source_exception import SourceError from dispatcher.source.constants import ( UBUNTU_APT_SOURCES_LIST, UBUNTU_APT_SOURCES_LIST_D, @@ -17,7 +18,14 @@ SourceParameters, ) from dispatcher.source.source_manager import ApplicationSourceManager, OsSourceManager -from inbm_common_lib.shell_runner import PseudoShellRunner +from dispatcher.source.linux_gpg_key import remove_gpg_key_if_exists, add_gpg_key + +from inbm_common_lib.utility import ( + get_canonical_representation_of_path, + remove_file, + move_file, + create_file_with_contents, +) logger = logging.getLogger(__name__) @@ -27,9 +35,14 @@ def __init__(self) -> None: pass def add(self, parameters: SourceParameters) -> None: - """Adds a source in the Ubuntu OS source file /etc/apt/sources.list""" - # TODO: Add functionality to add a source file in Ubuntu to /etc/apt/sources.list file - logger.debug(f"sources: {parameters.sources}") + """Adds sources in the Ubuntu OS source file /etc/apt/sources.list""" + + try: + with open(UBUNTU_APT_SOURCES_LIST, "a") as file: + for source in parameters.sources: + file.write(f"{source}\n") + except OSError as e: + raise SourceError(f"Error adding sources: {e}") from e def list(self) -> list[str]: """List deb and deb-src lines in /etc/apt/sources.list""" @@ -44,8 +57,7 @@ def list(self) -> list[str]: line for line in lines if line.startswith("deb ") or line.startswith("deb-src ") ] except OSError as e: - logger.error(f"Error opening source file: {e}") - raise DispatcherException(f"Error opening source file: {e}") from e + raise SourceError(f"Error opening source file: {e}") from e def remove(self, parameters: SourceParameters) -> None: """Removes a source in the Ubuntu OS source file /etc/apt/sources.list""" @@ -66,14 +78,19 @@ def remove(self, parameters: SourceParameters) -> None: logger.debug(f"Removed source: {line}") except OSError as e: - # Wrap any OSError exceptions in a DispatcherException and re-raise. - logger.error(f"Error occurred while trying to remove sources: {e}") - raise DispatcherException(f"Error occurred while trying to remove sources: {e}") from e + raise SourceError(f"Error occurred while trying to remove sources: {e}") from e def update(self, parameters: SourceParameters) -> None: - """Updates a source in the Ubuntu OS source file /etc/apt/sources.list""" - # TODO: Add functionality to update a source in Ubuntu file under /etc/apt/sources.list file - logger.debug(f"sources: {parameters.sources}") + """Updates a source in the Ubuntu OS source file /etc/apt/sources.list + + This will overwrite the file and add the listed sources.""" + + try: + with open(UBUNTU_APT_SOURCES_LIST, "w") as file: + for source in parameters.sources: + file.write(f"{source}\n") + except OSError as e: + raise SourceError(f"Error adding sources: {e}") from e class UbuntuApplicationSourceManager(ApplicationSourceManager): @@ -81,8 +98,16 @@ def __init__(self) -> None: pass def add(self, parameters: ApplicationAddSourceParameters) -> None: - """Adds new application source along with its key""" - pass + # Step 1: Add key + add_gpg_key(parameters.gpg_key_uri, parameters.gpg_key_name) + + # Step 2: Add the source + try: + create_file_with_contents( + os.path.join(UBUNTU_APT_SOURCES_LIST_D, parameters.file_name), parameters.sources + ) + except (IOError, OSError) as e: + raise SourceError(f"Error adding application source list: {e}") def list(self) -> list[ApplicationSourceList]: """List Ubuntu Application source lists under /etc/apt/sources.list.d""" @@ -106,36 +131,38 @@ def list(self) -> list[ApplicationSourceList]: sources.append(new_source) return sources except OSError as e: - logger.error(f"Error listing application sources: {e}") - raise DispatcherException(f"Error listing application sources: {e}") from e + raise SourceError(f"Error listing application sources: {e}") from e def remove(self, parameters: ApplicationRemoveSourceParameters) -> None: - """Removes a source file from the Ubuntu source file list under /etc/apt/sources.list.d""" + """Removes a source file from the Ubuntu source file list under /etc/apt/sources.list.d + @parameters: dataclass parameters for ApplicationRemoveSourceParameters + """ # Remove the GPG key - try: - stdout, stderr, exit_code = PseudoShellRunner().run( - f"gpg --list-keys {parameters.gpg_key_id}" - ) - - # If the key exists, try to remove it - if exit_code == 0: - stdout, stderr, exit_code = PseudoShellRunner().run( - f"gpg --delete-key {parameters.gpg_key_id}" - ) - if exit_code != 0: - raise DispatcherException("Error deleting GPG key: " + (stderr or stdout)) - - except OSError as e: - logger.error(f"Error checking or deleting GPG key: {e}") - raise DispatcherException(f"Error checking or deleting GPG key: {e}") from e + remove_gpg_key_if_exists(parameters.gpg_key_name) # Remove the file under /etc/apt/sources.list.d try: - os.remove(UBUNTU_APT_SOURCES_LIST_D + "/" + parameters.file_name) + if ( + os.path.sep in parameters.file_name + or parameters.file_name == ".." + or parameters.file_name == "." + ): + raise SourceError(f"Invalid file name: {parameters.file_name}") + + if not remove_file( + get_canonical_representation_of_path( + os.path.join(UBUNTU_APT_SOURCES_LIST_D, parameters.file_name) + ) + ): + raise SourceError(f"Error removing file: {parameters.file_name}") except OSError as e: - raise DispatcherException(f"Error removing file: {e}") from e + raise SourceError(f"Error removing file: {e}") from e def update(self, parameters: ApplicationUpdateSourceParameters) -> None: """Updates a source file in Ubuntu OS source file list under /etc/apt/sources.list.d""" - # TODO: Add functionality to update a Ubuntu source file under /etc/apt/sources.list.d - logger.debug(f"file_name: {parameters.file_name}, source: {parameters.sources[0]}") + try: + create_file_with_contents( + os.path.join(UBUNTU_APT_SOURCES_LIST_D, parameters.file_name), parameters.sources + ) + except IOError as e: + raise SourceError(f"Error occurred while trying to update sources: {e}") from e diff --git a/inbm/dispatcher-agent/fpm-template/etc/apparmor.d/usr.bin.inbm-dispatcher b/inbm/dispatcher-agent/fpm-template/etc/apparmor.d/usr.bin.inbm-dispatcher index 7ee31bc6b..6582d7c8e 100644 --- a/inbm/dispatcher-agent/fpm-template/etc/apparmor.d/usr.bin.inbm-dispatcher +++ b/inbm/dispatcher-agent/fpm-template/etc/apparmor.d/usr.bin.inbm-dispatcher @@ -59,7 +59,8 @@ /etc/apt/sources.list.bak rw, /etc/apt/sources.list.d/* rw, /etc/apt/sources.list.d/ rw, - /usr/bin/gpg px, + /usr/bin/gpg rUx, + /bin/gpg rUx, /etc/dispatcher_state rw, /etc/lsb-release r, /etc/magic r, @@ -151,6 +152,7 @@ /usr/share/configuration-agent/iotg_inb_schema.xsd l, /usr/share/configuration-agent/iotg_inb_schema.xsd.dpkg-tmp l, /usr/share/diagnostic-agent/ w, + /usr/share/keyrings/*.gpg w, /usr/share/telemetry-agent/ w, /usr/share/dispatcher-agent/*.json r, /usr/share/dispatcher-agent/manifest_schema.xsd r, @@ -178,6 +180,8 @@ /usr/share/trtl/trtl.xsd r, /usr/share/trtl/ w, /usr/share/trtl/* rw, + /usr/share/keyrings/ rw, + /usr/share/keyrings/* rw, /var/cache/manageability/ rw, /var/cache/manageability/** rw, /etc/intel-manageability/public/configuration-agent/ w, diff --git a/inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd b/inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd index 21c326407..6a8fad59b 100644 --- a/inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd +++ b/inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd @@ -322,16 +322,22 @@ - - + + - - + + + + + + + + @@ -341,10 +347,17 @@ - + - + + + + + + + + @@ -357,14 +370,14 @@ - + - + @@ -375,7 +388,6 @@ - diff --git a/inbm/dispatcher-agent/pyproject.toml b/inbm/dispatcher-agent/pyproject.toml index 705eac0b5..3e82c3d9e 100644 --- a/inbm/dispatcher-agent/pyproject.toml +++ b/inbm/dispatcher-agent/pyproject.toml @@ -40,7 +40,7 @@ max_tuple_unpack_length = 4 [tool.flakeheaven.plugins] "flake8-annotations" = ["-*", "+ANN001", "+ANN2??", "+ANN301"] -"wemake-python-styleguide" = ["-*", "+WPS2??", "-WPS220", "-WPS223", "-WPS222", "-WPS236", "-WPS237", "-WPS238"] # WPS2?? is complexity; 220 is deep nesting; 223 is elifs; 222 is logic; 236 is # of variables unpacked in a tuple; 237 is complex format string; 238 is # of raises in a function +"wemake-python-styleguide" = ["-*", "+WPS2??", "-WPS220", "-WPS223", "-WPS222", "-WPS225", "-WPS236", "-WPS237", "-WPS238"] # WPS2?? is complexity; 220 is deep nesting; 223 is elifs; 222 is logic; 236 is # of variables unpacked in a tuple; 237 is complex format string; 238 is # of raises in a function "flake8-bandit" = ["+*", "-S404", "-S603"] [tool.pytest.ini_options] diff --git a/inbm/dispatcher-agent/tests/unit/sota/test_mender_util.py b/inbm/dispatcher-agent/tests/unit/sota/test_mender_util.py index 1c7dd4e53..c17df2687 100644 --- a/inbm/dispatcher-agent/tests/unit/sota/test_mender_util.py +++ b/inbm/dispatcher-agent/tests/unit/sota/test_mender_util.py @@ -1,8 +1,5 @@ -import os -import tempfile import unittest -import fixtures -from unittest.mock import patch, mock_open +from unittest.mock import patch from dispatcher.sota.mender_util import * diff --git a/inbm/dispatcher-agent/tests/unit/source/test_linux_gpg_key.py b/inbm/dispatcher-agent/tests/unit/source/test_linux_gpg_key.py new file mode 100644 index 000000000..011ae5c58 --- /dev/null +++ b/inbm/dispatcher-agent/tests/unit/source/test_linux_gpg_key.py @@ -0,0 +1,75 @@ +import pytest +from unittest.mock import mock_open, patch + +from requests import RequestException +from dispatcher.dispatcher_exception import DispatcherException +from dispatcher.source.linux_gpg_key import add_gpg_key, remove_gpg_key_if_exists +from dispatcher.source.source_exception import SourceError + + +class TestLinuxGpgKey: + def test_remove_gpg_key_success(self, mocker): + mocker.patch("os.path.join", return_value="dummy/path/to/key") + mocker.patch("os.path.exists", return_value=True) + mock_remove = mocker.patch("os.remove") + + remove_gpg_key_if_exists("mock_key_name") + + mock_remove.assert_called_once_with("dummy/path/to/key") + + def test_remove_gpg_key_os_error(self, mocker): + mocker.patch("os.path.join", return_value="dummy/path/to/key") + mocker.patch("os.path.exists", return_value=True) + + mocker.patch("os.remove", side_effect=OSError("Mock OS Error")) + + with pytest.raises(SourceError) as e: + remove_gpg_key_if_exists("mock_key_name") + + assert "Error checking or deleting GPG key: mock_key_name" in str(e.value) + + def test_add_gpg_key_success(self, mocker): + mock_get = mocker.patch("dispatcher.source.linux_gpg_key.requests.get", autospec=True) + mocker.patch( + "dispatcher.source.linux_gpg_key.requests.Response.raise_for_status", autospec=True + ) + mock_get.return_value.content = b"some-gpg-key-data" + + mock_run = mocker.patch("dispatcher.source.linux_gpg_key.subprocess.run", autospec=True) + + remote_key_path = "https://example.com/key.gpg" + key_store_path = "/etc/apt/trusted.gpg.d/my_key.gpg" + add_gpg_key(remote_key_path, key_store_path) + + mock_get.assert_called_once_with(remote_key_path) + mock_run.assert_called_once_with( + ["/usr/bin/gpg", "--dearmor", "--output", key_store_path], + input="some-gpg-key-data", + check=True, + text=True, + shell=False, + ) + + def test_add_gpg_key_http_error(self, mocker): + # Mock requests.get() to raise an HTTP error + mocker.patch("dispatcher.source.linux_gpg_key.requests.get", side_effect=RequestException) + + # Execute the function and assert that SourceError is raised + with pytest.raises(SourceError): + add_gpg_key("https://example.com/key.gpg", "/etc/apt/trusted.gpg.d/my_key.gpg") + + # @patch("dispatcher.source.linux_gpg_key.PseudoShellRunner.run", + # return_value=[{"", "", 0}, {"", "GPG Delete Error", 1}]) + # def test_raise_when_unable_delete_gpg_key(self, mocker): + # shell_runner_mock = mocker.patch( + # "dispatcher.source.linux_gpg_key.PseudoShellRunner") + # shell_runner_instance = shell_runner_mock.return_value + # if isinstance([{"", "", 0}, {"", "GPG Delete Error", 1}], list): # Simulate sequence of command runs + # shell_runner_instance.run.side_effect = [ + # (stdout, stderr, code) for stdout, stderr, code in [{"", "", 0}, {"", "GPG Delete Error", 1}] + # ] + # with patch('dispatcher.source.linux_gpg_key.PseudoShellRunner.run') as mock_run: + # mock_run.return_value.json.side_effect = [{"", "", 0}, {"", "GPG Delete Error", 1}] + # with pytest.raises(DispatcherException) as ex: + # remove_gpg_key("123456A0") + # assert str(ex.value) == "Error deleting GPG key: GPG Delete Error" diff --git a/inbm/dispatcher-agent/tests/unit/source/test_source_command.py b/inbm/dispatcher-agent/tests/unit/source/test_source_command.py index 28676a20e..a0885ee4f 100644 --- a/inbm/dispatcher-agent/tests/unit/source/test_source_command.py +++ b/inbm/dispatcher-agent/tests/unit/source/test_source_command.py @@ -7,7 +7,14 @@ import pytest from dispatcher.common.result_constants import Result -from dispatcher.source.constants import ApplicationRemoveSourceParameters, OsType, SourceParameters +from dispatcher.source.constants import ( + ApplicationAddSourceParameters, + ApplicationRemoveSourceParameters, + ApplicationSourceList, + ApplicationUpdateSourceParameters, + OsType, + SourceParameters, +) from dispatcher.source.source_command import do_source_command from inbm_lib.xmlhandler import XmlHandler @@ -45,8 +52,8 @@ def _factory(xml): """, "dispatcher.source.source_command.create_application_source_manager", - {"foo.list": ["source1", "source2"]}, - '{"foo.list": ["source1", "source2"]}', + [ApplicationSourceList(name="source_name", sources=["line1", "line2"])], + '[{"name": "source_name", "sources": ["line1", "line2"]}]', ), ], ) @@ -84,14 +91,14 @@ def test_do_source_command_list( """ source - 46C1680FC119E61A501811823A319F932D945953 + intel-gpu-jammy.gpg intel-gpu-jammy.list """, "dispatcher.source.source_command.create_application_source_manager", OsType.Ubuntu, ApplicationRemoveSourceParameters( - gpg_key_id="46C1680FC119E61A501811823A319F932D945953", + gpg_key_name="intel-gpu-jammy.gpg", file_name="intel-gpu-jammy.list", ), ), @@ -111,3 +118,130 @@ def test_do_source_command_remove( mock_manager.remove.assert_called_once_with(expected_call) assert result == Result(status=200, message="SUCCESS") + + +@pytest.mark.parametrize( + "xml, manager_mock, os_type, expected_call", + [ + ( + """ + + source + + + + sourceA + sourceB + + + + """, + "dispatcher.source.source_command.create_os_source_manager", + OsType.Ubuntu, + SourceParameters(sources=["sourceA", "sourceB"]), + ), + ( + """ + + source + + + + gpguri + keyname + + + + sourceA + sourceB + + repofilename + + + + """, + "dispatcher.source.source_command.create_application_source_manager", + OsType.Ubuntu, + ApplicationAddSourceParameters( + file_name="repofilename", + gpg_key_name="keyname", + gpg_key_uri="gpguri", + sources=["sourceA", "sourceB"], + ), + ), + ], +) +def test_do_source_command_add( + mocker, xml_handler_factory, xml, manager_mock, os_type, expected_call +): + xml_handler = xml_handler_factory(xml) + + mock_manager = mocker.Mock() + mock_manager.add.return_value = None + + mocker.patch(manager_mock, return_value=mock_manager) + + result = do_source_command(xml_handler, os_type) + + mock_manager.add.assert_called_once_with(expected_call) + assert result == Result(status=200, message="SUCCESS") + + +@pytest.mark.parametrize( + "xml, manager_mock, os_type, expected_call", + [ + ( + """ + + source + + + + source1 + source2 + + + + """, + "dispatcher.source.source_command.create_os_source_manager", + OsType.Ubuntu, + SourceParameters(sources=["source1", "source2"]), + ), + ( + """ + + source + + + + + source1 + source2 + + filename + + + + """, + "dispatcher.source.source_command.create_application_source_manager", + OsType.Ubuntu, + ApplicationUpdateSourceParameters( + file_name="filename", sources=["source1", "source2"] + ), + ), + ], +) +def test_do_source_command_update( + mocker, xml_handler_factory, xml, manager_mock, os_type, expected_call +): + xml_handler = xml_handler_factory(xml) + + mock_manager = mocker.Mock() + mock_manager.update.return_value = None + + mocker.patch(manager_mock, return_value=mock_manager) + + result = do_source_command(xml_handler, os_type) + + mock_manager.update.assert_called_once_with(expected_call) + assert result == Result(status=200, message="SUCCESS") diff --git a/inbm/dispatcher-agent/tests/unit/source/test_ubuntu_source_cmd.py b/inbm/dispatcher-agent/tests/unit/source/test_ubuntu_source_cmd.py index 23dfd0c53..9206154f9 100644 --- a/inbm/dispatcher-agent/tests/unit/source/test_ubuntu_source_cmd.py +++ b/inbm/dispatcher-agent/tests/unit/source/test_ubuntu_source_cmd.py @@ -1,17 +1,25 @@ from unittest import mock import pytest from unittest.mock import mock_open, patch -from dispatcher.dispatcher_exception import DispatcherException +from dispatcher.source.source_exception import SourceError from dispatcher.source.constants import ( UBUNTU_APT_SOURCES_LIST_D, + UBUNTU_APT_SOURCES_LIST, ApplicationRemoveSourceParameters, SourceParameters, + ApplicationUpdateSourceParameters, ) from dispatcher.source.ubuntu_source_manager import ( UbuntuApplicationSourceManager, UbuntuOsSourceManager, ) +APP_SOURCE = [ + "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] " + "https://repositories.intel.com/gpu/ubuntu jammy unified", + "deb-src https://repo.zabbix.com/zabbix/5.0/ubuntu jammy main", +] + @pytest.fixture def sources_list_content(): @@ -45,7 +53,7 @@ def test_list(self, sources_list_content): def test_list_with_oserror_exception(self): with patch("builtins.open", side_effect=OSError): command = UbuntuOsSourceManager() - with pytest.raises(DispatcherException) as exc_info: + with pytest.raises(SourceError) as exc_info: command.list() assert "Error opening source file" in str(exc_info.value) @@ -111,14 +119,100 @@ def write_side_effect(*args, **kwargs): mo.return_value.write.side_effect = write_side_effect - with patch("builtins.open", mo), pytest.raises(DispatcherException) as exc_info: + with patch("builtins.open", mo), pytest.raises(SourceError) as exc_info: manager = UbuntuOsSourceManager() manager.remove(sources_to_remove) assert "Error occurred while trying to remove sources" in str(exc_info.value) + def test_ubuntu_os_source_manager_add_success(self): + test_sources = [ + "deb http://archive.ubuntu.com/ubuntu focal main", + "deb-src http://archive.ubuntu.com/ubuntu focal main", + ] + parameters = SourceParameters(sources=test_sources) + + manager = UbuntuOsSourceManager() + + m = mock_open() + with patch("builtins.open", m): + manager.add(parameters) + + m.assert_called_once_with(UBUNTU_APT_SOURCES_LIST, "a") + m().write.assert_any_call(f"{test_sources[0]}\n") + m().write.assert_any_call(f"{test_sources[1]}\n") + + def test_ubuntu_os_source_manager_add_error(self): + test_sources = [ + "deb http://archive.ubuntu.com/ubuntu focal main", + "deb-src http://archive.ubuntu.com/ubuntu focal main", + ] + parameters = SourceParameters(sources=test_sources) + + manager = UbuntuOsSourceManager() + + m = mock_open() + m.side_effect = OSError("Permission denied") + with patch("builtins.open", m): + with pytest.raises(SourceError) as e: + manager.add(parameters) + assert str(e.value) == "Error adding sources: Permission denied" + + def test_update_sources_success(self): + mock_sources = [ + "deb http://archive.ubuntu.com/ubuntu/ bionic universe", + "deb http://archive.ubuntu.com/ubuntu/ bionic-updates universe", + ] + parameters = SourceParameters(sources=mock_sources) + manager = UbuntuOsSourceManager() + mock_file = mock_open() + + # Act & Assert + with patch("builtins.open", mock_file): + manager.update(parameters) + mock_file.assert_called_once_with(UBUNTU_APT_SOURCES_LIST, "w") + mock_file().write.assert_has_calls( + [mock.call(f"{source}\n") for source in mock_sources] + ) + + def test_update_sources_os_error(self): + # Arrange + parameters = SourceParameters(sources=["source"]) + manager = UbuntuOsSourceManager() + mock_file = mock_open() + + # Simulate an OSError + mock_file.side_effect = OSError("Mocked error") + + # Act & Assert + with patch("builtins.open", mock_file): + with pytest.raises(SourceError) as excinfo: + manager.update(parameters) + assert "Error adding sources: Mocked error" in str(excinfo.value) + class TestUbuntuApplicationSourceManager: + @patch("dispatcher.source.ubuntu_source_manager.move_file") + def test_update_app_source_successfully(self, mock_move): + try: + params = ApplicationUpdateSourceParameters( + file_name="intel-gpu-jammy.list", sources=APP_SOURCE + ) + command = UbuntuApplicationSourceManager() + with patch("builtins.open", new_callable=mock_open()) as m: + command.update(params) + except SourceError as err: + assert False, f"'UbuntuApplicationSourceManager.update' raised an exception {err}" + + # def test_raises_exception_on_io_error_during_update_app_source_ubuntu(self): + # params = ApplicationUpdateSourceParameters(file_name="intel-gpu-jammy.list", + # sources=APP_SOURCE) + # command = UbuntuApplicationSourceManager() + # with pytest.raises(SourceError) as exc_info: + # with patch('builtins.open', new_callable=mock_open(), side_effect=OSError): + # command.update(params) + # assert "Error while writing file: " in str(exc_info.value) + def test_list(self, sources_list_d_content): with patch("glob.glob", return_value=["/etc/apt/sources.list.d/example.list"]), patch( "builtins.open", mock_open(read_data=sources_list_d_content) @@ -136,74 +230,56 @@ def test_list_raises_exception(self): "builtins.open", side_effect=OSError ): command = UbuntuApplicationSourceManager() - with pytest.raises(DispatcherException) as exc_info: + with pytest.raises(SourceError) as exc_info: command.list() assert "Error listing application sources" in str(exc_info.value) - @pytest.mark.parametrize( - "gpg_key_id, file_name, gpg_run_side_effect, gpg_key_exists, expected_except", - [ - # Case: Successful removal of GPG key and source file - ("123456A0", "example_source.list", [("", "", 0), ("", "", 0)], True, None), - # Case: GPG key does not exist, but the source file is still removed - ("123456A1", "example_source.list", [("", "No such key", 1)], False, None), - # Case: GPG exists but fails to delete, expect DispatcherException - ( - "123456A2", - "example_source.list", - [("", "", 0), ("", "GPG Delete Error", 1)], - True, - DispatcherException, - ), - # Case: OSError on checking GPG key, expect DispatcherException - ("123456A3", "example_source.list", OSError("GPG Error"), True, DispatcherException), - # Case: OSError on removing the file, expect DispatcherException - ( - "123456A4", - "example_source.list", - [("", "", 0), ("", "", 0)], - True, - DispatcherException, - ), - ], - ) - def test_ubuntu_application_source_manager_remove( - self, gpg_key_id, file_name, gpg_run_side_effect, gpg_key_exists, expected_except, mocker + @patch("dispatcher.source.ubuntu_source_manager.remove_file", return_value=True) + @patch("dispatcher.source.ubuntu_source_manager.remove_gpg_key_if_exists") + def test_successfully_remove_gpg_key_and_source_list( + self, mock_remove_gpg_key, mock_remove_file ): - parameters = ApplicationRemoveSourceParameters(gpg_key_id=gpg_key_id, file_name=file_name) + parameters = ApplicationRemoveSourceParameters( + gpg_key_name="example_source.gpg", file_name="example_source.list" + ) + command = UbuntuApplicationSourceManager() + try: + command.remove(parameters) + except SourceError: + self.fail("Remove GPG key raised DispatcherException unexpectedly!") - shell_runner_mock = mocker.patch( - "dispatcher.source.ubuntu_source_manager.PseudoShellRunner" + @patch("dispatcher.source.ubuntu_source_manager.remove_gpg_key_if_exists") + def test_raises_when_space_check_fails(self, mock_remove_gpg_key): + parameters = ApplicationRemoveSourceParameters( + gpg_key_name="example_source.gpg", file_name="../example_source.list" ) - shell_runner_instance = shell_runner_mock.return_value - if isinstance(gpg_run_side_effect, list): # Simulate sequence of command runs - shell_runner_instance.run.side_effect = [ - (stdout, stderr, code) for stdout, stderr, code in gpg_run_side_effect - ] - else: # Directly raise OSError - shell_runner_instance.run.side_effect = gpg_run_side_effect - - # Mock os.remove based on whether we expect an exception for file removal or not - os_remove_mock = mocker.patch("os.remove") - if expected_except is DispatcherException and gpg_run_side_effect == [ - ("", "", 0), - ("", "", 0), - ]: - os_remove_mock.side_effect = OSError("File could not be removed") - - if expected_except: - # If we expect an exception, check that it is raised - with pytest.raises(expected_except): - command = UbuntuApplicationSourceManager() - command.remove(parameters) - else: - # If no exception is expected, perform the operation and assert mocks are called as expected - command = UbuntuApplicationSourceManager() + command = UbuntuApplicationSourceManager() + with pytest.raises(SourceError) as ex: command.remove(parameters) + assert str(ex.value) == "Invalid file name: ../example_source.list" - expected_gpg_calls = [mocker.call(f"gpg --list-keys {gpg_key_id}")] - if gpg_key_exists: - expected_gpg_calls.append(mocker.call(f"gpg --delete-key {gpg_key_id}")) - shell_runner_instance.run.assert_has_calls(expected_gpg_calls) + @patch("dispatcher.source.ubuntu_source_manager.remove_file", return_value=False) + @patch("dispatcher.source.ubuntu_source_manager.remove_gpg_key_if_exists") + def test_raises_when_unable_to_remove_file(self, mock_remove_gpg_key, mock_remove_file): + parameters = ApplicationRemoveSourceParameters( + gpg_key_name="example_source.gpg", file_name="example_source.list" + ) + command = UbuntuApplicationSourceManager() + with pytest.raises(SourceError) as ex: + command.remove(parameters) + assert str(ex.value) == "Error removing file: example_source.list" - os_remove_mock.assert_called_once_with(UBUNTU_APT_SOURCES_LIST_D + "/" + file_name) + @patch( + "dispatcher.source.ubuntu_source_manager.os.path.join", + side_effect=OSError("unable to join path"), + ) + @patch("dispatcher.source.ubuntu_source_manager.remove_file", return_value=False) + @patch("dispatcher.source.ubuntu_source_manager.remove_gpg_key_if_exists") + def test_raises_on_os_error(self, mock_remove_gpg_key, mock_remove_file, mock_os_error): + parameters = ApplicationRemoveSourceParameters( + gpg_key_name="example_source.gpg", file_name="example_source.list" + ) + command = UbuntuApplicationSourceManager() + with pytest.raises(SourceError) as ex: + command.remove(parameters) + assert str(ex.value) == "Error removing file: unable to join path" diff --git a/inbm/dispatcher-agent/tests/unit/test_configuration_helper.py b/inbm/dispatcher-agent/tests/unit/test_configuration_helper.py index 5e33a5cbc..2342d3742 100644 --- a/inbm/dispatcher-agent/tests/unit/test_configuration_helper.py +++ b/inbm/dispatcher-agent/tests/unit/test_configuration_helper.py @@ -4,11 +4,8 @@ from dispatcher.packagemanager import memory_repo from dispatcher.dispatcher_exception import DispatcherException from inbm_lib.xmlhandler import XmlHandler -from unittest.mock import patch, MagicMock import os -from typing import Any - TEST_SCHEMA_LOCATION = os.path.join(os.path.dirname(__file__), '../../fpm-template/usr/share/dispatcher-agent/' 'manifest_schema.xsd') diff --git a/inbm/dispatcher-agent/tests/unit/test_dispatcher.py b/inbm/dispatcher-agent/tests/unit/test_dispatcher.py index 881ec3d8c..914448f74 100644 --- a/inbm/dispatcher-agent/tests/unit/test_dispatcher.py +++ b/inbm/dispatcher-agent/tests/unit/test_dispatcher.py @@ -384,6 +384,17 @@ def test_do_install_can_call_do_source_command(self, mock_workload_orchestration_func.assert_called() mock_do_source_command.assert_called_once() + def test_abc(self, mock_logging: Any): + xml = """\ + + + source + + """ + + d = TestDispatcher._build_dispatcher() + d.do_install(xml=xml, schema_location=TEST_SCHEMA_LOCATION) + @patch('dispatcher.config.config_operation.ConfigOperation._do_config_install_load') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.connect') @patch('inbm_lib.mqttclient.mqtt.mqtt.Client.subscribe') diff --git a/inbm/dispatcher-agent/tests/unit/test_update_logger.py b/inbm/dispatcher-agent/tests/unit/test_update_logger.py index e19d5bf9e..a12c6170e 100644 --- a/inbm/dispatcher-agent/tests/unit/test_update_logger.py +++ b/inbm/dispatcher-agent/tests/unit/test_update_logger.py @@ -4,8 +4,7 @@ from unittest.mock import patch from dispatcher.update_logger import UpdateLogger -from inbm_lib.constants import LOG_FILE, OTA_PENDING, OTA_SUCCESS, OTA_FAIL, FORMAT_VERSION -from inbm_lib.path_prefixes import INTEL_MANAGEABILITY_CACHE_PATH_PREFIX +from inbm_lib.constants import LOG_FILE, OTA_PENDING, OTA_SUCCESS, FAIL, FORMAT_VERSION SOTA_MANIFEST = 'ota
sotaremote
update' @@ -22,7 +21,7 @@ def test_set_time(self) -> None: @patch('dispatcher.update_logger.UpdateLogger.write_log_file') def test_save_log(self, mock_write_log_file) -> None: - expected_status = OTA_FAIL + expected_status = FAIL self.update_logger._time = datetime.datetime(2023, 12, 25, 00, 00, 00, 000000) expected_error = '{"status": 302, "message": "OTA FAILURE"}' expected_metadata = 'ota
aotaremote
updateapplicationhttp://security.ubuntu.com/ubuntu/pool/main/n/net-tools/net-tools_1.60-25ubuntu2.1_amd64.debno' diff --git a/inbm/dispatcher-agent/tests/unit/test_xmlparser.py b/inbm/dispatcher-agent/tests/unit/test_xmlparser.py index 961cee599..ec9ed840f 100644 --- a/inbm/dispatcher-agent/tests/unit/test_xmlparser.py +++ b/inbm/dispatcher-agent/tests/unit/test_xmlparser.py @@ -103,11 +103,13 @@ def test_application_source_with_add_passes_validation(self) -> None: - + - + + + @@ -124,10 +126,12 @@ def test_application_source_with_update_passes_validation(self) -> None: source - - - - + + + foo + + bar + @@ -143,10 +147,10 @@ def test_application_source_with_remove_passes_validation(self) -> None: - + name.gpg - + name.list diff --git a/inbm/dockerfiles/Dockerfile-check.m4 b/inbm/dockerfiles/Dockerfile-check.m4 index 86f507320..bd929afa0 100644 --- a/inbm/dockerfiles/Dockerfile-check.m4 +++ b/inbm/dockerfiles/Dockerfile-check.m4 @@ -56,6 +56,8 @@ RUN source /venv-py3/bin/activate && \ FROM venv-py3 as test-inbm-lib WORKDIR /src/inbm-lib +# for unit test +COPY inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd /src/inbm/dispatcher-agent/fpm-template/usr/share/dispatcher-agent/manifest_schema.xsd RUN source /venv-py3/bin/activate && \ cd /src/inbm-lib && \ set -o pipefail && \ diff --git a/inbm/autopep8.sh b/inbm/format-python.sh similarity index 100% rename from inbm/autopep8.sh rename to inbm/format-python.sh diff --git a/inbm/integration-reloaded/launchers/source-test.sh b/inbm/integration-reloaded/launchers/source-test.sh new file mode 100755 index 000000000..8acca7bd1 --- /dev/null +++ b/inbm/integration-reloaded/launchers/source-test.sh @@ -0,0 +1,26 @@ +#!/bin/bash +set -euxo pipefail + +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +. "$DIR"/../../integration-common/util/tc-messages.sh + +run_vagrant_provision_test() { + test_with_command "$1" vagrant provision --provision-with \""$1"\" +} + +vagrant_provision() { + vagrant provision --provision-with "$1" +} + +suite_started SOURCE + +cleanup() { + suite_finished SOURCE +} +trap cleanup 0 + +"$DIR"/vagrant-up.sh +"$DIR"/vagrant-wait-for-up.sh + +test_started "SOURCE test" +vagrant ssh -c "sudo /test/source/SOURCE.sh" diff --git a/inbm/integration-reloaded/run-quick.sh b/inbm/integration-reloaded/run-quick.sh index eb564e6a6..df6d4ca2c 100755 --- a/inbm/integration-reloaded/run-quick.sh +++ b/inbm/integration-reloaded/run-quick.sh @@ -18,6 +18,7 @@ trap general 0 launchers/install-framework-quick.sh launchers/update-system.sh +launchers/source-test.sh launchers/aota-test.sh launchers/sota-quick-tests.sh launchers/fota-test.sh diff --git a/inbm/integration-reloaded/run-slow.sh b/inbm/integration-reloaded/run-slow.sh index 1a2d565b0..112a86203 100755 --- a/inbm/integration-reloaded/run-slow.sh +++ b/inbm/integration-reloaded/run-slow.sh @@ -20,6 +20,7 @@ launchers/update-system.sh launchers/install-framework-slow.sh launchers/setup-servers.sh # This should happen after any docker uninstalls. +launchers/source-test.sh launchers/aota-test.sh launchers/sota-quick-tests.sh launchers/fota-test.sh diff --git a/inbm/integration-reloaded/test/source/SOURCE.sh b/inbm/integration-reloaded/test/source/SOURCE.sh new file mode 100755 index 000000000..c19cef04d --- /dev/null +++ b/inbm/integration-reloaded/test/source/SOURCE.sh @@ -0,0 +1,74 @@ +#!/bin/bash +set -e +set -x + +source /scripts/test_util.sh + +trap 'kill -9 $(jobs -p) || true' EXIT + +APT_SOURCES="/etc/apt/sources.list" +BAK_APT_SOURCES="$APT_SOURCES.bak" +FAKE_SOURCE="deb test123" +UPDATE_SOURCE1="deb test456" +UPDATE_SOURCE2="deb test789" +OPERA_KEY_URI="https://deb.opera.com/archive.key" +OPERA_KEY_NAME="opera.gpg" +OPERA_SOURCES="deb [arch=amd64 signed-by=/usr/share/keyrings/$OPERA_KEY_NAME] https://deb.opera.com/opera-stable/ stable non-free" +OPERA_LIST="opera.list" +NEW_APP_SOURCE="deb newsource" + +cp "$APT_SOURCES" "$BAK_APT_SOURCES" + +test_failed() { + cp "$BAK_APT_SOURCES" "$APT_SOURCES" + rm -f "/usr/share/keyrings/$OPERA_KEY_NAME" + rm -f "/etc/apt/sources.list.d/$OPERA_LIST" + echo "Return code: $?" + echo "TEST FAILED!!!" +} +trap test_failed ERR + +echo "Starting source test." | systemd-cat + +# OS tests +inbc source os add --sources "$FAKE_SOURCE" +grep "$FAKE_SOURCE" "$APT_SOURCES" +inbc source os list 2>&1 | grep "$FAKE_SOURCE" +inbc source os remove --sources "$FAKE_SOURCE" +grep "$FAKE_SOURCE" "$APT_SOURCES" && exit 1 + +inbc source os add --sources "$FAKE_SOURCE" +inbc source os update --sources "$UPDATE_SOURCE1" "$UPDATE_SOURCE2" +grep "$FAKE_SOURCE" "$APT_SOURCES" && exit 1 +grep "$UPDATE_SOURCE1" "$APT_SOURCES" +grep "$UPDATE_SOURCE2" "$APT_SOURCES" + +cp "$BAK_APT_SOURCES" "$APT_SOURCES" + +# Application tests +rm -f "/usr/share/keyrings/$OPERA_KEY_NAME" +rm -f "/etc/apt/sources.list.d/$OPERA_LIST" +inbc source application add --gpgKeyUri "$OPERA_KEY_URI" --gpgKeyName "$OPERA_KEY_NAME" --sources "$OPERA_SOURCES" --filename "$OPERA_LIST" +if [ ! -e "/usr/share/keyrings/$OPERA_KEY_NAME" ]; then + echo "Error: The file '/usr/share/keyrings/$OPERA_KEY_NAME' does not exist!" + exit 1 +fi +inbc source application list 2>&1 | grep "$OPERA_KEY_NAME" +inbc source application remove --gpgKeyName "$OPERA_KEY_NAME" --filename "$OPERA_LIST" + +if inbc source application list 2>&1 | grep -q "$OPERA_KEY_NAME"; then + echo "Error: $OPERA_KEY_NAME should not be present in the application list after removal" + exit 1 +fi + +inbc source application add --gpgKeyUri "$OPERA_KEY_URI" --gpgKeyName "$OPERA_KEY_NAME" --sources "$OPERA_SOURCES" --filename "$OPERA_LIST" +inbc source application update --sources "$NEW_APP_SOURCE" --filename "$OPERA_LIST" +inbc source application list 2>&1 | grep "$NEW_APP_SOURCE" + +if inbc source application list 2>&1 | grep -q "$OPERA_KEY_NAME"; then + echo "Error: $OPERA_KEY_NAME should not be present in the application list after update" + exit 1 +fi +inbc source application remove --gpgKeyName "$OPERA_KEY_NAME" --filename "$OPERA_LIST" +rm -f "/usr/share/keyrings/$OPERA_KEY_NAME" +rm -f "/etc/apt/sources.list.d/$OPERA_LIST" diff --git a/pull_request_template.md b/pull_request_template.md index 64099daf5..94f4fd216 100644 --- a/pull_request_template.md +++ b/pull_request_template.md @@ -30,7 +30,7 @@ Reference URL for issue tracking (JIRA/HSD/Github): **\** - [ ] PR change contains code related to security - [ ] PR introduces changes that breaks compatibility with other modules (If YES, please provide description) - [ ] Specific instructions or information for code reviewers (If any): -- [ ] Run 'go fmt' or autopep8 as applicable. +- [ ] Run 'go fmt' or format-python.sh as applicable. - [ ] New/modified methods and functions should have type annotations on signatures as applicable - [ ] New/modified methods must have appropriate doc strings (language dependent)