dcnm_image_policy (ready to merge) (#272)

* dcnm_image_policy: work in progress initial commit contains a sort-of working merged state * Rename Payload to Config2Payload, more... 1. Add Payload2Config 2. Make Payload a base class for both Config2Payload and Payload2Config 3. Shorten filenames in module_utils/image_policy/*.py so as not to duplicate the directory name. 4. Add unit tests for ApiEndpoints, Config2Payload, and Payload2Config * remove unused file * merged state: differing policy parameters are now updated We merge want into have with want taking precedence. * saving work in progress (not working currently) * Committing changes for today for backup * Rename Policy* to ImagePolicy*, refactor, more... 1. Rename classes from Policy* to ImagePolicy* to differentiate them from NDFC Policies 2. Refactor ImagePolicyUpdate* and ImagePolicyCreate* to deduplicate common code and simplify the commit() method in each. * Run thru black/isort * Add state property to MockAnsibleModule, more... Run thru black/isort * Align functionality and docstring * Make subclass of ImagePolicyCommon, more... 1. Modify fail_json() calls to include self.failed_result from ImagePolicyCommon() 2. Run thru black / isort * Add handler for query state, more... Also: 1. Add logging to all classes 2. Run thru linters 3. Rename Task() to ImagePolicyTask() to distinguish from other task classses in the log 4. Move result.py to dcnm_image_upgrade module_utils/common. We'll commit it there. 5. ImagePolicyReplaceBulk().payloads() remove unused var 6. Remove unused method_name vars from all methods that don't call fail_json(), since the logger now provides the method name. 7. Config2Payload().commit() added query state to conditional to avoid building unrelated payload. 8. ParamsSpec() - added _build_params_spec_for_query_state() method. * Standardize result processing, more... ImagePolicyTask: - update_diff_and_response() - new method - handle_replaced_state() - standardize result processing - handle_deleted_state() - rename delete to instance - handle_delete_state() - leverage update_diff_and_response() - leverage ImagePolicyQuery() - Use ImagePolicyTaskResult() rather than Result() - delete_policies_not_in_want() - leverage update_diff_and_response() - handle_query_state() - leverage ImagePolicyQuery() - handle_query_state() - leverage update_diff_and_response() - send_need_create() - leverage update_diff_and_response() - send_need_update() - leverage update_diff_and_response() ImagePolicyUpdate: - Standardize diff, result, reponse handling - Inject self.action into diff elements ImagePolicyReplaceBulk: - Standardize diff, result, reponse handling - Inject self.action into diff elements - Remove ImagePolicyUpdate() class ParamsSpec: - _build_params_spec_for_overridden_state() - new method ImagePolicyDelete: - Standardize diff, result, reponse handling - Inject self.action into diff elements ImagePolicyCreate: - Standardize diff, result, reponse handling - Inject self.action into diff elements ImagePolicyCommon: - Add properties for failed, response, response_current, result, result_current - Remove properties for send_interval, timeout, and unit_test - Leveage ImagePolicyTaskResult.failed_result for failed_result property * Remove parameters dict This was used in early development with MockAnsibleModule, but is no longer needed. * Initial unit tests and fixtures, more... Adding some initial unit tests. As a result of these, found several issues which are address in the tested code for this commit. Within tests/unit/modules/dcnm/dcnm_image_policy: - utils.py: Add fixtures, add comments for unused fixtures to remove later - test_image_policy_create.py: initial unit tests for ImagePolicyCreate() - test_image_policy_create_bulk.py: initial unit tests for ImagePolicyCreateBulk() - Add json data for above test cases Within module_utils/image_policy - update.py: refactor, run through linters, test changes - replace.py: refactor, run through linters, test changes - params_spec.py: run though linters - delete.py: run through linters - create.py: run through linters, refactor - common.py: self.module should be self.ansible_module * Forgot to commit this with initial commit. * Remove unused imports, add query.py, add image_policies.py For now, copy ImagePolicies() class (image_policies.py) from dcnm_image_upgrade, since the imports for this class within dcnm_image_policy won't work until dcnm_image_upgrade PR is merged. Need to rethink the location of common image_mgmt classes... * Forgot to add copied ImagePolicies() class, more... Rename test_image_policy_api_endpoints.py to test_image_policy_endpoints.py to reflect the filename being tested, rather than the class name. Fix ImagePolicies() import in create.py * Fix import issues, more... Several files that are shared with dcnm_image_upgrade (and will be committed with dcnm_image_upgrade) are needed for sanity and unit tests to pass for dcnm_image_policy. I've added a README_MERGE_TODO.txt file in module_utils/common that lists these files. We need to remove these files from this branch after committing dcnm_image_upgrade. * Add a documentation string * Fix yaml linter issues * Fix one more yaml linter issue * Fix argument_spec to define config as a list * Fix yaml linter errors and run through black/isort * Add dcnm_image_policy.py to tests/sanity/ignore* for missing-gplv3-license Also, fix module name in documentation. * Fix PEP8 issues * Remove unused file * Fix lazy formatting, remove unused file * Fix pylint bad-option-value * fail_json if image policy ref_count is non-zero For delete, update operations, an image policy cannot be attached to any devices. We now verify that this is the case, and fail_json with a message to use the dcnm_image_upgrade module to detach the image policies before trying to delete or update them. * Fix pylint issue * Add initial integration tests for merged and deleted state * Remove one extra blank line at the end of file to appease yamllint * Add integration tests for overridden state * Initial integration tests for replaced state * Initial integration tests for query state * 81% unit test coverage for ImagePolicyCreateBulk() Also: 1. Add more debug logs to module_utils/image_policy/create.py 2. Consistent var names in utils.py * Refactor tests to use global MockImagePolicies() * Fix linter objection over too few blank lines * 100% unit test coverage for create.py * Fix pylint issues, more... ImagePolicyCreateCommon(): set self.response_current and self.result_current directly from dcnm_send() and _handle_response() respectively. * 74% coverage for delete.py, more... ImagePolicyDelete.policy_names: add verification that policy_name is a string. ImagePolicyDelete: make image_policies private (self._image_policies) ImagePolicyDelete._get_policies_to_delete: Modify to set self._policies_to_delete instead of returning policies_to_delete (easier for writing unit tests). utils.py, MockImagePolicies: Add ref_count property utils.py, MockImagePolicies: remove *args from all_policies property ImagePolicyCommon._verify_image_policy_ref_count: skip ref_count == None ImagePolicyCommon._verify_image_policy_ref_count: use .items() to simplify for loop. * 100% unit test coverage for delete.py * 100% unit test coverage for query.py, more... Also: 1. Add TEST_NOTES for all json fixture files 2. Modify TEST_NOTES for consistency between fixture files * Forgot to commit image_policies.py and query.py Changes include: - ImagePolicies: remove response, response_data, and ressult properties so that these are inherited from ImagePolicyCommon - ImagePolicyQuery: Set _policies_to_query in __init__ - ImagePolicyQuery: Set failed to False in __init__ - ImagePolicyQuery: make image_policies private (i.e. self._image_policies) - ImagePolicyQuery: policy_names setter. Add checks for empty list, and list containing other than string values. - ImagePolicyQuery._get_policies_to_query: set self._policies_to_query rather than returning _policies_to_query (easier for unit tests). - ImagePolicyQuery.commit: Don't re-instantiate ImagePolicies, use instance from __init__(). * 100% unit test coverage for replace.py Also: ImagePolicyReplaceBulk: initialize verb/path in __init__() ImagePolicyReplaceBulk: add _verify_payload to check that mandatory keys are present in all payloads ImagePolicyReplaceBulk.payloads: refactor to leverage _verify_payloads() ImagePolicyReplaceBulk.default_policy: privatize method name -> _default_policy ImagePolicyReplaceBulk.image_policies: privatize instance name -> _image_policies ImagePolicyReplaceBulk.image_policies: Add comments to clarify some code ImagePolicyReplaceBulk.image_policies: Add debug statements ImagePolicyReplaceBulk._send_payloads: directly set self._result_current and self._response_current from the calls to _handle_response and dcnm_send respectively ImagePolicyReplaceBulk._send_payloads: out of paranoia, use copy.deepcopy() in assignments ImagePolicyReplaceBulk._process_responses: set self.failed as appropriate ImagePolicyCommon: add some debug logs ImagePolicyCreateCommon: out of paranoia, use copy.deepcopy() in assignments ImagePolicyDelete: Remove TODO from commit() docstring * Update comments and fix task name * 99% unit test coverage for update.py Also: - MockImagePolicies: modified to correctly handle ref_count and name when the policy does not exist in all_policies - delete.py: Added Summary to all unit tests - update.py: make self.image_policies private (self._image_policies) - update.py: fix typo in fail_json message - update.py: fix docstring for _build_payloads_to_commit() - update.py: in _build_payloads_to_commit() remove ref_count, imageName, and platformPolicies from the merged payload since these are not valid parameters for the edit-policy endpoint (these ARE returned by the controller, hence we have to remove them). - update.py: _send_payloads(): directly set self.response_current, and self.result_current from the return values of their respective calls. - update.py: _send_payloads(): out of paranoia, deepcopy the response, result, payload when assigning values to diff_*, response_* and result_*. - update.py, ImagePolicyUpdate.payload setter: we needed to set payloads as well here. fixed. - replace.py: move _default_policy() into ImagePolicyCommon() * Fix pylint issues * 100% unit test coverage for update.py * Improve docstrings and comments * Add unit test to bring create.py to 100% coverage * Add unit tests for payload and config setter error handling 90% unit test coverage for payload.py * payload.py: Unit tests for empty config and payload, more... This brings unit test coverage for payload.py to 98% Also: Rename unit test fixture data keys from "...00XXX" to "---00XXXa" for consistency with other unit test fixture data. Fix test_image_policy_payload_00221 to not to use the fixture data from test_image_policy_payload_00220. test_image_policy_payload_00221: Fix fixture data to include the "type" key. * payload.py: add unit test for ansible states query and deleted This brings unit test coverage for payload.py to 100% * ImagePolicyCommon: add unit tests for _get_response(), more... This brings unit test coverage for image_policy_common.py to 79% Also: Update docstring for _handle_post_put_delete_response to indicate that RETURN_CODE is not considered when determining the result and to better describe the logic. * ImagePolicyCommon: add unit tests, more... This brings coverage for image_policy_common.py to 89% Add unit tests for: - make_boolean() - make_none() - @Changed - @diff - @failed - @failed_result * Fix PEP8 errors. * 100% unit test coverage for image_policy_common.py Add unit tests for: - ImagePolicyCommon._handle_unknown_request_verbs() - ImagePolicyCommon.response_current - ImagePolicyCommon.response - ImagePolicyCommon.response_data - ImagePolicyCommon.result - ImagePolicyCommon.result_current For consistency, update fail_json messages and match values for: - ImagePolicyCommon.changed - ImagePolicyCommon.diff - ImagePolicyCommon.failed * Modifications based on Mike's initial review of dcnm_image_policy.py * Fix ImagePolicyTaskResult() docstring (Usage section) The original example code was overwriting the result instance. * ImagePolicyTaskResult: 100% unit test coverage * Update integration tests with new include syntax 'include' is deprecated in Ansible roles and is now 'include_tasks' * First attempt at documentation * antsibull-docs generated doc fails rstcheck Need to ping Mike to remember how we generate docs... ERROR: docs/cisco.dcnm.dcnm_image_policy_module.rst:37:0: Unknown directive type "rst-class". * Implement check_mode * Fix PEP8 errors and add check_mode to MockAnsibleModule * testing potential fix for image_policy unit test failures * Modifications to fix (hopefully) unit tests after implementing check_mode * Fix PEP8 too many black lines * Update docs * Forgot to commit dcnm_image_policy.py with the updated DOCUMENTATION section. * Fix validate-modules invalid-documentation * 2nd try: fixing invalid-documentation format. * Remove redundant REST request query.py: ImagePolicyQuery.commit(): remove redundant refresh() call create.py: remove commented line * Align rseults output with existing modules, more... Leverage Results() class for results generation Leveage RestSend() class in all other classes Update all unit and integration tests * Forgot to commit the unit tests * Fix PEP8 errors * Remove unused library as it's causing an import error * Fix python 3.8 TypeError * Update docstring to reflect current results output. * Fix blank line with whitespace in docstring. * For deleted state, delete all policies if config is None Also: dcnm_image_policy.py: Common().get_want() was referencing a non-existant property (Results().module_result) in exit_json(). Added an "ok_result" property to Results to avoid an error. * Fix documentation to remove required from config --------- Co-authored-by: Mike Wiebe <[email protected]>
CiscoDevNet · Mar 27, 2024 · 6b8e628 · 6b8e628
1 parent d223336
commit 6b8e628
Show file tree

Hide file tree

Showing 67 changed files with 14,296 additions and 0 deletions.
diff --git a/__init__.py b/__init__.py
diff --git a/docs/cisco.dcnm.dcnm_image_policy_module.rst b/docs/cisco.dcnm.dcnm_image_policy_module.rst
diff --git a/plugins/module_utils/common/README_MERGE_TODO.txt b/plugins/module_utils/common/README_MERGE_TODO.txt
@@ -0,0 +1,9 @@
+The following files in this directory need to be deleted prior to merging since these are being committed with dcnm_image_upgrade.
+
+The single source of truth for these is dcnm_image_upgrade.
+
+log.py
+logging_config.json
+merge_dicts.py
+params_merge_defaults.py
+params_validate.py
diff --git a/plugins/module_utils/image_policy/__init__.py b/plugins/module_utils/image_policy/__init__.py
diff --git a/plugins/module_utils/image_policy/common.py b/plugins/module_utils/image_policy/common.py
@@ -0,0 +1,228 @@
+# Copyright (c) 2024 Cisco and/or its affiliates.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import, division, print_function
+
+__metaclass__ = type
+__author__ = "Allen Robel"
+
+import inspect
+import logging
+from typing import Any, Dict
+
+
+class ImagePolicyCommon:
+    """
+    Common methods used by the other classes supporting
+    dcnm_image_policy module
+
+    Usage (where ansible_module is an instance of
+    AnsibleModule or MockAnsibleModule):
+
+    class MyClass(ImagePolicyCommon):
+        def __init__(self, module):
+            super().__init__(module)
+        ...
+    """
+
+    def __init__(self, ansible_module):
+        self.class_name = self.__class__.__name__
+
+        self.log = logging.getLogger(f"dcnm.{self.class_name}")
+        self.log.debug("ENTERED ImagePolicyCommon()")
+
+        self.ansible_module = ansible_module
+        self.check_mode = self.ansible_module.check_mode
+        self.state = ansible_module.params["state"]
+
+        self.params = ansible_module.params
+
+        self.properties: Dict[str, Any] = {}
+        self.properties["results"] = None
+
+    def _verify_image_policy_ref_count(self, instance, policy_names):
+        """
+        instance: ImagePolicies() instance
+        policy_names: list of policy names
+
+        Verify that all image policies in policy_names have a
+        ref_count of 0 (i.e. no devices are using the policy).
+
+        If the ref_count is greater than 0, fail_json with a message
+        indicating that the policy, or policies, must be detached from
+        all devices before it/they can be deleted.
+        """
+        method_name = inspect.stack()[0][3]
+        _non_zero_ref_counts = {}
+        for policy_name in policy_names:
+            instance.policy_name = policy_name
+            msg = f"instance.policy_name: {instance.policy_name}, "
+            msg += f"instance.ref_count: {instance.ref_count}."
+            self.log.debug(msg)
+            # If the policy does not exist on the controller, the ref_count
+            # will be None. We skip these too.
+            if instance.ref_count in [0, None]:
+                continue
+            _non_zero_ref_counts[policy_name] = instance.ref_count
+        if len(_non_zero_ref_counts) == 0:
+            return
+        msg = f"{self.class_name}.{method_name}: "
+        msg += "One or more policies have devices attached. "
+        msg += "Detach these policies from all devices first using "
+        msg += "the dcnm_image_upgrade module, with state == deleted. "
+        for policy_name, ref_count in _non_zero_ref_counts.items():
+            msg += f"policy_name: {policy_name}, "
+            msg += f"ref_count: {ref_count}. "
+        self.ansible_module.fail_json(msg, **self.results.failed_result)
+
+    def _default_policy(self, policy_name):
+        """
+        Return a default policy payload for policy name.
+        """
+        method_name = inspect.stack()[0][3]
+        if not isinstance(policy_name, str):
+            msg = f"{self.class_name}.{method_name}: "
+            msg += "policy_name must be a string. "
+            msg += f"Got type {type(policy_name).__name__} for "
+            msg += f"value {policy_name}."
+            self.log.debug(msg)
+            self.ansible_module.fail_json(msg, **self.results.failed_result)
+
+        policy = {
+            "agnostic": False,
+            "epldImgName": "",
+            "nxosVersion": "",
+            "packageName": "",
+            "platform": "",
+            "policyDescr": "",
+            "policyName": policy_name,
+            "policyType": "PLATFORM",
+            "rpmimages": "",
+        }
+        return policy
+
+    def _handle_response(self, response, verb):
+        """
+        Call the appropriate handler for response based on verb
+        """
+        if verb == "GET":
+            return self._handle_get_response(response)
+        if verb in {"POST", "PUT", "DELETE"}:
+            return self._handle_post_put_delete_response(response)
+        return self._handle_unknown_request_verbs(response, verb)
+
+    def _handle_unknown_request_verbs(self, response, verb):
+        method_name = inspect.stack()[0][3]
+
+        msg = f"{self.class_name}.{method_name}: "
+        msg += f"Unknown request verb ({verb}) for response {response}."
+        self.ansible_module.fail_json(msg)
+
+    def _handle_get_response(self, response):
+        """
+        Caller:
+            - self._handle_response()
+        Handle controller responses to GET requests
+        Returns: dict() with the following keys:
+        - found:
+            - False, if request error was "Not found" and RETURN_CODE == 404
+            - True otherwise
+        - success:
+            - False if RETURN_CODE != 200 or MESSAGE != "OK"
+            - True otherwise
+        """
+        result = {}
+        success_return_codes = {200, 404}
+        if (
+            response.get("RETURN_CODE") == 404
+            and response.get("MESSAGE") == "Not Found"
+        ):
+            result["found"] = False
+            result["success"] = True
+            return result
+        if (
+            response.get("RETURN_CODE") not in success_return_codes
+            or response.get("MESSAGE") != "OK"
+        ):
+            result["found"] = False
+            result["success"] = False
+            return result
+        result["found"] = True
+        result["success"] = True
+        return result
+
+    def _handle_post_put_delete_response(self, response):
+        """
+        Caller:
+            - self.self._handle_response()
+
+        Handle POST, PUT, DELETE responses from the controller.
+
+        Returns: dict() with the following keys:
+        - changed:
+            - True if changes were made to by the controller
+                - ERROR key is not present
+                - MESSAGE == "OK"
+            - False otherwise
+        - success:
+            - False if MESSAGE != "OK" or ERROR key is present
+            - True otherwise
+        """
+        result = {}
+        if response.get("ERROR") is not None:
+            result["success"] = False
+            result["changed"] = False
+            return result
+        if response.get("MESSAGE") != "OK" and response.get("MESSAGE") is not None:
+            result["success"] = False
+            result["changed"] = False
+            return result
+        result["success"] = True
+        result["changed"] = True
+        return result
+
+    def make_boolean(self, value):
+        """
+        Return value converted to boolean, if possible.
+        Return value, if value cannot be converted.
+        """
+        if isinstance(value, bool):
+            return value
+        if isinstance(value, str):
+            if value.lower() in ["true", "yes"]:
+                return True
+            if value.lower() in ["false", "no"]:
+                return False
+        return value
+
+    def make_none(self, value):
+        """
+        Return None if value is an empty string, or a string
+        representation of a None type
+        Return value otherwise
+        """
+        if value in ["", "none", "None", "NONE", "null", "Null", "NULL"]:
+            return None
+        return value
+
+    @property
+    def results(self):
+        """
+        An instance of the Results class.
+        """
+        return self.properties["results"]
+
+    @results.setter
+    def results(self, value):
+        self.properties["results"] = value