fix: Add about Python req. and fix ansible-bootstrap (#22)

fix: Add about Python req. and fix ansible-bootstrap

Add some extra information about the management of Python packages and
how they are added to the projects requirements.txt Python package file.

Improve the 'ansible-requirements' devbox command to add traceability
information, and the merge process strips any comments.

Ensure we always establish py venv because of devbox.

Also fix the ansible-bootstrap command to ensure the requirements files
are in place, if not already in place.

Minor wording and spelling fixed as well.

README.md

@@ -32,7 +32,7 @@ This plugin has the baseline of tools needed for a DevOps project.
 
 This includes:
 
-* Python, including a virtual environment if `requirements.txt` is present in the root of the repository
+* Python, and activation of a virtual environment if `requirements.txt` is present in the root of the repository using this plugin
 * Pre-commit and linting hooks
 * tenv, tflint and terraform-docs for working with terraform or tofu
 * Taskfile
@@ -41,21 +41,25 @@ Assumptions:
 
 * You use either Bash or Zsh as your shell.
 
-#### Python and pip
+#### Python and virtual environments
 
 The plugin creates a python virtual environment, activates it and installs pip requirements if `requirements.txt` exists in the root of the project. If your requirements file is located elsewhere, you can create a "root" requirements.txt with a link to your other requirements file.
 
 Example `requirement.txt`:
 
+```requirements.txt
+-r customlocation/requirements.txt
 ```
--r configure/requirements.txt
-```
+
+Taskfile tasks, or devbox scripts might add to the base `requirements.txt` in the projects to ensure other requirements files are loaded, like the `bootstrap-taskfile` or `bootstrap-ansible` that add the line `-r configure/requirements.txt` into `requirements.txt`.
+
+Any Python package requirements needed for the devbox plugin itself, are managed in [`config/devbox-requirements.txt`](config/devbox-requirements.txt) and additionally installed during devbox init hooks, separately from the project requirements that might come from other sources or other devbox commands that manage those.
 
 ##### Requirements in repositories vs. defaults from devbox
 
-There has been created a devbox task, that merges an eventual `configure/requirements.txt.local` with a `requirements.txt.dist` from this devbox repository. This means that we can give some default python requirements from here, and not need to handle requirements in the platofmr repository. Updates for the requirements can be done easy from here.
+There has been created a devbox task, that merges an eventual `configure/requirements.txt.local` with a `requirements.txt.dist` from this devbox repository. This means that we can give some default python requirements from here, and not need to handle requirements in the infrastructure repositories. Updates for the requirements can be done easily from here.
 
-Update the `configure/requirements.txt` by running `devbox run ansible-requirements` from the root of the platform repository. This `configure/requirements.txt` must then be included in `requirements.txt` as [described above](#python-and-pip).
+Update the `configure/requirements.txt` by running `devbox run ansible-requirements` from the root of the platform repository. This `configure/requirements.txt` must then be included in `requirements.txt` as [described above](#python-and-virtual-environments).
 
 #### Terraform
 

devbox-plugins/base-config/config/ansible-requirements-helper.sh

@@ -7,13 +7,25 @@ set -e
 
 # If we have a configure/requirements.txt.local file, we merge it with the ansible-requirements.txt.dist file.
 if [ -e configure/requirements.txt.local ]; then
+    # remove old file first, if it exists because merge_requirements will not overwrite and create new ones
     rm -f requirements-merged.txt
+    # produced requirements-merged.txt in the root of the repository
     merge_requirements {{ .Virtenv }}/ansible-requirements.txt.dist configure/requirements.txt.local
-    mv requirements-merged.txt configure/requirements.txt
+
+    # Add little helper and traceability msg and output to final file and then add merged result
+    cat << EOF > configure/requirements.txt
+# This file is the result of a merge of requirements files, always taking the union of Python packages, and the last version of each package.
+# The merge is done by our devops tools, and a devbox run command merging two files:
+# -  {{ .Virtenv }}/ansible-requirements.txt.dist
+# - configure/requirements.txt.local
+# NOTICE comments are stripped out, so check the two files for comments if in doubt of why packages are included.
+EOF
+    cat requirements-merged.txt >> configure/requirements.txt
+    rm requirements-merged.txt
 else
-    # Or else we simply just use the ansible-requirements.txt.dist file provided from devbox.
+    # Or else we simply just use the ansible-requirements.txt.dist file provided from devbox (this also contains comments and traceability information)
     cp {{ .Virtenv }}/ansible-requirements.txt.dist configure/requirements.txt
 fi
 
 # By touching the file, devbox will re-initialize itself, and update environments.
-touch devbox.json
+touch devbox.json

devbox-plugins/base-config/config/ansible-requirements.txt.dist

@@ -1,3 +1,8 @@
+# This files comes from our devops tools base-config plugin, and the authoritative source of our standard packages needed
+# for working with Ansible.
+# The files is supposed to contain our common selection of packages
+# and are used also to keep projects aligned by running specific devbox run command to distribute and possible merge with local
+# requirements files
 ansible-core==2.17.1
 # For the Hetzner Cloud dynamic inventory plugin:
 # https://docs.ansible.com/ansible/latest/collections/hetzner/hcloud/hcloud_inventory.html#ansible-collections-hetzner-hcloud-hcloud-inventory-requirements

devbox-plugins/base-config/config/devbox-requirements.txt

@@ -1,3 +1,5 @@
+# Used for the ansible-requirements devbox run command
+#
 # https://pypi.org/project/merge-requirements/
 # It seems unmaintained, and is very old from 2016, however the newer tool like pyreq-merger at https://github.com/mhristodor/pyreq-merger doesn't even work because the distribution is failing due to missing .version.txt file it seems he forgot to distribute with the tool. Maybe to output help or usage with the version of tool?
 # Those two tools was however the only ones we could find covering our most important use-cases about upgrading packages with locked versions, found in two requirements files.

devbox-plugins/base-config/plugin.json

@@ -62,10 +62,9 @@
   },
   "shell": {
     "init_hook": [
-      // create virtual environment if requirements.txt exists
+      // always create virtual environment to get devbox requirements installed
       // 'activate' works for bash and zsh
-      "if [ -f requirements.txt ];then . $VENV_DIR/bin/activate;fi",
-
+      ". $VENV_DIR/bin/activate",
       // Install tools, that devbox needs.
       "pip install -r {{ .Virtenv }}/devbox-requirements.txt",
 
@@ -122,6 +121,16 @@
       "bootstrap-ansible": [
         "echo 'Configures a Ansible project and merges the standard configuration with your local configuration.'",
         "mkdir -p configure/",
+
+        // Also add requirements needed to work with Ansible and install python packages
+        // First of if we use Ansible, we need a project requirements.txt file is not already there
+        "if ! [ -f requirements.txt ]; then touch requirements.txt; fi",
+        // Then we need to point it to the requirements file used in Ansible from the configure directory
+        "if ! crudini --get requirements.txt \"\" \"-r configure/requirements.txt\";then echo -e '\n\n# Add required Python packages for Ansible use\n-r configure/requirements.txt' >> requirements.txt; fi",
+
+        // Reuse another devbox run script to ensure we have a default Ansible requirements.txt file, merged with possible local ones
+        "devbox run ansible-requirements",
+
         "if ! [ -f configure/ansible.cfg ];then echo 'Creating empty ansible.cfg file if not exists.'; fi",
         "if ! [ -f configure/ansible.cfg ];then cp -v {{ .Virtenv }}/ansible.cfg configure/ansible.cfg; fi",
         "echo 'Merging ansible.cfg.dist into ansible.cfg.'",