Add support for AlmaLinux 8/9 and Rocky Linux 8 (#440)

* Add support for AlmaLinux and Rocky Linux in the Ansible playbooks.
* Switch the default Azure VM image to AlmaLinux 9.
* Add an Azure-specific cloud-init file for AlmaLinux 9 to install `rsync` at first boot, as the image does not seem to install it.
* Add an Ansible task to stop and disable `firewalld` for AlmaLinux 9, to allow RPC communication within the cluster.
* Use Python to install Ansible on the proxy for non-CentOS OS to ensure a current version of Ansible is used. This avoids problems when the Ansible package in the EL repo is outdated or requires a newer Python than is supported by the given OS.
* While using Rocky Linux 9 is not blocked, there are some external issues with the Azure Linux agent and this OS version.
* These changes have been tested on Azure with various OS images used for the cluster nodes and the "proxy". Currently "known good" OS versions for Azure include Fedora 39, CentOS 7.9, AlmaLinux 8/9, Rocky Linux 8.
* Update references to CentOS 7 in the context of EC2 as the default image for EC2 had already been changed to a Fedora 35 AMI.

README.md

@@ -110,13 +110,7 @@ set by the `cluster.username` property.
 
 ### Configuring the AMI
 
-You might also need to configure the `aws_ami` property in [muchos.props]. Muchos by default uses a free
-CentOS 7 image that is hosted in the AWS marketplace but managed by the
-CentOS organization. If you have never used this image in EC2 before, you will need to go to the
-[CentOS 7 product page][centos7] to accept the software terms. If this is not done, you will get an
-error when you try to launch your cluster. By default, the `aws_ami` property is set to an AMI in `us-east-1`.
-You will need to changes this value if a newer image has been released or if you are running in different region
-than `us-east-1`.
+You might also need to configure the `aws_ami` property in [muchos.props]. Muchos by default uses a Fedora 35 image for EC2. By default, the `aws_ami` property is set to this Fedora 35 AMI in `us-east-1`. You will need to change this value if a newer image has been released or if you are running in different region than `us-east-1`.
 
 ### Launching the cluster
 
@@ -144,11 +138,8 @@ this file will be appended on each node to the `~/.ssh/authorized_keys` file for
 `cluster.username` property.  You will also need to ensure you have authenticated to Azure and set the target
 subscription using the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli?view=azure-cli-latest).
 
-Muchos by default uses a CentOS 7 image that is hosted in the Azure marketplace. The Azure Linux Agent is already
+Muchos by default uses an AlmaLinux 9 image that is hosted in the Azure marketplace. The Azure Linux Agent is already
 pre-installed on the Azure Marketplace images and is typically available from the distribution's package repository.
-Azure requires that the publishers of the endorsed Linux distributions regularly update their images in the Azure
-Marketplace with the latest patches and security fixes, at a quarterly or faster cadence. Updated images in the Azure
-Marketplace are available automatically to customers as new versions of an image SKU.
 
 Edit the values in the sections within [muchos.props] as below
 Under the `general` section, edit following values as per your configuration
@@ -168,7 +159,7 @@ Under the `azure` section, edit following values as per your configuration:
 * `azure_image_reference` allows you to specify the Azure image SKU in the format as shown below.
   ```bash
   offer|publisher|sku|version|image_id|
-  Ex: CentOS|OpenLogic|7_9|latest||
+  Ex: almalinux-x86_64|almalinux|9-gen2|latest||
   ```
   For more information on using other images, refer to [Azure images](docs/azure-image-reference.md).
 * `azure_proxy_image_reference` allows you to specify the Azure image SKU that will be used for the optional proxy machine.
@@ -373,7 +364,6 @@ Muchos is powered by the following projects:
  * [azure-cli] - The Azure CLI is a command-line tool for managing Azure resources.
  * [ansible-azure] - Ansible includes a suite of modules for interacting with Azure Resource Manager.
 
-[centos7]: https://aws.amazon.com/marketplace/pp/B00O7WM7QW
 [aws-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
 [awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
 [awscli-config]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration

ansible/roles/azure/files/cloud-init-alma9.yml

@@ -0,0 +1,24 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Install packages specific to AlmaLinux 9
+packages:
+ - rsync

ansible/roles/common/tasks/main.yml

@@ -21,7 +21,7 @@
   retries: 10
   delay: 15
   until: epelresult is not failed
-  when: ansible_facts['distribution'] == "CentOS"
+  when: ansible_facts['distribution'] in ["CentOS", "AlmaLinux", "Rocky"]
 - name: "install packages"
   yum:
     name:
@@ -50,19 +50,19 @@
   delay: 15
   until: yumresult_centos7 is not failed
   when: (ansible_facts['distribution'] == "CentOS") and (ansible_facts['distribution_major_version'] == "7")
-- name: "Install packages specific to CentOS 8"
+- name: "Install packages specific to AlmaLinux 8 / 9, Rocky Linux 8 / 9"
   yum:
     name:
       - python3-policycoreutils
       - collectd-disk
       - collectd-write_http
       - make
     state: present
-  register: yumresult_centos8
+  register: yumresult_os89
   retries: 10
   delay: 15
-  until: yumresult_centos8 is not failed
-  when: (ansible_facts['distribution'] == "CentOS") and (ansible_facts['distribution_major_version'] == "8")
+  until: yumresult_os89 is not failed
+  when: (ansible_facts['distribution'] in ["AlmaLinux", "Rocky"]) and (ansible_facts['distribution_major_version'] in ["8", "9"])
 - name: "Install packages specific to Fedora"
   yum:
     name:
@@ -76,6 +76,13 @@
   delay: 15
   until: yumresult_fedora is not failed
   when: ansible_facts['distribution'] == "Fedora"
+# Disable firewalld on AlmaLinux 9
+- name: "Stop and disable firewalld on AlmaLinux 9"
+  service:
+    name: "firewalld"
+    state: stopped
+    enabled: false
+  when: (ansible_facts['distribution'] == "AlmaLinux") and (ansible_facts['distribution_major_version'] == "9")
 - name: "get exact jdk folder path"
   find:
     file_type: directory

ansible/scripts/install_ansible.sh

@@ -16,24 +16,30 @@
 # limitations under the License.
 #
 
-
 bin=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
 base_dir=$( cd "$( dirname "$bin" )" && pwd )
 
 set -e
 
 # enable yum epel repo
 os_id=$(grep '^ID=' /etc/os-release | cut -d'=' -f2 | tr -d '"')
-if [[ $os_id = "centos" ]]; then
+if [[ $os_id = "centos" || $os_id = "almalinux" || $os_id = "rocky" ]]; then
   is_installed_epel_release="rpm -q --quiet epel-release"
   install_epel_release="sudo yum install -q -y epel-release"
   for i in {1..10}; do ($is_installed_epel_release || $install_epel_release) && break || sleep 15; done
 fi
 
 # install ansible
-is_installed_ansible="rpm -q --quiet ansible"
-install_ansible="sudo yum install -q -y ansible"
-for i in {1..10}; do ($is_installed_ansible || $install_ansible) && break || sleep 15; done
+if [[ $os_id = "centos" ]]; then
+  is_installed_ansible="rpm -q --quiet ansible"
+  install_ansible="sudo yum install -q -y ansible"
+  for i in {1..10}; do ($is_installed_ansible || $install_ansible) && break || sleep 15; done
+fi
+
+if [[ $os_id = "fedora" || $os_id = "almalinux" || $os_id = "rocky" ]]; then
+  python3 -m ensurepip
+  pip3 install --quiet --user ansible
+fi
 
 # setup user-specific ansible configuration
 if [[ ! -h ~/.ansible.cfg ]]; then
@@ -44,6 +50,8 @@ fi
 
 # setup ansible hosts
 if [[ ! -h /etc/ansible/hosts ]]; then
+  # If Ansible was installed using Python, the /etc/ansible folder needs to be created manually
+  sudo mkdir -p /etc/ansible
   cd /etc/ansible
   sudo rm -f hosts
   sudo ln -s $base_dir/conf/hosts hosts

conf/muchos.props.example

@@ -129,19 +129,18 @@ use_multiple_vmss = False
 vmss_priority = None
 # Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|image_id|
 # Please refer 'Launching an Azure cluster' section of the README before making changes
-azure_image_reference = CentOS|OpenLogic|7_9|latest||
+azure_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||
 # Image payment plan information - values required only if the image requires payment plan info.
 # The format of this configuration line is plan_name|product|publisher|
 azure_image_plan = |||
 # Cloud init file to use when creating VMs with the above image reference.
-# Currently, a CentOS 7.9 specific cloud-init file is used. In the future, different files can be used
-# as needed for different images like Alma / Rocky / Fedora.
-azure_image_cloud_init_file = cloud-init-centos79.yml
+# Currently, an Alma Linux 9 specific cloud-init file is used. Different files can be used if needed.
+azure_image_cloud_init_file = cloud-init-alma9.yml
 # Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|image_id|
 # This is the image that will be used for the proxy machine (if specified by azure_proxy_host). If
 # this is not set, then the value of azure_image_reference will be used on the proxy.
 # Please refer 'Launching an Azure cluster' section of the README before making changes
-#azure_proxy_image_reference = CentOS|OpenLogic|7_9|latest||
+#azure_proxy_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||
 # Proxy image payment plan information - required only if the proxy image requires payment plan info.
 # If this is not set, the value of azure_image_plan will be used on the proxy.
 # The format of this configuration line is plan_name|product|publisher|

docs/azure-image-reference.md

@@ -7,11 +7,22 @@ The sections below describe the various image related configurations in `muchos.
 ## azure_image_reference
 `azure_image_reference` is a pipe-delimited string in the format `offer|publisher|sku|version|image_id|`. The trailing pipe character is intentional.
 
-* For Azure Marketplace images, the values for the fields `offer|publisher|sku|version` can be obtained from the Azure portal, or by using the Azure CLI commands as shown later. For example, the CentOS 7.9 image currently used as the default in Muchos is specified as:
+* For Azure Marketplace images, the values for the fields `offer|publisher|sku|version` can be obtained from the Azure portal, or by using the Azure CLI commands as shown later. For example, the AlmaLinux 9 image which is used as the default for Azure clusters in Muchos is specified as:
 
-    `CentOS|OpenLogic|7_9|latest||`
+    `azure_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||`
 
-  In the above case, since it's a marketplace image, the last value of `image_id` is empty.
+  In the above case, since it's a marketplace image, the last value of `image_id` is empty. In addition, an Alma Linux 9 specific cloud-init file is specified. This cloud-init file installs `rsync` which seems to not be included in Alma Linux 9 images:
+
+    `azure_image_cloud_init_file = cloud-init-alma9.yml`
+
+  The Rocky Linux images in Azure require plan information to be supplied. Currently, the Rocky Linux 8 image has been verified to work correctly. For using Rocky Linux 8 instead of Alma Linux, here is what you can use in `muchos.props`. The image plan information is mandatory for this image:
+
+    * `azure_image_reference = rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|free|latest||`
+    * `azure_image_plan = free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+
+  In addition, you might need to manually view and accept the terms of use for the Rocky Linux image, before being able to use it within Muchos. See the `azure_image_plan` section below for details.
+
+  Note: The Rocky Linux 9 image on Azure has an issue with data disks not being correctly mapped to paths under /dev/disk/azure/scsi1. This is either an image issue, or an issue with the Microsoft Azure Linux Guest Agent. Till the issue is diagnosed and fixed, we do not recommend using the Azure Rocky Linux 9 image at this point in time.
 
 * It is also possible to use a [custom Azure image](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/imaging) with Muchos. For using an image from an Azure Compute Gallery ("Shared Image Gallery"), the full resource ID of the image should be specified for `image_id` and the other fields should not be specified. For example:
 
@@ -28,11 +39,13 @@ The sections below describe the various image related configurations in `muchos.
 ## azure_image_plan
 `azure_image_plan` is only needed when working with images which require payment plan information to be supplied when a VM or VMSS is being created using that image. The format of this configuration is `plan_name|product|publisher|`. Plan information for the images published by a given publisher can easily be queried by using the Azure CLI. For example, to query the plan information for a Rocky Linux image in Azure:
 
-    `az vm image show --urn "erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest" --query "plan"`
+   `az vm image show --urn "erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest" --query "plan"`
 
 Then using that information, `azure_image_plan` can be configured as below in muchos.props:
 
-    `azure_image_plan = free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+   `azure_image_plan = free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+
+More information about purchase plans, and accepting terms, etc. is available [here](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/cli-ps-findimage#check-the-purchase-plan-information).
 
 ## azure_image_cloud_init_file
 `azure_image_cloud_init_file` is used to optionally specify the name of a cloud-init file to be used. Only specify the filename here, and make sure that the file exists under the `ansible/roles/azure/files` directory in this repo.
@@ -48,21 +61,20 @@ By default, these configurations are commented out in the muchos.props file, and
 # Other useful commands
 You can run the below Azure CLI command to determine the list of SKU's available for a given product and publisher in a given region:
 
-```bash
-az vm image list-skus -l <region> -f AlmaLinux -p AlmaLinux -o table
-```
+  `az vm image list-skus -l <region> -f AlmaLinux -p AlmaLinux -o table`
+
 For illustration, provided a sample output that displays the sku list from `westus2` region. The sku name `8-gen1, 8-gen2` refer to [Azure VMs generations](https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2). 
 
-```bash
-$ az vm image list-skus -l westus2 -f AlmaLinux -p AlmaLinux -o table
-Location    Name
-----------  --------
-westus2     8-gen1
-westus2     8-gen2
-westus2     8_4
-westus2     8_4-gen2
-westus2     8_5
-westus2     8_5-gen2
-westus2     9-gen1
-westus2     9-gen2
-```
+  ```bash
+  $ az vm image list-skus -l westus2 -f AlmaLinux -p AlmaLinux -o table
+  Location    Name
+  ----------  --------
+  westus2     8-gen1
+  westus2     8-gen2
+  westus2     8_4
+  westus2     8_4-gen2
+  westus2     8_5
+  westus2     8_5-gen2
+  westus2     9-gen1
+  westus2     9-gen2
+  ```

docs/azure-ubuntu-1804.md

@@ -1,9 +1,9 @@
 Tips for running Muchos for Azure under Ubuntu 18.04
 ----------------------------------------------------
 
-Muchos sets up a Centos cluster by default, but it does not have to run in Centos.  If you
-wish to run Muchos under Ubuntu 18.04 and have it set up an Azure cluster, then
-the following steps can get you on your way.
+For Azure, Muchos sets up an AlmaLinux 9 based cluster by default, but Muchos itself can
+be run on other flavors of Linux. If you wish to run Muchos under Ubuntu 18.04 and have it
+set up an Azure cluster, then the following steps can get you on your way.
 
 ```bash
 # Install Azure CLI.  See the Azure documentation.
@@ -17,10 +17,10 @@ sudo apt install python3-pip
 sudo pip3 install -r lib/requirements.txt
 # Current versions of Ansible separate out the Azure-specific modules into a
 # separate "collection". To install that, and associated pre-requisites, a helper
-# script has been provided. Please be sure to execute this script:
+# script has been provided. Please be sure to execute these scripts:
 ./scripts/install-ansible-for-azure
+./scripts/install-ansible-collections
 ```
 
-A virtual python environment is not needed in Ubuntu.  The instructions that
-mention that are targeted for Centos 7.  The version of Python 3 and pip3 that
-ship with Ubuntu 18.04 suffice.
+A virtual python environment is not needed in Ubuntu, but can be useful considering
+the variety of dependencies installed locally for Azure.

lib/muchos/config/azure.py

@@ -202,7 +202,7 @@ def azure_fileshare_password(self):
         return self.get("azure", "azure_fileshare_password")
 
     @ansible_host_var
-    @default("CentOS|OpenLogic|7_9|latest|")
+    @default("almalinux-x86_64|almalinux|9-gen2|latest||")
     def azure_image_reference(self):
         return self.get("azure", "azure_image_reference")
 

lib/muchos/util.py

@@ -32,15 +32,14 @@ def __init__(self, arch, ephemeral=1, has_nvme=False):
         self.has_nvme = has_nvme
 
 
-AMI_HELP_MSG = """PLEASE NOTE - If you have accepted the software terms for CentOS 7 and still get an error,
-this could be due to CentOS releasing new images of CentOS 7.  When this occurs, the old images
-are no longer available to new users.  If you think this is the case, go to the CentOS 7 product
-page on AWS Marketplace at the URL below to find the latest AMI:
-
-    https://aws.amazon.com/marketplace/pp/B00O7WM7QW
-
-On the product page, find the latest AMI ID for your EC2 region. This should be used to set the 'aws_ami'
-property in your muchos.props.  After setting the 'aws_ami' property, run the launch command again.
+AMI_HELP_MSG = """PLEASE NOTE - If you have accepted the software terms for the selected AMI and still get an error,
+this could be due to the publisher releasing new images of that OS. When this occurs, the old images
+are no longer available to new users. You will then need to visit the publisher's page and find the latest AMI ID
+for your EC2 region. This should be used to set the 'aws_ami' property in your muchos.props.
+After setting the 'aws_ami' property, run the launch command again.
+
+The Fedora Project also makes available AMIs for their Fedora Cloud releases. The list of Fedora Cloud AMIs
+can be viewed at https://fedoraproject.org/cloud/download and clicking on the AWS link."
 """  # noqa
 
 instance_types = {