Reference Architecture 23.07

New Components/Features:
- Support for 5th Gen Intel® Xeon® Scalable processors requires the NDA version of the Intel® QuickAssist Technology (Intel® QAT) drivers
- Support for 12th Gen Intel® Core™ processors
- Support for Intel Atom® x6000e series processors
- Support for 5G O-RAN security with NETCONF server/client authentication (Intel® Software Guard Extensions (Intel® SGX))
- Support Rancher RKE2 Kubernetes distribution for the access edge configuration profile with Ubuntu. (v1.26.2+rke2r1)
- Support for Intel® FPGA SmartNIC WSN6050 Platform for video production
- Enabled Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card and made it available in the host machine, which requires the NDA version of image
- Support for Intel® SGX in VMRA by upgrading QEMU and libvirt
- Support for Key Management Reference Application (KMRA) in Virtual Machine Reference Architecture (VMRA)
- Support for KMRA on 4th Gen Intel® Xeon® Scalable processors on production SKUs
- Implemented Intel® SGX-enabled Istio in VMs
- Support for Cilium eBPF Dataplane on Microsoft Azure Kubernetes Service (AKS)
- Updates to Kubernetes version and tools used to deploy on Microsoft Azure Kubernetes Service (AKS) and Amazon Web Service (AWS) Elastic Kubernetes Service (EKS) in Cloud Reference System Architecture
- Implemented support and option for Intel® QuickAssist Technology (Intel® QAT) in-tree versus out-of-tree drivers and libraries
- Integrated Intel® oneAPI Base Toolkit (Base Kit) 2023.1 and Intel® AI Analytics Toolkit (AI Kit) 2023.1.1
- Integrated FFmpeg with cartwheel (Intel GitHub 2023q1 release)
- Added two Configuration Profiles:
    On-Premises SW Defined Factory Configuration Profile for industrial use cases
    On-Premises VSS Configuration Profile for Video Structuring Server (VSS) use cases
- RHEL 9.2 as base OS
- RHEL 9.2 RT as base OS
- Ubuntu 22.04.2 as base OS

Updates/Changes:
- Version upgraded for the vast majority of Reference System components (See elsewhere in this document for complete BOM and versions)
- Notable updates:
    FlexRAN™ to v23.03
    Kubernetes* to v1.26.3
    CPU Control Plane Plugin for Kubernetes* to v0.1.2
    Telemetry Aware Scheduling to v0.5.0
    GPU Aware Scheduling to v0.5.2
    Intel® Power Manager to v2.2
    Service Mesh Istio to v1.18.1
    Intel® Managed Distribution of Istio* Service Mesh to v1.18.0-intel.0
    Data Plane Development Kit (DPDK) to v23.05
    Open vSwitch with DPDK to 3.11
    Traffic Analytics Development Kit (TADK) to 23.03
    OpenSSL to openssl-3.1.0
    Intel® Data Center GPU Flex Series driver to 20230519 release
    SR-IOV FEC Operator to 2.7
    Intel® Platform Telemetry Insights to 23.07 (with license)
- Kubespray* is provided via ansible-galaxy collection instead of git submodule

Updates/Changes made for the Reference System 23.02.1 minor release:
- Stack Validation:
    Test cases created for Anuket Reference Architecture Kubernetes Component Level Architecture specifications, to be used for Anuket Reference Conformance Kubernetes and Project Sylva Stack validation
    Test cases created for Device Plugins Single Root IO Virtualization (SR-IOV) Data Plane Development Kit and Multus Container Network Interface
    Cloud Native Network Function (CNF) Validation:
    Test case to check CNF allocation of SR-IOV devices like virtual functions of network adapters or accelerators, to be used for Project Sylva CNF validation
- Added workaround for building the Intel® Ethernet Operator and SR-IOV FEC (Forward-Error Correction) Operator
- Resolved issue regarding the user-space CNI by disabling Vector Packet Processing (VPP)
- Removed dependency of Intel® QuickAssist Technology (Intel® QAT) on OpenSSL to allow independent deployment of Crypto libraries

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- 5th Gen Intel® Xeon® Scalable processors (XCC, MCC)
- 4th Gen Intel® Xeon® Scalable processor with Intel® vRAN Boost up to 32 cores
- Intel® FPGA SmartNIC WSN6050 Platform
- 12th Gen Intel® Core™ processors
- Intel Atom® x6000e series processors
- Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card

Removed Support:
- Discontinued supporting Cloud Native Data Plane (CNDP)
- Discontinued supporting RHEL 9.0 as base OS
- Discontinued supporting RHEL 8.6 RT as base OS

Known Limitations/Restrictions:
- Intel® Data Center GPU Flex Series, CPU Control Plane Plugin for Kubernetes, Intel® Media SDK (only Docker runtime) are only supported on Ubuntu OS
- FlexRAN™ container support is limited to v22.07, Ubuntu 22.04 base OS, and only on 3rd Gen Intel® Xeon® Scalable processors
- MinIO is supported only with CRI-O runtime
- Only in-tree Intel® QuickAssist Technology (Intel® QAT) and Intel® Ethernet Network Adapter E810 drivers supported on RHEL 9.2
- Intel® Ethernet Network Adapter E810 in-tree driver does not support VF function on RHEL 9.2, which impacts XRAN mode test in FlexRAN™ application
- Intel® QuickAssist Technology (Intel® QAT) is not supported on Rocky Linux 9.1 on 5th Gen Intel® Xeon® Scalable processors
- Intel® Data Streaming Accelerator (Intel® DSA) may not work on some older (earlier stepping) CPUs on RHEL 9.2
- UserSpace CNI with VPP is not supported
- Rancher only supported for containerd.
- CAdvisor not supported on CRI-O runtime

Co-authored-by: Alek Du <[email protected]>
Co-authored-by: Ali Shah, Syed Faraz <[email protected]>
Co-authored-by: Benedikt, Jan <[email protected]>
Co-authored-by: Fiala, Jiri <[email protected]>
Co-authored-by: Gherghe, Calin <[email protected]>
Co-authored-by: Hu, Hao <[email protected]>
Co-authored-by: Jiang, Renzhi <[email protected]>
Co-authored-by: Kasanic, Denis <[email protected]>
Co-authored-by: Liu, Rachel A <[email protected]>
Co-authored-by: Long, Zhifang <[email protected]>
Co-authored-by: MacGillivray, Mac <[email protected]>
Co-authored-by: Musial, Michal <[email protected]>
Co-authored-by: Park, Seungweon <[email protected]>
Co-authored-by: Pedersen, Michael <[email protected]>
Co-authored-by: Prokes, Jiri <[email protected]>
Co-authored-by: Ren, Shu <[email protected]>
Co-authored-by: Vrana, Roman <[email protected]>
Co-authored-by: Xu, Guoshu <[email protected]>
Co-authored-by: Zenov, Mykyta <[email protected]>

Makefile

@@ -3,7 +3,7 @@ ARCH ?= 'icx'
 NIC ?= 'cvl'
 MIRRORS ?= false
 PLAYBOOKS_DIRS = playbooks playbooks/infra playbooks/intel
-PLAYBOOK_NAMES = access basic full_nfv on_prem regional_dc remote_fp storage build_your_own
+PLAYBOOK_NAMES = access basic full_nfv on_prem on_prem_vss on_prem_sw_defined_factory regional_dc remote_fp build_your_own
 
 # set default target available with simple 'make' command
 .DEFAULT_GOAL := examples

Pipfile

@@ -6,7 +6,7 @@ name = "pypi"
 [packages]
 ansible = "~=5.7.1"
 "ansible-core" = "~=2.12"
-cryptography = "~=39.0"
+cryptography = "~=41.0"
 jinja2 = "~=3.1"
 netaddr = "~=0.7.19"
 pbr = "~=5.4"

README.md

@@ -8,13 +8,7 @@ The software provided here is for reference only and not intended for production
 
 **_NOTE:_** Instruction provided bellow are prepared for deployment done under root user by default. If you want to do deployment under non-root user then read [this](docs/rootless_deployment.md) file first and then continue with following steps under that non-root user.
 
-1. Initialize git submodules to download Kubespray code.
-
-    ```bash
-    git submodule update --init
-    ```
-
-2. Decide which configuration profile you want to use and export environmental variable.
+1. Decide which configuration profile you want to use and export environmental variable.
    > **_NOTE:_** It will be used only to ease execution of the steps listed below.
     - For **Kubernetes Basic Infrastructure** deployment:
 
@@ -46,13 +40,25 @@ The software provided here is for reference only and not intended for production
         export PROFILE=on_prem
         ```
 
+    - For **Kubernetes Infrastructure On Customer Premises for VSS** deployment:
+
+        ```bash
+        export PROFILE=on_prem_vss
+        ```
+
+    - For **Kubernetes Infrastructure On Customer Premises for SW-Defined Factory** deployment:
+
+        ```bash
+        export PROFILE=on_prem_sw_defined_factory
+        ```
+
     - For **Kubernetes Build-Your-Own Infrastructure** deployment:
 
         ```bash
         export PROFILE=build_your_own
         ```
 
-3. Install dependencies using one of the following methods
+2. Install python dependencies using one of the following methods
 
     a) Non-invasive virtual environment using pipenv
 
@@ -79,12 +85,18 @@ The software provided here is for reference only and not intended for production
     pip3 install -r requirements.txt
     ```
 
+3. Install ansible collection dependencies with following command:
+
+    ```bash
+    ansible-galaxy install -r collections/requirements.yml
+    ```
+
 4. Generate example host_vars, group_vars and inventory files for Intel Container Experience Kits profiles.
 
    > **_NOTE:_** It is **highly recommended** to read [this](docs/generate_profiles.md) file before profiles generation.
 
     ```bash
-    make examples ARCH=<skl,clx,**icx**,spr> NIC=<fvl,**cvl**>
+    make examples ARCH=<atom,core,**icx**,spr> NIC=<fvl,**cvl**>
     ```
 
 5. Copy example inventory file to the project root dir.
@@ -145,7 +157,7 @@ The software provided here is for reference only and not intended for production
       Needed details are at least dataplane_interfaces
       For more details see [VM case configuration guide](docs/vm_config_guide.md)
 
-9. **Required:** Apply bug fix patch for Kubespray submodule (for RHEL 8+).
+9. **Mandatory:** Apply patch for Kubespray collection.
 
     ```bash
     ansible-playbook -i inventory.ini playbooks/k8s/patch_kubespray.yml
@@ -154,7 +166,9 @@ The software provided here is for reference only and not intended for production
 10. Execute `ansible-playbook`.
 
     > **_NOTE:_** For Cloud case this step is not used. See the [cloud/](cloud/) directory for more details
-
+    
+    > **_NOTE:_** It is recommended to use "--flush-cache" (e.g. "ansible-playbook -i --flush-cache inventory.ini playbooks/remote_fp.yml") when executing ansible-playbook in order to avoid unknown issues such as skip of tasks/roles, unable to update previous run inventory details, etc.
+              
     ```bash
     ansible-playbook -i inventory.ini playbooks/${PROFILE}.yml
     ```
@@ -178,6 +192,7 @@ Refer to the documentation linked below to see configuration details for selecte
 - [VM multinode setup guide](docs/vm_multinode_setup_guide.md)
 - [VM cluster expansion guide](docs/vm_cluster_expansion_guide.md)
 - [Non-root deployment guide](docs/rootless_deployment.md)
+
 ## Prerequisites and Requirements
 
 - Required packages on the target servers: **Python3**.

action_plugins/cpupin.py

@@ -17,6 +17,7 @@
 
 # Make coding more python3-ish, this is required for contributions to Ansible
 from __future__ import (absolute_import, division, print_function)
+
 __metaclass__ = type
 
 import re
@@ -37,6 +38,7 @@
 
 class ActionModule(ActionBase):
     """cpupin action plugin implementation"""
+
     def __init__(self, task, connection, play_context, loader, templar, shared_loader_obj):
         super().__init__(task, connection, play_context, loader, templar, shared_loader_obj)
         # CPUs allocated for host OS
@@ -165,7 +167,7 @@ def run(self, tmp=None, task_vars=None):
         if not self.pinning and self.alloc_all and int(self.number) != 0:
             msg = "You have to set parameter 'cpu_total:' to '0' when 'alloc_all: true' is used"
 
-        if not self.pinning and self.alloc_all and ( self.cpus or self.numa ):
+        if not self.pinning and self.alloc_all and (self.cpus or self.numa):
             msg = "'cpus' and 'numa' can't be used with 'alloc_all: true'"
 
         if self.pinning and not self.alloc_all and (not self.cpus or not self.numa):
@@ -179,8 +181,8 @@ def run(self, tmp=None, task_vars=None):
 
         if self.pinning and self.alloc_all and (not self.cpus or self.numa):
             msg = ("When using parameters pinning=true and alloc_all=true, 'numa' parameter is None"
-                ", 'cpus' parameter have to be prepared in advance e.g.: via running module with "
-                "pinning=false")
+                   ", 'cpus' parameter have to be prepared in advance e.g.: via running module with "
+                   "pinning=false")
 
         if msg:
             raise AnsibleActionFail(msg)
@@ -475,7 +477,6 @@ def _allocate_all_cpus(self, task_vars):
         self.cpu_list.sort()
         return task_vars
 
-
     def _allocate_cpus(self, task_vars):
         """ Allocate required number of CPUs
 
@@ -485,7 +486,7 @@ def _allocate_cpus(self, task_vars):
 
         # Select random NUMA
         if not self.numa:
-            self.numa = random.choice(self.numa_nodes) # nosec B311 # pseudo random is not used for security purposes
+            self.numa = random.choice(self.numa_nodes)  # nosec B311 # pseudo random is not used for security purposes
 
         if not self.cpus:
             self.cpu_list = self._select_cpus(task_vars['numa_nodes_cpus'], self.number, self.numa)
@@ -622,7 +623,7 @@ def _pin_cpus(self):
                                   f"{to_native(emupin_result['stderr'].strip())}'")
 
         if not self.alloc_all:
-        # Update VM NUMA alignment
+            # Update VM NUMA alignment
             cmd_numa = f"virsh numatune {self.name} --nodeset {self.numa} --live --config"
             numa_result = self._low_level_execute_command(cmd_numa)
             if numa_result['rc'] != 0:

ansible.cfg

@@ -16,6 +16,8 @@ fact_caching_timeout = 7200
 
 action_plugins = ./action_plugins:~/.ansible/plugins/action:/usr/share/ansible/plugins/action
 library = ./library
+roles_path = roles
+collections_path = ./collections
 
 log_path = ./.ansible_last_run.log
 display_args_to_stdout = False

cloud/README.md

@@ -14,13 +14,11 @@ Cloud RA allows for deploying Intel Container Experience Kits on managed Kuberne
 
 - Python 3.8+
 
-- Azure CLI 2.46.0+ ([Install Guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt))
+- Azure CLI 2.50.0+ ([Install Guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt))
 
-- Azure CLI "aks-preview" extesnion ([Install Guide](https://learn.microsoft.com/en-us/cli/azure/azure-cli-extensions-overview)). Needed for enabling SGX.
+- AWS CLI 2.12.7+ ([Install Guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html))
 
-- AWS CLI 2.11.0+ ([Install Guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html))
-
-- Terraform 1.3.9+
+- Terraform 1.5.2+
 
 - Docker 20.10.17+
 
@@ -83,8 +81,8 @@ azureConfig:
   sg_whitelist_cidr_blocks: []
   enable_proximity_placement: true
   aks:
-    kubernetes_version: "1.25"
-    cni: "kubenet" # Possible values are: kubenet, cilium, cilium-ebpf
+    kubernetes_version: "1.26"
+    cni: "kubenet" # Possible values are: kubenet, cilium
     enable_sgx: false # Requires DCsv series instances in one of node pools
     default_node_pool:
       subnet_name: "subnet_a"
@@ -122,7 +120,7 @@ awsConfig:
   sg_whitelist_cidr_blocks: []
   ecr_repositories: []
   eks:
-    kubernetes_version: "1.24"
+    kubernetes_version: "1.26"
     subnets: ["subnet_a", "subnet_b"]
     custom_ami: "ubuntu" # Comment out this line to use Amazon Linux 2 OS
     node_groups:

cloud/cwdf.py

@@ -1,8 +1,10 @@
-import click
-from cwdf import compose_terraform, compose_cloudcli
 import os
+
+import click
 import paramiko
 
+from cwdf_util import compose_terraform, compose_cloudcli
+
 
 @click.group()
 def cli():
@@ -45,6 +47,7 @@ def generate_terraform(cwdf_config, ssh_public_key, generate_keys, job_id, creat
     tf_manifest = compose_terraform(cwdf_config, job_id, ssh_public_key, create_ansible_host, create_container_registry)
     click.echo(tf_manifest)
 
+
 @click.command()
 @click.option('--deployment_dir', help='Path to deployment directory', required=True)
 @click.option('--cwdf_config', help='Path to CWDF yaml config file', required=True)