shard: total number of Sharding on the node. Sharding allows sharing the same GPU on multiple jobs. The total number of shards is evenly distributed across all GPUs on the node.For some cloud providers, it possible to define additional attributes. The following sections present the available attributes per provider.
diff --git a/matrix/index.html b/matrix/index.html index cbca6677..2ea579f4 100644 --- a/matrix/index.html +++ b/matrix/index.html @@ -16,7 +16,7 @@ - + diff --git a/search/search_index.json b/search/search_index.json index bb9bcd61..156858a4 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Magic Castle Documentation","text":""},{"location":"#1-setup","title":"1. Setup","text":"To use Magic Castle you will need:
To install Terraform, follow the tutorial or go directly on Terraform download page.
You can verify Terraform was properly installed by looking at the version in a terminal:
terraform version\nAWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, environment variables, representing your AWS Access Key and AWS Secret Key: export AWS_ACCESS_KEY_ID=\"an-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"a-secret-key\"\nReference: AWS Provider - Environment Variables
"},{"location":"#122-google-cloud","title":"1.2.2 Google Cloud","text":"gcloud auth application-default loginaz loginReference : Azure Provider: Authenticating using the Azure CLI
"},{"location":"#124-openstack-ovh","title":"1.2.4 OpenStack / OVH","text":"Download your OpenStack Open RC file. It is project-specific and contains the credentials used by Terraform to communicate with OpenStack API. To download, using OpenStack web page go to: Project \u2192 API Access, then click on Download OpenStack RC File then right-click on OpenStack RC File (Identity API v3), Save Link as..., and save the file.
In a terminal located in the same folder as your OpenStack RC file, source the OpenStack RC file:
source *-openrc.sh\nOnce you are authenticated with your cloud provider, you should be able to communicate with its API. This section lists for each provider some instructions to test this.
"},{"location":"#131-aws","title":"1.3.1 AWS","text":"test_aws.tf with the following content: provider \"aws\" {\n region = \"us-east-1\"\n}\n\ndata \"aws_ec2_instance_type\" \"example\" {\n instance_type = \"t2.micro\"\n}\nterraform init\nterraform plan\nNo changes. Your infrastructure matches the configuration.\nError: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.\nIn a terminal, enter:
gcloud projects list\nPROJECT_ID NAME PROJECT_NUMBER\nTake note of the project_id of the Google Cloud project you want to use, you will need it later.
In a terminal, enter:
az account show\n{\n \"environmentName\": \"AzureCloud\",\n \"homeTenantId\": \"98467e3b-33c2-4a34-928b-ed254db26890\",\n \"id\": \"4dda857e-1d61-457f-b0f0-e8c784d1fb20\",\n \"isDefault\": true,\n \"managedByTenants\": [],\n \"name\": \"Pay-As-You-Go\",\n \"state\": \"Enabled\",\n \"tenantId\": \"495fc59f-96d9-4c3f-9c78-7a7b5f33d962\",\n \"user\": {\n \"name\": \"user@example.com\",\n \"type\": \"user\"\n }\n}\ntest_os.tf with the following content: terraform {\n required_providers {\n openstack = {\n source = \"terraform-provider-openstack/openstack\"\n }\n }\n}\ndata \"openstack_identity_auth_scope_v3\" \"scope\" {\n name = \"my_scope\"\n}\nterraform init\nterraform plan\nNo changes. Your infrastructure matches the configuration.\nError: Error creating OpenStack identity client:\nThe default quotas set by Amazon are sufficient to build the Magic Castle AWS examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Amazon EC2 service quotas.
"},{"location":"#142-google-cloud","title":"1.4.2 Google Cloud","text":"The default quotas set by Google Cloud are sufficient to build the Magic Castle GCP examples. To increase the limits, or request access to special resources like GPUs, refer to Google Compute Engine Resource quotas.
"},{"location":"#143-microsoft-azure","title":"1.4.3 Microsoft Azure","text":"The default quotas set by Microsoft Azure are sufficient to build the Magic Castle Azure examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Azure subscription and service limits, quotas, and constraints.
"},{"location":"#144-openstack","title":"1.4.4 OpenStack","text":"Minimum project requirements:
Note 1: Magic Castle supposes the OpenStack project comes with a network, a subnet and a router already initialized. If any of these components is missing, you will need to create them manually before launching terraform.
The default quotas set by OVH are sufficient to build the Magic Castle OVH examples. To increase the limits, or request access to special resources like GPUs, refer to OVHcloud - Increasing Public Cloud quotas.
"},{"location":"#2-cloud-cluster-architecture-overview","title":"2. Cloud Cluster Architecture Overview","text":""},{"location":"#3-initialization","title":"3. Initialization","text":""},{"location":"#31-main-file","title":"3.1 Main File","text":"tar xvf magic_castle*.tar.gzmv magic_castle* hulkcd hulkThe file main.tf contains Terraform modules and outputs. Modules are files that define a set of resources that will be configured based on the inputs provided in the module block. Outputs are used to tell Terraform which variables of our module we would like to be shown on the screen once the resources have been instantiated.
This file will be our main canvas to design our new clusters. As long as the module block parameters suffice to our need, we will be able to limit our configuration to this sole file. Further customization will be addressed during the second part of the workshop.
"},{"location":"#32-terraform","title":"3.2 Terraform","text":"Terraform fetches the plugins required to interact with the cloud provider defined by our main.tf once when we initialize. To initialize, enter the following command:
terraform init\nThe initialization is specific to the folder where you are currently located. The initialization process looks at all .tf files and fetches the plugins required to build the resources defined in these files. If you replace some or all .tf files inside a folder that has already been initialized, just call the command again to make sure you have all plugins.
The initialization process creates a .terraform folder at the root of your current folder. You do not need to look at its content for now.
Once Terraform folder has been initialized, it is possible to fetch the newest version of the modules used by calling:
terraform init -upgrade\nIn the main.tf file, there is a module named after your cloud provider, i.e.: module \"openstack\". This module corresponds to the high-level infrastructure of your cluster.
The following sections describes each variable that can be used to customize the deployed infrastructure and its configuration. Optional variables can be absent from the example module. The order of the variables does not matter, but the following sections are ordered as the variables appear in the examples.
"},{"location":"#41-source","title":"4.1 source","text":"The first line of the module block indicates to Terraform where it can find the files that define the resources that will compose your cluster. In the releases, this variable is a relative path to the cloud provider folder (i.e.: ./aws).
Requirement: Must be a path to a local folder containing the Magic Castle Terraform files for the cloud provider of your choice. It can also be a git repository. Refer to Terraform documentation on module source for more information.
Post build modification effect: terraform init will have to be called again and the next terraform apply might propose changes if the infrastructure describe by the new module is different.
Magic Castle configuration management is handled by Puppet. The Puppet configuration files are stored in a git repository. This is typically ComputeCanada/puppet-magic_castle repository on GitHub.
Leave this variable to its current value to deploy a vanilla Magic Castle cluster.
If you wish to customize the instances' role assignment, add services, or develop new features for Magic Castle, fork the ComputeCanada/puppet-magic_castle and point this variable to your fork's URL. For more information on Magic Castle puppet configuration customization, refer to MC developer documentation.
Requirement: Must be a valid HTTPS URL to a git repository describing a Puppet environment compatible with Magic Castle. If the repo is private, generate an access token with a permission to read the repo content, and provide the token in the config_git_url like this:
config_git_url = \"https://oauth2:${oauth-key-goes-here}@domain.com/username/repo.git\"\nPost build modification effect: no effect. To change the Puppet configuration source, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#43-config_version","title":"4.3 config_version","text":"Since Magic Cluster configuration is managed with git, it is possible to specify which version of the configuration you wish to use. Typically, it will match the version number of the release you have downloaded (i.e: 9.3).
Requirement: Must refer to a git commit, tag or branch existing in the git repository pointed by config_git_url.
Post build modification effect: none. To change the Puppet configuration version, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#44-cluster_name","title":"4.4 cluster_name","text":"Defines the ClusterName variable in slurm.conf and the name of the cluster in the Slurm accounting database (see slurm.conf documentation).
Requirement: Must be lowercase alphanumeric characters and start with a letter. It can include dashes. cluster_name must be 40 characters or less.
Post build modification effect: destroy and re-create all instances at next terraform apply.
Defines
resolv.conf search domain as int.{cluster_name}.{domain}Optional modules following the current module in the example main.tf can be used to register DNS records in relation to your cluster if the DNS zone of this domain is administered by one of the supported providers. Refer to section 6. DNS Configuration for more details.
Requirements:
[a-z]([-a-z0-9]*[a-z0-9]), concatenated with periods.*.domain. IN A x.x.x.x exists for that domain. You can verify no such record exist with dig: dig +short '*.${domain}'\nPost build modification effect: destroy and re-create all instances at next terraform apply.
Defines the name of the image that will be used as the base image for the cluster nodes.
You can use a custom image if you wish, but configuration management should be mainly done through Puppet. Image customization is mostly envisioned as a way to accelerate the configuration process by applying the security patches and OS updates in advance.
To specify a different image for an instance type, use the image instance attribute
Requirements: the operating system on the image must be from the RedHat family. This includes CentOS (8, 9), Rocky Linux (8, 9), and AlmaLinux (8, 9).
Post build modification effect: none. If this variable is modified, existing instances will ignore the change and future instances will use the new value.
"},{"location":"#461-aws","title":"4.6.1 AWS","text":"The image field needs to correspond to the Amazon Machine Image (AMI) ID. AMI IDs are specific to regions and architectures. Make sure to use the right ID for the region and CPU architecture you are using (i.e: x86_64).
To find out which AMI ID you need to use, refer to - AlmaLinux OS Amazon Web Services AMIs - CentOS list of official images available on the AWS Marketplace - Rocky Linux
Note: Before you can use the AMI, you will need to accept the usage terms and subscribe to the image on AWS Marketplace. On your first deployment, you will be presented an error similar to this one:
\u2502 Error: Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=cvugziknvmxgqna9noibqnnsy\n\u2502 status code: 401, request id: 1f04a85a-f16a-41c6-82b5-342dc3dd6a3d\n\u2502\n\u2502 on aws/infrastructure.tf line 67, in resource \"aws_instance\" \"instances\":\n\u2502 67: resource \"aws_instance\" \"instances\" {\nClick to Subscribe yellow button."},{"location":"#462-microsoft-azure","title":"4.6.2 Microsoft Azure","text":"The image field for Azure can either be a string or a map.
A string image specification will correspond to the image id. Image ids can be retrieved using the following command-line:
az image builder list\nA map image specification needs to contain the following fields publisher, offer sku, and optionally version. The map is used to specify images found in Azure Marketplace. Here is an example:
{\n publisher = \"OpenLogic\",\n offer = \"CentOS-CI\",\n sku = \"7-CI\"\n}\nThe image name can be a regular expression. If more than one image is returned by the query to OpenStack, the most recent is selected.
"},{"location":"#47-instances","title":"4.7 instances","text":"The instances variable is a map that defines the virtual machines that will form the cluster. The map' keys define the hostnames and the values are the attributes of the virtual machines.
Each instance is identified by a unique hostname. An instance's hostname is written as the key followed by its index (1-based). The following map:
instances = {\n mgmt = { type = \"p2-4gb\", tags = [...] },\n login = { type = \"p2-4gb\", count = 1, tags = [...] },\n node = { type = \"c2-15gb-31\", count = 2, tags = [...] },\n gpu-node = { type = \"gpu2.large\", count = 3, tags = [...] },\n}\nmgmt1\nlogin1\nnode1\nnode2\ngpu-node1\ngpu-node2\ngpu-node3\nHostnames must follow a set of rules, from hostname man page:
Valid characters for hostnames are ASCII letters from a to z, the digits from 0 to 9, and the hyphen (-). A hostname may not start with a hyphen.
Two attributes are expected to be defined for each instance: 1. type: name for varying combinations of CPU, memory, GPU, etc. (i.e: t2.medium); 2. tags: list of labels that defines the role of the instance.
Tags are used in the Terraform code to identify if devices (volume, network) need to be attached to an instance, while in Puppet code tags are used to identify roles of the instances.
Terraform tags:
login: identify instances accessible with SSH from Internet and pointed by the domain name A recordspool: identify instances created only when their hostname appears in the var.pool list.proxy: identify instances accessible with HTTP/HTTPS and pointed by the vhost A recordspublic: identify instances that need to have a public ip address reachable from Internetpuppet: identify instances configured as Puppet serversspot: identify instances that are to be spawned as spot/preemptible instances. This tag is supported in AWS, Azure and GCP. It is ignored by OpenStack and OVH.efa: attach an Elastic Fabric Adapter network interface to the instance. This tag is supported in AWS.Puppet tags expected by the puppet-magic_castle environment.
login: identify a login instance (minimum: 2 CPUs, 2GB RAM)mgmt: identify a management instance i.e: FreeIPA server, Slurm controller, Slurm DB (minimum: 2 CPUs, 6GB RAM)nfs: identify the instance that acts as an NFS server.node: identify a compute node instance (minimum: 1 CPUs, 2GB RAM)pool: when combined with node, it identifies compute nodes that Slurm can resume/suspend to meet workload demand.proxy: identify the instance that executes the Caddy reverse proxy and JupyterHub.In the Magic Castle Puppet environment, an instance cannot be tagged as mgmt and proxy.
You are free to define your own additional tags.
"},{"location":"#472-optional-attributes","title":"4.7.2 Optional attributes","text":"Optional attributes can be defined:
count: number of virtual machines with this combination of hostname prefix, type and tags to create (default: 1).image: specification of the image to use for this instance type. (default: global image value). Refer to section 10.12 - Create a compute node image to learn how this attribute can be leveraged to accelerate compute node configuration.disk_type: type of the instance's root disk (default: see the next table).
disk_type disk_size (GiB) Azure Premium_LRS 30 AWS gp2 10 GCP pd-ssd 20 OpenStack null 10 OVH null 10 disk_size: size in gibibytes (GiB) of the instance's root disk containing the operating system and service software (default: see the previous table).
mig: map of NVIDIA Multi-Instance GPU (MIG) short profile names and count used to partition the instances' GPU, example for an A100: mig = { \"1g.5gb\" = 2, \"2g.10gb\" = 1, \"3g.20gb\" = 1 }\nFor some cloud providers, it possible to define additional attributes. The following sections present the available attributes per provider.
"},{"location":"#aws","title":"AWS","text":"For instances with the spot tags, these attributes can also be set:
wait_for_fulfillment (default: true)spot_type (default: permanent)instance_interruption_behavior (default: stop)spot_price (default: not set)block_duration_minutes (default: not set) [note 1] For more information on these attributes, refer to aws_spot_instance_request argument referenceNote 1: block_duration_minutes is not available to new AWS accounts or accounts without billing history - AWS EC2 Spot Instance requests. When not available, its usage can trigger quota errors like this:
Error requesting spot instances: MaxSpotInstanceCountExceeded: Max spot instance count exceeded\nFor instances with the spot tags, these attributes can also be set:
max_bid_price (default: not set)eviction_policy (default: Deallocate) For more information on these attributes, refer to azurerm_linux_virtual_machine argument referencegpu_type: name of the GPU model to attach to the instance. Refer to Google Cloud documentation for the list of available models per regiongpu_count: number of GPUs of the gpu_type model to attach to the instanceModifying any part of the map after the cluster is built will only affect the type of instances associated with what was modified at the next terraform apply.
The volumes variable is a map that defines the block devices that should be attached to instances that have the corresponding key in their list of tags. To each instance with the tag, unique block devices are attached, no multi-instance attachment is supported.
Each volume in map is defined a key corresponding to its and a map of attributes:
size: size of the block device in GB.type (optional): type of volume to use. Default value per provider:Premium_LRSgp2pd-ssdnullnullVolumes with a tag that have no corresponding instance will not be created.
In the following example:
instances = {\u00a0\n server = { type = \"p4-6gb\", tags = [\"nfs\"] }\n}\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 100 }\n scratch = { size = 100 }\n }\n mds = {\n oss1 = { size = 500 }\n oss2 = { size = 500 }\n }\n}\nThe instance server1 will have three volumes attached to it. The volumes tagged mds are not created since no instances have the corresponding tag.
To define an infrastructure with no volumes, set the volumes variable to an empty map:
volumes = {}\nPost build modification effect: destruction of the corresponding volumes and attachments, and creation of new empty volumes and attachments. If an no instance with a corresponding tag exist following modifications, the volumes will be deleted.
"},{"location":"#49-public_keys","title":"4.9 public_keys","text":"List of SSH public keys that will have access to your cluster sudoer account.
Post build modification effect: trigger scp of hieradata files at next terraform apply. The sudoer account authorized_keys file will be updated by each instance's Puppet agent following the copy of the hieradata files.
default value: 0
Defines how many guest user accounts will be created in FreeIPA. Each user account shares the same randomly generated password. The usernames are defined as userX where X is a number between 1 and the value of nb_users (zero-padded, i.e.: user01 if X < 100, user1 if X < 10).
If an NFS NFS home volume is defined, each user will have a home folder on a shared NFS storage hosted on the NFS server node.
User accounts do not have sudoer privileges. If you wish to use sudo, you will have to login using the sudoer account and the SSH keys listed in public_keys.
If you would like to add a user account after the cluster is built, refer to section 10.3 and 10.4.
Requirement: Must be an integer, minimum value is 0.
Post build modification effect: trigger scp of hieradata files at next terraform apply. If nb_users is increased, new guest accounts will be created during the following Puppet run on mgmt1. If nb_users is decreased, it will have no effect: the guest accounts already created will be left intact.
default value: 4 random words separated by dots
Defines the password for the guest user accounts instead of using a randomly generated one.
Requirement: Minimum length 8 characters.
The password can be provided in a PKCS7 encrypted form. Refer to sub-section 4.15 eyaml_key for instructions on how to encrypt the password.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Password of all guest accounts will be changed to match the new password value.
default value: centos
Defines the username of the account with sudo privileges. The account ssh authorized keys are configured with the SSH public keys with public_keys.
Post build modification effect: none. To change sudoer username, destroy the cluster or redefine the value of profile::base::sudoer_username in hieradata.
default value: empty string
Defines custom variable values that are injected in the Puppet hieradata file. Useful to override common configuration of Puppet classes.
List of useful examples:
profile::base::admin_email: \"me@example.org\"\nprofile::fail2ban::ignore_ip: ['132.203.0.0/16', '8.8.8.8']\njupyterhub::enable_otp_auth: false\nprometheus::alertmanager::route:\n group_by:\n - 'alertname'\n - 'cluster'\n - 'service'\n group_wait: '5s'\n group_interval: '5m'\n repeat_interval: '3h'\n receiver: 'slack'\n\nprometheus::alertmanager::receivers:\n - name: 'slack'\n slack_configs:\n - api_url: 'https://hooks.slack.com/services/ABCDEFG123456'\n channel: \"#channel\"\n send_resolved: true\n username: 'username'\nRefer to the following Puppet modules' documentation to know more about the key-values that can be defined:
The file created from this string can be found on the Puppet server as /etc/puppetlabs/data/user_data.yaml
Requirement: The string needs to respect the YAML syntax.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
default_value: Empty string
Defines the path to a directory containing a hierarchy of YAML data files. The hierarchy is copied on the Puppet server in /etc/puppetlabs/data/user_data.
Hierarchy structure:
<dir>/hostnames/<hostname>/*.yaml<dir>/hostnames/<hostname>.yaml<dir>/prefixes/<prefix>/*.yaml<dir>/prefixes/<prefix>.yaml<dir>/*.yamlFor more information on hieradata, refer to section 4.13 hieradata (optional).
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
default value: empty string
Defines the private RSA key required to decrypt the values encrypted with hiera-eyaml PKCS7. This key will be copied on the Puppet server.
Post build modification effect: trigger scp of private key file at next terraform apply.
If you plan to track the cluster configuration files in git (i.e:main.tf, user_data.yaml), it would be a good idea to encrypt the sensitive property values.
Magic Castle uses hiera-eyaml to provide a per-value encryption of sensitive properties to be used by Puppet.
The private key and its corresponding public key wrapped in a X509 certificate can be generated with openssl:
openssl req -x509 -nodes -newkey rsa:2048 -keyout private_key.pkcs7.pem -out public_key.pkcs7.pem -batch\nor with eyaml:
eyaml createkeys --pkcs7-public-key=public_key.pkcs7.pem --pkcs7-private-key=private_key.pkcs7.pem\nTo encrypt a sensitive property with openssl:
echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | openssl base64 -A | xargs printf \"ENC[PKCS7,%s]\\n\"\nTo encrypt a sensitive property with eyaml:
eyaml encrypt -s 'your-secret' --pkcs7-public-key=public_key.pkcs7.pem -o string\nTo provide the value of this variable via Terraform Cloud, encode the private key content with base64:
openssl base64 -A -in private_key.pkcs7.pem\nDefine a variable in your main.tf:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n}\nThen make sure to decode it before passing it to the cloud provider module:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n eyaml_key = base64decode(var.tfc_eyaml_key)\n ...\n}\ndefault value:
{\n ssh = { \"from_port\" = 22, \"to_port\" = 22, tag = \"login\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n http = { \"from_port\" = 80, \"to_port\" = 80, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n https = { \"from_port\" = 443, \"to_port\" = 443, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n globus = { \"from_port\" = 2811, \"to_port\" = 2811, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"54.237.254.192/29\" },\n myproxy = { \"from_port\" = 7512, \"to_port\" = 7512, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n gridftp = { \"from_port\" = 50000, \"to_port\" = 51000, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" }\n}\nDefines a map of firewall rules that control external traffic to the public nodes. Each rule is defined as a map of key-value pairs and has to be assigned a unique name:
from_port (req.): the lower part of the allowed port range, valid integer value needs to be between 1 and 65535.to_port (req.): the higher part of the allowed port range, valid integer value needs to be between 1 and 65535.tag (req.): instances with this tag will be assigned this firewall rule.ethertype (opt. default: \"IPv4\"): the layer 3 protocol type (\"IPv4\" or \"IPv6\").protocol (opt. default: \"tcp\"): the layer 4 protocol type.cidr (opt. default: \"0.0.0.0/0\"): the remote CIDR, the value needs to be a valid CIDR (i.e. 192.168.0.0/16).If you would like Magic Castle to be able to transfer files and update the state of the cluster in Puppet, make sure there exists at least one effective firewall rule where from_port <= 22 <= to_port and for which the external IP address of the machine that executes Terraform is in the CIDR range (i.e: cidr = \"0.0.0.0/0\" being the most permissive). This corresponds to the ssh rule in the default firewall rule map. This guarantees that Terraform will be able to use SSH to connect to the cluster from anywhere. For more information about this requirement, refer to Magic Castle's bastion tag computation code.
Post build modification effect: modify the cloud provider firewall rules at next terraform apply.
default_value: \"alliance\"
Defines the scientific software environment that users have access when they login. Possible values are:
\"alliance\" / \"computecanada\": Digital Alliance Research Alliance of Canada scientific software environment (previously Compute Canada environment)\"eessi\": European Environment for Scientific Software Installation (EESSI)null / \"\": no scientific software environmentPost build modification effect: trigger scp of hieradata files at next terraform apply.
default_value: []
Defines a list of hostnames with the tag \"pool\" that have to be online. This variable is typically managed by the workload scheduler through Terraform API. For more information, refer to Enable Magic Castle Autoscaling
Post build modification effect: pool tagged hosts with name present in the list will be instantiated, others will stay uninstantiated or will be destroyed if previously instantiated.
default_value = false
If true, the base image packages will not be upgraded during the first boot. By default, all packages are upgraded.
Post build modification effect: No effect on currently built instances. Ones created after the modification will take into consideration the new value of the parameter to determine whether they should upgrade the base image packages or not.
"},{"location":"#421-puppetfile-optional","title":"4.21 puppetfile (optional)","text":"default_value = \"\"
Defines a second Puppetfile used to install complementary modules with r10k.
Post build modification effect: trigger scp of Puppetfile at next terraform apply. Each instance's Puppet agent will be reloaded following the installation of the new modules.
Defines the label of the AWS EC2 region where the cluster will be created (i.e.: us-east-2).
Requirement: Must be in the list of available EC2 regions.
Post build modification effect: rebuild of all resources at next terraform apply.
default value: None
Defines the label of the data center inside the AWS region where the cluster will be created (i.e.: us-east-2a). If left blank, it chosen at random amongst the availability zones of the selected region.
Requirement: Must be in a valid availability zone for the selected region. Refer to AWS documentation to find out how list the availability zones.
"},{"location":"#52-microsoft-azure","title":"5.2 Microsoft Azure","text":""},{"location":"#521-location","title":"5.2.1 location","text":"Defines the label of the Azure location where the cluster will be created (i.e.: eastus).
Requirement: Must be a valid Azure location. To get the list of available location, you can use Azure CLI : az account list-locations -o table.
Post build modification effect: rebuild of all resources at next terraform apply.
default value: None
Defines the name of an already created resource group to use. Terraform will no longer attempt to manage a resource group for Magic Castle if this variable is defined and will instead create all resources within the provided resource group. Define this if you wish to use an already created resource group or you do not have a subscription-level access to create and destroy resource groups.
Post build modification effect: rebuild of all instances at next terraform apply.
default value:
{\n name = null\n product = null\n publisher = null\n}\nPurchase plan information for Azure Marketplace image. Certain images from Azure Marketplace requires a terms acceptance or a fee to be used. When using this kind of image, you must supply the plan details.
For example, to use the official AlmaLinux image, you have to first add it to your account. Then to use it with Magic Castle, you must supply the following plan information:
plan = {\n name = \"8_7\"\n product = \"almalinux\"\n publisher = \"almalinux\"\n}\nDefines the label of the unique identifier associated with the Google Cloud project in which the resources will be created. It needs to corresponds to GCP project ID, which is composed of the project name and a randomly assigned number.
Requirement: Must be a valid Google Cloud project ID.
Post build modification effect: rebuild of all resources at next terraform apply.
Defines the name of the specific geographical location where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud region. Refer to Google Cloud documentation for the list of available regions and their characteristics.
"},{"location":"#533-zone-optional","title":"5.3.3 zone (optional)","text":"default value: None
Defines the name of the zone within the region where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud zone. Refer to Google Cloud documentation for the list of available zones and their characteristics.
"},{"location":"#54-openstack-and-ovh","title":"5.4 OpenStack and OVH","text":""},{"location":"#541-os_floating_ips-optional","title":"5.4.1 os_floating_ips (optional)","text":"default value: {}
Defines a map as an association of instance names (key) to pre-allocated floating ip addresses (value). Example:
os_floating_ips = {\n login1 = \"132.213.13.59\"\n login2 = \"132.213.13.25\"\n }\nThis variable can be useful if you manage your DNS manually and you would like the keep the same domain name for your cluster at each build.
Post build modification effect: change the floating ips assigned to the public instances.
"},{"location":"#542-os_ext_network-optional","title":"5.4.2 os_ext_network (optional)","text":"default value: None
Defines the name of the external network that provides the floating ips. Define this only if your OpenStack cloud provides multiple external networks, otherwise, Terraform can find it automatically.
Post build modification effect: change the floating ips assigned to the public nodes.
"},{"location":"#544-subnet_id-optional","title":"5.4.4 subnet_id (optional)","text":"default value: None
Defines the ID of the internal IPV4 subnet to which the instances are connected. Define this if you have or intend to have more than one subnets defined in your OpenStack project. Otherwise, Terraform can find it automatically. Can be used to force a v4 subnet when both v4 and v6 exist.
Post build modification effect: rebuild of all instances at next terraform apply.
Some functionalities in Magic Castle require the registration of DNS records under the cluster name in the selected domain. This includes web services like JupyterHub, Mokey and FreeIPA web portal.
If your domain DNS records are managed by one of the supported providers, follow the instructions in the corresponding sections to have the cluster's DNS records created and tracked by Magic Castle.
If your DNS provider is not supported, you can manually create the records. Refer to the subsection 6.3 for more details.
"},{"location":"#61-cloudflare","title":"6.1 Cloudflare","text":"dns module for Cloudflare in your main.tf.output \"hostnames\" block.terraform init.CLOUDFLARE_EMAIL and CLOUDFLARE_API_KEY, where CLOUDFLARE_EMAIL is your Cloudflare account email address and CLOUDFLARE_API_KEY is your account Global API Key available in your Cloudflare profile.If you prefer using an API token instead of the global API key, you will need to configure a token with the following four permissions with the Cloudflare API Token interface.
Section Subsection Permission Zone DNS EditInstead of step 5, export only CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_API_TOKEN, and CLOUDFLARE_DNS_API_TOKEN equal to the API token generated previously.
requirement: Install the Google Cloud SDK
gcloud auth application-default logindns module for Google Cloud in your main.tf.output \"hostnames\" block.main.tf's dns module, configure the variables project and zone_name with their respective values as defined by your Google Cloud project.terraform init.If your DNS provider is not currently supported by Magic Castle, you can create the DNS records manually.
Magic Castle provides a module that creates a text file with the DNS records that can then be imported manually in your DNS zone. To use this module, add the following snippet to your main.tf:
module \"dns\" {\n source = \"./dns/txt\"\n name = module.openstack.cluster_name\n domain = module.openstack.domain\n public_instances = module.openstack.public_instances\n}\nFind and replace openstack in the previous snippet by your cloud provider of choice if not OpenStack (i.e: aws, gcp, etc.).
The file will be created after the terraform apply in the same folder as your main.tf and will be named as ${name}.${domain}.txt.
Magic Castle DNS module creates SSHFP records for all instances with a public ip address. These records can be used by SSH clients to verify the SSH host keys of the server. If DNSSEC is enabled for the domain and the SSH client is correctly configured, no host key confirmation will be prompted when connecting to the server.
For more information on how to activate DNSSEC, refer to your DNS provider documentation:
To setup an SSH client to use SSHFP records, add
VerifyHostKeyDNS yes\n~/.ssh/config)."},{"location":"#7-planning","title":"7. Planning","text":"Once your initial cluster configuration is done, you can initiate a planning phase where you will ask Terraform to communicate with your cloud provider and verify that your cluster can be built as it is described by the main.tf configuration file.
Terraform should now be able to communicate with your cloud provider. To test your configuration file, enter the following command
terraform plan\nThis command will validate the syntax of your configuration file and communicate with the provider, but it will not create new resources. It is only a dry-run. If Terraform does not report any error, you can move to the next step. Otherwise, read the errors and fix your configuration file accordingly.
"},{"location":"#8-deployment","title":"8. Deployment","text":"To create the resources defined by your main, enter the following command
terraform apply\nThe command will produce the same output as the plan command, but after the output it will ask for a confirmation to perform the proposed actions. Enter yes.
Terraform will then proceed to create the resources defined by the configuration file. It should take a few minutes. Once the creation process is completed, Terraform will output the guest account usernames and password, the sudoer username and the floating ip of the login node.
Warning: although the instance creation process is finished once Terraform outputs the connection information, you will not be able to connect and use the cluster immediately. The instance creation is only the first phase of the cluster-building process. The configuration: the creation of the user accounts, installation of FreeIPA, Slurm, configuration of JupyterHub, etc.; takes around 15 minutes after the instances are created.
Once it is booted, you can follow an instance configuration process by looking at:
/var/log/cloud-init-output.logjournalctl -u puppetIf unexpected problems occur during configuration, you can provide these logs to the authors of Magic Castle to help you debug.
"},{"location":"#81-deployment-customization","title":"8.1 Deployment Customization","text":"You can modify the main.tf at any point of your cluster's life and apply the modifications while it is running.
Warning: Depending on the variables you modify, Terraform might destroy some or all resources, and create new ones. The effects of modifying each variable are detailed in the subsections of Configuration.
For example, to increase the number of computes nodes by one. Open main.tf, add 1 to node's count , save the document and call
terraform apply\nTerraform will analyze the difference between the current state and the future state, and plan the creation of a single new instance. If you accept the action plan, the instance will be created, provisioned and eventually automatically add to the Slurm cluster configuration.
You could do the opposite and reduce the number of compute nodes to 0.
"},{"location":"#9-destruction","title":"9. Destruction","text":"Once you're done working with your cluster and you would like to recover the resources, in the same folder as main.tf, enter:
terraform destroy -refresh=false\nThe -refresh=false\u00a0flag is to avoid an issue where one or many of the data sources return no results and stall the cluster destruction with a message like the following:
Error: Your query returned no results. Please change your search criteria and try again.\nAs for apply, Terraform will output a plan that you will have to confirm by entering yes.
Warning: once the cluster is destroyed, nothing will be left, even the shared storage will be erased.
"},{"location":"#91-instance-destruction","title":"9.1 Instance Destruction","text":"It is possible to destroy only the instances and keep the rest of the infrastructure like the floating ip, the volumes, the generated SSH host key, etc. To do so, set the count value of the instance type you wish to destroy to 0.
"},{"location":"#92-reset","title":"9.2 Reset","text":"On some occasions, it is desirable to rebuild some of the instances from scratch. Using terraform taint, you can designate resources that will be rebuilt at next application of the plan.
To rebuild the first login node :
terraform taint 'module.openstack.openstack_compute_instance_v2.instances[\"login1\"]'\nterraform apply\nOnce the cluster is online and configured, you can modify its configuration as you see fit. We list here how to do most commonly asked for customizations.
Some customizations are done from the Puppet server instance (puppet). To connect to the puppet server, follow these steps:
ssh -A centos@cluster_ip. Replace centos by the value of sudoer_username if it is different.ssh puppetNote on Google Cloud: In GCP, OS Login lets you use Compute Engine IAM roles to manage SSH access to Linux instances. This feature is incompatible with Magic Castle. Therefore, it is turned off in the instances metadata (enable-oslogin=\"FALSE\"). The only account with sudoer rights that can log in the cluster is configured by the variable sudoer_username (default: centos).
If you plan to modify configuration files manually, you will need to disable Puppet. Otherwise, you might find out that your modifications have disappeared in a 30-minute window.
Puppet executes a run every 30 minutes and at reboot. To disable puppet:
sudo puppet agent --disable \"<MESSAGE>\"\nRefer to section 4.11.
"},{"location":"#103-add-ldap-users","title":"10.3 Add LDAP Users","text":"Users can be added to Magic Castle LDAP database (FreeIPA) with either one of the following methods: hieradata, command-line, and Mokey web-portal. Each method is presented in the following subsections.
New LDAP users are automatically assigned a home folder on NFS.
Magic Castle determines if an LDAP user should be member of a Slurm account based on its POSIX groups. When a user is added to a POSIX group, a daemon try to match the group name to the following regular expression:
(ctb|def|rpp|rrg)-[a-z0-9_-]*\nIf there is a match, the user will be added to a Slurm account with the same name, and will gain access to the corresponding project folder under /project.
Note: The regular expression represents how Compute Canada names its resources allocation. The regular expression can be redefined, see profile::accounts:::project_regex
Using the hieradata variable in the main.tf, it is possible to define LDAP users.
Examples of LDAP user definition with hieradata are provided in puppet-magic_castle documentation.
"},{"location":"#1032-command-line","title":"10.3.2 Command-Line","text":"To add a user account after the cluster is built, log in mgmt1 and call:
kinit admin\nIPA_GUEST_PASSWD=<new_user_passwd> /sbin/ipa_create_user.py <username> [--group <group_name>]\nkdestroy\nIf user sign-up with Mokey is enabled, users can create their own account at
https://mokey.yourcluster.domain.tld/auth/signup\nIt is possible that an administrator is required to enable the account with Mokey. You can access the administrative panel of FreeIPA at :
https://ipa.yourcluster.domain.tld/\nThe FreeIPA administrator credentials can be retrieved from an encrypted file on the Puppet server. Refer to section 10.14 to know how.
"},{"location":"#104-increase-the-number-of-guest-accounts","title":"10.4 Increase the Number of Guest Accounts","text":"To increase the number of guest accounts after creating the cluster with Terraform, simply increase the value of nb_users, then call :
terraform apply\nEach instance's Puppet agent will be reloaded following the copy of the hieradata files, and the new accounts will be created.
"},{"location":"#105-restrict-ssh-access","title":"10.5 Restrict SSH Access","text":"By default, instances tagged login have their port 22 opened to entire world. If you know the range of ip addresses that will connect to your cluster, we strongly recommend that you limit the access to port 22 to this range.
To limit the access to port 22, refer to section 4.14 firewall_rules, and replace the cidr of the ssh rule to match the range of ip addresses that have be the allowed to connect to the cluster. If there are more than one range, create multiple rules with distinct names.
The default Python kernel corresponds to the Python installed in /opt/ipython-kernel. Each compute node has its own copy of the environment. To add packages to this environment, add the following lines to hieradata in main.tf:
jupyterhub::kernel::venv::packages:\n - package_A\n - package_B\n - package_C\nand replace package_* by the packages you need to install. Then call:
terraform apply\nNo longer supported
"},{"location":"#108-recovering-from-puppet-rebuild","title":"10.8 Recovering from puppet rebuild","text":"The modifications of some of the parameters in the main.tf file can trigger the rebuild of the puppet instance. This instance hosts the Puppet Server on which depends the Puppet agent of the other instances. When puppet is rebuilt, the other Puppet agents cease to recognize Puppet Server identity since the Puppet Server identity and certificates have been regenerated.
To fix the Puppet agents, you will need to apply the following commands on each instance other than puppet once puppet is rebuilt:
sudo systemctl stop puppet\nsudo rm -rf /etc/puppetlabs/puppet/ssl/\nsudo systemctl start puppet\nThen, on puppet, you will need to sign the new certificate requests made by the instances. First, you can list the requests:
sudo /opt/puppetlabs/bin/puppetserver ca list\nThen, if every instance is listed, you can sign all requests:
sudo /opt/puppetlabs/bin/puppetserver ca sign --all\nIf you prefer, you can sign individual request by specifying their name:
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname NAME[,NAME]\nLogin nodes run fail2ban, an intrusion prevention software that protects login nodes from brute-force attacks. fail2ban is configured to ban ip addresses that attempted to login 20 times and failed in a window of 60 minutes. The ban time is 24 hours.
In the context of a workshop with SSH novices, the 20-attempt rule might be triggered, resulting in participants banned and puzzled, which is a bad start for a workshop. There are solutions to mitigate this problem.
"},{"location":"#1091-define-a-list-of-ip-addresses-that-can-never-be-banned","title":"10.9.1 Define a list of ip addresses that can never be banned","text":"fail2ban keeps a list of ip addresses that are allowed to fail to login without risking jail time. To add an ip address to that list, add the following lines to the variable hieradata\u00a0in main.tf:
profile::fail2ban::ignoreip:\n - x.x.x.x\n - y.y.y.y\nx.x.x.x and y.y.y.y are ip addresses you want to add to the ignore list. The ip addresses can be written using CIDR notations. The ignore ip list on Magic Castle already includes 127.0.0.1/8 and the cluster subnet CIDR. Once the line is added, call:
terraform apply\nfail2ban rule that banned ip addresses that failed to connect with SSH can be disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::jails: ['ssh-ban-root']\nOnce the line is added, call:
terraform apply\nfail2ban ban ip addresses by adding rules to iptables. To remove these rules, you need to tell fail2ban to unban the ips.
To list the ip addresses that are banned, execute the following command:
sudo fail2ban-client status ssh-route\nTo unban ip addresses, enter the following command followed by the ip addresses you want to unban:
sudo fail2ban-client set ssh-route unbanip\nWhile this is not recommended, fail2ban can be completely disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::service_ensure: 'stopped'\nthen call :
terraform apply\nSELinux can be set in permissive mode to debug new workflows that would be prevented by SELinux from working properly. To do so, add the following line to the variable hieradata\u00a0in main.tf:
selinux::mode: 'permissive'\nWhen scaling the compute node pool, either manually by changing the count or automatically with Slurm autoscale, it can become beneficial to reduce the time spent configuring the machine when it boots for the first time, hence reducing the time requires before it becomes available in Slurm. One way to achieve this is to clone the root disk of a fully configured compute node and use it as the base image of future compute nodes.
This process has three steps:
The following subsection explains how to accomplish each step.
Warning: While it will work in most cases, avoid re-using the compute node image of a previous deployment. The preparation steps cleans most of the deployment specific configuration and secrets, but there is no guarantee that the configuration will be entirely compatible with a different deployment.
"},{"location":"#10121-prepare-the-volume-for-cloning","title":"10.12.1 Prepare the volume for cloning","text":"The environment puppet-magic_castle installs a script that prepares the volume for cloning named prepare4image.sh.
To make sure a node is ready for cloning, open its puppet agent log and validate the catalog was successfully applied at least once:
journalctl -u puppet | grep \"Applied catalog\"\nTo prepare the volume for cloning, execute the following line while connected to the compute node:
sudo /usr/sbin/prepare4image.sh\nBe aware that, since it is preferable for the instance to be powered off when cloning its volume, the script halts the machine once it is completed. Therefore, after executing prepare4image.sh, you will be disconnected from the instance.
The script prepare4image.sh executes the following steps in order:
slurmd)/etc/etc/fstab/var/log/message contentOnce the instance is powered off, access your cloud provider dashboard, find the instance and follow the provider's instructions to create the image.
Note down the name/id of the image you created, it will be needed during the next step.
"},{"location":"#10123-configure-magic-castle-terraform-code-to-use-the-new-image","title":"10.12.3 Configure Magic Castle Terraform code to use the new image","text":"Edit your main.tf and add image = \"name-or-id-of-your-image\" to the dictionary defining the instance. The instance previously powered off will be powered on and future non-instantiated machines will use the image at the next execution of terraform apply.
If the cluster is composed of heterogeneous compute nodes, it is possible to create an image for each type of compute nodes. Here is an example with Google Cloud
instances = {\n mgmt = { type = \"n2-standard-2\", tags = [\"puppet\", \"mgmt\", \"nfs\"], count = 1 }\n login = { type = \"n2-standard-2\", tags = [\"login\", \"public\", \"proxy\"], count = 1 }\n node = {\n type = \"n2-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n image = \"rocky-mc-cpu-node\"\n }\n gpu = {\n type = \"n1-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n gpu_type = \"nvidia-tesla-t4\"\n gpu_count = 1\n image = \"rocky-mc-gpu-node\"\n }\n}\nDuring the cloud-init initialization phase, bootstrap.sh script is executed. This script generates a set of encrypted secret values that are required by the Magic Castle Puppet environment:
profile::consul::acl_api_tokenprofile::freeipa::mokey::passwordprofile::freeipa::server::admin_passwordprofile::freeipa::server::ds_passwordprofile::slurm::accounting::passwordprofile::slurm::base::munge_keyTo read or change the value of one of these keys, use eyaml edit command on the puppet host, like this:
sudo /opt/puppetlabs/puppet/bin/eyaml edit \\\n --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/boot_private_key.pkcs7.pem \\\n --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/boot_public_key.pkcs7.pem \\\n /etc/puppetlabs/code/environments/production/data/bootstrap.yaml\nIt is also possible to redefine the values of these keys by adding the key-value pair to the hieradata configuration file. Refer to section 4.13 hieradata. User defined values take precedence over boot generated values in the Magic Castle Puppet data hierarchy.
"},{"location":"#1014-expand-a-volume","title":"10.14 Expand a volume","text":"Volumes defined in the volumes map can be expanded at will. To enable online extension of a volume, add enable_resize = true to its specs map. You can then increase the size at will. The corresponding volume will be expanded by the cloud provider and the filesystem will be extended by Puppet.
You can modify the Terraform module files in the folder named after your cloud provider (e.g: gcp, openstack, aws, etc.)
Figure 1 (below) illustrates how Magic Castle is structured to provide a unified interface between multiple cloud providers. Each blue block is a file or a module, while white blocks are variables or resources. Arrows indicate variables or resources that contribute to the definition of the linked variables or resources. The figure can be read as a flow-chart from top to bottom. Some resources and variables have been left out of the chart to avoid cluttering it further.
Figure 1. Magic Castle Terraform Project Structure
main.tf: User provides the instances and volumes structure they wants as _map_s. instances = {\n mgmt = { type = \"p4-7.5gb\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"p2-3.75gb\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"p2-3.75gb\", tags = [\"node\"], count = 2 }\n}\n\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 500 }\n scratch = { size = 500 }\n }\n}\ncommon/design:
instances map is expanded to form a new map where each entry represents a single host. instances = {\n mgmt1 = {\n type = \"p2-3.75gb\"\n tags = [\"puppet\", \"mgmt\", \"nfs\"]\n }\n login1 = {\n type = \"p2-3.75gb\"\n tags = [\"login\", \"public\", \"proxy\"]\n }\n node1 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n node2 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n}\nvolumes map is expanded to form a new map where each entry represent a single volume volumes = {\n mgmt1-nfs-home = { size = 100 }\n mgmt1-nfs-project = { size = 100 }\n mgmt1-nfs-scratch = { size = 500 }\n}\nnetwork.tf: the instances map from common/design is used to generate a network interface (nic) for each host, and a public ip address for each host with the public tag.
resource \"provider_network_interface\" \"nic\" {\n for_each = module.design.instances\n ...\n}\ncommon/configuration: for each host in instances, a cloud-init yaml config that includes puppetservers is generated. These configs are outputted to a user_data map where the keys are the hostnames.
user_data = {\n for key, values in var.instances :\n key => templatefile(\"${path.module}/puppet.yaml\", { ... })\n}\ninfrastructure.tf: for each host in instances, an instance resource as defined by the selected cloud provider is generated. Each instance is initially configured by its user_data cloud-init yaml config.
resource \"provider_instance\" \"instances\" {\n for_each = module.design.instance\n user_data = module.instance_config.user_data[each.key]\n ...\n}\ninfrastructure.tf: for each volume in volumes, a block device as defined by the selected cloud provider is generated and attached it to its matching instance using an attachment resource.
resource \"provider_volume\" \"volumes\" {\n for_each = module.design.volumes\n size = each.value.size\n ...\n}\nresource \"provider_attachment\" \"attachments\" {\n for_each = module.design.volumes\n instance_id = provider_instance.instances[each.value.instance].id\n volume_id = provider_volume.volumes[each.key].id\n ...\n}\ninfrastructure.tf: the created instances' information are consolidated in a map named inventory.
inventory = {\n mgmt1 = {\n public_ip = \"\"\n local_ip = \"10.0.0.1\"\n id = \"abc1213-123-1231\"\n tags = [\"mgmt\", \"puppet\", \"nfs\"]\n }\n ...\n}\ncommon/provision: the information from created instances is consolidated and written in a yaml file namedterraform_data.yaml that is uploaded on the Puppet server as part of the hieradata.
resource \"terraform_data\" \"deploy_puppetserver_files\" {\n ...\n provisioner \"file\" {\n content = var.terraform_data\n destination = \"terraform_data.yaml\"\n }\n ...\n}\noutputs.tf: the information of all instances that have a public address are output as a map named public_instances.
In the previous section, we have used generic resource name when writing HCL code that defines these resources. The following table indicate what resource is used for each provider based on its role in the cluster.
Resource AWS Azure Google Cloud Platform OpenStack OVH network aws_vpc azurerm_virtual_network google_compute_network prebuilt openstack_networking_network_v2 subnet aws_subnet azurerm_subnet google_compute_subnetwork prebuilt openstack_networking_subnet_v2 router aws_route not used google_compute_router built-in not used nat aws_internet_gateway not used google_compute_router_nat built-in not used firewall aws_security_group azurerm_network_security_group google_compute_firewall openstack_compute_secgroup_v2 openstack_compute_secgroup_v2 nic aws_network_interface azurerm_network_interface google_compute_address openstack_networking_port_v2 openstack_networking_port_v2 public ip aws_eip azurerm_public_ip google_compute_address openstack_networking_floatingip_v2 openstack_networking_network_v2 instance aws_instance azurerm_linux_virtual_machine google_compute_instance openstack_compute_instance_v2 openstack_compute_instance_v2 volume aws_ebs_volume azurerm_managed_disk google_compute_disk openstack_blockstorage_volume_v3 openstack_blockstorage_volume_v3 attachment aws_volume_attachment azurerm_virtual_machine_data_disk_attachment google_compute_attached_disk openstack_compute_volume_attach_v2 openstack_compute_volume_attach_v2"},{"location":"design/#using-reference-design-to-extend-for-a-new-cloud-provider","title":"Using reference design to extend for a new cloud provider","text":"Magic Castle currently supports five cloud providers, but its design makes it easy to add new providers. This section presents a step-by-step guide to add a new cloud provider support to Magic Castle.
Identify the resources. Using the Resource per provider table, read the cloud provider Terraform documentation, and identify the name for each resource in the table.
Check minimum requirements. Once all resources have been identified, you should be able to determine if the cloud provider can be used to deploy Magic Castle. If you found a name for each resource listed in table, the cloud provider can be supported. If some resources are missing, you will need to read the provider's documentation to determine if the absence of the resource can be compensated for somehow.
Initialize the provider folder. Create a folder named after the provider. In this folder, create two symlinks, one pointing to common/variables.tf and the other to common/outputs.tf. These files define the interface common to all providers supported by Magic Castle.
Define cloud provider specifics variables. Create a file named after your provider provider_name.tf\u00a0and define variables that are required by the provider but not common to all providers, for example the availability zone or the region. In this file, define two local variables named cloud_provider and cloud_region.
Initialize the infrastructure. Create a file named infrastructure.tf. In this file:
provider \"provider_name\" {\n region = var.region\n}\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\nCreate the networking infrastructure. Create a file named network.tf and define the network, subnet, router, nat, firewall, nic and public ip resources using the module.design.instances map.
Create the volumes. In infrastructure.tf, define the volumes resource using module.design.volumes.
Consolidate the instances' information. In infrastructure.tf, define a local variable named inventory that will be a map containing the following keys for each instance: public_ip, local_ip, prefix, tags, and specs (#cpu, #gpus, ram, volumes). For the volumes, you need to provide the paths under which the volumes will be found on the instances to which they are attached. This is typically derived from the volume id. Here is an example:
volumes = contains(keys(module.design.volume_per_instance), x) ? {\n for pv_key, pv_values in var.volumes:\n pv_key => {\n for name, specs in pv_values:\n name => [\"/dev/disk/by-id/*${substr(provider.volumes[\"${x}-${pv_key}-${name}\"].id, 0, 20)}\"]\n } if contains(values.tags, pv_key)\n } : {}\nCreate the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n public_keys = var.public_keys\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\nCreate the instances. In infrastructure.tf, define the instances resource using module.design.instances_to_build for the instance attributes and module.configuration.user_data for the initial configuration.
Attach the volumes. In infrastructure.tf, define the attachments resource using module.design.volumes and refer to the attribute each.value.instance to retrieve the instance's id to which the volume needs to be attached.
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=cloud_provider_instance_resource.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\nInclude the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\nIdentify the resources. For Digital Ocean, Oracle Cloud and Alibaba Cloud, we get the following resource mapping: | Resource | Digital Ocean | Oracle Cloud | Alibaba Cloud | | ----------- | :-------------------- | :-------------------- | :-------------------- | | network | digitalocean_vpc | oci_core_vcn | alicloud_vpc | | subnet | built in vpc | oci_subnet | alicloud_vswitch | | router | n/a | oci_core_route_table | built in vpc | | nat | n/a | oci_core_internet_gateway | alicloud_nat_gateway | | firewall | digitalocean_firewall | oci_core_security_list | alicloud_security_group | | nic | n/a | built in instance | alicloud_network_interface | | public ip | digitalocean_floating_ip | built in instance | alicloud_eip | | instance | digitalocean_droplet | oci_core_instance | alicloud_instance | | volume | digitalocean_volume | oci_core_volume | alicloud_disk | | attachment | digitalocean_volume_attachment | oci_core_volume_attachment | alicloud_disk_attachment |
Check minimum requirements. In the preceding table, we can see Digital Ocean does not have the ability to define a network interface. The documentation also leads us to conclude that it is not possible to define the private ip address of the instances before creating them. Because the Puppet server ip address is required before generating the cloud-init YAML config for all instances, including the Puppet server itself, this means it impossible to use Digital Ocean to spawn a Magic Castle cluster. Oracle Cloud presents the same issue, however, after reading the instance documentation, we find that it is possible to define a static ip address as a string in the instance attribute. It would therefore be possible to create a datastructure in Terraform that would associate each instance hostname with an ip address in the subnet CIDR. Alibaba cloud has an answer for each resource, so we will use this provider in the following steps.
Initialize the provider folder. In a terminal:
git clone https://github.com/ComputeCanada/magic_castle.git\ncd magic_castle\nmkdir alicloud\ncd aliclcoud\nln -s ../common/{variables,outputs}.tf .\nDefine cloud provider specifics variables. Add the following to a new file alicloud.tf:
variable \"region\" { }\nlocals {\n cloud_provider = \"alicloud\"\n cloud_region = var.region\n}\nInitialize the infrastructure. Add the following to a new file infrastructure.tf:
provider \"alicloud\" {\n region = var.region\n}\n\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\nCreate the networking infrastructure. network.tf base template:
resource \"alicloud_vpc\" \"network\" { }\nresource \"alicloud_vswitch\" \"subnet\" { }\nresource \"alicloud_nat_gateway\" \"nat\" { }\nresource \"alicloud_security_group\" \"firewall\" { }\nresource \"alicloud_security_group_rule\" \"allow_in_services\" { }\nresource \"alicloud_security_group\" \"allow_any_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_ingress_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_egress_inside_vpc\" { }\nresource \"alicloud_network_interface\" \"nic\" { }\nresource \"alicloud_eip\" \"public_ip\" { }\nresource \"alicloud_eip_association\" \"eip_asso\" { }\nCreate the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk\" \"volumes\" {\n for_each = module.design.volumes\n}\nConsolidate the instances' information. Add the following snippet to infrastructure.tf:
locals {\n inventory = { for x, values in module.design.instances :\n x => {\n public_ip = contains(values[\"tags\"], \"public\") ? alicloud_eip.public_ip[x].public_ip : \"\"\n local_ip = alicloud_network_interface.nic[x].private_ip\n tags = values[\"tags\"]\n id = alicloud_instance.instances[x].id\n specs = {\n cpus = ...\n gpus = ...\n ram = ...\n volumes = contains(keys(module.design.volume_per_instance), x) ? {\n for pv_key, pv_values in var.volumes:\n pv_key => {\n for name, specs in pv_values:\n name => [\"/dev/disk/by-id/virtio-${replace(alicloud_disk.volumes[\"${x}-${pv_key}-${name}\"].id, \"d-\", \"\")}\"]\n } if contains(values.tags, pv_key)\n } : {}\n }\n }\n }\n}\nCreate the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n public_keys = var.public_keys\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\nCreate the instances. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_instance\" \"instances\" {\n for_each = module.design.instances\n}\nAttach the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk_attachment\" \"attachments\" {\n for_each = module.design.volumes\n}\nIdentify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=alicloud_instance.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\nInclude the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n}\nOnce your new provider is written, you can write an example that will use the module to spawn a Magic Castle cluster with that provider.
module \"alicloud\" {\n source = \"./alicloud\"\n config_git_url = \"https://github.com/ComputeCanada/puppet-magic_castle.git\"\n config_version = \"main\"\n\n cluster_name = \"new\"\n domain = \"my.cloud\"\n image = \"centos_7_9_x64_20G_alibase_20210318.vhd\"\n nb_users = 10\n\n instances = {\n mgmt = { type = \"ecs.g6.large\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"ecs.g6.large\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"ecs.g6.large\", tags = [\"node\"], count = 1 }\n }\n\n volumes = {\n nfs = {\n home = { size = 10 }\n project = { size = 50 }\n scratch = { size = 50 }\n }\n }\n\n public_keys = [file(\"~/.ssh/id_rsa.pub\")]\n\n # Alicloud specifics\n region = \"us-west-1\"\n}\nTo develop for Magic Castle you will need: * Terraform (>= 1.4.0) * git * Access to a Cloud (e.g.: Compute Canada Arbutus) * Ability to communicate with the cloud provider API from your computer * A cloud project with enough room for the resource described in section Magic Caslte Doc 1.1. * [optional] Puppet Development Kit (PDK)
"},{"location":"developers/#2-where-to-start","title":"2. Where to start","text":"The Magic Castle project is defined by Terraform infrastructure-as-code component that is responsible of generating a cluster architecture in a cloud and a Puppet environment component that configures the cluster instances based on their role.
If you wish to add device, an instance, add a new networking interface or a filesystem, you will most likely need to develop some Terraform code. The project structure for Terraform code is described in the reference design document. The document also describes how one could work with current Magic Castle code to add support for another cloud provider.
If you wish to add a service to one of the Puppet environments, install a new software, modify an instance configuration or role, you will most likely need to develop some Puppet code. The following section provides more details on the Puppet environments available and how to develop them.
"},{"location":"developers/#3-puppet-environment","title":"3. Puppet environment","text":"Magic Castle Terraform code initialized every instances to be a Puppet agent and an instance with the tag puppet as the Puppet main server. On the Puppet main server, there is a folder containing the configuration code for the instances of the cluster, this folder is called a Puppet environment and it is pulled from GitHub during the initial configuration of the Puppet main server.
The source of that environment is provided to Terraform using the variable config_git_url.
A repository describing a Magic Castle Puppet environment must contain at the least the following files and folders:
config_git_repo\n\u2523 Puppetfile\n\u2523 environment.conf\n\u2523 hiera.yaml\n\u2517 data\n \u2517 common.yaml\n\u2517 manifests/\n \u2517 site.pp\nPuppetfile specifies the Puppet modules that need to be installed in the environment.environment.conf overrides the primary server default settings for the environment.hiera.yaml configures an ordered list of YAML file data sources.data/common.yaml is common data source for the instances part of hierarchy defined by hiera.yaml.manifests/site.pp defines how each instance will be configured based on their hostname and/or tags.An example of a bare-bone Magic Castle Puppet environment is available on GitHub: MagicCastle/puppet-environment, while the Puppet environment that replicates a Compute Canada HPC cluster is named ComputeCanada/puppet-magic_castle.
"},{"location":"developers/#terraform_datayaml-a-bridge-between-terraform-and-puppet","title":"terraform_data.yaml: a bridge between Terraform and Puppet","text":"To provide information on the deployed resources and the value of the input parameters, Magic Castle Terraform code uploads to the Puppet main server a file named terraform_data.yaml, in the folder /etc/puppetlabs/data/. There is also a symlink created in /etc/puppetlabs/code/environment/production/data/ to ease its usage inside the Puppet environment.
When included in the data hierarchy (hiera.yaml), terraform_data.yaml can provide information about the instances, the volumes and the variables set by the user through the main.tf file. The file has the following structure:
---\nterraform:\n data:\n cluster_name: \"\"\n domain_name: \"\"\n guest_passwd: \"\"\n nb_users: \"\"\n public_keys: []\n sudoer_username: \"\"\n instances:\n host1:\n hostkeys:\n rsa: \"\"\n ed25519: \"\"\n local_ip: \"x.x.x.x\"\n prefix: \"host\"\n public_ip: \"\"\n specs:\n \"cpus\": 0\n \"gpus\": 0\n \"ram\": 0\n tags:\n - \"tag_1\"\n - \"tag_2\"\n tag_ip:\n tag_1:\n - x.x.x.x\n tag_2:\n - x.x.x.x\n volumes:\n volume_tag1:\n volume_1:\n - \"/dev/disk/by-id/123-*\"\n volume_2:\n - \"/dev/disk/by-id/123-abc-*\"\nThe values provided by terraform_data.yaml can be accessed in Puppet by using the lookup() function. For example, to access an instance's list of tags:
lookup(\"terraform.instances.${::hostname}.tags\")\nalias() function. For example, to define the number of guest accounts using the value of nb_users, we could add this to common.yaml profile::accounts::guests::nb_accounts: \"%{alias('terraform.data.nb_users')}\"\nThe configuration of each instance is defined in manifests/site.pp file of the Puppet environment. In this file, it is possible to define a configuration based on an instance hostname
node \"mgmt1\" { }\ndefault node : node default {\n $instance_tags = lookup(\"terraform.instances.${::hostname}.tags\")\n if 'tag_1' in $instances_tags { }\n}\nIt is possible to define Puppet resource directly in site.pp. However, above a certain level of complexity, which can be reach fairly quickly, it is preferable to define classes and include these classes in site.pp based on the node hostname or tags.
Classes can be defined in the Puppet environment under the following path: site/profile/manifests. These classes are named profile classes and the philosophy behind it is explained in Puppet documentation. Because these classes are defined in site/profile, their name has to start with the prefix profile::.
It is also possible to include classes defined externally and installed using the Puppetfile. These classes installed by r10k can be found in the modules folder of the Puppet environment.
To test new additions to puppet.yaml, it is possible to execute cloud-init phases manually. There are four steps that can be executed sequentially: init local, init modules config and modules final. Here are the corresponding commands to execute each step:
cloud-init init --local\ncloud-init init\ncloud-init modules --mode=config\ncloud-init modules --mode=final\nIt is also possible to clean a cloud-init execution and have it execute again at next reboot. To do so, enter the following command:
cloud-init clean\n-r to the previous command to reboot the instance once cloud-init has finishing cleaning."},{"location":"developers/#42-selinux","title":"4.2 SELinux","text":"SELinux is enabled on every instances of a Magic Castle cluster. Some applications do not provide SELinux policies which can lead to their malfunctionning when SELinux is enabled. It is possible to track down the reasons why SELinux is preventing an application to work properly using the command-line tool ausearch.
If you suspect application app-a to be denied by SELinux to work properly, run the following command as root:
ausearch -c app-a --raw | grep denied\nTo see all requests denied by SELinux:
ausearch --raw | grep denied\nSometime, the denials are hidden from regular logging. To display all denials, run the following command as root:
semodule --disable_dontaudit --build\nOnce you have found the denials that are the cause of the problem, you can create a new policy to allow the requests that were previously denied with the following command:
ausearch -c app-a --raw | grep denied | audit2allow -a -M app-a\nFinally, you can install the generated policy using the command provided by auditallow.
If you need to tweak an existing enforcement file and you want to recompile the policy package, you can with the following commands:
checkmodule -M -m -o my_policy.mod my_policy.te\nsemodule_package -o my_policy.pp -m my_policy.mod\nTo build a release, use the script release.sh located at the root of Magic Castle git repo.
Usage: release.sh VERSION [provider ...]\nreleases where it was called. The VERSION argument is expected to correspond to git tag in the puppet-magic_castle repo. It could also be a branch name or a commit. If the provider optional argument is left blank, release files will be built for all providers currently supported by Magic Castle.
Examples:
$ ./release.sh main openstack\n$ ./release.sh 5.8 gcp$ ./release.sh 5.7 azure ovh\n* The documentation provides instructions on how to add support for other cloud providers.
"},{"location":"matrix/#supported-operating-systems","title":"Supported operating systems","text":"Name CentOS 7 CentOS 8 Rocky Linux 8 AlmaLinux 8 Debian 10 Ubuntu 18 Ubuntu 20 Windows 10 AWS ParallelCluster yes yes yes yes yes no yes no Azure CycleCloud yes yes yes yes yes no yes - Azure HPC On-Demand Platform yes no no yes no yes no yes Google HPC-Toolkit yes no no no no no no no Cluster in the Cloud no yes no no no no no no ElastiCluster yes yes yes yes no no no no Magic Castle no yes yes yes no no no no On-Demand Data Centre - - - - - - - - Slurm on GCP yes no yes no yes no yes no"},{"location":"matrix/#supported-job-schedulers","title":"Supported job schedulers","text":"Name AwsBatch Grid Engine HTCondor Moab Open PBS PBS Pro Slurm AWS ParallelCluster yes no no no no no yes Azure CycleCloud no yes yes no no yes yes Azure HPC On-Demand Platform no no no no yes no yes Google HPC-Toolkit no no no no no no yes Cluster in the Cloud no no no no no no yes ElastiCluster no yes no no no no yes Magic Castle no no no no no no yes On-Demand Data Centre no no no yes no no no Slurm on GCP no no no no no no yes"},{"location":"matrix/#technologies","title":"Technologies","text":"Name Infrastructure configuration Programming languages Configuration management Scientific software AWS ParallelCluster CLI generating YAML Python Chef Spack Azure CycleCloud WebUI or CLI + templates Python Chef Bring your own Azure HPC On-Demand Platform YAML files + shell scripts Shell, Terraform Ansible, Packer CVMFS Cluster in the Cloud CLI generating Terraform code Python, Terraform Ansible, Packer EESSI ElastiCluster CLI interpreting an INI file Python, Shell Ansible Bring your own Google HPC-Toolkit CLI generating Terraform code Go, Terraform Ansible, Packer Spack Magic Castle Terraform modules Terraform Puppet CC-CVMFS, EESSI On-Demand Data Centre - - - - Slurm GCP Terraform modules Terraform Ansible, Packer Spack"},{"location":"sequence/","title":"Magic Castle Sequence Diagrams","text":"The following sequence diagrams illustrate the inner working of Magic Castle once terraform apply is called. Some details were left out of the diagrams, but every diagram is followed by references to the code files that were used to build it.
puppet-magic_castle.git does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.magic_castle:/common/design/main.tfmagic_castle:/openstack/network-1.tfmagic_castle:/openstack/network-2.tfmagic_castle:/common/configuration/main.tfmagic_castle:/openstack/infrastructure.tfmagic_castle:/common/provision/main.tfmagic_castle:/dns/cloudflare/main.tfconfig_git_url repo does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.magic_castle:/common/configuration/puppet.yamlpuppet-magic_castle:/manifests/site.pppuppet-magic_castle:/profile/manifests/base.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/freeipa.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/cvmfs.pppuppet-magic_castle:/profile/manifests/slurm.ppThis document explains how to use Magic Castle with Terraform Cloud.
"},{"location":"terraform_cloud/#what-is-terraform-cloud","title":"What is Terraform Cloud?","text":"Terraform Cloud is HashiCorp\u2019s managed service that allows to provision infrastructure using a web browser or a REST API instead of the command-line. This also means that the provisioned infrastructure parameters can be modified by a team and the state is stored in the cloud instead of a local machine.
When provisioning in commercial cloud, Terraform Cloud can also provide a cost estimate of the resources.
"},{"location":"terraform_cloud/#getting-started-with-terraform-cloud","title":"Getting started with Terraform Cloud","text":"main.tf available for the cloud of your choicemain.tfYou will be redirected automatically to your new workspace.
"},{"location":"terraform_cloud/#providing-cloud-provider-credentials-to-terraform-cloud","title":"Providing cloud provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your cloud provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each cloud provider and how to retrieve the values of the variable from the provider.
If you plan on using these environment variables with multiple workspaces, it is recommended to create a credential variable set in Terraform Cloud.
"},{"location":"terraform_cloud/#aws","title":"AWS","text":"You need to define these environment variables: - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY (sensitive)
The value of these variables can either correspond to the value of access key created on the AWS Security Credentials - Access keys page, or you can add user dedicated to Terraform Cloud in AWS IAM Users, and use its access key.
"},{"location":"terraform_cloud/#azure","title":"Azure","text":"You need to define these environment variables: - ARM_CLIENT_ID - ARM_CLIENT_SECRET (sensitive) - ARM_SUBSCRIPTION_ID - ARM_TENANT_ID
Refer to Terraform Azure Provider - Creating a Service Principal to know how to create a Service Principal and retrieve the values for these environment variables.
"},{"location":"terraform_cloud/#google-cloud","title":"Google Cloud","text":"You need to define this environment variable: - GOOGLE_CLOUD_KEYFILE_JSON (sensitive)
The value of the variable will be the content of a Google Cloud service account JSON key file expressed a single line string. Example:
{\"type\": \"service_account\",\"project_id\": \"project-id-1234\",\"private_key_id\": \"abcd1234\",...}\nYou can use jq to format the string from the JSON file provided by Google:
jq . -c project-name-123456-abcdefjg.json\nYou need to define these environment variables: - OS_AUTH_URL - OS_PROJECT_ID - OS_REGION_NAME - OS_INTERFACE - OS_IDENTITY_API_VERSION - OS_USER_DOMAIN_NAME - OS_USERNAME - OS_PASSWORD (sensitive)
Apart from OS_PASSWORD, the values for these variables are available in OpenStack RC file provided for your project.
If you prefer to use OpenStack application credentials, you need to define at least these variables: - OS_AUTH_TYPE\u00a0 - OS_AUTH_URL - OS_APPLICATION_CREDENTIAL_ID - OS_APPLICATION_CREDENTIAL_SECRET
and potentially these too: - OS_IDENTITY_API_VERSION\u00a0 - OS_REGION_NAME - OS_INTERFACE
The values for these variables are available in OpenStack RC file provided when creating the application credentials.
"},{"location":"terraform_cloud/#providing-dns-provider-credentials-to-terraform-cloud","title":"Providing DNS provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your DNS provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each DNS provider and how to retrieve the values of the variable from the provider.
"},{"location":"terraform_cloud/#cloudflare","title":"CloudFlare","text":"Refer to DNS - CloudFlare section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#google-cloud-dns","title":"Google Cloud DNS","text":"Refer to DNS - Google Cloud section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#managing-magic-castle-variables-with-terraform-cloud-ui","title":"Managing Magic Castle variables with Terraform Cloud UI","text":"It is possible to use Terraform Cloud web interface to define variable values in your main.tf. For example, you could want to define a guest password without writing it directly in main.tf to avoid displaying publicly.
To manage a variable with Terraform Cloud: 1. edit your main.tf to define the variables you want to manage. In the following example, we want to manage the number of nodes and the guest password.
Add the variables at the beginning of the `main.tf`:\n ```hcl\n variable \"nb_nodes\" {}\n variable \"password\" {}\n ```\n\nThen replace the static value by the variable in our `main.tf`,\n\ncompute node count\n ```hcl\n node = { type = \"p2-3gb\", tags = [\"node\"], count = var.nb_nodes }\n ```\nguest password\n ```hcl\n guest_passwd = var.password\n ```\nmain.tf. Check \"Sensitive\" if the variable content should not never be shown in the UI or the API.You may edit the variables at any point of your cluster lifetime.
"},{"location":"terraform_cloud/#applying-changes","title":"Applying changes","text":"To create your cluster, apply changes made to your main.tf or the variables, you will need to queue a plan. When you push to the default branch of the linked git repository, a plan will be automatically created. You can also create a plan manually. To do so, click on the \"Queue plan manually\" button inside your workspace, then \"Queue plan\".
Once the plan has been successfully created, you can apply it using the \"Runs\" section. Click on the latest queued plan, then on the \"Apply plan\" button at the bottom of the plan page.
"},{"location":"terraform_cloud/#auto-apply","title":"Auto apply","text":"It is possible to apply automatically a successful plan. Go in the \"Settings\" section, and under \"Apply method\" select \"Auto apply\". Any following successful plan will then be automatically applied.
"},{"location":"terraform_cloud/#magic-castle-terraform-cloud-and-the-cli","title":"Magic Castle, Terraform Cloud and the CLI","text":"Terraform cloud only allows to apply or destroy the plan as stated in the main.tf, but sometimes it can be useful to run some other terraform commands that are only available through the command-line interface, for example terraform taint.
It is possible to import the terraform state of a cluster on your local computer and then use the CLI on it.
Log in Terraform cloud:
terraform login\nCreate a folder where the terraform state will be stored:
mkdir my-cluster-1\nCreate a file named cloud.tf with the following content in your cluster folder:
terraform {\n cloud {\n organization = \"REPLACE-BY-YOUR-TF-CLOUD-ORG\"\n workspaces {\n name = \"REPLACE-BY-THE-NAME-OF-YOUR-WORKSPACE\"\n }\n }\n}\norganization and name with the appropriate value for your cluster. Initialize the folder and retrieve the state:
terraform init\nTo confirm the workspace has been properly imported locally, you can list the resources using:
terraform state list\nMagic Castle in combination with Terraform Cloud (TFE) can be configured to give Slurm the ability to create and destroy instances based on the job queue content.
To enable this feature: 1. Create a TFE API Token and save it somewhere safe.
1.1. If you subscribe to Terraform Cloud Team & Governance plan, you can generate\na [Team API Token](https://www.terraform.io/cloud-docs/users-teams-organizations/api-tokens#team-api-tokens).\nThe team associated with this token requires no access to organization and can be secret.\nIt does not have to include any member. Team API token is preferable as its permissions can be\nrestricted to the minimum required for autoscale purpose.\nCreate a workspace in TFE
2.1. Make sure the repo is private as it will contain the API token.
2.2. If you generated a Team API Token in 1, provide access to the workspace to the team:
2.3 In Configure settings, under Advanced options, for Apply method, select Auto apply.
Create the environment variables of the cloud provider credentials in TFE
pool in TFE. Set value to [] and check HCL.data.yaml in your git repo with the following content: yaml\u00a0 --- profile::slurm::controller::tfe_token: <TFE API token> profile::slurm::controller::tfe_workspace: <TFE workspace id> Complete the file by replacing <TFE API TOKEN> with the token generated at step 1 and <TFE workspace id> (i.e.: ws-...) by the id of the workspace created at step 2. It is recommended to encrypt the TFE API token before committing data.yaml in git. Refer to section 4.15 of README.md to know how to encrypt the token.data.yaml in git and push.Modify main.tf:
main.tf.variable \"pool\" { description = \"Slurm pool of compute nodes\" }\ninstances with the tags pool and node. These are the nodes that Slurm will able to create and destroy.pool = var.pool\npublic_keys =, replace [file(\"~/.ssh/id_rsa.pub\")] by a list of SSH public keys that will have admin access to the cluster.public_keys = ..., add hieradata = file(\"data.yaml\").Go to your workspace in TFE, click on Actions -> Start a new run -> Plan and apply -> Start run. Then, click on \"Confirm & Apply\" and \"Confirm Plan\".
To reduce the time required for compute nodes to become available in Slurm, consider creating a compute node image.
JupyterHub will time out by default after 300 seconds if a node is not spawned yet. Since it may take longer than this to spawn a node, even with an image created, consider increasing the timeout by adding the following to your YAML configuration file:
jupyterhub::jupyterhub_config_hash:\n SlurmFormSpawner:\n start_timeout: 900\nSlurm 23 adds the possibility for sinfo to report nodes that are not yet spawned. This is useful if you want JupyterHub to be aware of those nodes, for example if you want to allow to use GPU nodes without keeping them online at all time. To use that version of Slurm, add the following to your YAML configuration file:
profile::slurm::base::slurm_version: '23.02'\nIf after enabling autoscaling with Terraform Cloud for your Magic Castle cluster, the number of nodes does not increase when submitting jobs, verify the following points:
profile::slurm::controller::tfe_workspace in data.yaml.squeue on the cluster, and verify the reasons why jobs are still in the queue. If under the column (Reason), there is the keyword ReqNodeNotAvail, it implies Slurm tried to boot the listed nodes, but they would not show up before the timeout, therefore Slurm marked them as down. It can happen if your cloud provider is slow to build the instances, or following a configuration problem like in 2. When Slurm marks a node as down, a trace is left in slurmctld's log - using zgrep on the slurm controller node (typically mgmt1): sudo zgrep \"marking down\" /var/log/slurm/slurmctld.log*\nsudo /opt/software/slurm/bin/scontrol update nodename=node[Y-Z] state=IDLE\nnode[Y-Z] by the hostname range listed next to ReqNodeNotAvail in squeue.mgmt1:/var/log/slurm, look for errors in the file slurm_resume.log.To use Magic Castle you will need:
To install Terraform, follow the tutorial or go directly on Terraform download page.
You can verify Terraform was properly installed by looking at the version in a terminal:
terraform version\nAWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, environment variables, representing your AWS Access Key and AWS Secret Key: export AWS_ACCESS_KEY_ID=\"an-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"a-secret-key\"\nReference: AWS Provider - Environment Variables
"},{"location":"#122-google-cloud","title":"1.2.2 Google Cloud","text":"gcloud auth application-default loginaz loginReference : Azure Provider: Authenticating using the Azure CLI
"},{"location":"#124-openstack-ovh","title":"1.2.4 OpenStack / OVH","text":"Download your OpenStack Open RC file. It is project-specific and contains the credentials used by Terraform to communicate with OpenStack API. To download, using OpenStack web page go to: Project \u2192 API Access, then click on Download OpenStack RC File then right-click on OpenStack RC File (Identity API v3), Save Link as..., and save the file.
In a terminal located in the same folder as your OpenStack RC file, source the OpenStack RC file:
source *-openrc.sh\nOnce you are authenticated with your cloud provider, you should be able to communicate with its API. This section lists for each provider some instructions to test this.
"},{"location":"#131-aws","title":"1.3.1 AWS","text":"test_aws.tf with the following content: provider \"aws\" {\n region = \"us-east-1\"\n}\n\ndata \"aws_ec2_instance_type\" \"example\" {\n instance_type = \"t2.micro\"\n}\nterraform init\nterraform plan\nNo changes. Your infrastructure matches the configuration.\nError: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.\nIn a terminal, enter:
gcloud projects list\nPROJECT_ID NAME PROJECT_NUMBER\nTake note of the project_id of the Google Cloud project you want to use, you will need it later.
In a terminal, enter:
az account show\n{\n \"environmentName\": \"AzureCloud\",\n \"homeTenantId\": \"98467e3b-33c2-4a34-928b-ed254db26890\",\n \"id\": \"4dda857e-1d61-457f-b0f0-e8c784d1fb20\",\n \"isDefault\": true,\n \"managedByTenants\": [],\n \"name\": \"Pay-As-You-Go\",\n \"state\": \"Enabled\",\n \"tenantId\": \"495fc59f-96d9-4c3f-9c78-7a7b5f33d962\",\n \"user\": {\n \"name\": \"user@example.com\",\n \"type\": \"user\"\n }\n}\ntest_os.tf with the following content: terraform {\n required_providers {\n openstack = {\n source = \"terraform-provider-openstack/openstack\"\n }\n }\n}\ndata \"openstack_identity_auth_scope_v3\" \"scope\" {\n name = \"my_scope\"\n}\nterraform init\nterraform plan\nNo changes. Your infrastructure matches the configuration.\nError: Error creating OpenStack identity client:\nThe default quotas set by Amazon are sufficient to build the Magic Castle AWS examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Amazon EC2 service quotas.
"},{"location":"#142-google-cloud","title":"1.4.2 Google Cloud","text":"The default quotas set by Google Cloud are sufficient to build the Magic Castle GCP examples. To increase the limits, or request access to special resources like GPUs, refer to Google Compute Engine Resource quotas.
"},{"location":"#143-microsoft-azure","title":"1.4.3 Microsoft Azure","text":"The default quotas set by Microsoft Azure are sufficient to build the Magic Castle Azure examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Azure subscription and service limits, quotas, and constraints.
"},{"location":"#144-openstack","title":"1.4.4 OpenStack","text":"Minimum project requirements:
Note 1: Magic Castle supposes the OpenStack project comes with a network, a subnet and a router already initialized. If any of these components is missing, you will need to create them manually before launching terraform.
The default quotas set by OVH are sufficient to build the Magic Castle OVH examples. To increase the limits, or request access to special resources like GPUs, refer to OVHcloud - Increasing Public Cloud quotas.
"},{"location":"#2-cloud-cluster-architecture-overview","title":"2. Cloud Cluster Architecture Overview","text":""},{"location":"#3-initialization","title":"3. Initialization","text":""},{"location":"#31-main-file","title":"3.1 Main File","text":"tar xvf magic_castle*.tar.gzmv magic_castle* hulkcd hulkThe file main.tf contains Terraform modules and outputs. Modules are files that define a set of resources that will be configured based on the inputs provided in the module block. Outputs are used to tell Terraform which variables of our module we would like to be shown on the screen once the resources have been instantiated.
This file will be our main canvas to design our new clusters. As long as the module block parameters suffice to our need, we will be able to limit our configuration to this sole file. Further customization will be addressed during the second part of the workshop.
"},{"location":"#32-terraform","title":"3.2 Terraform","text":"Terraform fetches the plugins required to interact with the cloud provider defined by our main.tf once when we initialize. To initialize, enter the following command:
terraform init\nThe initialization is specific to the folder where you are currently located. The initialization process looks at all .tf files and fetches the plugins required to build the resources defined in these files. If you replace some or all .tf files inside a folder that has already been initialized, just call the command again to make sure you have all plugins.
The initialization process creates a .terraform folder at the root of your current folder. You do not need to look at its content for now.
Once Terraform folder has been initialized, it is possible to fetch the newest version of the modules used by calling:
terraform init -upgrade\nIn the main.tf file, there is a module named after your cloud provider, i.e.: module \"openstack\". This module corresponds to the high-level infrastructure of your cluster.
The following sections describes each variable that can be used to customize the deployed infrastructure and its configuration. Optional variables can be absent from the example module. The order of the variables does not matter, but the following sections are ordered as the variables appear in the examples.
"},{"location":"#41-source","title":"4.1 source","text":"The first line of the module block indicates to Terraform where it can find the files that define the resources that will compose your cluster. In the releases, this variable is a relative path to the cloud provider folder (i.e.: ./aws).
Requirement: Must be a path to a local folder containing the Magic Castle Terraform files for the cloud provider of your choice. It can also be a git repository. Refer to Terraform documentation on module source for more information.
Post build modification effect: terraform init will have to be called again and the next terraform apply might propose changes if the infrastructure describe by the new module is different.
Magic Castle configuration management is handled by Puppet. The Puppet configuration files are stored in a git repository. This is typically ComputeCanada/puppet-magic_castle repository on GitHub.
Leave this variable to its current value to deploy a vanilla Magic Castle cluster.
If you wish to customize the instances' role assignment, add services, or develop new features for Magic Castle, fork the ComputeCanada/puppet-magic_castle and point this variable to your fork's URL. For more information on Magic Castle puppet configuration customization, refer to MC developer documentation.
Requirement: Must be a valid HTTPS URL to a git repository describing a Puppet environment compatible with Magic Castle. If the repo is private, generate an access token with a permission to read the repo content, and provide the token in the config_git_url like this:
config_git_url = \"https://oauth2:${oauth-key-goes-here}@domain.com/username/repo.git\"\nPost build modification effect: no effect. To change the Puppet configuration source, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#43-config_version","title":"4.3 config_version","text":"Since Magic Cluster configuration is managed with git, it is possible to specify which version of the configuration you wish to use. Typically, it will match the version number of the release you have downloaded (i.e: 9.3).
Requirement: Must refer to a git commit, tag or branch existing in the git repository pointed by config_git_url.
Post build modification effect: none. To change the Puppet configuration version, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#44-cluster_name","title":"4.4 cluster_name","text":"Defines the ClusterName variable in slurm.conf and the name of the cluster in the Slurm accounting database (see slurm.conf documentation).
Requirement: Must be lowercase alphanumeric characters and start with a letter. It can include dashes. cluster_name must be 40 characters or less.
Post build modification effect: destroy and re-create all instances at next terraform apply.
Defines
resolv.conf search domain as int.{cluster_name}.{domain}Optional modules following the current module in the example main.tf can be used to register DNS records in relation to your cluster if the DNS zone of this domain is administered by one of the supported providers. Refer to section 6. DNS Configuration for more details.
Requirements:
[a-z]([-a-z0-9]*[a-z0-9]), concatenated with periods.*.domain. IN A x.x.x.x exists for that domain. You can verify no such record exist with dig: dig +short '*.${domain}'\nPost build modification effect: destroy and re-create all instances at next terraform apply.
Defines the name of the image that will be used as the base image for the cluster nodes.
You can use a custom image if you wish, but configuration management should be mainly done through Puppet. Image customization is mostly envisioned as a way to accelerate the configuration process by applying the security patches and OS updates in advance.
To specify a different image for an instance type, use the image instance attribute
Requirements: the operating system on the image must be from the RedHat family. This includes CentOS (8, 9), Rocky Linux (8, 9), and AlmaLinux (8, 9).
Post build modification effect: none. If this variable is modified, existing instances will ignore the change and future instances will use the new value.
"},{"location":"#461-aws","title":"4.6.1 AWS","text":"The image field needs to correspond to the Amazon Machine Image (AMI) ID. AMI IDs are specific to regions and architectures. Make sure to use the right ID for the region and CPU architecture you are using (i.e: x86_64).
To find out which AMI ID you need to use, refer to - AlmaLinux OS Amazon Web Services AMIs - CentOS list of official images available on the AWS Marketplace - Rocky Linux
Note: Before you can use the AMI, you will need to accept the usage terms and subscribe to the image on AWS Marketplace. On your first deployment, you will be presented an error similar to this one:
\u2502 Error: Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=cvugziknvmxgqna9noibqnnsy\n\u2502 status code: 401, request id: 1f04a85a-f16a-41c6-82b5-342dc3dd6a3d\n\u2502\n\u2502 on aws/infrastructure.tf line 67, in resource \"aws_instance\" \"instances\":\n\u2502 67: resource \"aws_instance\" \"instances\" {\nClick to Subscribe yellow button."},{"location":"#462-microsoft-azure","title":"4.6.2 Microsoft Azure","text":"The image field for Azure can either be a string or a map.
A string image specification will correspond to the image id. Image ids can be retrieved using the following command-line:
az image builder list\nA map image specification needs to contain the following fields publisher, offer sku, and optionally version. The map is used to specify images found in Azure Marketplace. Here is an example:
{\n publisher = \"OpenLogic\",\n offer = \"CentOS-CI\",\n sku = \"7-CI\"\n}\nThe image name can be a regular expression. If more than one image is returned by the query to OpenStack, the most recent is selected.
"},{"location":"#47-instances","title":"4.7 instances","text":"The instances variable is a map that defines the virtual machines that will form the cluster. The map' keys define the hostnames and the values are the attributes of the virtual machines.
Each instance is identified by a unique hostname. An instance's hostname is written as the key followed by its index (1-based). The following map:
instances = {\n mgmt = { type = \"p2-4gb\", tags = [...] },\n login = { type = \"p2-4gb\", count = 1, tags = [...] },\n node = { type = \"c2-15gb-31\", count = 2, tags = [...] },\n gpu-node = { type = \"gpu2.large\", count = 3, tags = [...] },\n}\nmgmt1\nlogin1\nnode1\nnode2\ngpu-node1\ngpu-node2\ngpu-node3\nHostnames must follow a set of rules, from hostname man page:
Valid characters for hostnames are ASCII letters from a to z, the digits from 0 to 9, and the hyphen (-). A hostname may not start with a hyphen.
Two attributes are expected to be defined for each instance: 1. type: name for varying combinations of CPU, memory, GPU, etc. (i.e: t2.medium); 2. tags: list of labels that defines the role of the instance.
Tags are used in the Terraform code to identify if devices (volume, network) need to be attached to an instance, while in Puppet code tags are used to identify roles of the instances.
Terraform tags:
login: identify instances accessible with SSH from Internet and pointed by the domain name A recordspool: identify instances created only when their hostname appears in the var.pool list.proxy: identify instances accessible with HTTP/HTTPS and pointed by the vhost A recordspublic: identify instances that need to have a public ip address reachable from Internetpuppet: identify instances configured as Puppet serversspot: identify instances that are to be spawned as spot/preemptible instances. This tag is supported in AWS, Azure and GCP. It is ignored by OpenStack and OVH.efa: attach an Elastic Fabric Adapter network interface to the instance. This tag is supported in AWS.Puppet tags expected by the puppet-magic_castle environment.
login: identify a login instance (minimum: 2 CPUs, 2GB RAM)mgmt: identify a management instance i.e: FreeIPA server, Slurm controller, Slurm DB (minimum: 2 CPUs, 6GB RAM)nfs: identify the instance that acts as an NFS server.node: identify a compute node instance (minimum: 1 CPUs, 2GB RAM)pool: when combined with node, it identifies compute nodes that Slurm can resume/suspend to meet workload demand.proxy: identify the instance that executes the Caddy reverse proxy and JupyterHub.In the Magic Castle Puppet environment, an instance cannot be tagged as mgmt and proxy.
You are free to define your own additional tags.
"},{"location":"#472-optional-attributes","title":"4.7.2 Optional attributes","text":"Optional attributes can be defined:
count: number of virtual machines with this combination of hostname prefix, type and tags to create (default: 1).image: specification of the image to use for this instance type. (default: global image value). Refer to section 10.12 - Create a compute node image to learn how this attribute can be leveraged to accelerate compute node configuration.disk_type: type of the instance's root disk (default: see the next table).
disk_type disk_size (GiB) Azure Premium_LRS 30 AWS gp2 10 GCP pd-ssd 20 OpenStack null 10 OVH null 10 disk_size: size in gibibytes (GiB) of the instance's root disk containing the operating system and service software (default: see the previous table).
mig: map of NVIDIA Multi-Instance GPU (MIG) short profile names and count used to partition the instances' GPU, example for an A100: mig = { \"1g.5gb\" = 2, \"2g.10gb\" = 1, \"3g.20gb\" = 1 }\nshard: total number of Sharding on the node. Sharding allows sharing the same GPU on multiple jobs. The total number of shards is evenly distributed across all GPUs on the node.For some cloud providers, it possible to define additional attributes. The following sections present the available attributes per provider.
"},{"location":"#aws","title":"AWS","text":"For instances with the spot tags, these attributes can also be set:
wait_for_fulfillment (default: true)spot_type (default: permanent)instance_interruption_behavior (default: stop)spot_price (default: not set)block_duration_minutes (default: not set) [note 1] For more information on these attributes, refer to aws_spot_instance_request argument referenceNote 1: block_duration_minutes is not available to new AWS accounts or accounts without billing history - AWS EC2 Spot Instance requests. When not available, its usage can trigger quota errors like this:
Error requesting spot instances: MaxSpotInstanceCountExceeded: Max spot instance count exceeded\nFor instances with the spot tags, these attributes can also be set:
max_bid_price (default: not set)eviction_policy (default: Deallocate) For more information on these attributes, refer to azurerm_linux_virtual_machine argument referencegpu_type: name of the GPU model to attach to the instance. Refer to Google Cloud documentation for the list of available models per regiongpu_count: number of GPUs of the gpu_type model to attach to the instanceModifying any part of the map after the cluster is built will only affect the type of instances associated with what was modified at the next terraform apply.
The volumes variable is a map that defines the block devices that should be attached to instances that have the corresponding key in their list of tags. To each instance with the tag, unique block devices are attached, no multi-instance attachment is supported.
Each volume in map is defined a key corresponding to its and a map of attributes:
size: size of the block device in GB.type (optional): type of volume to use. Default value per provider:Premium_LRSgp2pd-ssdnullnullVolumes with a tag that have no corresponding instance will not be created.
In the following example:
instances = {\u00a0\n server = { type = \"p4-6gb\", tags = [\"nfs\"] }\n}\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 100 }\n scratch = { size = 100 }\n }\n mds = {\n oss1 = { size = 500 }\n oss2 = { size = 500 }\n }\n}\nThe instance server1 will have three volumes attached to it. The volumes tagged mds are not created since no instances have the corresponding tag.
To define an infrastructure with no volumes, set the volumes variable to an empty map:
volumes = {}\nPost build modification effect: destruction of the corresponding volumes and attachments, and creation of new empty volumes and attachments. If an no instance with a corresponding tag exist following modifications, the volumes will be deleted.
"},{"location":"#49-public_keys","title":"4.9 public_keys","text":"List of SSH public keys that will have access to your cluster sudoer account.
Post build modification effect: trigger scp of hieradata files at next terraform apply. The sudoer account authorized_keys file will be updated by each instance's Puppet agent following the copy of the hieradata files.
default value: 0
Defines how many guest user accounts will be created in FreeIPA. Each user account shares the same randomly generated password. The usernames are defined as userX where X is a number between 1 and the value of nb_users (zero-padded, i.e.: user01 if X < 100, user1 if X < 10).
If an NFS NFS home volume is defined, each user will have a home folder on a shared NFS storage hosted on the NFS server node.
User accounts do not have sudoer privileges. If you wish to use sudo, you will have to login using the sudoer account and the SSH keys listed in public_keys.
If you would like to add a user account after the cluster is built, refer to section 10.3 and 10.4.
Requirement: Must be an integer, minimum value is 0.
Post build modification effect: trigger scp of hieradata files at next terraform apply. If nb_users is increased, new guest accounts will be created during the following Puppet run on mgmt1. If nb_users is decreased, it will have no effect: the guest accounts already created will be left intact.
default value: 4 random words separated by dots
Defines the password for the guest user accounts instead of using a randomly generated one.
Requirement: Minimum length 8 characters.
The password can be provided in a PKCS7 encrypted form. Refer to sub-section 4.15 eyaml_key for instructions on how to encrypt the password.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Password of all guest accounts will be changed to match the new password value.
default value: centos
Defines the username of the account with sudo privileges. The account ssh authorized keys are configured with the SSH public keys with public_keys.
Post build modification effect: none. To change sudoer username, destroy the cluster or redefine the value of profile::base::sudoer_username in hieradata.
default value: empty string
Defines custom variable values that are injected in the Puppet hieradata file. Useful to override common configuration of Puppet classes.
List of useful examples:
profile::base::admin_email: \"me@example.org\"\nprofile::fail2ban::ignore_ip: ['132.203.0.0/16', '8.8.8.8']\njupyterhub::enable_otp_auth: false\nprometheus::alertmanager::route:\n group_by:\n - 'alertname'\n - 'cluster'\n - 'service'\n group_wait: '5s'\n group_interval: '5m'\n repeat_interval: '3h'\n receiver: 'slack'\n\nprometheus::alertmanager::receivers:\n - name: 'slack'\n slack_configs:\n - api_url: 'https://hooks.slack.com/services/ABCDEFG123456'\n channel: \"#channel\"\n send_resolved: true\n username: 'username'\nRefer to the following Puppet modules' documentation to know more about the key-values that can be defined:
The file created from this string can be found on the Puppet server as /etc/puppetlabs/data/user_data.yaml
Requirement: The string needs to respect the YAML syntax.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
default_value: Empty string
Defines the path to a directory containing a hierarchy of YAML data files. The hierarchy is copied on the Puppet server in /etc/puppetlabs/data/user_data.
Hierarchy structure:
<dir>/hostnames/<hostname>/*.yaml<dir>/hostnames/<hostname>.yaml<dir>/prefixes/<prefix>/*.yaml<dir>/prefixes/<prefix>.yaml<dir>/*.yamlFor more information on hieradata, refer to section 4.13 hieradata (optional).
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
default value: empty string
Defines the private RSA key required to decrypt the values encrypted with hiera-eyaml PKCS7. This key will be copied on the Puppet server.
Post build modification effect: trigger scp of private key file at next terraform apply.
If you plan to track the cluster configuration files in git (i.e:main.tf, user_data.yaml), it would be a good idea to encrypt the sensitive property values.
Magic Castle uses hiera-eyaml to provide a per-value encryption of sensitive properties to be used by Puppet.
The private key and its corresponding public key wrapped in a X509 certificate can be generated with openssl:
openssl req -x509 -nodes -newkey rsa:2048 -keyout private_key.pkcs7.pem -out public_key.pkcs7.pem -batch\nor with eyaml:
eyaml createkeys --pkcs7-public-key=public_key.pkcs7.pem --pkcs7-private-key=private_key.pkcs7.pem\nTo encrypt a sensitive property with openssl:
echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | openssl base64 -A | xargs printf \"ENC[PKCS7,%s]\\n\"\nTo encrypt a sensitive property with eyaml:
eyaml encrypt -s 'your-secret' --pkcs7-public-key=public_key.pkcs7.pem -o string\nTo provide the value of this variable via Terraform Cloud, encode the private key content with base64:
openssl base64 -A -in private_key.pkcs7.pem\nDefine a variable in your main.tf:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n}\nThen make sure to decode it before passing it to the cloud provider module:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n eyaml_key = base64decode(var.tfc_eyaml_key)\n ...\n}\ndefault value:
{\n ssh = { \"from_port\" = 22, \"to_port\" = 22, tag = \"login\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n http = { \"from_port\" = 80, \"to_port\" = 80, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n https = { \"from_port\" = 443, \"to_port\" = 443, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n globus = { \"from_port\" = 2811, \"to_port\" = 2811, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"54.237.254.192/29\" },\n myproxy = { \"from_port\" = 7512, \"to_port\" = 7512, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n gridftp = { \"from_port\" = 50000, \"to_port\" = 51000, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" }\n}\nDefines a map of firewall rules that control external traffic to the public nodes. Each rule is defined as a map of key-value pairs and has to be assigned a unique name:
from_port (req.): the lower part of the allowed port range, valid integer value needs to be between 1 and 65535.to_port (req.): the higher part of the allowed port range, valid integer value needs to be between 1 and 65535.tag (req.): instances with this tag will be assigned this firewall rule.ethertype (opt. default: \"IPv4\"): the layer 3 protocol type (\"IPv4\" or \"IPv6\").protocol (opt. default: \"tcp\"): the layer 4 protocol type.cidr (opt. default: \"0.0.0.0/0\"): the remote CIDR, the value needs to be a valid CIDR (i.e. 192.168.0.0/16).If you would like Magic Castle to be able to transfer files and update the state of the cluster in Puppet, make sure there exists at least one effective firewall rule where from_port <= 22 <= to_port and for which the external IP address of the machine that executes Terraform is in the CIDR range (i.e: cidr = \"0.0.0.0/0\" being the most permissive). This corresponds to the ssh rule in the default firewall rule map. This guarantees that Terraform will be able to use SSH to connect to the cluster from anywhere. For more information about this requirement, refer to Magic Castle's bastion tag computation code.
Post build modification effect: modify the cloud provider firewall rules at next terraform apply.
default_value: \"alliance\"
Defines the scientific software environment that users have access when they login. Possible values are:
\"alliance\" / \"computecanada\": Digital Alliance Research Alliance of Canada scientific software environment (previously Compute Canada environment)\"eessi\": European Environment for Scientific Software Installation (EESSI)null / \"\": no scientific software environmentPost build modification effect: trigger scp of hieradata files at next terraform apply.
default_value: []
Defines a list of hostnames with the tag \"pool\" that have to be online. This variable is typically managed by the workload scheduler through Terraform API. For more information, refer to Enable Magic Castle Autoscaling
Post build modification effect: pool tagged hosts with name present in the list will be instantiated, others will stay uninstantiated or will be destroyed if previously instantiated.
default_value = false
If true, the base image packages will not be upgraded during the first boot. By default, all packages are upgraded.
Post build modification effect: No effect on currently built instances. Ones created after the modification will take into consideration the new value of the parameter to determine whether they should upgrade the base image packages or not.
"},{"location":"#421-puppetfile-optional","title":"4.21 puppetfile (optional)","text":"default_value = \"\"
Defines a second Puppetfile used to install complementary modules with r10k.
Post build modification effect: trigger scp of Puppetfile at next terraform apply. Each instance's Puppet agent will be reloaded following the installation of the new modules.
Defines the label of the AWS EC2 region where the cluster will be created (i.e.: us-east-2).
Requirement: Must be in the list of available EC2 regions.
Post build modification effect: rebuild of all resources at next terraform apply.
default value: None
Defines the label of the data center inside the AWS region where the cluster will be created (i.e.: us-east-2a). If left blank, it chosen at random amongst the availability zones of the selected region.
Requirement: Must be in a valid availability zone for the selected region. Refer to AWS documentation to find out how list the availability zones.
"},{"location":"#52-microsoft-azure","title":"5.2 Microsoft Azure","text":""},{"location":"#521-location","title":"5.2.1 location","text":"Defines the label of the Azure location where the cluster will be created (i.e.: eastus).
Requirement: Must be a valid Azure location. To get the list of available location, you can use Azure CLI : az account list-locations -o table.
Post build modification effect: rebuild of all resources at next terraform apply.
default value: None
Defines the name of an already created resource group to use. Terraform will no longer attempt to manage a resource group for Magic Castle if this variable is defined and will instead create all resources within the provided resource group. Define this if you wish to use an already created resource group or you do not have a subscription-level access to create and destroy resource groups.
Post build modification effect: rebuild of all instances at next terraform apply.
default value:
{\n name = null\n product = null\n publisher = null\n}\nPurchase plan information for Azure Marketplace image. Certain images from Azure Marketplace requires a terms acceptance or a fee to be used. When using this kind of image, you must supply the plan details.
For example, to use the official AlmaLinux image, you have to first add it to your account. Then to use it with Magic Castle, you must supply the following plan information:
plan = {\n name = \"8_7\"\n product = \"almalinux\"\n publisher = \"almalinux\"\n}\nDefines the label of the unique identifier associated with the Google Cloud project in which the resources will be created. It needs to corresponds to GCP project ID, which is composed of the project name and a randomly assigned number.
Requirement: Must be a valid Google Cloud project ID.
Post build modification effect: rebuild of all resources at next terraform apply.
Defines the name of the specific geographical location where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud region. Refer to Google Cloud documentation for the list of available regions and their characteristics.
"},{"location":"#533-zone-optional","title":"5.3.3 zone (optional)","text":"default value: None
Defines the name of the zone within the region where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud zone. Refer to Google Cloud documentation for the list of available zones and their characteristics.
"},{"location":"#54-openstack-and-ovh","title":"5.4 OpenStack and OVH","text":""},{"location":"#541-os_floating_ips-optional","title":"5.4.1 os_floating_ips (optional)","text":"default value: {}
Defines a map as an association of instance names (key) to pre-allocated floating ip addresses (value). Example:
os_floating_ips = {\n login1 = \"132.213.13.59\"\n login2 = \"132.213.13.25\"\n }\nThis variable can be useful if you manage your DNS manually and you would like the keep the same domain name for your cluster at each build.
Post build modification effect: change the floating ips assigned to the public instances.
"},{"location":"#542-os_ext_network-optional","title":"5.4.2 os_ext_network (optional)","text":"default value: None
Defines the name of the external network that provides the floating ips. Define this only if your OpenStack cloud provides multiple external networks, otherwise, Terraform can find it automatically.
Post build modification effect: change the floating ips assigned to the public nodes.
"},{"location":"#544-subnet_id-optional","title":"5.4.4 subnet_id (optional)","text":"default value: None
Defines the ID of the internal IPV4 subnet to which the instances are connected. Define this if you have or intend to have more than one subnets defined in your OpenStack project. Otherwise, Terraform can find it automatically. Can be used to force a v4 subnet when both v4 and v6 exist.
Post build modification effect: rebuild of all instances at next terraform apply.
Some functionalities in Magic Castle require the registration of DNS records under the cluster name in the selected domain. This includes web services like JupyterHub, Mokey and FreeIPA web portal.
If your domain DNS records are managed by one of the supported providers, follow the instructions in the corresponding sections to have the cluster's DNS records created and tracked by Magic Castle.
If your DNS provider is not supported, you can manually create the records. Refer to the subsection 6.3 for more details.
"},{"location":"#61-cloudflare","title":"6.1 Cloudflare","text":"dns module for Cloudflare in your main.tf.output \"hostnames\" block.terraform init.CLOUDFLARE_EMAIL and CLOUDFLARE_API_KEY, where CLOUDFLARE_EMAIL is your Cloudflare account email address and CLOUDFLARE_API_KEY is your account Global API Key available in your Cloudflare profile.If you prefer using an API token instead of the global API key, you will need to configure a token with the following four permissions with the Cloudflare API Token interface.
Section Subsection Permission Zone DNS EditInstead of step 5, export only CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_API_TOKEN, and CLOUDFLARE_DNS_API_TOKEN equal to the API token generated previously.
requirement: Install the Google Cloud SDK
gcloud auth application-default logindns module for Google Cloud in your main.tf.output \"hostnames\" block.main.tf's dns module, configure the variables project and zone_name with their respective values as defined by your Google Cloud project.terraform init.If your DNS provider is not currently supported by Magic Castle, you can create the DNS records manually.
Magic Castle provides a module that creates a text file with the DNS records that can then be imported manually in your DNS zone. To use this module, add the following snippet to your main.tf:
module \"dns\" {\n source = \"./dns/txt\"\n name = module.openstack.cluster_name\n domain = module.openstack.domain\n public_instances = module.openstack.public_instances\n}\nFind and replace openstack in the previous snippet by your cloud provider of choice if not OpenStack (i.e: aws, gcp, etc.).
The file will be created after the terraform apply in the same folder as your main.tf and will be named as ${name}.${domain}.txt.
Magic Castle DNS module creates SSHFP records for all instances with a public ip address. These records can be used by SSH clients to verify the SSH host keys of the server. If DNSSEC is enabled for the domain and the SSH client is correctly configured, no host key confirmation will be prompted when connecting to the server.
For more information on how to activate DNSSEC, refer to your DNS provider documentation:
To setup an SSH client to use SSHFP records, add
VerifyHostKeyDNS yes\n~/.ssh/config)."},{"location":"#7-planning","title":"7. Planning","text":"Once your initial cluster configuration is done, you can initiate a planning phase where you will ask Terraform to communicate with your cloud provider and verify that your cluster can be built as it is described by the main.tf configuration file.
Terraform should now be able to communicate with your cloud provider. To test your configuration file, enter the following command
terraform plan\nThis command will validate the syntax of your configuration file and communicate with the provider, but it will not create new resources. It is only a dry-run. If Terraform does not report any error, you can move to the next step. Otherwise, read the errors and fix your configuration file accordingly.
"},{"location":"#8-deployment","title":"8. Deployment","text":"To create the resources defined by your main, enter the following command
terraform apply\nThe command will produce the same output as the plan command, but after the output it will ask for a confirmation to perform the proposed actions. Enter yes.
Terraform will then proceed to create the resources defined by the configuration file. It should take a few minutes. Once the creation process is completed, Terraform will output the guest account usernames and password, the sudoer username and the floating ip of the login node.
Warning: although the instance creation process is finished once Terraform outputs the connection information, you will not be able to connect and use the cluster immediately. The instance creation is only the first phase of the cluster-building process. The configuration: the creation of the user accounts, installation of FreeIPA, Slurm, configuration of JupyterHub, etc.; takes around 15 minutes after the instances are created.
Once it is booted, you can follow an instance configuration process by looking at:
/var/log/cloud-init-output.logjournalctl -u puppetIf unexpected problems occur during configuration, you can provide these logs to the authors of Magic Castle to help you debug.
"},{"location":"#81-deployment-customization","title":"8.1 Deployment Customization","text":"You can modify the main.tf at any point of your cluster's life and apply the modifications while it is running.
Warning: Depending on the variables you modify, Terraform might destroy some or all resources, and create new ones. The effects of modifying each variable are detailed in the subsections of Configuration.
For example, to increase the number of computes nodes by one. Open main.tf, add 1 to node's count , save the document and call
terraform apply\nTerraform will analyze the difference between the current state and the future state, and plan the creation of a single new instance. If you accept the action plan, the instance will be created, provisioned and eventually automatically add to the Slurm cluster configuration.
You could do the opposite and reduce the number of compute nodes to 0.
"},{"location":"#9-destruction","title":"9. Destruction","text":"Once you're done working with your cluster and you would like to recover the resources, in the same folder as main.tf, enter:
terraform destroy -refresh=false\nThe -refresh=false\u00a0flag is to avoid an issue where one or many of the data sources return no results and stall the cluster destruction with a message like the following:
Error: Your query returned no results. Please change your search criteria and try again.\nAs for apply, Terraform will output a plan that you will have to confirm by entering yes.
Warning: once the cluster is destroyed, nothing will be left, even the shared storage will be erased.
"},{"location":"#91-instance-destruction","title":"9.1 Instance Destruction","text":"It is possible to destroy only the instances and keep the rest of the infrastructure like the floating ip, the volumes, the generated SSH host key, etc. To do so, set the count value of the instance type you wish to destroy to 0.
"},{"location":"#92-reset","title":"9.2 Reset","text":"On some occasions, it is desirable to rebuild some of the instances from scratch. Using terraform taint, you can designate resources that will be rebuilt at next application of the plan.
To rebuild the first login node :
terraform taint 'module.openstack.openstack_compute_instance_v2.instances[\"login1\"]'\nterraform apply\nOnce the cluster is online and configured, you can modify its configuration as you see fit. We list here how to do most commonly asked for customizations.
Some customizations are done from the Puppet server instance (puppet). To connect to the puppet server, follow these steps:
ssh -A centos@cluster_ip. Replace centos by the value of sudoer_username if it is different.ssh puppetNote on Google Cloud: In GCP, OS Login lets you use Compute Engine IAM roles to manage SSH access to Linux instances. This feature is incompatible with Magic Castle. Therefore, it is turned off in the instances metadata (enable-oslogin=\"FALSE\"). The only account with sudoer rights that can log in the cluster is configured by the variable sudoer_username (default: centos).
If you plan to modify configuration files manually, you will need to disable Puppet. Otherwise, you might find out that your modifications have disappeared in a 30-minute window.
Puppet executes a run every 30 minutes and at reboot. To disable puppet:
sudo puppet agent --disable \"<MESSAGE>\"\nRefer to section 4.11.
"},{"location":"#103-add-ldap-users","title":"10.3 Add LDAP Users","text":"Users can be added to Magic Castle LDAP database (FreeIPA) with either one of the following methods: hieradata, command-line, and Mokey web-portal. Each method is presented in the following subsections.
New LDAP users are automatically assigned a home folder on NFS.
Magic Castle determines if an LDAP user should be member of a Slurm account based on its POSIX groups. When a user is added to a POSIX group, a daemon try to match the group name to the following regular expression:
(ctb|def|rpp|rrg)-[a-z0-9_-]*\nIf there is a match, the user will be added to a Slurm account with the same name, and will gain access to the corresponding project folder under /project.
Note: The regular expression represents how Compute Canada names its resources allocation. The regular expression can be redefined, see profile::accounts:::project_regex
Using the hieradata variable in the main.tf, it is possible to define LDAP users.
Examples of LDAP user definition with hieradata are provided in puppet-magic_castle documentation.
"},{"location":"#1032-command-line","title":"10.3.2 Command-Line","text":"To add a user account after the cluster is built, log in mgmt1 and call:
kinit admin\nIPA_GUEST_PASSWD=<new_user_passwd> /sbin/ipa_create_user.py <username> [--group <group_name>]\nkdestroy\nIf user sign-up with Mokey is enabled, users can create their own account at
https://mokey.yourcluster.domain.tld/auth/signup\nIt is possible that an administrator is required to enable the account with Mokey. You can access the administrative panel of FreeIPA at :
https://ipa.yourcluster.domain.tld/\nThe FreeIPA administrator credentials can be retrieved from an encrypted file on the Puppet server. Refer to section 10.14 to know how.
"},{"location":"#104-increase-the-number-of-guest-accounts","title":"10.4 Increase the Number of Guest Accounts","text":"To increase the number of guest accounts after creating the cluster with Terraform, simply increase the value of nb_users, then call :
terraform apply\nEach instance's Puppet agent will be reloaded following the copy of the hieradata files, and the new accounts will be created.
"},{"location":"#105-restrict-ssh-access","title":"10.5 Restrict SSH Access","text":"By default, instances tagged login have their port 22 opened to entire world. If you know the range of ip addresses that will connect to your cluster, we strongly recommend that you limit the access to port 22 to this range.
To limit the access to port 22, refer to section 4.14 firewall_rules, and replace the cidr of the ssh rule to match the range of ip addresses that have be the allowed to connect to the cluster. If there are more than one range, create multiple rules with distinct names.
The default Python kernel corresponds to the Python installed in /opt/ipython-kernel. Each compute node has its own copy of the environment. To add packages to this environment, add the following lines to hieradata in main.tf:
jupyterhub::kernel::venv::packages:\n - package_A\n - package_B\n - package_C\nand replace package_* by the packages you need to install. Then call:
terraform apply\nNo longer supported
"},{"location":"#108-recovering-from-puppet-rebuild","title":"10.8 Recovering from puppet rebuild","text":"The modifications of some of the parameters in the main.tf file can trigger the rebuild of the puppet instance. This instance hosts the Puppet Server on which depends the Puppet agent of the other instances. When puppet is rebuilt, the other Puppet agents cease to recognize Puppet Server identity since the Puppet Server identity and certificates have been regenerated.
To fix the Puppet agents, you will need to apply the following commands on each instance other than puppet once puppet is rebuilt:
sudo systemctl stop puppet\nsudo rm -rf /etc/puppetlabs/puppet/ssl/\nsudo systemctl start puppet\nThen, on puppet, you will need to sign the new certificate requests made by the instances. First, you can list the requests:
sudo /opt/puppetlabs/bin/puppetserver ca list\nThen, if every instance is listed, you can sign all requests:
sudo /opt/puppetlabs/bin/puppetserver ca sign --all\nIf you prefer, you can sign individual request by specifying their name:
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname NAME[,NAME]\nLogin nodes run fail2ban, an intrusion prevention software that protects login nodes from brute-force attacks. fail2ban is configured to ban ip addresses that attempted to login 20 times and failed in a window of 60 minutes. The ban time is 24 hours.
In the context of a workshop with SSH novices, the 20-attempt rule might be triggered, resulting in participants banned and puzzled, which is a bad start for a workshop. There are solutions to mitigate this problem.
"},{"location":"#1091-define-a-list-of-ip-addresses-that-can-never-be-banned","title":"10.9.1 Define a list of ip addresses that can never be banned","text":"fail2ban keeps a list of ip addresses that are allowed to fail to login without risking jail time. To add an ip address to that list, add the following lines to the variable hieradata\u00a0in main.tf:
profile::fail2ban::ignoreip:\n - x.x.x.x\n - y.y.y.y\nx.x.x.x and y.y.y.y are ip addresses you want to add to the ignore list. The ip addresses can be written using CIDR notations. The ignore ip list on Magic Castle already includes 127.0.0.1/8 and the cluster subnet CIDR. Once the line is added, call:
terraform apply\nfail2ban rule that banned ip addresses that failed to connect with SSH can be disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::jails: ['ssh-ban-root']\nOnce the line is added, call:
terraform apply\nfail2ban ban ip addresses by adding rules to iptables. To remove these rules, you need to tell fail2ban to unban the ips.
To list the ip addresses that are banned, execute the following command:
sudo fail2ban-client status ssh-route\nTo unban ip addresses, enter the following command followed by the ip addresses you want to unban:
sudo fail2ban-client set ssh-route unbanip\nWhile this is not recommended, fail2ban can be completely disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::service_ensure: 'stopped'\nthen call :
terraform apply\nSELinux can be set in permissive mode to debug new workflows that would be prevented by SELinux from working properly. To do so, add the following line to the variable hieradata\u00a0in main.tf:
selinux::mode: 'permissive'\nWhen scaling the compute node pool, either manually by changing the count or automatically with Slurm autoscale, it can become beneficial to reduce the time spent configuring the machine when it boots for the first time, hence reducing the time requires before it becomes available in Slurm. One way to achieve this is to clone the root disk of a fully configured compute node and use it as the base image of future compute nodes.
This process has three steps:
The following subsection explains how to accomplish each step.
Warning: While it will work in most cases, avoid re-using the compute node image of a previous deployment. The preparation steps cleans most of the deployment specific configuration and secrets, but there is no guarantee that the configuration will be entirely compatible with a different deployment.
"},{"location":"#10121-prepare-the-volume-for-cloning","title":"10.12.1 Prepare the volume for cloning","text":"The environment puppet-magic_castle installs a script that prepares the volume for cloning named prepare4image.sh.
To make sure a node is ready for cloning, open its puppet agent log and validate the catalog was successfully applied at least once:
journalctl -u puppet | grep \"Applied catalog\"\nTo prepare the volume for cloning, execute the following line while connected to the compute node:
sudo /usr/sbin/prepare4image.sh\nBe aware that, since it is preferable for the instance to be powered off when cloning its volume, the script halts the machine once it is completed. Therefore, after executing prepare4image.sh, you will be disconnected from the instance.
The script prepare4image.sh executes the following steps in order:
slurmd)/etc/etc/fstab/var/log/message contentOnce the instance is powered off, access your cloud provider dashboard, find the instance and follow the provider's instructions to create the image.
Note down the name/id of the image you created, it will be needed during the next step.
"},{"location":"#10123-configure-magic-castle-terraform-code-to-use-the-new-image","title":"10.12.3 Configure Magic Castle Terraform code to use the new image","text":"Edit your main.tf and add image = \"name-or-id-of-your-image\" to the dictionary defining the instance. The instance previously powered off will be powered on and future non-instantiated machines will use the image at the next execution of terraform apply.
If the cluster is composed of heterogeneous compute nodes, it is possible to create an image for each type of compute nodes. Here is an example with Google Cloud
instances = {\n mgmt = { type = \"n2-standard-2\", tags = [\"puppet\", \"mgmt\", \"nfs\"], count = 1 }\n login = { type = \"n2-standard-2\", tags = [\"login\", \"public\", \"proxy\"], count = 1 }\n node = {\n type = \"n2-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n image = \"rocky-mc-cpu-node\"\n }\n gpu = {\n type = \"n1-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n gpu_type = \"nvidia-tesla-t4\"\n gpu_count = 1\n image = \"rocky-mc-gpu-node\"\n }\n}\nDuring the cloud-init initialization phase, bootstrap.sh script is executed. This script generates a set of encrypted secret values that are required by the Magic Castle Puppet environment:
profile::consul::acl_api_tokenprofile::freeipa::mokey::passwordprofile::freeipa::server::admin_passwordprofile::freeipa::server::ds_passwordprofile::slurm::accounting::passwordprofile::slurm::base::munge_keyTo read or change the value of one of these keys, use eyaml edit command on the puppet host, like this:
sudo /opt/puppetlabs/puppet/bin/eyaml edit \\\n --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/boot_private_key.pkcs7.pem \\\n --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/boot_public_key.pkcs7.pem \\\n /etc/puppetlabs/code/environments/production/data/bootstrap.yaml\nIt is also possible to redefine the values of these keys by adding the key-value pair to the hieradata configuration file. Refer to section 4.13 hieradata. User defined values take precedence over boot generated values in the Magic Castle Puppet data hierarchy.
"},{"location":"#1014-expand-a-volume","title":"10.14 Expand a volume","text":"Volumes defined in the volumes map can be expanded at will. To enable online extension of a volume, add enable_resize = true to its specs map. You can then increase the size at will. The corresponding volume will be expanded by the cloud provider and the filesystem will be extended by Puppet.
You can modify the Terraform module files in the folder named after your cloud provider (e.g: gcp, openstack, aws, etc.)
Figure 1 (below) illustrates how Magic Castle is structured to provide a unified interface between multiple cloud providers. Each blue block is a file or a module, while white blocks are variables or resources. Arrows indicate variables or resources that contribute to the definition of the linked variables or resources. The figure can be read as a flow-chart from top to bottom. Some resources and variables have been left out of the chart to avoid cluttering it further.
Figure 1. Magic Castle Terraform Project Structure
main.tf: User provides the instances and volumes structure they wants as _map_s. instances = {\n mgmt = { type = \"p4-7.5gb\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"p2-3.75gb\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"p2-3.75gb\", tags = [\"node\"], count = 2 }\n}\n\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 500 }\n scratch = { size = 500 }\n }\n}\ncommon/design:
instances map is expanded to form a new map where each entry represents a single host. instances = {\n mgmt1 = {\n type = \"p2-3.75gb\"\n tags = [\"puppet\", \"mgmt\", \"nfs\"]\n }\n login1 = {\n type = \"p2-3.75gb\"\n tags = [\"login\", \"public\", \"proxy\"]\n }\n node1 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n node2 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n}\nvolumes map is expanded to form a new map where each entry represent a single volume volumes = {\n mgmt1-nfs-home = { size = 100 }\n mgmt1-nfs-project = { size = 100 }\n mgmt1-nfs-scratch = { size = 500 }\n}\nnetwork.tf: the instances map from common/design is used to generate a network interface (nic) for each host, and a public ip address for each host with the public tag.
resource \"provider_network_interface\" \"nic\" {\n for_each = module.design.instances\n ...\n}\ncommon/configuration: for each host in instances, a cloud-init yaml config that includes puppetservers is generated. These configs are outputted to a user_data map where the keys are the hostnames.
user_data = {\n for key, values in var.instances :\n key => templatefile(\"${path.module}/puppet.yaml\", { ... })\n}\ninfrastructure.tf: for each host in instances, an instance resource as defined by the selected cloud provider is generated. Each instance is initially configured by its user_data cloud-init yaml config.
resource \"provider_instance\" \"instances\" {\n for_each = module.design.instance\n user_data = module.instance_config.user_data[each.key]\n ...\n}\ninfrastructure.tf: for each volume in volumes, a block device as defined by the selected cloud provider is generated and attached it to its matching instance using an attachment resource.
resource \"provider_volume\" \"volumes\" {\n for_each = module.design.volumes\n size = each.value.size\n ...\n}\nresource \"provider_attachment\" \"attachments\" {\n for_each = module.design.volumes\n instance_id = provider_instance.instances[each.value.instance].id\n volume_id = provider_volume.volumes[each.key].id\n ...\n}\ninfrastructure.tf: the created instances' information are consolidated in a map named inventory.
inventory = {\n mgmt1 = {\n public_ip = \"\"\n local_ip = \"10.0.0.1\"\n id = \"abc1213-123-1231\"\n tags = [\"mgmt\", \"puppet\", \"nfs\"]\n }\n ...\n}\ncommon/provision: the information from created instances is consolidated and written in a yaml file namedterraform_data.yaml that is uploaded on the Puppet server as part of the hieradata.
resource \"terraform_data\" \"deploy_puppetserver_files\" {\n ...\n provisioner \"file\" {\n content = var.terraform_data\n destination = \"terraform_data.yaml\"\n }\n ...\n}\noutputs.tf: the information of all instances that have a public address are output as a map named public_instances.
In the previous section, we have used generic resource name when writing HCL code that defines these resources. The following table indicate what resource is used for each provider based on its role in the cluster.
Resource AWS Azure Google Cloud Platform OpenStack OVH network aws_vpc azurerm_virtual_network google_compute_network prebuilt openstack_networking_network_v2 subnet aws_subnet azurerm_subnet google_compute_subnetwork prebuilt openstack_networking_subnet_v2 router aws_route not used google_compute_router built-in not used nat aws_internet_gateway not used google_compute_router_nat built-in not used firewall aws_security_group azurerm_network_security_group google_compute_firewall openstack_compute_secgroup_v2 openstack_compute_secgroup_v2 nic aws_network_interface azurerm_network_interface google_compute_address openstack_networking_port_v2 openstack_networking_port_v2 public ip aws_eip azurerm_public_ip google_compute_address openstack_networking_floatingip_v2 openstack_networking_network_v2 instance aws_instance azurerm_linux_virtual_machine google_compute_instance openstack_compute_instance_v2 openstack_compute_instance_v2 volume aws_ebs_volume azurerm_managed_disk google_compute_disk openstack_blockstorage_volume_v3 openstack_blockstorage_volume_v3 attachment aws_volume_attachment azurerm_virtual_machine_data_disk_attachment google_compute_attached_disk openstack_compute_volume_attach_v2 openstack_compute_volume_attach_v2"},{"location":"design/#using-reference-design-to-extend-for-a-new-cloud-provider","title":"Using reference design to extend for a new cloud provider","text":"Magic Castle currently supports five cloud providers, but its design makes it easy to add new providers. This section presents a step-by-step guide to add a new cloud provider support to Magic Castle.
Identify the resources. Using the Resource per provider table, read the cloud provider Terraform documentation, and identify the name for each resource in the table.
Check minimum requirements. Once all resources have been identified, you should be able to determine if the cloud provider can be used to deploy Magic Castle. If you found a name for each resource listed in table, the cloud provider can be supported. If some resources are missing, you will need to read the provider's documentation to determine if the absence of the resource can be compensated for somehow.
Initialize the provider folder. Create a folder named after the provider. In this folder, create two symlinks, one pointing to common/variables.tf and the other to common/outputs.tf. These files define the interface common to all providers supported by Magic Castle.
Define cloud provider specifics variables. Create a file named after your provider provider_name.tf\u00a0and define variables that are required by the provider but not common to all providers, for example the availability zone or the region. In this file, define two local variables named cloud_provider and cloud_region.
Initialize the infrastructure. Create a file named infrastructure.tf. In this file:
provider \"provider_name\" {\n region = var.region\n}\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\nCreate the networking infrastructure. Create a file named network.tf and define the network, subnet, router, nat, firewall, nic and public ip resources using the module.design.instances map.
Create the volumes. In infrastructure.tf, define the volumes resource using module.design.volumes.
Consolidate the instances' information. In infrastructure.tf, define a local variable named inventory that will be a map containing the following keys for each instance: public_ip, local_ip, prefix, tags, and specs (#cpu, #gpus, ram, volumes). For the volumes, you need to provide the paths under which the volumes will be found on the instances to which they are attached. This is typically derived from the volume id. Here is an example:
volumes = contains(keys(module.design.volume_per_instance), x) ? {\n for pv_key, pv_values in var.volumes:\n pv_key => {\n for name, specs in pv_values:\n name => [\"/dev/disk/by-id/*${substr(provider.volumes[\"${x}-${pv_key}-${name}\"].id, 0, 20)}\"]\n } if contains(values.tags, pv_key)\n } : {}\nCreate the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n public_keys = var.public_keys\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\nCreate the instances. In infrastructure.tf, define the instances resource using module.design.instances_to_build for the instance attributes and module.configuration.user_data for the initial configuration.
Attach the volumes. In infrastructure.tf, define the attachments resource using module.design.volumes and refer to the attribute each.value.instance to retrieve the instance's id to which the volume needs to be attached.
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=cloud_provider_instance_resource.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\nInclude the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\nIdentify the resources. For Digital Ocean, Oracle Cloud and Alibaba Cloud, we get the following resource mapping: | Resource | Digital Ocean | Oracle Cloud | Alibaba Cloud | | ----------- | :-------------------- | :-------------------- | :-------------------- | | network | digitalocean_vpc | oci_core_vcn | alicloud_vpc | | subnet | built in vpc | oci_subnet | alicloud_vswitch | | router | n/a | oci_core_route_table | built in vpc | | nat | n/a | oci_core_internet_gateway | alicloud_nat_gateway | | firewall | digitalocean_firewall | oci_core_security_list | alicloud_security_group | | nic | n/a | built in instance | alicloud_network_interface | | public ip | digitalocean_floating_ip | built in instance | alicloud_eip | | instance | digitalocean_droplet | oci_core_instance | alicloud_instance | | volume | digitalocean_volume | oci_core_volume | alicloud_disk | | attachment | digitalocean_volume_attachment | oci_core_volume_attachment | alicloud_disk_attachment |
Check minimum requirements. In the preceding table, we can see Digital Ocean does not have the ability to define a network interface. The documentation also leads us to conclude that it is not possible to define the private ip address of the instances before creating them. Because the Puppet server ip address is required before generating the cloud-init YAML config for all instances, including the Puppet server itself, this means it impossible to use Digital Ocean to spawn a Magic Castle cluster. Oracle Cloud presents the same issue, however, after reading the instance documentation, we find that it is possible to define a static ip address as a string in the instance attribute. It would therefore be possible to create a datastructure in Terraform that would associate each instance hostname with an ip address in the subnet CIDR. Alibaba cloud has an answer for each resource, so we will use this provider in the following steps.
Initialize the provider folder. In a terminal:
git clone https://github.com/ComputeCanada/magic_castle.git\ncd magic_castle\nmkdir alicloud\ncd aliclcoud\nln -s ../common/{variables,outputs}.tf .\nDefine cloud provider specifics variables. Add the following to a new file alicloud.tf:
variable \"region\" { }\nlocals {\n cloud_provider = \"alicloud\"\n cloud_region = var.region\n}\nInitialize the infrastructure. Add the following to a new file infrastructure.tf:
provider \"alicloud\" {\n region = var.region\n}\n\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\nCreate the networking infrastructure. network.tf base template:
resource \"alicloud_vpc\" \"network\" { }\nresource \"alicloud_vswitch\" \"subnet\" { }\nresource \"alicloud_nat_gateway\" \"nat\" { }\nresource \"alicloud_security_group\" \"firewall\" { }\nresource \"alicloud_security_group_rule\" \"allow_in_services\" { }\nresource \"alicloud_security_group\" \"allow_any_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_ingress_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_egress_inside_vpc\" { }\nresource \"alicloud_network_interface\" \"nic\" { }\nresource \"alicloud_eip\" \"public_ip\" { }\nresource \"alicloud_eip_association\" \"eip_asso\" { }\nCreate the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk\" \"volumes\" {\n for_each = module.design.volumes\n}\nConsolidate the instances' information. Add the following snippet to infrastructure.tf:
locals {\n inventory = { for x, values in module.design.instances :\n x => {\n public_ip = contains(values[\"tags\"], \"public\") ? alicloud_eip.public_ip[x].public_ip : \"\"\n local_ip = alicloud_network_interface.nic[x].private_ip\n tags = values[\"tags\"]\n id = alicloud_instance.instances[x].id\n specs = {\n cpus = ...\n gpus = ...\n ram = ...\n volumes = contains(keys(module.design.volume_per_instance), x) ? {\n for pv_key, pv_values in var.volumes:\n pv_key => {\n for name, specs in pv_values:\n name => [\"/dev/disk/by-id/virtio-${replace(alicloud_disk.volumes[\"${x}-${pv_key}-${name}\"].id, \"d-\", \"\")}\"]\n } if contains(values.tags, pv_key)\n } : {}\n }\n }\n }\n}\nCreate the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n public_keys = var.public_keys\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\nCreate the instances. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_instance\" \"instances\" {\n for_each = module.design.instances\n}\nAttach the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk_attachment\" \"attachments\" {\n for_each = module.design.volumes\n}\nIdentify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=alicloud_instance.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\nInclude the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n}\nOnce your new provider is written, you can write an example that will use the module to spawn a Magic Castle cluster with that provider.
module \"alicloud\" {\n source = \"./alicloud\"\n config_git_url = \"https://github.com/ComputeCanada/puppet-magic_castle.git\"\n config_version = \"main\"\n\n cluster_name = \"new\"\n domain = \"my.cloud\"\n image = \"centos_7_9_x64_20G_alibase_20210318.vhd\"\n nb_users = 10\n\n instances = {\n mgmt = { type = \"ecs.g6.large\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"ecs.g6.large\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"ecs.g6.large\", tags = [\"node\"], count = 1 }\n }\n\n volumes = {\n nfs = {\n home = { size = 10 }\n project = { size = 50 }\n scratch = { size = 50 }\n }\n }\n\n public_keys = [file(\"~/.ssh/id_rsa.pub\")]\n\n # Alicloud specifics\n region = \"us-west-1\"\n}\nTo develop for Magic Castle you will need: * Terraform (>= 1.4.0) * git * Access to a Cloud (e.g.: Compute Canada Arbutus) * Ability to communicate with the cloud provider API from your computer * A cloud project with enough room for the resource described in section Magic Caslte Doc 1.1. * [optional] Puppet Development Kit (PDK)
"},{"location":"developers/#2-where-to-start","title":"2. Where to start","text":"The Magic Castle project is defined by Terraform infrastructure-as-code component that is responsible of generating a cluster architecture in a cloud and a Puppet environment component that configures the cluster instances based on their role.
If you wish to add device, an instance, add a new networking interface or a filesystem, you will most likely need to develop some Terraform code. The project structure for Terraform code is described in the reference design document. The document also describes how one could work with current Magic Castle code to add support for another cloud provider.
If you wish to add a service to one of the Puppet environments, install a new software, modify an instance configuration or role, you will most likely need to develop some Puppet code. The following section provides more details on the Puppet environments available and how to develop them.
"},{"location":"developers/#3-puppet-environment","title":"3. Puppet environment","text":"Magic Castle Terraform code initialized every instances to be a Puppet agent and an instance with the tag puppet as the Puppet main server. On the Puppet main server, there is a folder containing the configuration code for the instances of the cluster, this folder is called a Puppet environment and it is pulled from GitHub during the initial configuration of the Puppet main server.
The source of that environment is provided to Terraform using the variable config_git_url.
A repository describing a Magic Castle Puppet environment must contain at the least the following files and folders:
config_git_repo\n\u2523 Puppetfile\n\u2523 environment.conf\n\u2523 hiera.yaml\n\u2517 data\n \u2517 common.yaml\n\u2517 manifests/\n \u2517 site.pp\nPuppetfile specifies the Puppet modules that need to be installed in the environment.environment.conf overrides the primary server default settings for the environment.hiera.yaml configures an ordered list of YAML file data sources.data/common.yaml is common data source for the instances part of hierarchy defined by hiera.yaml.manifests/site.pp defines how each instance will be configured based on their hostname and/or tags.An example of a bare-bone Magic Castle Puppet environment is available on GitHub: MagicCastle/puppet-environment, while the Puppet environment that replicates a Compute Canada HPC cluster is named ComputeCanada/puppet-magic_castle.
"},{"location":"developers/#terraform_datayaml-a-bridge-between-terraform-and-puppet","title":"terraform_data.yaml: a bridge between Terraform and Puppet","text":"To provide information on the deployed resources and the value of the input parameters, Magic Castle Terraform code uploads to the Puppet main server a file named terraform_data.yaml, in the folder /etc/puppetlabs/data/. There is also a symlink created in /etc/puppetlabs/code/environment/production/data/ to ease its usage inside the Puppet environment.
When included in the data hierarchy (hiera.yaml), terraform_data.yaml can provide information about the instances, the volumes and the variables set by the user through the main.tf file. The file has the following structure:
---\nterraform:\n data:\n cluster_name: \"\"\n domain_name: \"\"\n guest_passwd: \"\"\n nb_users: \"\"\n public_keys: []\n sudoer_username: \"\"\n instances:\n host1:\n hostkeys:\n rsa: \"\"\n ed25519: \"\"\n local_ip: \"x.x.x.x\"\n prefix: \"host\"\n public_ip: \"\"\n specs:\n \"cpus\": 0\n \"gpus\": 0\n \"ram\": 0\n tags:\n - \"tag_1\"\n - \"tag_2\"\n tag_ip:\n tag_1:\n - x.x.x.x\n tag_2:\n - x.x.x.x\n volumes:\n volume_tag1:\n volume_1:\n - \"/dev/disk/by-id/123-*\"\n volume_2:\n - \"/dev/disk/by-id/123-abc-*\"\nThe values provided by terraform_data.yaml can be accessed in Puppet by using the lookup() function. For example, to access an instance's list of tags:
lookup(\"terraform.instances.${::hostname}.tags\")\nalias() function. For example, to define the number of guest accounts using the value of nb_users, we could add this to common.yaml profile::accounts::guests::nb_accounts: \"%{alias('terraform.data.nb_users')}\"\nThe configuration of each instance is defined in manifests/site.pp file of the Puppet environment. In this file, it is possible to define a configuration based on an instance hostname
node \"mgmt1\" { }\ndefault node : node default {\n $instance_tags = lookup(\"terraform.instances.${::hostname}.tags\")\n if 'tag_1' in $instances_tags { }\n}\nIt is possible to define Puppet resource directly in site.pp. However, above a certain level of complexity, which can be reach fairly quickly, it is preferable to define classes and include these classes in site.pp based on the node hostname or tags.
Classes can be defined in the Puppet environment under the following path: site/profile/manifests. These classes are named profile classes and the philosophy behind it is explained in Puppet documentation. Because these classes are defined in site/profile, their name has to start with the prefix profile::.
It is also possible to include classes defined externally and installed using the Puppetfile. These classes installed by r10k can be found in the modules folder of the Puppet environment.
To test new additions to puppet.yaml, it is possible to execute cloud-init phases manually. There are four steps that can be executed sequentially: init local, init modules config and modules final. Here are the corresponding commands to execute each step:
cloud-init init --local\ncloud-init init\ncloud-init modules --mode=config\ncloud-init modules --mode=final\nIt is also possible to clean a cloud-init execution and have it execute again at next reboot. To do so, enter the following command:
cloud-init clean\n-r to the previous command to reboot the instance once cloud-init has finishing cleaning."},{"location":"developers/#42-selinux","title":"4.2 SELinux","text":"SELinux is enabled on every instances of a Magic Castle cluster. Some applications do not provide SELinux policies which can lead to their malfunctionning when SELinux is enabled. It is possible to track down the reasons why SELinux is preventing an application to work properly using the command-line tool ausearch.
If you suspect application app-a to be denied by SELinux to work properly, run the following command as root:
ausearch -c app-a --raw | grep denied\nTo see all requests denied by SELinux:
ausearch --raw | grep denied\nSometime, the denials are hidden from regular logging. To display all denials, run the following command as root:
semodule --disable_dontaudit --build\nOnce you have found the denials that are the cause of the problem, you can create a new policy to allow the requests that were previously denied with the following command:
ausearch -c app-a --raw | grep denied | audit2allow -a -M app-a\nFinally, you can install the generated policy using the command provided by auditallow.
If you need to tweak an existing enforcement file and you want to recompile the policy package, you can with the following commands:
checkmodule -M -m -o my_policy.mod my_policy.te\nsemodule_package -o my_policy.pp -m my_policy.mod\nTo build a release, use the script release.sh located at the root of Magic Castle git repo.
Usage: release.sh VERSION [provider ...]\nreleases where it was called. The VERSION argument is expected to correspond to git tag in the puppet-magic_castle repo. It could also be a branch name or a commit. If the provider optional argument is left blank, release files will be built for all providers currently supported by Magic Castle.
Examples:
$ ./release.sh main openstack\n$ ./release.sh 5.8 gcp$ ./release.sh 5.7 azure ovh\n* The documentation provides instructions on how to add support for other cloud providers.
"},{"location":"matrix/#supported-operating-systems","title":"Supported operating systems","text":"Name CentOS 7 CentOS 8 Rocky Linux 8 AlmaLinux 8 Debian 10 Ubuntu 18 Ubuntu 20 Windows 10 AWS ParallelCluster yes yes yes yes yes no yes no Azure CycleCloud yes yes yes yes yes no yes - Azure HPC On-Demand Platform yes no no yes no yes no yes Google HPC-Toolkit yes no no no no no no no Cluster in the Cloud no yes no no no no no no ElastiCluster yes yes yes yes no no no no Magic Castle no yes yes yes no no no no On-Demand Data Centre - - - - - - - - Slurm on GCP yes no yes no yes no yes no"},{"location":"matrix/#supported-job-schedulers","title":"Supported job schedulers","text":"Name AwsBatch Grid Engine HTCondor Moab Open PBS PBS Pro Slurm AWS ParallelCluster yes no no no no no yes Azure CycleCloud no yes yes no no yes yes Azure HPC On-Demand Platform no no no no yes no yes Google HPC-Toolkit no no no no no no yes Cluster in the Cloud no no no no no no yes ElastiCluster no yes no no no no yes Magic Castle no no no no no no yes On-Demand Data Centre no no no yes no no no Slurm on GCP no no no no no no yes"},{"location":"matrix/#technologies","title":"Technologies","text":"Name Infrastructure configuration Programming languages Configuration management Scientific software AWS ParallelCluster CLI generating YAML Python Chef Spack Azure CycleCloud WebUI or CLI + templates Python Chef Bring your own Azure HPC On-Demand Platform YAML files + shell scripts Shell, Terraform Ansible, Packer CVMFS Cluster in the Cloud CLI generating Terraform code Python, Terraform Ansible, Packer EESSI ElastiCluster CLI interpreting an INI file Python, Shell Ansible Bring your own Google HPC-Toolkit CLI generating Terraform code Go, Terraform Ansible, Packer Spack Magic Castle Terraform modules Terraform Puppet CC-CVMFS, EESSI On-Demand Data Centre - - - - Slurm GCP Terraform modules Terraform Ansible, Packer Spack"},{"location":"sequence/","title":"Magic Castle Sequence Diagrams","text":"The following sequence diagrams illustrate the inner working of Magic Castle once terraform apply is called. Some details were left out of the diagrams, but every diagram is followed by references to the code files that were used to build it.
puppet-magic_castle.git does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.magic_castle:/common/design/main.tfmagic_castle:/openstack/network-1.tfmagic_castle:/openstack/network-2.tfmagic_castle:/common/configuration/main.tfmagic_castle:/openstack/infrastructure.tfmagic_castle:/common/provision/main.tfmagic_castle:/dns/cloudflare/main.tfconfig_git_url repo does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.magic_castle:/common/configuration/puppet.yamlpuppet-magic_castle:/manifests/site.pppuppet-magic_castle:/profile/manifests/base.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/freeipa.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/cvmfs.pppuppet-magic_castle:/profile/manifests/slurm.ppThis document explains how to use Magic Castle with Terraform Cloud.
"},{"location":"terraform_cloud/#what-is-terraform-cloud","title":"What is Terraform Cloud?","text":"Terraform Cloud is HashiCorp\u2019s managed service that allows to provision infrastructure using a web browser or a REST API instead of the command-line. This also means that the provisioned infrastructure parameters can be modified by a team and the state is stored in the cloud instead of a local machine.
When provisioning in commercial cloud, Terraform Cloud can also provide a cost estimate of the resources.
"},{"location":"terraform_cloud/#getting-started-with-terraform-cloud","title":"Getting started with Terraform Cloud","text":"main.tf available for the cloud of your choicemain.tfYou will be redirected automatically to your new workspace.
"},{"location":"terraform_cloud/#providing-cloud-provider-credentials-to-terraform-cloud","title":"Providing cloud provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your cloud provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each cloud provider and how to retrieve the values of the variable from the provider.
If you plan on using these environment variables with multiple workspaces, it is recommended to create a credential variable set in Terraform Cloud.
"},{"location":"terraform_cloud/#aws","title":"AWS","text":"You need to define these environment variables: - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY (sensitive)
The value of these variables can either correspond to the value of access key created on the AWS Security Credentials - Access keys page, or you can add user dedicated to Terraform Cloud in AWS IAM Users, and use its access key.
"},{"location":"terraform_cloud/#azure","title":"Azure","text":"You need to define these environment variables: - ARM_CLIENT_ID - ARM_CLIENT_SECRET (sensitive) - ARM_SUBSCRIPTION_ID - ARM_TENANT_ID
Refer to Terraform Azure Provider - Creating a Service Principal to know how to create a Service Principal and retrieve the values for these environment variables.
"},{"location":"terraform_cloud/#google-cloud","title":"Google Cloud","text":"You need to define this environment variable: - GOOGLE_CLOUD_KEYFILE_JSON (sensitive)
The value of the variable will be the content of a Google Cloud service account JSON key file expressed a single line string. Example:
{\"type\": \"service_account\",\"project_id\": \"project-id-1234\",\"private_key_id\": \"abcd1234\",...}\nYou can use jq to format the string from the JSON file provided by Google:
jq . -c project-name-123456-abcdefjg.json\nYou need to define these environment variables: - OS_AUTH_URL - OS_PROJECT_ID - OS_REGION_NAME - OS_INTERFACE - OS_IDENTITY_API_VERSION - OS_USER_DOMAIN_NAME - OS_USERNAME - OS_PASSWORD (sensitive)
Apart from OS_PASSWORD, the values for these variables are available in OpenStack RC file provided for your project.
If you prefer to use OpenStack application credentials, you need to define at least these variables: - OS_AUTH_TYPE\u00a0 - OS_AUTH_URL - OS_APPLICATION_CREDENTIAL_ID - OS_APPLICATION_CREDENTIAL_SECRET
and potentially these too: - OS_IDENTITY_API_VERSION\u00a0 - OS_REGION_NAME - OS_INTERFACE
The values for these variables are available in OpenStack RC file provided when creating the application credentials.
"},{"location":"terraform_cloud/#providing-dns-provider-credentials-to-terraform-cloud","title":"Providing DNS provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your DNS provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each DNS provider and how to retrieve the values of the variable from the provider.
"},{"location":"terraform_cloud/#cloudflare","title":"CloudFlare","text":"Refer to DNS - CloudFlare section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#google-cloud-dns","title":"Google Cloud DNS","text":"Refer to DNS - Google Cloud section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#managing-magic-castle-variables-with-terraform-cloud-ui","title":"Managing Magic Castle variables with Terraform Cloud UI","text":"It is possible to use Terraform Cloud web interface to define variable values in your main.tf. For example, you could want to define a guest password without writing it directly in main.tf to avoid displaying publicly.
To manage a variable with Terraform Cloud: 1. edit your main.tf to define the variables you want to manage. In the following example, we want to manage the number of nodes and the guest password.
Add the variables at the beginning of the `main.tf`:\n ```hcl\n variable \"nb_nodes\" {}\n variable \"password\" {}\n ```\n\nThen replace the static value by the variable in our `main.tf`,\n\ncompute node count\n ```hcl\n node = { type = \"p2-3gb\", tags = [\"node\"], count = var.nb_nodes }\n ```\nguest password\n ```hcl\n guest_passwd = var.password\n ```\nmain.tf. Check \"Sensitive\" if the variable content should not never be shown in the UI or the API.You may edit the variables at any point of your cluster lifetime.
"},{"location":"terraform_cloud/#applying-changes","title":"Applying changes","text":"To create your cluster, apply changes made to your main.tf or the variables, you will need to queue a plan. When you push to the default branch of the linked git repository, a plan will be automatically created. You can also create a plan manually. To do so, click on the \"Queue plan manually\" button inside your workspace, then \"Queue plan\".
Once the plan has been successfully created, you can apply it using the \"Runs\" section. Click on the latest queued plan, then on the \"Apply plan\" button at the bottom of the plan page.
"},{"location":"terraform_cloud/#auto-apply","title":"Auto apply","text":"It is possible to apply automatically a successful plan. Go in the \"Settings\" section, and under \"Apply method\" select \"Auto apply\". Any following successful plan will then be automatically applied.
"},{"location":"terraform_cloud/#magic-castle-terraform-cloud-and-the-cli","title":"Magic Castle, Terraform Cloud and the CLI","text":"Terraform cloud only allows to apply or destroy the plan as stated in the main.tf, but sometimes it can be useful to run some other terraform commands that are only available through the command-line interface, for example terraform taint.
It is possible to import the terraform state of a cluster on your local computer and then use the CLI on it.
Log in Terraform cloud:
terraform login\nCreate a folder where the terraform state will be stored:
mkdir my-cluster-1\nCreate a file named cloud.tf with the following content in your cluster folder:
terraform {\n cloud {\n organization = \"REPLACE-BY-YOUR-TF-CLOUD-ORG\"\n workspaces {\n name = \"REPLACE-BY-THE-NAME-OF-YOUR-WORKSPACE\"\n }\n }\n}\norganization and name with the appropriate value for your cluster. Initialize the folder and retrieve the state:
terraform init\nTo confirm the workspace has been properly imported locally, you can list the resources using:
terraform state list\nMagic Castle in combination with Terraform Cloud (TFE) can be configured to give Slurm the ability to create and destroy instances based on the job queue content.
To enable this feature: 1. Create a TFE API Token and save it somewhere safe.
1.1. If you subscribe to Terraform Cloud Team & Governance plan, you can generate\na [Team API Token](https://www.terraform.io/cloud-docs/users-teams-organizations/api-tokens#team-api-tokens).\nThe team associated with this token requires no access to organization and can be secret.\nIt does not have to include any member. Team API token is preferable as its permissions can be\nrestricted to the minimum required for autoscale purpose.\nCreate a workspace in TFE
2.1. Make sure the repo is private as it will contain the API token.
2.2. If you generated a Team API Token in 1, provide access to the workspace to the team:
2.3 In Configure settings, under Advanced options, for Apply method, select Auto apply.
Create the environment variables of the cloud provider credentials in TFE
pool in TFE. Set value to [] and check HCL.data.yaml in your git repo with the following content: yaml\u00a0 --- profile::slurm::controller::tfe_token: <TFE API token> profile::slurm::controller::tfe_workspace: <TFE workspace id> Complete the file by replacing <TFE API TOKEN> with the token generated at step 1 and <TFE workspace id> (i.e.: ws-...) by the id of the workspace created at step 2. It is recommended to encrypt the TFE API token before committing data.yaml in git. Refer to section 4.15 of README.md to know how to encrypt the token.data.yaml in git and push.Modify main.tf:
main.tf.variable \"pool\" { description = \"Slurm pool of compute nodes\" }\ninstances with the tags pool and node. These are the nodes that Slurm will able to create and destroy.pool = var.pool\npublic_keys =, replace [file(\"~/.ssh/id_rsa.pub\")] by a list of SSH public keys that will have admin access to the cluster.public_keys = ..., add hieradata = file(\"data.yaml\").Go to your workspace in TFE, click on Actions -> Start a new run -> Plan and apply -> Start run. Then, click on \"Confirm & Apply\" and \"Confirm Plan\".
To reduce the time required for compute nodes to become available in Slurm, consider creating a compute node image.
JupyterHub will time out by default after 300 seconds if a node is not spawned yet. Since it may take longer than this to spawn a node, even with an image created, consider increasing the timeout by adding the following to your YAML configuration file:
jupyterhub::jupyterhub_config_hash:\n SlurmFormSpawner:\n start_timeout: 900\nSlurm 23 adds the possibility for sinfo to report nodes that are not yet spawned. This is useful if you want JupyterHub to be aware of those nodes, for example if you want to allow to use GPU nodes without keeping them online at all time. To use that version of Slurm, add the following to your YAML configuration file:
profile::slurm::base::slurm_version: '23.02'\nIf after enabling autoscaling with Terraform Cloud for your Magic Castle cluster, the number of nodes does not increase when submitting jobs, verify the following points:
profile::slurm::controller::tfe_workspace in data.yaml.squeue on the cluster, and verify the reasons why jobs are still in the queue. If under the column (Reason), there is the keyword ReqNodeNotAvail, it implies Slurm tried to boot the listed nodes, but they would not show up before the timeout, therefore Slurm marked them as down. It can happen if your cloud provider is slow to build the instances, or following a configuration problem like in 2. When Slurm marks a node as down, a trace is left in slurmctld's log - using zgrep on the slurm controller node (typically mgmt1): sudo zgrep \"marking down\" /var/log/slurm/slurmctld.log*\nsudo /opt/software/slurm/bin/scontrol update nodename=node[Y-Z] state=IDLE\nnode[Y-Z] by the hostname range listed next to ReqNodeNotAvail in squeue.mgmt1:/var/log/slurm, look for errors in the file slurm_resume.log.