echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | base64 -w0 | xargs printf "ENC[PKCS7,%s]\n"
+or with eyaml:
+eyaml createkeys --pkcs7-public-key=public_key.pkcs7.pem --pkcs7-private-key=private_key.pkcs7.pem
+
4.14.2 Encrypting sensitive properties
+To encrypt a sensitive property with openssl:
+
echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | openssl base64 -A | xargs printf "ENC[PKCS7,%s]\n"
+
To encrypt a sensitive property with eyaml:
+
eyaml encrypt -s 'your-secret' --pkcs7-public-key=public_key.pkcs7.pem -o string
To learn more about public_key.pkcs7.pem and how it can be generated before the cluster creation, refer to
-section 10.13 Generate and replace Puppet hieradata encryption keys.
-4.14 firewall_rules (optional)
+
+To provide the value of this variable via Terraform Cloud, encode the private key content with base64:
+openssl base64 -A -in private_key.pkcs7.pem
+
Define a variable in your main.tf:
+variable "tfc_eyaml_key" {}
+module "openstack" {
+ ...
+}
+
Then make sure to decode it before passing it to the cloud provider module:
+variable "tfc_eyaml_key" {}
+module "openstack" {
+ ...
+ eyaml_key = base64decode(var.tfc_eyaml_key)
+ ...
+}
+
4.15 firewall_rules (optional)
default value:
{
ssh = { "from_port" = 22, "to_port" = 22, tag = "login", "protocol" = "tcp", "cidr" = "0.0.0.0/0" },
@@ -3430,7 +3488,7 @@ 4.14 firewall_rules (optional)
about this requirement, refer to Magic Castle's
bastion tag computation code.
Post build modification effect: modify the cloud provider firewall rules at next terraform apply.
-4.15 generate_ssh_key (optional)
+4.16 generate_ssh_key (optional)
default_value: false
If true, Terraform will generate an ssh key pair that would then be used when copying file with Terraform
file-provisioner. The public key will be added to the sudoer account authorized keys.
@@ -3447,7 +3505,7 @@ 4.15 generate_ssh_key (optional)
from the sudoer account authorized_keys file at next
Puppet agent run.
-4.16 software_stack (optional)
+4.17 software_stack (optional)
default_value: "alliance"
Defines the scientific software environment that users have access when they login.
Possible values are:
@@ -3457,7 +3515,7 @@ 4.16 software_stack (optional)
null / "": no scientific software environment
Post build modification effect: trigger scp of hieradata files at next terraform apply.
-4.17 pool (optional)
+4.18 pool (optional)
default_value: []
Defines a list of hostnames with the tag "pool" that have to be online. This variable is typically
managed by the workload scheduler through Terraform API. For more information, refer to
@@ -3465,14 +3523,14 @@
4.17 pool (optional)
Post build modification effect: pool tagged hosts with name present in the list
will be instantiated, others will stay uninstantiated or will be destroyed
if previously instantiated.
-4.18 skip_upgrade (optional)
+4.19 skip_upgrade (optional)
default_value = false
If true, the base image packages will not be upgraded during the first boot. By default,
all packages are upgraded.
Post build modification effect: No effect on currently built instances. Ones created
after the modification will take into consideration the new value of the parameter to determine
whether they should upgrade the base image packages or not.
-4.19 puppetfile (optional)
+4.20 puppetfile (optional)
default_value = ""
Defines a complement of modules to install with librarian puppet when initializing the Puppet environment
on the first boot of the Puppet server. If the provided string include the
@@ -4083,46 +4141,7 @@
10.13 Generate and replace Puppet hieradata encryption keys
-During the Puppet server initial boot, a pair of hiera-eyaml encryptions keys are generated in
-/opt/puppetlabs/puppet/eyaml:
-
-private_key.pkcs7.pempublic_key.pkcs7.pem
-To encrypt the values before creating the cluster, the encryptions keys can be generated beforehand and then transferred on the Puppet server.
-The keys can be generated with eyaml:
-
-or openssl:
-
openssl req -x509 -nodes -days 100000 -newkey rsa:2048 -keyout private_key.pkcs7.pem -out public_key.pkcs7.pem -subj '/'
-
The resulting public key can then be used to encrypt secrets, while the private and the public keys have to be transferred on the Puppet server to allow it to decrypt the values. In the following command examples, replace
-puppet by the hostname of your puppetserver (i.e.: mgmt1).
-
-- Transfer the keys on the Puppet server using SCP with SSH jumphost
-
scp -J centos@cluster.yourdomain.cloud {public,private}_key.pkcs7.pem centos@puppet:~/
-
-- Replace the existing keys by the one transferred:
-
ssh -J centos@cluster.yourdomain.cloud centos@puppet sudo cp {public,private}_key.pkcs7.pem /etc/puppetlabs/puppet/eyaml
-
-- Remove the keys from the admin account home folder:
-
ssh -J centos@cluster.yourdomain.cloud centos@puppet rm {public,private}_key.pkcs7.pem
-
-
-To backup the encryption keys from an existing Puppet server:
-
-- Create a readable copy of the encryption keys in the sudoer home account
-
ssh -J centos@cluster.yourdomain.cloud centos@puppet 'sudo rsync --owner --group --chown=centos:centos /etc/puppetlabs/puppet/eyaml/{public,private}_key.pkcs7.pem ~/'
-
-- Transfer the files locally:
-
scp -J centos@cluster.yourdomain.cloud centos@puppet:~/{public,private}_key.pkcs7.pem .
-
-- Remove the keys from the sudoer account home folder:
-
ssh -J centos@cluster.yourdomain.cloud centos@puppet rm {public,private}_key.pkcs7.pem
-
-
-10.14 Read and edit secret values generated at boot
+10.13 Read and edit secret values generated at boot
During the cloud-init initialization phase,
bootstrap.sh
script is executed. This script generates a set of encrypted secret values that are required
@@ -4146,7 +4165,7 @@
10.14 Read and edit
the hieradata configuration file. Refer to section 4.13 hieradata.
User defined values take precedence over boot generated values in the Magic Castle
Puppet data hierarchy.
-10.15 Expand a volume
+10.14 Expand a volume
Volumes defined in the volumes map can be expanded at will. After their creation, you can
increase their size in the main.tf then call terraform apply and the associated block
device will be expanded.
diff --git a/search/search_index.json b/search/search_index.json
index 1e7fb0e5..4f724241 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Magic Castle Documentation","text":""},{"location":"#1-setup","title":"1. Setup","text":"To use Magic Castle you will need:
- Terraform (>= 1.4.0)
- Authenticated access to a cloud
- Ability to communicate with the cloud provider API from your computer
- A project with operational limits meeting the requirements described in Quotas subsection.
- ssh-agent running and tracking your SSH key
"},{"location":"#11-terraform","title":"1.1 Terraform","text":"To install Terraform, follow the tutorial or go directly on Terraform download page.
You can verify Terraform was properly installed by looking at the version in a terminal:
terraform version\n
"},{"location":"#12-authentication","title":"1.2 Authentication","text":""},{"location":"#121-amazon-web-services-aws","title":"1.2.1 Amazon Web Services (AWS)","text":" - Go to AWS - My Security Credentials
- Create a new access key.
- In a terminal, export
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, environment variables, representing your AWS Access Key and AWS Secret Key: export AWS_ACCESS_KEY_ID=\"an-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"a-secret-key\"\n
Reference: AWS Provider - Environment Variables
"},{"location":"#122-google-cloud","title":"1.2.2 Google Cloud","text":" - Install the Google Cloud SDK
- In a terminal, enter :
gcloud auth application-default login
"},{"location":"#123-microsoft-azure","title":"1.2.3 Microsoft Azure","text":" - Install Azure CLI
- In a terminal, enter :
az login
Reference : Azure Provider: Authenticating using the Azure CLI
"},{"location":"#124-openstack-ovh","title":"1.2.4 OpenStack / OVH","text":" -
Download your OpenStack Open RC file. It is project-specific and contains the credentials used by Terraform to communicate with OpenStack API. To download, using OpenStack web page go to: Project \u2192 API Access, then click on Download OpenStack RC File then right-click on OpenStack RC File (Identity API v3), Save Link as..., and save the file.
-
In a terminal located in the same folder as your OpenStack RC file, source the OpenStack RC file:
source *-openrc.sh\n
This command will ask for a password, enter your OpenStack password.
"},{"location":"#13-cloud-api","title":"1.3 Cloud API","text":"Once you are authenticated with your cloud provider, you should be able to communicate with its API. This section lists for each provider some instructions to test this.
"},{"location":"#131-aws","title":"1.3.1 AWS","text":" - In a dedicated temporary folder, create a file named
test_aws.tf with the following content: provider \"aws\" {\n region = \"us-east-1\"\n}\n\ndata \"aws_ec2_instance_type\" \"example\" {\n instance_type = \"t2.micro\"\n}\n
- In a terminal, move to where the file is located, then:
terraform init\n
- Finally, test terraform communication with AWS:
terraform plan\n
If everything is configured properly, terraform will output: No changes. Your infrastructure matches the configuration.\n
Otherwise, it will output: Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.\n
- You can delete the temporary folder and its content.
"},{"location":"#132-google-cloud","title":"1.3.2 Google Cloud","text":"In a terminal, enter:
gcloud projects list\n
It should output a table with 3 columns PROJECT_ID NAME PROJECT_NUMBER\n
Take note of the project_id of the Google Cloud project you want to use, you will need it later.
"},{"location":"#133-microsoft-azure","title":"1.3.3 Microsoft Azure","text":"In a terminal, enter:
az account show\n
It should output a JSON dictionary similar to this: {\n \"environmentName\": \"AzureCloud\",\n \"homeTenantId\": \"98467e3b-33c2-4a34-928b-ed254db26890\",\n \"id\": \"4dda857e-1d61-457f-b0f0-e8c784d1fb20\",\n \"isDefault\": true,\n \"managedByTenants\": [],\n \"name\": \"Pay-As-You-Go\",\n \"state\": \"Enabled\",\n \"tenantId\": \"495fc59f-96d9-4c3f-9c78-7a7b5f33d962\",\n \"user\": {\n \"name\": \"user@example.com\",\n \"type\": \"user\"\n }\n}\n
"},{"location":"#134-openstack-ovh","title":"1.3.4 OpenStack / OVH","text":" - In a dedicated temporary folder, create a file named
test_os.tf with the following content: terraform {\n required_providers {\n openstack = {\n source = \"terraform-provider-openstack/openstack\"\n }\n }\n}\ndata \"openstack_identity_auth_scope_v3\" \"scope\" {\n name = \"my_scope\"\n}\n
- In a terminal, move to where the file is located, then:
terraform init\n
- Finally, test terraform communication with OpenStack:
terraform plan\n
If everything is configured properly, terraform will output: No changes. Your infrastructure matches the configuration.\n
Otherwise, it will output: Error: Error creating OpenStack identity client:\n
if the OpenStack cloud API cannot be reached. - You can delete the temporary folder and its content.
"},{"location":"#14-quotas","title":"1.4 Quotas","text":""},{"location":"#141-aws","title":"1.4.1 AWS","text":"The default quotas set by Amazon are sufficient to build the Magic Castle AWS examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Amazon EC2 service quotas.
"},{"location":"#142-google-cloud","title":"1.4.2 Google Cloud","text":"The default quotas set by Google Cloud are sufficient to build the Magic Castle GCP examples. To increase the limits, or request access to special resources like GPUs, refer to Google Compute Engine Resource quotas.
"},{"location":"#143-microsoft-azure","title":"1.4.3 Microsoft Azure","text":"The default quotas set by Microsoft Azure are sufficient to build the Magic Castle Azure examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Azure subscription and service limits, quotas, and constraints.
"},{"location":"#144-openstack","title":"1.4.4 OpenStack","text":"Minimum project requirements:
- 1 floating IP
- 3 security groups
- 1 network (see note 1)
- 1 subnet (see note 1)
- 1 router (see note 1)
- 3 volumes
- 3 instances
- 8 VCPUs
- 7 neutron ports
- 12 GB of RAM
- 8 security rules
- 80 GB of volume storage
Note 1: Magic Castle supposes the OpenStack project comes with a network, a subnet and a router already initialized. If any of these components is missing, you will need to create them manually before launching terraform.
- Create and manage networks, JUSUF user documentation
- Create and manage network - UI, OpenStack Documentation
- Create and manage network - CLI, OpenStack Documentation
"},{"location":"#145-ovh","title":"1.4.5 OVH","text":"The default quotas set by OVH are sufficient to build the Magic Castle OVH examples. To increase the limits, or request access to special resources like GPUs, refer to OVHcloud - Increasing Public Cloud quotas.
"},{"location":"#15-ssh-agent","title":"1.5 ssh-agent","text":"To transfer configuration files, Terraform will connect to your cluster using SSH. To avoid providing your private key to Terraform directly, you will have to add it to the authentication agent, ssh-agent.
To learn how to start ssh-agent and add keys, refer to ssh-agent - How to configure, forwarding, protocol.
Note 1: If you own more than one key pair, make sure the private key added to ssh-agent corresponds to the public key that will be granted access to your cluster (refer to section 4.9 public_keys).
Note 2: If you have no wish to use ssh-agent, you can configure Magic Castle to generate a key pair specific to your cluster. The public key will be written in the sudoer authorized_keys and Terraform will be able to connect the cluster using the corresponding private key. For more information, refer to section 4.15 generate_ssh_key.
"},{"location":"#2-cloud-cluster-architecture-overview","title":"2. Cloud Cluster Architecture Overview","text":""},{"location":"#3-initialization","title":"3. Initialization","text":""},{"location":"#31-main-file","title":"3.1 Main File","text":" - Go to https://github.com/ComputeCanada/magic_castle/releases.
- Download the latest release of Magic Castle for your cloud provider.
- Open a Terminal.
- Uncompress the release:
tar xvf magic_castle*.tar.gz - Rename the release folder after your favourite superhero:
mv magic_castle* hulk - Move inside the folder:
cd hulk
The file main.tf contains Terraform modules and outputs. Modules are files that define a set of resources that will be configured based on the inputs provided in the module block. Outputs are used to tell Terraform which variables of our module we would like to be shown on the screen once the resources have been instantiated.
This file will be our main canvas to design our new clusters. As long as the module block parameters suffice to our need, we will be able to limit our configuration to this sole file. Further customization will be addressed during the second part of the workshop.
"},{"location":"#32-terraform","title":"3.2 Terraform","text":"Terraform fetches the plugins required to interact with the cloud provider defined by our main.tf once when we initialize. To initialize, enter the following command:
terraform init\n
The initialization is specific to the folder where you are currently located. The initialization process looks at all .tf files and fetches the plugins required to build the resources defined in these files. If you replace some or all .tf files inside a folder that has already been initialized, just call the command again to make sure you have all plugins.
The initialization process creates a .terraform folder at the root of your current folder. You do not need to look at its content for now.
"},{"location":"#321-terraform-modules-upgrade","title":"3.2.1 Terraform Modules Upgrade","text":"Once Terraform folder has been initialized, it is possible to fetch the newest version of the modules used by calling:
terraform init -upgrade\n
"},{"location":"#4-configuration","title":"4. Configuration","text":"In the main.tf file, there is a module named after your cloud provider, i.e.: module \"openstack\". This module corresponds to the high-level infrastructure of your cluster.
The following sections describes each variable that can be used to customize the deployed infrastructure and its configuration. Optional variables can be absent from the example module. The order of the variables does not matter, but the following sections are ordered as the variables appear in the examples.
"},{"location":"#41-source","title":"4.1 source","text":"The first line of the module block indicates to Terraform where it can find the files that define the resources that will compose your cluster. In the releases, this variable is a relative path to the cloud provider folder (i.e.: ./aws).
Requirement: Must be a path to a local folder containing the Magic Castle Terraform files for the cloud provider of your choice. It can also be a git repository. Refer to Terraform documentation on module source for more information.
Post build modification effect: terraform init will have to be called again and the next terraform apply might propose changes if the infrastructure describe by the new module is different.
"},{"location":"#42-config_git_url","title":"4.2 config_git_url","text":"Magic Castle configuration management is handled by Puppet. The Puppet configuration files are stored in a git repository. This is typically ComputeCanada/puppet-magic_castle repository on GitHub.
Leave this variable to its current value to deploy a vanilla Magic Castle cluster.
If you wish to customize the instances' role assignment, add services, or develop new features for Magic Castle, fork the ComputeCanada/puppet-magic_castle and point this variable to your fork's URL. For more information on Magic Castle puppet configuration customization, refer to MC developer documentation.
Requirement: Must be a valid HTTPS URL to a git repository describing a Puppet environment compatible with Magic Castle. If the repo is private, generate an access token with a permission to read the repo content, and provide the token in the config_git_url like this:
config_git_url = \"https://oauth2:${oauth-key-goes-here}@domain.com/username/repo.git\"\n
This works for GitHub and GitLab (including community edition). Post build modification effect: no effect. To change the Puppet configuration source, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#43-config_version","title":"4.3 config_version","text":"Since Magic Cluster configuration is managed with git, it is possible to specify which version of the configuration you wish to use. Typically, it will match the version number of the release you have downloaded (i.e: 9.3).
Requirement: Must refer to a git commit, tag or branch existing in the git repository pointed by config_git_url.
Post build modification effect: none. To change the Puppet configuration version, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#44-cluster_name","title":"4.4 cluster_name","text":"Defines the ClusterName variable in slurm.conf and the name of the cluster in the Slurm accounting database (see slurm.conf documentation).
Requirement: Must be lowercase alphanumeric characters and start with a letter. It can include dashes. cluster_name must be 40 characters or less.
Post build modification effect: destroy and re-create all instances at next terraform apply.
"},{"location":"#45-domain","title":"4.5 domain","text":"Defines
- the Kerberos realm name when initializing FreeIPA.
- the internal domain name and the
resolv.conf search domain as int.{cluster_name}.{domain}
Optional modules following the current module in the example main.tf can be used to register DNS records in relation to your cluster if the DNS zone of this domain is administered by one of the supported providers. Refer to section 6. DNS Configuration and SSL Certificates for more details.
Requirements:
Post build modification effect: destroy and re-create all instances at next terraform apply.
"},{"location":"#46-image","title":"4.6 image","text":"Defines the name of the image that will be used as the base image for the cluster nodes.
You can use a custom image if you wish, but configuration management should be mainly done through Puppet. Image customization is mostly envisioned as a way to accelerate the configuration process by applying the security patches and OS updates in advance.
To specify a different image for an instance type, use the image instance attribute
Requirements: the operating system on the image must be from the RedHat family. This includes CentOS (7, 8), Rocky Linux (8), and AlmaLinux (8).
Post build modification effect: none. If this variable is modified, existing instances will ignore the change and future instances will use the new value.
"},{"location":"#461-aws","title":"4.6.1 AWS","text":"The image field needs to correspond to the Amazon Machine Image (AMI) ID. AMI IDs are specific to regions and architectures. Make sure to use the right ID for the region and CPU architecture you are using (i.e: x86_64).
To find out which AMI ID you need to use, refer to - AlmaLinux OS Amazon Web Services AMIs - CentOS list of official images available on the AWS Marketplace - Rocky Linux
Note: Before you can use the AMI, you will need to accept the usage terms and subscribe to the image on AWS Marketplace. On your first deployment, you will be presented an error similar to this one:
\u2502 Error: Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=cvugziknvmxgqna9noibqnnsy\n\u2502 status code: 401, request id: 1f04a85a-f16a-41c6-82b5-342dc3dd6a3d\n\u2502\n\u2502 on aws/infrastructure.tf line 67, in resource \"aws_instance\" \"instances\":\n\u2502 67: resource \"aws_instance\" \"instances\" {\n
To accept the terms and fix the error, visit the link provided in the error output, then click on the Click to Subscribe yellow button."},{"location":"#462-microsoft-azure","title":"4.6.2 Microsoft Azure","text":"The image field for Azure can either be a string or a map.
A string image specification will correspond to the image id. Image ids can be retrieved using the following command-line:
az image builder list\n
A map image specification needs to contain the following fields publisher, offer sku, and optionally version. The map is used to specify images found in Azure Marketplace. Here is an example:
{\n publisher = \"OpenLogic\",\n offer = \"CentOS-CI\",\n sku = \"7-CI\"\n}\n
"},{"location":"#463-openstack","title":"4.6.3 OpenStack","text":"The image name can be a regular expression. If more than one image is returned by the query to OpenStack, the most recent is selected.
"},{"location":"#47-instances","title":"4.7 instances","text":"The instances variable is a map that defines the virtual machines that will form the cluster. The map' keys define the hostnames and the values are the attributes of the virtual machines.
Each instance is identified by a unique hostname. An instance's hostname is written as the key followed by its index (1-based). The following map:
instances = {\n mgmt = { type = \"p2-4gb\", tags = [...] },\n login = { type = \"p2-4gb\", count = 1, tags = [...] },\n node = { type = \"c2-15gb-31\", count = 2, tags = [...] },\n gpu-node = { type = \"gpu2.large\", count = 3, tags = [...] },\n}\n
will spawn instances with the following hostnames: mgmt1\nlogin1\nnode1\nnode2\ngpu-node1\ngpu-node2\ngpu-node3\n
Hostnames must follow a set of rules, from hostname man page:
Valid characters for hostnames are ASCII letters from a to z, the digits from 0 to 9, and the hyphen (-). A hostname may not start with a hyphen.
Two attributes are expected to be defined for each instance: 1. type: name for varying combinations of CPU, memory, GPU, etc. (i.e: t2.medium); 2. tags: list of labels that defines the role of the instance.
"},{"location":"#471-tags","title":"4.7.1 tags","text":"Tags are used in the Terraform code to identify if devices (volume, network) need to be attached to an instance, while in Puppet code tags are used to identify roles of the instances.
Terraform tags:
login: identify instances accessible with SSH from Internet and pointed by the domain name A recordspool: identify instances created only when their hostname appears in the var.pool list.proxy: identify instances accessible with HTTP/HTTPS and pointed by the vhost A recordspublic: identify instances that need to have a public ip address reachable from Internetpuppet: identify instances configured as Puppet serversspot: identify instances that are to be spawned as spot/preemptible instances. This tag is supported in AWS, Azure and GCP. It is ignored by OpenStack and OVH.efa: attach an Elastic Fabric Adapter network interface to the instance. This tag is supported in AWS.ssl: identify instances that receive a copy of the SSL wildcard certificate for the domain
Puppet tags expected by the puppet-magic_castle environment.
login: identify a login instance (minimum: 2 CPUs, 2GB RAM)mgmt: identify a management instance i.e: FreeIPA server, Slurm controller, Slurm DB (minimum: 2 CPUs, 6GB RAM)nfs: identify the instance that acts as an NFS server.node: identify a compute node instance (minimum: 1 CPUs, 2GB RAM)pool: when combined with node, it identifies compute nodes that Slurm can resume/suspend to meet workload demand.proxy: identify the instance that executes the Caddy reverse proxy and JupyterHub.
In the Magic Castle Puppet environment, an instance cannot be tagged as mgmt and proxy.
You are free to define your own additional tags.
"},{"location":"#472-optional-attributes","title":"4.7.2 Optional attributes","text":"Optional attributes can be defined:
count: number of virtual machines with this combination of hostname prefix, type and tags to create (default: 1).image: specification of the image to use for this instance type. (default: global image value). Refer to section 10.12 - Create a compute node image to learn how this attribute can be leveraged to accelerate compute node configuration.-
disk_type: type of the instance's root disk (default: see the next table).
Provider disk_type disk_size (GiB) Azure Premium_LRS 30 AWS gp2 10 GCP pd-ssd 20 OpenStack null 10 OVH null 10 -
disk_size: size in gibibytes (GiB) of the instance's root disk containing the operating system and service software (default: see the previous table).
mig: map of NVIDIA Multi-Instance GPU (MIG) short profile names and count used to partition the instances' GPU, example for an A100: mig = { \"1g.5gb\" = 2, \"2g.10gb\" = 1, \"3g.20gb\" = 1 }\n
This is only functional with MIG supported GPUs, and with x86-64 processors (see NVIDIA/mig-parted issue #30).
For some cloud providers, it possible to define additional attributes. The following sections present the available attributes per provider.
"},{"location":"#aws","title":"AWS","text":"For instances with the spot tags, these attributes can also be set:
wait_for_fulfillment (default: true)spot_type (default: permanent)instance_interruption_behavior (default: stop)spot_price (default: not set)block_duration_minutes (default: not set) [note 1] For more information on these attributes, refer to aws_spot_instance_request argument reference
Note 1: block_duration_minutes is not available to new AWS accounts or accounts without billing history - AWS EC2 Spot Instance requests. When not available, its usage can trigger quota errors like this:
Error requesting spot instances: MaxSpotInstanceCountExceeded: Max spot instance count exceeded\n
"},{"location":"#azure","title":"Azure","text":"For instances with the spot tags, these attributes can also be set:
max_bid_price (default: not set)eviction_policy (default: Deallocate) For more information on these attributes, refer to azurerm_linux_virtual_machine argument reference
"},{"location":"#gcp","title":"GCP","text":" gpu_type: name of the GPU model to attach to the instance. Refer to Google Cloud documentation for the list of available models per regiongpu_count: number of GPUs of the gpu_type model to attach to the instance
"},{"location":"#473-post-build-modification-effect","title":"4.7.3 Post build modification effect","text":"Modifying any part of the map after the cluster is built will only affect the type of instances associated with what was modified at the next terraform apply.
"},{"location":"#48-volumes","title":"4.8 volumes","text":"The volumes variable is a map that defines the block devices that should be attached to instances that have the corresponding key in their list of tags. To each instance with the tag, unique block devices are attached, no multi-instance attachment is supported.
Each volume in map is defined a key corresponding to its and a map of attributes:
size: size of the block device in GB.type (optional): type of volume to use. Default value per provider:- Azure:
Premium_LRS - AWS:
gp2 - GCP:
pd-ssd - OpenStack:
null - OVH:
null
Volumes with a tag that have no corresponding instance will not be created.
In the following example:
instances = {\u00a0\n server = { type = \"p4-6gb\", tags = [\"nfs\"] }\n}\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 100 }\n scratch = { size = 100 }\n }\n mds = {\n oss1 = { size = 500 }\n oss2 = { size = 500 }\n }\n}\n
The instance server1 will have three volumes attached to it. The volumes tagged mds are not created since no instances have the corresponding tag.
To define an infrastructure with no volumes, set the volumes variable to an empty map:
volumes = {}\n
Post build modification effect: destruction of the corresponding volumes and attachments, and creation of new empty volumes and attachments. If an no instance with a corresponding tag exist following modifications, the volumes will be deleted.
"},{"location":"#49-public_keys","title":"4.9 public_keys","text":"List of SSH public keys that will have access to your cluster sudoer account.
Note 1: You need to add the private key associated with one of the public keys to your local authentication agent (i.e: ssh-add) for Terraform to be able to copy Puppet configuration files with scp on the cluster. Otherwise, Magic Castle can create a key pair for unique to this cluster, see section 4.15 - generate_ssh_key (optional).
Post build modification effect: trigger scp of hieradata files at next terraform apply. The sudoer account authorized_keys file will be updated by each instance's Puppet agent following the copy of the hieradata files.
"},{"location":"#410-nb_users-optional","title":"4.10 nb_users (optional)","text":"default value: 0
Defines how many guest user accounts will be created in FreeIPA. Each user account shares the same randomly generated password. The usernames are defined as userX where X is a number between 1 and the value of nb_users (zero-padded, i.e.: user01 if X < 100, user1 if X < 10).
If an NFS NFS home volume is defined, each user will have a home folder on a shared NFS storage hosted on the NFS server node.
User accounts do not have sudoer privileges. If you wish to use sudo, you will have to login using the sudoer account and the SSH keys listed in public_keys.
If you would like to add a user account after the cluster is built, refer to section 10.3 and 10.4.
Requirement: Must be an integer, minimum value is 0.
Post build modification effect: trigger scp of hieradata files at next terraform apply. If nb_users is increased, new guest accounts will be created during the following Puppet run on mgmt1. If nb_users is decreased, it will have no effect: the guest accounts already created will be left intact.
"},{"location":"#411-guest_passwd-optional","title":"4.11 guest_passwd (optional)","text":"default value: 4 random words separated by dots
Defines the password for the guest user accounts instead of using a randomly generated one.
Requirement: Minimum length 8 characters.
The password can be provided in a PKCS7 encrypted form. Refer to sub-section 4.13.1 Encrypting hieradata secrets for instructions on how to encrypt the password.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Password of all guest accounts will be changed to match the new password value.
"},{"location":"#412-sudoer_username-optional","title":"4.12 sudoer_username (optional)","text":"default value: centos
Defines the username of the account with sudo privileges. The account ssh authorized keys are configured with the SSH public keys with public_keys.
Post build modification effect: none. To change sudoer username, destroy the cluster or redefine the value of profile::base::sudoer_username in hieradata.
"},{"location":"#413-hieradata-optional","title":"4.13 hieradata (optional)","text":"default value: empty string
Defines custom variable values that are injected in the Puppet hieradata file. Useful to override common configuration of Puppet classes.
List of useful examples:
- Receive logs of Puppet runs with changes to your email, add the following line to the string:
profile::base::admin_email: \"me@example.org\"\n
- Define ip addresses that can never be banned by fail2ban:
profile::fail2ban::ignore_ip: ['132.203.0.0/16', '8.8.8.8']\n
- Remove one-time password field from JupyterHub login page:
jupyterhub::enable_otp_auth: false\n
- Setup AlertManager to receive Prometheus alerts on Slack:
prometheus::alertmanager::route:\n group_by:\n - 'alertname'\n - 'cluster'\n - 'service'\n group_wait: '5s'\n group_interval: '5m'\n repeat_interval: '3h'\n receiver: 'slack'\n\nprometheus::alertmanager::receivers:\n - name: 'slack'\n slack_configs:\n - api_url: 'https://hooks.slack.com/services/ABCDEFG123456'\n channel: \"#channel\"\n send_resolved: true\n username: 'username'\n
Refer to the following Puppet modules' documentation to know more about the key-values that can be defined:
- puppet-magic_castle
- puppet-jupyterhub
- puppet-prometheus
The file created from this string can be found on puppet as
/etc/puppetlabs/data/user_data.yaml\n
Requirement: The string needs to respect the YAML syntax.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
"},{"location":"#4131-encrypting-hieradata-secrets","title":"4.13.1. Encrypting hieradata secrets","text":"If you plan to track the cluster configuration files in git (i.e:main.tf, user_data.yaml), it would be a good idea to encrypt the sensitive property values.
Magic Castle uses Puppet hiera-eyaml to provide a per-value encryption of sensitive properties to be used by Puppet.
To encrypt the data, you need to access the eyaml public certificate file of your cluster. This file is located on the Puppet server at /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem. With the public certificate file, you can encrypt the values with eyaml:
/opt/puppetlabs/puppet/bin/eyaml encrypt -s 'your-secret' --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem -o string\n
You can encrypt the value remotely using SSH jump host:
ssh -J centos@your-cluster.yourdomain.cloud centos@puppet /opt/puppetlabs/puppet/bin/eyaml encrypt -s 'your-secret' --pkcs7-public-key=/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem -o string\n
In the preceding command, replace puppet by the hostname of your puppetserver (i.e.: mgmt1). The openssl command-line can also be used to encrypt a value with the certificate file:
echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | base64 -w0 | xargs printf \"ENC[PKCS7,%s]\\n\"\n
To learn more about public_key.pkcs7.pem and how it can be generated before the cluster creation, refer to section 10.13 Generate and replace Puppet hieradata encryption keys.
"},{"location":"#414-firewall_rules-optional","title":"4.14 firewall_rules (optional)","text":"default value:
{\n ssh = { \"from_port\" = 22, \"to_port\" = 22, tag = \"login\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n http = { \"from_port\" = 80, \"to_port\" = 80, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n https = { \"from_port\" = 443, \"to_port\" = 443, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n globus = { \"from_port\" = 2811, \"to_port\" = 2811, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"54.237.254.192/29\" },\n myproxy = { \"from_port\" = 7512, \"to_port\" = 7512, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n gridftp = { \"from_port\" = 50000, \"to_port\" = 51000, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" }\n}\n
Defines a map of firewall rules that control external traffic to the public nodes. Each rule is defined as a map of key-value pairs and has to be assigned a unique name:
from_port (req.): the lower part of the allowed port range, valid integer value needs to be between 1 and 65535.to_port (req.): the higher part of the allowed port range, valid integer value needs to be between 1 and 65535.tag (req.): instances with this tag will be assigned this firewall rule.ethertype (opt. default: \"IPv4\"): the layer 3 protocol type (\"IPv4\" or \"IPv6\").protocol (opt. default: \"tcp\"): the layer 4 protocol type.cidr (opt. default: \"0.0.0.0/0\"): the remote CIDR, the value needs to be a valid CIDR (i.e. 192.168.0.0/16).
If you would like Magic Castle to be able to transfer files and update the state of the cluster in Puppet, make sure there exists at least one effective firewall rule where from_port <= 22 <= to_port and for which the external IP address of the machine that executes Terraform is in the CIDR range (i.e: cidr = \"0.0.0.0/0\" being the most permissive). This corresponds to the ssh rule in the default firewall rule map. This guarantees that Terraform will be able to use SSH to connect to the cluster from anywhere. For more information about this requirement, refer to Magic Castle's bastion tag computation code.
Post build modification effect: modify the cloud provider firewall rules at next terraform apply.
"},{"location":"#415-generate_ssh_key-optional","title":"4.15 generate_ssh_key (optional)","text":"default_value: false
If true, Terraform will generate an ssh key pair that would then be used when copying file with Terraform file-provisioner. The public key will be added to the sudoer account authorized keys.
This parameter is useful when Terraform does not have access to one of the private key associated with the public keys provided in public_keys.
Post build modification effect:
false -> true: will cause Terraform failure. Terraform will try to use the newly created private SSH key to connect to the cluster, while the corresponding public SSH key is yet registered with the sudoer account.true -> false: will trigger a scp of terraform_data.yaml at next terraform apply. The Terraform public SSH key will be removed from the sudoer account authorized_keys file at next Puppet agent run.
"},{"location":"#416-software_stack-optional","title":"4.16 software_stack (optional)","text":"default_value: \"alliance\"
Defines the scientific software environment that users have access when they login. Possible values are:
- default -
\"alliance\" / \"computecanada\": Digital Alliance Research Alliance of Canada scientific software environment (previously Compute Canada environment) \"eessi\": European Environment for Scientific Software Installation (EESSI)null / \"\": no scientific software environment
Post build modification effect: trigger scp of hieradata files at next terraform apply.
"},{"location":"#417-pool-optional","title":"4.17 pool (optional)","text":"default_value: []
Defines a list of hostnames with the tag \"pool\" that have to be online. This variable is typically managed by the workload scheduler through Terraform API. For more information, refer to Enable Magic Castle Autoscaling
Post build modification effect: pool tagged hosts with name present in the list will be instantiated, others will stay uninstantiated or will be destroyed if previously instantiated.
"},{"location":"#418-skip_upgrade-optional","title":"4.18 skip_upgrade (optional)","text":"default_value = false
If true, the base image packages will not be upgraded during the first boot. By default, all packages are upgraded.
Post build modification effect: No effect on currently built instances. Ones created after the modification will take into consideration the new value of the parameter to determine whether they should upgrade the base image packages or not.
"},{"location":"#419-puppetfile-optional","title":"4.19 puppetfile (optional)","text":"default_value = \"\"
Defines a complement of modules to install with librarian puppet when initializing the Puppet environment on the first boot of the Puppet server. If the provided string include the forge\u00a0setting, the provided content will replace entirely the Magic Castle environment's Puppetfile.
Post build modification effect: None. To modify the Puppetfile after the cluster is initialized, log on the Puppet server and modify /etc/puppetlabs/code/environments/production/Puppetfile.
"},{"location":"#5-cloud-specific-configuration","title":"5. Cloud Specific Configuration","text":""},{"location":"#51-amazon-web-services","title":"5.1 Amazon Web Services","text":""},{"location":"#511-region","title":"5.1.1 region","text":"Defines the label of the AWS EC2 region where the cluster will be created (i.e.: us-east-2).
Requirement: Must be in the list of available EC2 regions.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#512-availability_zone-optional","title":"5.1.2 availability_zone (optional)","text":"default value: None
Defines the label of the data center inside the AWS region where the cluster will be created (i.e.: us-east-2a). If left blank, it chosen at random amongst the availability zones of the selected region.
Requirement: Must be in a valid availability zone for the selected region. Refer to AWS documentation to find out how list the availability zones.
"},{"location":"#52-microsoft-azure","title":"5.2 Microsoft Azure","text":""},{"location":"#521-location","title":"5.2.1 location","text":"Defines the label of the Azure location where the cluster will be created (i.e.: eastus).
Requirement: Must be a valid Azure location. To get the list of available location, you can use Azure CLI : az account list-locations -o table.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#522-azure_resource_group-optional","title":"5.2.2 azure_resource_group (optional)","text":"default value: None
Defines the name of an already created resource group to use. Terraform will no longer attempt to manage a resource group for Magic Castle if this variable is defined and will instead create all resources within the provided resource group. Define this if you wish to use an already created resource group or you do not have a subscription-level access to create and destroy resource groups.
Post build modification effect: rebuild of all instances at next terraform apply.
"},{"location":"#523-plan-optional","title":"5.2.3 plan (optional)","text":"default value:
{\n name = null\n product = null\n publisher = null\n}\n
Purchase plan information for Azure Marketplace image. Certain images from Azure Marketplace requires a terms acceptance or a fee to be used. When using this kind of image, you must supply the plan details.
For example, to use the official AlmaLinux image, you have to first add it to your account. Then to use it with Magic Castle, you must supply the following plan information:
plan = {\n name = \"8_7\"\n product = \"almalinux\"\n publisher = \"almalinux\"\n}\n
"},{"location":"#53-google-cloud","title":"5.3 Google Cloud","text":""},{"location":"#531-project","title":"5.3.1 project","text":"Defines the label of the unique identifier associated with the Google Cloud project in which the resources will be created. It needs to corresponds to GCP project ID, which is composed of the project name and a randomly assigned number.
Requirement: Must be a valid Google Cloud project ID.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#532-region","title":"5.3.2 region","text":"Defines the name of the specific geographical location where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud region. Refer to Google Cloud documentation for the list of available regions and their characteristics.
"},{"location":"#533-zone-optional","title":"5.3.3 zone (optional)","text":"default value: None
Defines the name of the zone within the region where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud zone. Refer to Google Cloud documentation for the list of available zones and their characteristics.
"},{"location":"#54-openstack-and-ovh","title":"5.4 OpenStack and OVH","text":""},{"location":"#541-os_floating_ips-optional","title":"5.4.1 os_floating_ips (optional)","text":"default value: {}
Defines a map as an association of instance names (key) to pre-allocated floating ip addresses (value). Example:
os_floating_ips = {\n login1 = \"132.213.13.59\"\n login2 = \"132.213.13.25\"\n }\n
- instances tagged as public that have an entry in this map will be assigned the corresponding ip address;
- instances tagged as public that do not have an entry in this map will be assigned a floating ip managed by Terraform.
- instances not tagged as public that have an entry in this map will not be assigned a floating ip.
This variable can be useful if you manage your DNS manually and you would like the keep the same domain name for your cluster at each build.
Post build modification effect: change the floating ips assigned to the public instances.
"},{"location":"#542-os_ext_network-optional","title":"5.4.2 os_ext_network (optional)","text":"default value: None
Defines the name of the external network that provides the floating ips. Define this only if your OpenStack cloud provides multiple external networks, otherwise, Terraform can find it automatically.
Post build modification effect: change the floating ips assigned to the public nodes.
"},{"location":"#544-subnet_id-optional","title":"5.4.4 subnet_id (optional)","text":"default value: None
Defines the ID of the internal IPV4 subnet to which the instances are connected. Define this if you have or intend to have more than one subnets defined in your OpenStack project. Otherwise, Terraform can find it automatically. Can be used to force a v4 subnet when both v4 and v6 exist.
Post build modification effect: rebuild of all instances at next terraform apply.
"},{"location":"#6-dns-configuration-and-wildcard-ssl-certificate","title":"6. DNS Configuration and Wildcard SSL Certificate","text":"Some functionalities in Magic Castle require the registration of DNS records under the cluster name in the selected domain. This includes web services like JupyterHub, Mokey and FreeIPA web portal.
If your domain DNS records are managed by one of the supported providers, follow the instructions in the corresponding sections to have the cluster's DNS records created and tracked by Magic Castle.
If your DNS provider is not supported, you can manually create the records. Refer to the subsection 6.3 for more details.
Optionally, Magic Castle can issue with Let's encrypt DNS-01 challenge a wildcard certificate of the form *.${cluster_name}.${domain_name}. Refer to the subsection 6.4 for more information. If no wildcard certificate is issued, the reverse proxy server Caddy will automatically issue one certificate per virtual host.
"},{"location":"#61-cloudflare","title":"6.1 Cloudflare","text":" - Uncomment the
dns module for Cloudflare in your main.tf. - Uncomment the
output \"hostnames\" block. - Download and install the Cloudflare Terraform module:
terraform init. - Export the environment variables
CLOUDFLARE_EMAIL and CLOUDFLARE_API_KEY, where CLOUDFLARE_EMAIL is your Cloudflare account email address and CLOUDFLARE_API_KEY is your account Global API Key available in your Cloudflare profile.
"},{"location":"#612-cloudflare-api-token","title":"6.1.2 Cloudflare API Token","text":"If you prefer using an API token instead of the global API key, you will need to configure a token with the following four permissions with the Cloudflare API Token interface.
Section Subsection Permission Zone DNS Edit Instead of step 5, export only CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_API_TOKEN, and CLOUDFLARE_DNS_API_TOKEN equal to the API token generated previously.
"},{"location":"#62-google-cloud","title":"6.2 Google Cloud","text":"requirement: Install the Google Cloud SDK
- Login to your Google account with gcloud CLI :
gcloud auth application-default login - Uncomment the
dns module for Google Cloud in your main.tf. - Uncomment the
output \"hostnames\" block. - In
main.tf's dns module, configure the variables project and zone_name with their respective values as defined by your Google Cloud project. - Download and install the Google Cloud Terraform module:
terraform init.
"},{"location":"#63-unsupported-providers","title":"6.3 Unsupported providers","text":"If your DNS provider is not currently supported by Magic Castle, you can create the DNS records manually.
Magic Castle provides a module that creates a text file with the DNS records that can then be imported manually in your DNS zone. To use this module, add the following snippet to your main.tf:
module \"dns\" {\n source = \"./dns/txt\"\n name = module.openstack.cluster_name\n domain = module.openstack.domain\n public_instances = module.openstack.public_instances\n}\n
Find and replace openstack in the previous snippet by your cloud provider of choice if not OpenStack (i.e: aws, gcp, etc.).
The file will be created after the terraform apply in the same folder as your main.tf and will be named as ${name}.${domain}.txt.
"},{"location":"#64-wildcard-ssl-certificate","title":"6.4 Wildcard SSL Certificate","text":"Requirement: A private key associated with one of the public keys needs to be tracked (i.e: ssh-add) by the local authentication agent (i.e: ssh-agent). This module uses the ssh-agent tracked SSH keys to authenticate and to copy SSL certificate files to the proxy nodes after their creation.
To issue a wildcard SSL certificate with Magic Castle, modify the dns module call in your main.tf like this:
- add the input:
issue_wildcard_cert = true - add the input:
email = \"replace.by@your.email\"
"},{"location":"#641-acme-account-private-key","title":"6.4.1 ACME Account Private Key","text":"To create the wildcard SSL certificate associated with the domain name, Magic Castle creates a private key and register a new ACME account with this key. This account registration process is done for each new cluster. However, ACME limits the number of new accounts that can be created to a maximum of 10 per IP Address per 3 hours.
If you plan to create more than 10 clusters per 3 hours, we recommend registering an ACME account first and then provide its private key in PEM format to Magic Castle DNS module, using the acme_key_pem variable.
"},{"location":"#how-to-generate-an-acme-account-private-key","title":"How to Generate an ACME Account Private Key","text":"In a separate folder, create a file with the following content
terraform {\n required_version = \">= 1.2.1\"\n required_providers {\n acme = {\n source = \"vancluever/acme\"\n }\n tls = {\n source = \"hashicorp/tls\"\n }\n }\n}\n\nvariable \"email\" {}\n\nprovider \"acme\" {\n server_url = \"https://acme-v02.api.letsencrypt.org/directory\"\n}\nresource \"tls_private_key\" \"private_key\" {\n algorithm = \"RSA\"\n}\nresource \"acme_registration\" \"reg\" {\n account_key_pem = tls_private_key.private_key.private_key_pem\n email_address = var.email\n}\nresource \"local_file\" \"acme_key_pem\" {\n content = tls_private_key.private_key.private_key_pem\n filename = \"acme_key.pem\"\n}\n
In the same folder, enter the following commands and follow the instructions:
terraform init\nterraform apply\n
Once done, copy the file named acme_key.pem somewhere safe, and where you will be able to refer to later on. Then, when the time comes to create a new cluster, add the following variable to the DNS module in your main.tf:
acme_key_pem = file(\"path/to/your/acme_key.pem\")\n
"},{"location":"#642-issuing-the-wildcard-ssl-certificate-manually","title":"6.4.2 Issuing the wildcard SSL certificate manually","text":"Magic Castle generates with Let's Encrypt a wildcard certificate for *.cluster_name.domain. You can use certbot DNS-01 challenge plugin to generate the wildcard certificate.
You will then need to copy the certificate files in the proper location on each proxy node. The reverse proxy configuration expects the following files to exist:
/etc/letsencrypt/live/${domain_name}/fullchain.pem/etc/letsencrypt/live/${domain_name}/privkey.pem/etc/letsencrypt/live/${domain_name}/chain.pem
Refer to the reverse proxy configuration for more details.
"},{"location":"#65-sshfp-records-and-dnssec","title":"6.5 SSHFP records and DNSSEC","text":"Magic Castle DNS module creates SSHFP records for all instances with a public ip address. These records can be used by SSH clients to verify the SSH host keys of the server. If DNSSEC is enabled for the domain and the SSH client is correctly configured, no host key confirmation will be prompted when connecting to the server.
For more information on how to activate DNSSEC, refer to your DNS provider documentation:
- CloudFlare - Enable DNSSEC
- Google Cloud - Manage DNSSEC configuration
To setup an SSH client to use SSHFP records, add
VerifyHostKeyDNS yes\n
to its configuration file (i.e.: ~/.ssh/config)."},{"location":"#7-planning","title":"7. Planning","text":"Once your initial cluster configuration is done, you can initiate a planning phase where you will ask Terraform to communicate with your cloud provider and verify that your cluster can be built as it is described by the main.tf configuration file.
Terraform should now be able to communicate with your cloud provider. To test your configuration file, enter the following command
terraform plan\n
This command will validate the syntax of your configuration file and communicate with the provider, but it will not create new resources. It is only a dry-run. If Terraform does not report any error, you can move to the next step. Otherwise, read the errors and fix your configuration file accordingly.
"},{"location":"#8-deployment","title":"8. Deployment","text":"To create the resources defined by your main, enter the following command
terraform apply\n
The command will produce the same output as the plan command, but after the output it will ask for a confirmation to perform the proposed actions. Enter yes.
Terraform will then proceed to create the resources defined by the configuration file. It should take a few minutes. Once the creation process is completed, Terraform will output the guest account usernames and password, the sudoer username and the floating ip of the login node.
Warning: although the instance creation process is finished once Terraform outputs the connection information, you will not be able to connect and use the cluster immediately. The instance creation is only the first phase of the cluster-building process. The configuration: the creation of the user accounts, installation of FreeIPA, Slurm, configuration of JupyterHub, etc.; takes around 15 minutes after the instances are created.
Once it is booted, you can follow an instance configuration process by looking at:
/var/log/cloud-init-output.logjournalctl -u puppet
If unexpected problems occur during configuration, you can provide these logs to the authors of Magic Castle to help you debug.
"},{"location":"#81-deployment-customization","title":"8.1 Deployment Customization","text":"You can modify the main.tf at any point of your cluster's life and apply the modifications while it is running.
Warning: Depending on the variables you modify, Terraform might destroy some or all resources, and create new ones. The effects of modifying each variable are detailed in the subsections of Configuration.
For example, to increase the number of computes nodes by one. Open main.tf, add 1 to node's count , save the document and call
terraform apply\n
Terraform will analyze the difference between the current state and the future state, and plan the creation of a single new instance. If you accept the action plan, the instance will be created, provisioned and eventually automatically add to the Slurm cluster configuration.
You could do the opposite and reduce the number of compute nodes to 0.
"},{"location":"#9-destruction","title":"9. Destruction","text":"Once you're done working with your cluster and you would like to recover the resources, in the same folder as main.tf, enter:
terraform destroy -refresh=false\n
The -refresh=false\u00a0flag is to avoid an issue where one or many of the data sources return no results and stall the cluster destruction with a message like the following:
Error: Your query returned no results. Please change your search criteria and try again.\n
This type of error happens when for example the specified image no longer exists (see issue #40). As for apply, Terraform will output a plan that you will have to confirm by entering yes.
Warning: once the cluster is destroyed, nothing will be left, even the shared storage will be erased.
"},{"location":"#91-instance-destruction","title":"9.1 Instance Destruction","text":"It is possible to destroy only the instances and keep the rest of the infrastructure like the floating ip, the volumes, the generated SSH host key, etc. To do so, set the count value of the instance type you wish to destroy to 0.
"},{"location":"#92-reset","title":"9.2 Reset","text":"On some occasions, it is desirable to rebuild some of the instances from scratch. Using terraform taint, you can designate resources that will be rebuilt at next application of the plan.
To rebuild the first login node :
terraform taint 'module.openstack.openstack_compute_instance_v2.instances[\"login1\"]'\nterraform apply\n
"},{"location":"#10-customize-cluster-software-configuration","title":"10. Customize Cluster Software Configuration","text":"Once the cluster is online and configured, you can modify its configuration as you see fit. We list here how to do most commonly asked for customizations.
Some customizations are done from the Puppet server instance (puppet). To connect to the puppet server, follow these steps:
- Make sure your SSH key is loaded in your ssh-agent.
- SSH in your cluster with forwarding of the authentication agent connection enabled:
ssh -A centos@cluster_ip. Replace centos by the value of sudoer_username if it is different. - SSH in the Puppet server instance:
ssh puppet
Note on Google Cloud: In GCP, OS Login lets you use Compute Engine IAM roles to manage SSH access to Linux instances. This feature is incompatible with Magic Castle. Therefore, it is turned off in the instances metadata (enable-oslogin=\"FALSE\"). The only account with sudoer rights that can log in the cluster is configured by the variable sudoer_username (default: centos).
"},{"location":"#101-disable-puppet","title":"10.1 Disable Puppet","text":"If you plan to modify configuration files manually, you will need to disable Puppet. Otherwise, you might find out that your modifications have disappeared in a 30-minute window.
Puppet executes a run every 30 minutes and at reboot. To disable puppet:
sudo puppet agent --disable \"<MESSAGE>\"\n
"},{"location":"#102-replace-the-guest-account-password","title":"10.2 Replace the Guest Account Password","text":"Refer to section 4.11.
"},{"location":"#103-add-ldap-users","title":"10.3 Add LDAP Users","text":"Users can be added to Magic Castle LDAP database (FreeIPA) with either one of the following methods: hieradata, command-line, and Mokey web-portal. Each method is presented in the following subsections.
New LDAP users are automatically assigned a home folder on NFS.
Magic Castle determines if an LDAP user should be member of a Slurm account based on its POSIX groups. When a user is added to a POSIX group, a daemon try to match the group name to the following regular expression:
(ctb|def|rpp|rrg)-[a-z0-9_-]*\n
If there is a match, the user will be added to a Slurm account with the same name, and will gain access to the corresponding project folder under /project.
Note: The regular expression represents how Compute Canada names its resources allocation. The regular expression can be redefined, see profile::accounts:::project_regex
"},{"location":"#1031-hieradata","title":"10.3.1 hieradata","text":"Using the hieradata variable in the main.tf, it is possible to define LDAP users.
Examples of LDAP user definition with hieradata are provided in puppet-magic_castle documentation.
"},{"location":"#1032-command-line","title":"10.3.2 Command-Line","text":"To add a user account after the cluster is built, log in mgmt1 and call:
kinit admin\nIPA_GUEST_PASSWD=<new_user_passwd> /sbin/ipa_create_user.py <username> [--group <group_name>]\nkdestroy\n
"},{"location":"#1033-mokey","title":"10.3.3 Mokey","text":"If user sign-up with Mokey is enabled, users can create their own account at
https://mokey.yourcluster.domain.tld/auth/signup\n
It is possible that an administrator is required to enable the account with Mokey. You can access the administrative panel of FreeIPA at :
https://ipa.yourcluster.domain.tld/\n
The FreeIPA administrator credentials can be retrieved from an encrypted file on the Puppet server. Refer to section 10.14 to know how.
"},{"location":"#104-increase-the-number-of-guest-accounts","title":"10.4 Increase the Number of Guest Accounts","text":"To increase the number of guest accounts after creating the cluster with Terraform, simply increase the value of nb_users, then call :
terraform apply\n
Each instance's Puppet agent will be reloaded following the copy of the hieradata files, and the new accounts will be created.
"},{"location":"#105-restrict-ssh-access","title":"10.5 Restrict SSH Access","text":"By default, instances tagged login have their port 22 opened to entire world. If you know the range of ip addresses that will connect to your cluster, we strongly recommend that you limit the access to port 22 to this range.
To limit the access to port 22, refer to section 4.14 firewall_rules, and replace the cidr of the ssh rule to match the range of ip addresses that have be the allowed to connect to the cluster. If there are more than one range, create multiple rules with distinct names.
"},{"location":"#106-add-packages-to-jupyter-default-python-kernel","title":"10.6 Add Packages to Jupyter Default Python Kernel","text":"The default Python kernel corresponds to the Python installed in /opt/ipython-kernel. Each compute node has its own copy of the environment. To add packages to this environment, add the following lines to hieradata in main.tf:
jupyterhub::kernel::venv::packages:\n - package_A\n - package_B\n - package_C\n
and replace package_* by the packages you need to install. Then call:
terraform apply\n
"},{"location":"#107-activate-globus-endpoint","title":"10.7 Activate Globus Endpoint","text":"No longer supported
"},{"location":"#108-recovering-from-puppet-rebuild","title":"10.8 Recovering from puppet rebuild","text":"The modifications of some of the parameters in the main.tf file can trigger the rebuild of the puppet instance. This instance hosts the Puppet Server on which depends the Puppet agent of the other instances. When puppet is rebuilt, the other Puppet agents cease to recognize Puppet Server identity since the Puppet Server identity and certificates have been regenerated.
To fix the Puppet agents, you will need to apply the following commands on each instance other than puppet once puppet is rebuilt:
sudo systemctl stop puppet\nsudo rm -rf /etc/puppetlabs/puppet/ssl/\nsudo systemctl start puppet\n
Then, on puppet, you will need to sign the new certificate requests made by the instances. First, you can list the requests:
sudo /opt/puppetlabs/bin/puppetserver ca list\n
Then, if every instance is listed, you can sign all requests:
sudo /opt/puppetlabs/bin/puppetserver ca sign --all\n
If you prefer, you can sign individual request by specifying their name:
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname NAME[,NAME]\n
"},{"location":"#109-dealing-with-banned-ip-addresses-fail2ban","title":"10.9 Dealing with banned ip addresses (fail2ban)","text":"Login nodes run fail2ban, an intrusion prevention software that protects login nodes from brute-force attacks. fail2ban is configured to ban ip addresses that attempted to login 20 times and failed in a window of 60 minutes. The ban time is 24 hours.
In the context of a workshop with SSH novices, the 20-attempt rule might be triggered, resulting in participants banned and puzzled, which is a bad start for a workshop. There are solutions to mitigate this problem.
"},{"location":"#1091-define-a-list-of-ip-addresses-that-can-never-be-banned","title":"10.9.1 Define a list of ip addresses that can never be banned","text":"fail2ban keeps a list of ip addresses that are allowed to fail to login without risking jail time. To add an ip address to that list, add the following lines to the variable hieradata\u00a0in main.tf:
profile::fail2ban::ignoreip:\n - x.x.x.x\n - y.y.y.y\n
where x.x.x.x and y.y.y.y are ip addresses you want to add to the ignore list. The ip addresses can be written using CIDR notations. The ignore ip list on Magic Castle already includes 127.0.0.1/8 and the cluster subnet CIDR. Once the line is added, call:
terraform apply\n
"},{"location":"#1092-remove-fail2ban-ssh-route-jail","title":"10.9.2 Remove fail2ban ssh-route jail","text":"fail2ban rule that banned ip addresses that failed to connect with SSH can be disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::jails: ['ssh-ban-root']\n
This will keep the jail that automatically ban any ip that tries to login as root, and remove the ssh failed password jail. Once the line is added, call:
terraform apply\n
"},{"location":"#1093-unban-ip-addresses","title":"10.9.3 Unban ip addresses","text":"fail2ban ban ip addresses by adding rules to iptables. To remove these rules, you need to tell fail2ban to unban the ips.
To list the ip addresses that are banned, execute the following command:
sudo fail2ban-client status ssh-route\n
To unban ip addresses, enter the following command followed by the ip addresses you want to unban:
sudo fail2ban-client set ssh-route unbanip\n
"},{"location":"#1094-disable-fail2ban","title":"10.9.4 Disable fail2ban","text":"While this is not recommended, fail2ban can be completely disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::service_ensure: 'stopped'\n
then call :
terraform apply\n
"},{"location":"#1010-generate-a-new-ssl-certificate","title":"10.10 Generate a new SSL certificate","text":"The SSL certificate configured by the dns module is valid for 90 days. If you plan to use your cluster for more than 90 days, you will need to generate a new SSL certificate before the one installed on the cluster expires.
To generate a new certificate, use the following command on your computer:
terraform taint 'module.dns.module.acme.acme_certificate.certificate'\n
Then apply the modification:
terraform apply\n
The apply generates a new certificate, uploads it on the nodes that need it and reloads the reverse proxy if it is configured.
"},{"location":"#1011-set-selinux-in-permissive-mode","title":"10.11 Set SELinux in permissive mode","text":"SELinux can be set in permissive mode to debug new workflows that would be prevented by SELinux from working properly. To do so, add the following line to the variable hieradata\u00a0in main.tf:
selinux::mode: 'permissive'\n
"},{"location":"#1012-create-a-compute-node-image","title":"10.12 Create a compute node image","text":"When scaling the compute node pool, either manually by changing the count or automatically with Slurm autoscale, it can become beneficial to reduce the time spent configuring the machine when it boots for the first time, hence reducing the time requires before it becomes available in Slurm. One way to achieve this is to clone the root disk of a fully configured compute node and use it as the base image of future compute nodes.
This process has three steps:
- Prepare the volume for image cloning
- Create the image
- Configure Magic Castle Terraform code to use the new image
The following subsection explains how to accomplish each step.
Warning: While it will work in most cases, avoid re-using the compute node image of a previous deployment. The preparation steps cleans most of the deployment specific configuration and secrets, but there is no guarantee that the configuration will be entirely compatible with a different deployment.
"},{"location":"#10121-prepare-the-volume-for-cloning","title":"10.12.1 Prepare the volume for cloning","text":"The environment puppet-magic_castle installs a script that prepares the volume for cloning named prepare4image.sh.
To make sure a node is ready for cloning, open its puppet agent log and validate the catalog was successfully applied at least once:
journalctl -u puppet | grep \"Applied catalog\"\n
To prepare the volume for cloning, execute the following line while connected to the compute node:
sudo /usr/sbin/prepare4image.sh\n
Be aware that, since it is preferable for the instance to be powered off when cloning its volume, the script halts the machine once it is completed. Therefore, after executing prepare4image.sh, you will be disconnected from the instance.
The script prepare4image.sh executes the following steps in order:
- Stop and disable puppet agent
- Stop and disable slurm compute node daemon (
slurmd) - Stop and disable consul agent daemon
- Stop and disable consul-template daemon
- Unenroll the host from the IPA server
- Remove puppet agent configuration files in
/etc - Remove consul agent identification files
- Unmount NFS directories
- Remove NFS directories
/etc/fstab - Stop syslog
- Clear
/var/log/message content - Remove cloud-init's logs and artifacts so it can re-run
- Power off the machine
"},{"location":"#10122-create-the-image","title":"10.12.2 Create the image","text":"Once the instance is powered off, access your cloud provider dashboard, find the instance and follow the provider's instructions to create the image.
- AWS
- Azure
- GCP
- OpenStack
- OVH
Note down the name/id of the image you created, it will be needed during the next step.
"},{"location":"#10123-configure-magic-castle-terraform-code-to-use-the-new-image","title":"10.12.3 Configure Magic Castle Terraform code to use the new image","text":"Edit your main.tf and add image = \"name-or-id-of-your-image\" to the dictionary defining the instance. The instance previously powered off will be powered on and future non-instantiated machines will use the image at the next execution of terraform apply.
If the cluster is composed of heterogeneous compute nodes, it is possible to create an image for each type of compute nodes. Here is an example with Google Cloud
instances = {\n mgmt = { type = \"n2-standard-2\", tags = [\"puppet\", \"mgmt\", \"nfs\"], count = 1 }\n login = { type = \"n2-standard-2\", tags = [\"login\", \"public\", \"proxy\"], count = 1 }\n node = {\n type = \"n2-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n image = \"rocky-mc-cpu-node\"\n }\n gpu = {\n type = \"n1-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n gpu_type = \"nvidia-tesla-t4\"\n gpu_count = 1\n image = \"rocky-mc-gpu-node\"\n }\n}\n
"},{"location":"#1013-generate-and-replace-puppet-hieradata-encryption-keys","title":"10.13 Generate and replace Puppet hieradata encryption keys","text":"During the Puppet server initial boot, a pair of hiera-eyaml encryptions keys are generated in /opt/puppetlabs/puppet/eyaml:
private_key.pkcs7.pempublic_key.pkcs7.pem
To encrypt the values before creating the cluster, the encryptions keys can be generated beforehand and then transferred on the Puppet server.
The keys can be generated with eyaml:
eyaml createkeys\n
or openssl:
openssl req -x509 -nodes -days 100000 -newkey rsa:2048 -keyout private_key.pkcs7.pem -out public_key.pkcs7.pem -subj '/'\n
The resulting public key can then be used to encrypt secrets, while the private and the public keys have to be transferred on the Puppet server to allow it to decrypt the values. In the following command examples, replace puppet by the hostname of your puppetserver (i.e.: mgmt1).
- Transfer the keys on the Puppet server using SCP with SSH jumphost
scp -J centos@cluster.yourdomain.cloud {public,private}_key.pkcs7.pem centos@puppet:~/\n
- Replace the existing keys by the one transferred:
ssh -J centos@cluster.yourdomain.cloud centos@puppet sudo cp {public,private}_key.pkcs7.pem /etc/puppetlabs/puppet/eyaml\n
- Remove the keys from the admin account home folder:
ssh -J centos@cluster.yourdomain.cloud centos@puppet rm {public,private}_key.pkcs7.pem\n
To backup the encryption keys from an existing Puppet server:
- Create a readable copy of the encryption keys in the sudoer home account
ssh -J centos@cluster.yourdomain.cloud centos@puppet 'sudo rsync --owner --group --chown=centos:centos /etc/puppetlabs/puppet/eyaml/{public,private}_key.pkcs7.pem ~/'\n
- Transfer the files locally:
scp -J centos@cluster.yourdomain.cloud centos@puppet:~/{public,private}_key.pkcs7.pem .\n
- Remove the keys from the sudoer account home folder:
ssh -J centos@cluster.yourdomain.cloud centos@puppet rm {public,private}_key.pkcs7.pem\n
"},{"location":"#1014-read-and-edit-secret-values-generated-at-boot","title":"10.14 Read and edit secret values generated at boot","text":"During the cloud-init initialization phase, bootstrap.sh script is executed. This script generates a set of encrypted secret values that are required by the Magic Castle Puppet environment:
profile::consul::acl_api_tokenprofile::freeipa::mokey::passwordprofile::freeipa::server::admin_passwordprofile::freeipa::server::ds_passwordprofile::slurm::accounting::passwordprofile::slurm::base::munge_key
To read or change the value of one of these keys, use eyaml edit command on the puppet host, like this:
sudo /opt/puppetlabs/puppet/bin/eyaml edit \\\n --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/boot_private_key.pkcs7.pem \\\n --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/boot_public_key.pkcs7.pem \\\n /etc/puppetlabs/code/environments/production/data/bootstrap.yaml\n
It is also possible to redefine the values of these keys by adding the key-value pair to the hieradata configuration file. Refer to section 4.13 hieradata. User defined values take precedence over boot generated values in the Magic Castle Puppet data hierarchy.
"},{"location":"#1015-expand-a-volume","title":"10.15 Expand a volume","text":"Volumes defined in the volumes map can be expanded at will. After their creation, you can increase their size in the main.tf then call terraform apply and the associated block device will be expanded.
To benefit from the new storage, the following commands need to be ran as root on the instance to which the expanded volume is attached.
- Identify the physical volume path
pvscan\n
- Expand the physical volume
pvresize /dev/vdxyz # replace vdxyz by the volume identify at step 1\n
- Identify the volume group path
lvdisplay\n
- Expand the volume group using step volume group path identified
lvextend -l '+100%FREE' -r /dev/project_vg/project\n
- Resize the XFS filesystem:
xfs_growfs /dev/project_vg/project\n
"},{"location":"#11-customize-magic-castle-terraform-files","title":"11. Customize Magic Castle Terraform Files","text":"You can modify the Terraform module files in the folder named after your cloud provider (e.g: gcp, openstack, aws, etc.)
"},{"location":"design/","title":"Design","text":""},{"location":"design/#magic-castle-terraform-structure","title":"Magic Castle Terraform Structure","text":"Figure 1 (below) illustrates how Magic Castle is structured to provide a unified interface between multiple cloud providers. Each blue block is a file or a module, while white blocks are variables or resources. Arrows indicate variables or resources that contribute to the definition of the linked variables or resources. The figure can be read as a flow-chart from top to bottom. Some resources and variables have been left out of the chart to avoid cluttering it further.
Figure 1. Magic Castle Terraform Project Structure
main.tf: User provides the instances and volumes structure they wants as _map_s. instances = {\n mgmt = { type = \"p4-7.5gb\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"p2-3.75gb\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"p2-3.75gb\", tags = [\"node\"], count = 2 }\n}\n\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 500 }\n scratch = { size = 500 }\n }\n}\n
-
common/design:
- the
instances map is expanded to form a new map where each entry represents a single host. instances = {\n mgmt1 = {\n type = \"p2-3.75gb\"\n tags = [\"puppet\", \"mgmt\", \"nfs\"]\n }\n login1 = {\n type = \"p2-3.75gb\"\n tags = [\"login\", \"public\", \"proxy\"]\n }\n node1 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n node2 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n}\n
- the
volumes map is expanded to form a new map where each entry represent a single volume volumes = {\n mgmt1-nfs-home = { size = 100 }\n mgmt1-nfs-project = { size = 100 }\n mgmt1-nfs-scratch = { size = 500 }\n}\n
-
network.tf: the instances map from common/design is used to generate a network interface (nic) for each host, and a public ip address for each host with the public tag.
resource \"provider_network_interface\" \"nic\" {\n for_each = module.design.instances\n ...\n}\n
-
common/configuration: for each host in instances, a cloud-init yaml config that includes puppetservers is generated. These configs are outputted to a user_data map where the keys are the hostnames.
user_data = {\n for key, values in var.instances :\n key => templatefile(\"${path.module}/puppet.yaml\", { ... })\n}\n
-
infrastructure.tf: for each host in instances, an instance resource as defined by the selected cloud provider is generated. Each instance is initially configured by its user_data cloud-init yaml config.
resource \"provider_instance\" \"instances\" {\n for_each = module.design.instance\n user_data = module.instance_config.user_data[each.key]\n ...\n}\n
-
infrastructure.tf: for each volume in volumes, a block device as defined by the selected cloud provider is generated and attached it to its matching instance using an attachment resource.
resource \"provider_volume\" \"volumes\" {\n for_each = module.design.volumes\n size = each.value.size\n ...\n}\nresource \"provider_attachment\" \"attachments\" {\n for_each = module.design.volumes\n instance_id = provider_instance.instances[each.value.instance].id\n volume_id = provider_volume.volumes[each.key].id\n ...\n}\n
-
infrastructure.tf: the created instances' information are consolidated in a map named inventory.
inventory = {\n mgmt1 = {\n public_ip = \"\"\n local_ip = \"10.0.0.1\"\n id = \"abc1213-123-1231\"\n tags = [\"mgmt\", \"puppet\", \"nfs\"]\n }\n ...\n}\n
-
common/provision: the information from created instances is consolidated and written in a yaml file namedterraform_data.yaml that is uploaded on the Puppet server as part of the hieradata.
resource \"null_resource\" \"deploy_hieradata\" {\n ...\n provisioner \"file\" {\n content = var.terraform_data\n destination = \"terraform_data.yaml\"\n }\n ...\n}\n
-
outputs.tf: the information of all instances that have a public address are output as a map named public_instances.
"},{"location":"design/#resource-per-provider","title":"Resource per provider","text":"In the previous section, we have used generic resource name when writing HCL code that defines these resources. The following table indicate what resource is used for each provider based on its role in the cluster.
Resource AWS Azure Google Cloud Platform OpenStack OVH network aws_vpc azurerm_virtual_network google_compute_network prebuilt openstack_networking_network_v2 subnet aws_subnet azurerm_subnet google_compute_subnetwork prebuilt openstack_networking_subnet_v2 router aws_route not used google_compute_router built-in not used nat aws_internet_gateway not used google_compute_router_nat built-in not used firewall aws_security_group azurerm_network_security_group google_compute_firewall openstack_compute_secgroup_v2 openstack_compute_secgroup_v2 nic aws_network_interface azurerm_network_interface google_compute_address openstack_networking_port_v2 openstack_networking_port_v2 public ip aws_eip azurerm_public_ip google_compute_address openstack_networking_floatingip_v2 openstack_networking_network_v2 instance aws_instance azurerm_linux_virtual_machine google_compute_instance openstack_compute_instance_v2 openstack_compute_instance_v2 volume aws_ebs_volume azurerm_managed_disk google_compute_disk openstack_blockstorage_volume_v3 openstack_blockstorage_volume_v3 attachment aws_volume_attachment azurerm_virtual_machine_data_disk_attachment google_compute_attached_disk openstack_compute_volume_attach_v2 openstack_compute_volume_attach_v2"},{"location":"design/#using-reference-design-to-extend-for-a-new-cloud-provider","title":"Using reference design to extend for a new cloud provider","text":"Magic Castle currently supports five cloud providers, but its design makes it easy to add new providers. This section presents a step-by-step guide to add a new cloud provider support to Magic Castle.
-
Identify the resources. Using the Resource per provider table, read the cloud provider Terraform documentation, and identify the name for each resource in the table.
-
Check minimum requirements. Once all resources have been identified, you should be able to determine if the cloud provider can be used to deploy Magic Castle. If you found a name for each resource listed in table, the cloud provider can be supported. If some resources are missing, you will need to read the provider's documentation to determine if the absence of the resource can be compensated for somehow.
-
Initialize the provider folder. Create a folder named after the provider. In this folder, create two symlinks, one pointing to common/variables.tf and the other to common/outputs.tf. These files define the interface common to all providers supported by Magic Castle.
-
Define cloud provider specifics variables. Create a file named after your provider provider_name.tf\u00a0and define variables that are required by the provider but not common to all providers, for example the availability zone or the region. In this file, define two local variables named cloud_provider and cloud_region.
-
Initialize the infrastructure. Create a file named infrastructure.tf. In this file:
- define the provider block if it requires input parameters, i.e: var.region
provider \"provider_name\" {\n region = var.region\n}\n
- include the design module
module \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\n
-
Create the networking infrastructure. Create a file named network.tf and define the network, subnet, router, nat, firewall, nic and public ip resources using the module.design.instances map.
-
Create the volumes. In infrastructure.tf, define the volumes resource using module.design.volumes.
-
Consolidate the volume device information. In infrastructure.tf, define a local variable named volume_devices implementing the following logic in HCL. Replace the line starting by /dev/disk/by-id with the proper logic that would match the volume resource to its device path from within the instance to which it is attached.
volume_devices = {\n for ki, vi in var.volumes :\n ki => {\n for kj, vj in vi :\n kj => [for key, volume in module.design.volumes :\n \"/dev/disk/by-id/*${substr(provider_volume.volumes[\"${volume[\"instance\"]}-${ki}-${kj}\"].id, 0, 20)}\"\n if key == \"${volume[\"instance\"]}-${ki}-${kj}\"\n ]\n }\n}\n
-
Consolidate the instances' information. In infrastructure.tf, define a local variable named inventory that will be a map containing the following keys for each instance: public_ip, local_ip, prefix, tags, and specs (#cpu, #gpus, ram).
-
Create the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n generate_ssh_key = var.generate_ssh_key\n public_keys = var.public_keys\n volume_devices = local.volume_devices\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\n
-
Create the instances. In infrastructure.tf, define the instances resource using module.design.instances_to_build for the instance attributes and module.configuration.user_data for the initial configuration.
-
Attach the volumes. In infrastructure.tf, define the attachments resource using module.design.volumes and refer to the attribute each.value.instance to retrieve the instance's id to which the volume needs to be attached.
-
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=cloud_provider_instance_resource.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\n
-
Include the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\n
"},{"location":"design/#an-example","title":"An example","text":" -
Identify the resources. For Digital Ocean, Oracle Cloud and Alibaba Cloud, we get the following resource mapping: | Resource | Digital Ocean | Oracle Cloud | Alibaba Cloud | | ----------- | :-------------------- | :-------------------- | :-------------------- | | network | digitalocean_vpc | oci_core_vcn | alicloud_vpc | | subnet | built in vpc | oci_subnet | alicloud_vswitch | | router | n/a | oci_core_route_table | built in vpc | | nat | n/a | oci_core_internet_gateway | alicloud_nat_gateway | | firewall | digitalocean_firewall | oci_core_security_list | alicloud_security_group | | nic | n/a | built in instance | alicloud_network_interface | | public ip | digitalocean_floating_ip | built in instance | alicloud_eip | | instance | digitalocean_droplet | oci_core_instance | alicloud_instance | | volume | digitalocean_volume | oci_core_volume | alicloud_disk | | attachment | digitalocean_volume_attachment | oci_core_volume_attachment | alicloud_disk_attachment |
-
Check minimum requirements. In the preceding table, we can see Digital Ocean does not have the ability to define a network interface. The documentation also leads us to conclude that it is not possible to define the private ip address of the instances before creating them. Because the Puppet server ip address is required before generating the cloud-init YAML config for all instances, including the Puppet server itself, this means it impossible to use Digital Ocean to spawn a Magic Castle cluster. Oracle Cloud presents the same issue, however, after reading the instance documentation, we find that it is possible to define a static ip address as a string in the instance attribute. It would therefore be possible to create a datastructure in Terraform that would associate each instance hostname with an ip address in the subnet CIDR. Alibaba cloud has an answer for each resource, so we will use this provider in the following steps.
-
Initialize the provider folder. In a terminal:
git clone https://github.com/ComputeCanada/magic_castle.git\ncd magic_castle\nmkdir alicloud\ncd aliclcoud\nln -s ../common/{variables,outputs}.tf .\n
-
Define cloud provider specifics variables. Add the following to a new file alicloud.tf:
variable \"region\" { }\nlocals {\n cloud_provider = \"alicloud\"\n cloud_region = var.region\n}\n
-
Initialize the infrastructure. Add the following to a new file infrastructure.tf:
provider \"alicloud\" {\n region = var.region\n}\n\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\n
-
Create the networking infrastructure. network.tf base template:
resource \"alicloud_vpc\" \"network\" { }\nresource \"alicloud_vswitch\" \"subnet\" { }\nresource \"alicloud_nat_gateway\" \"nat\" { }\nresource \"alicloud_security_group\" \"firewall\" { }\nresource \"alicloud_security_group_rule\" \"allow_in_services\" { }\nresource \"alicloud_security_group\" \"allow_any_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_ingress_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_egress_inside_vpc\" { }\nresource \"alicloud_network_interface\" \"nic\" { }\nresource \"alicloud_eip\" \"public_ip\" { }\nresource \"alicloud_eip_association\" \"eip_asso\" { }\n
-
Create the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk\" \"volumes\" {\n for_each = module.design.volumes\n}\n
-
Consolidate the volume devices' information. Add the following snippet to infrastructure.tf:
volume_devices = {\n for ki, vi in var.volumes :\n ki => {\n for kj, vj in vi :\n kj => [for key, volume in module.design.volumes :\n \"/dev/disk/by-id/virtio-${replace(alicloud_disk.volumes[\"${volume[\"instance\"]}-${ki}-${kj}\"].id, \"d-\", \"\")}\"\n if key == \"${volume[\"instance\"]}-${ki}-${kj}\"\n ]\n }\n}\n
-
Consolidate the instances' information. Add the following snippet to infrastructure.tf:
locals {\n inventory = { for x, values in module.design.instances :\n x => {\n public_ip = contains(values[\"tags\"], \"public\") ? alicloud_eip.public_ip[x].public_ip : \"\"\n local_ip = alicloud_network_interface.nic[x].private_ip\n tags = values[\"tags\"]\n id = alicloud_instance.instances[x].id\n specs = {\n cpus = ...\n gpus = ...\n ram = ...\n }\n }\n }\n}\n
-
Create the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n generate_ssh_key = var.generate_ssh_key\n public_keys = var.public_keys\n volume_devices = local.volume_devices\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\n
-
Create the instances. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_instance\" \"instances\" {\n for_each = module.design.instances\n}\n
-
Attach the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk_attachment\" \"attachments\" {\n for_each = module.design.volumes\n}\n
-
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=alicloud_instance.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\n
-
Include the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\n
Once your new provider is written, you can write an example that will use the module to spawn a Magic Castle cluster with that provider.
module \"alicloud\" {\n source = \"./alicloud\"\n config_git_url = \"https://github.com/ComputeCanada/puppet-magic_castle.git\"\n config_version = \"main\"\n\n cluster_name = \"new\"\n domain = \"my.cloud\"\n image = \"centos_7_9_x64_20G_alibase_20210318.vhd\"\n nb_users = 10\n\n instances = {\n mgmt = { type = \"ecs.g6.large\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"ecs.g6.large\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"ecs.g6.large\", tags = [\"node\"], count = 1 }\n }\n\n volumes = {\n nfs = {\n home = { size = 10 }\n project = { size = 50 }\n scratch = { size = 50 }\n }\n }\n\n public_keys = [file(\"~/.ssh/id_rsa.pub\")]\n\n # Alicloud specifics\n region = \"us-west-1\"\n}\n
"},{"location":"developers/","title":"Magic Castle Developer Documentation","text":""},{"location":"developers/#table-of-content","title":"Table of Content","text":" - Setup
- Where to start
- Puppet environment
- Troubleshooting
- Release
"},{"location":"developers/#1-setup","title":"1. Setup","text":"To develop for Magic Castle you will need: * Terraform (>= 1.4.0) * git * Access to a Cloud (e.g.: Compute Canada Arbutus) * Ability to communicate with the cloud provider API from your computer * A cloud project with enough room for the resource described in section Magic Caslte Doc 1.1. * [optional] Puppet Development Kit (PDK)
"},{"location":"developers/#2-where-to-start","title":"2. Where to start","text":"The Magic Castle project is defined by Terraform infrastructure-as-code component that is responsible of generating a cluster architecture in a cloud and a Puppet environment component that configures the cluster instances based on their role.
If you wish to add device, an instance, add a new networking interface or a filesystem, you will most likely need to develop some Terraform code. The project structure for Terraform code is described in the reference design document. The document also describes how one could work with current Magic Castle code to add support for another cloud provider.
If you wish to add a service to one of the Puppet environments, install a new software, modify an instance configuration or role, you will most likely need to develop some Puppet code. The following section provides more details on the Puppet environments available and how to develop them.
"},{"location":"developers/#3-puppet-environment","title":"3. Puppet environment","text":"Magic Castle Terraform code initialized every instances to be a Puppet agent and an instance with the tag puppet as the Puppet main server. On the Puppet main server, there is a folder containing the configuration code for the instances of the cluster, this folder is called a Puppet environment and it is pulled from GitHub during the initial configuration of the Puppet main server.
The source of that environment is provided to Terraform using the variable config_git_url.
A repository describing a Magic Castle Puppet environment must contain at the least the following files and folders:
config_git_repo\n\u2523 Puppetfile\n\u2523 environment.conf\n\u2523 hiera.yaml\n\u2517 data\n \u2517 common.yaml\n\u2517 manifests/\n \u2517 site.pp\n
Puppetfile specifies the Puppet modules that need to be installed in the environment.environment.conf overrides the primary server default settings for the environment.hiera.yaml configures an ordered list of YAML file data sources.data/common.yaml is common data source for the instances part of hierarchy defined by hiera.yaml.manifests/site.pp defines how each instance will be configured based on their hostname and/or tags.
An example of a bare-bone Magic Castle Puppet environment is available on GitHub: MagicCastle/puppet-environment, while the Puppet environment that replicates a Compute Canada HPC cluster is named ComputeCanada/puppet-magic_castle.
"},{"location":"developers/#terraform_datayaml-a-bridge-between-terraform-and-puppet","title":"terraform_data.yaml: a bridge between Terraform and Puppet","text":"To provide information on the deployed resources and the value of the input parameters, Magic Castle Terraform code uploads to the Puppet main server a file named terraform_data.yaml, in the folder /etc/puppetlabs/data/. There is also a symlink created in /etc/puppetlabs/code/environment/production/data/ to ease its usage inside the Puppet environment.
When included in the data hierarchy (hiera.yaml), terraform_data.yaml can provide information about the instances, the volumes and the variables set by the user through the main.tf file. The file has the following structure:
---\nterraform:\n data:\n cluster_name: \"\"\n domain_name: \"\"\n guest_passwd: \"\"\n nb_users: \"\"\n public_keys: []\n sudoer_username: \"\"\n instances:\n host1:\n hostkeys:\n rsa: \"\"\n ed25519: \"\"\n local_ip: \"x.x.x.x\"\n prefix: \"host\"\n public_ip: \"\"\n specs:\n \"cpus\": 0\n \"gpus\": 0\n \"ram\": 0\n tags:\n - \"tag_1\"\n - \"tag_2\"\n tag_ip:\n tag_1:\n - x.x.x.x\n tag_2:\n - x.x.x.x\n volumes:\n volume_tag1:\n volume_1:\n - \"/dev/disk/by-id/123-*\"\n volume_2:\n - \"/dev/disk/by-id/123-abc-*\"\n
The values provided by terraform_data.yaml can be accessed in Puppet by using the lookup() function. For example, to access an instance's list of tags:
lookup(\"terraform.instances.${::hostname}.tags\")\n
The data source can also be used to define a key in another data source YAML file by using the alias() function. For example, to define the number of guest accounts using the value of nb_users, we could add this to common.yaml profile::accounts::guests::nb_accounts: \"%{alias('terraform.data.nb_users')}\"\n
"},{"location":"developers/#configuring-instances-sitepp-and-classes","title":"Configuring instances: site.pp and classes","text":"The configuration of each instance is defined in manifests/site.pp file of the Puppet environment. In this file, it is possible to define a configuration based on an instance hostname
node \"mgmt1\" { }\n
or using the instance tags by defining the configuration for the default node : node default {\n $instance_tags = lookup(\"terraform.instances.${::hostname}.tags\")\n if 'tag_1' in $instances_tags { }\n}\n
It is possible to define Puppet resource directly in site.pp. However, above a certain level of complexity, which can be reach fairly quickly, it is preferable to define classes and include these classes in site.pp based on the node hostname or tags.
Classes can be defined in the Puppet environment under the following path: site/profile/manifests. These classes are named profile classes and the philosophy behind it is explained in Puppet documentation. Because these classes are defined in site/profile, their name has to start with the prefix profile::.
It is also possible to include classes defined externally and installed using the Puppetfile. These classes installed by r10k can be found in the modules folder of the Puppet environment.
"},{"location":"developers/#4-troubleshooting","title":"4. Troubleshooting","text":""},{"location":"developers/#41-cloud-init","title":"4.1 cloud-init","text":"To test new additions to puppet.yaml, it is possible to execute cloud-init phases manually. There are four steps that can be executed sequentially: init local, init modules config and modules final. Here are the corresponding commands to execute each step:
cloud-init init --local\ncloud-init init\ncloud-init modules --mode=config\ncloud-init modules --mode=final\n
It is also possible to clean a cloud-init execution and have it execute again at next reboot. To do so, enter the following command:
cloud-init clean\n
Add -r to the previous command to reboot the instance once cloud-init has finishing cleaning."},{"location":"developers/#42-selinux","title":"4.2 SELinux","text":"SELinux is enabled on every instances of a Magic Castle cluster. Some applications do not provide SELinux policies which can lead to their malfunctionning when SELinux is enabled. It is possible to track down the reasons why SELinux is preventing an application to work properly using the command-line tool ausearch.
If you suspect application app-a to be denied by SELinux to work properly, run the following command as root:
ausearch -c app-a --raw | grep denied\n
To see all requests denied by SELinux:
ausearch --raw | grep denied\n
Sometime, the denials are hidden from regular logging. To display all denials, run the following command as root:
semodule --disable_dontaudit --build\n
then re-execute the application that is not working properly. Once you have found the denials that are the cause of the problem, you can create a new policy to allow the requests that were previously denied with the following command:
ausearch -c app-a --raw | grep denied | audit2allow -a -M app-a\n
Finally, you can install the generated policy using the command provided by auditallow.
"},{"location":"developers/#building-the-policy-package-file-pp-from-the-enforcement-file-te","title":"Building the policy package file (.pp) from the enforcement file (.te)","text":"If you need to tweak an existing enforcement file and you want to recompile the policy package, you can with the following commands:
checkmodule -M -m -o my_policy.mod my_policy.te\nsemodule_package -o my_policy.pp -m my_policy.mod\n
"},{"location":"developers/#references","title":"References","text":" - https://wiki.gentoo.org/wiki/SELinux
- https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details
"},{"location":"developers/#5-release","title":"5. Release","text":"To build a release, use the script release.sh located at the root of Magic Castle git repo.
Usage: release.sh VERSION [provider ...]\n
The script creates a folder named releases where it was called. The VERSION argument is expected to correspond to git tag in the puppet-magic_castle repo. It could also be a branch name or a commit. If the provider optional argument is left blank, release files will be built for all providers currently supported by Magic Castle.
Examples:
"},{"location":"matrix/","title":"Comparison of Cloud HPC Cluster Projects","text":"Name Creator First public release date Software license AWS ParallelCluster AWS November 12, 2018 Apache License v2 Azure CycleCloud Microsoft October 17, 2018 MIT License Azure HPC On-Demand Platform Microsoft April 23, 2021 MIT License Cluster in the Cloud Matt Williams - University of Bristol March 27, 2019 MIT License ElastiCluster Riccardo Murri - University of Zurich July 17, 2013 GPLv3 Google HPC-Toolkit Google May 26, 2022 Apache License v2 Magic Castle F\u00e9lix-Antoine Fortin - Compute Canada August 26, 2019 MIT License On-Demand Data Centre Adaptive Computing - - Slurm on GCP SchedMD March 14, 2018 Apache License v2"},{"location":"matrix/#supported-cloud-providers","title":"Supported cloud providers","text":"Name Alibaba Cloud AWS Azure Google Cloud IBM Cloud OpenStack Oracle Cloud OVH\u00a0 AWS ParallelCluster no yes no no no no no no Azure CycleCloud no no yes no no no no no Azure HPC On-Demand Platform no no yes no no no no no Cluster-in-the-Cloud no yes no yes no no yes no ElastiCluster* no yes yes yes no yes no - Google HPC-Toolkit no no no yes no no no no Magic Castle* no yes yes yes no yes no yes On-Demand Data Centre yes yes yes yes no no yes no Slurm on GCP no no no yes no no no no * The documentation provides instructions on how to add support for other cloud providers.
"},{"location":"matrix/#supported-operating-systems","title":"Supported operating systems","text":"Name CentOS 7 CentOS 8 Rocky Linux 8 AlmaLinux 8 Debian 10 Ubuntu 18 Ubuntu 20 Windows 10 AWS ParallelCluster yes yes yes yes yes no yes no Azure CycleCloud yes yes yes yes yes no yes - Azure HPC On-Demand Platform yes no no yes no yes no yes Google HPC-Toolkit yes no no no no no no no Cluster in the Cloud no yes no no no no no no ElastiCluster yes yes yes yes no no no no Magic Castle yes yes yes yes no no no no On-Demand Data Centre - - - - - - - - Slurm on GCP yes no no no yes no yes no"},{"location":"matrix/#supported-job-schedulers","title":"Supported job schedulers","text":"Name AwsBatch Grid Engine HTCondor Moab Open PBS PBS Pro Slurm AWS ParallelCluster yes no no no no no yes Azure CycleCloud no yes yes no no yes yes Azure HPC On-Demand Platform no no no no yes no yes Google HPC-Toolkit no no no no no no yes Cluster in the Cloud no no no no no no yes ElastiCluster no yes no no no no yes Magic Castle no no no no no no yes On-Demand Data Centre no no no yes no no no Slurm on GCP no no no no no no yes"},{"location":"matrix/#technologies","title":"Technologies","text":"Name Infrastructure configuration Programming languages Configuration management Scientific software AWS ParallelCluster CLI generating YAML Python Chef Spack Azure CycleCloud WebUI or CLI + templates Python Chef Bring your own Azure HPC On-Demand Platform YAML files + shell scripts Shell, Terraform Ansible, Packer CVMFS Cluster in the Cloud CLI generating Terraform code Python, Terraform Ansible, Packer EESSI ElastiCluster CLI interpreting an INI file Python, Shell Ansible Bring your own Google HPC-Toolkit CLI generating Terraform code Go, Terraform Ansible, Packer Spack Magic Castle Terraform modules Terraform Puppet CC-CVMFS, EESSI On-Demand Data Centre - - - - Slurm GCP Terraform modules Terraform Ansible, Packer Spack"},{"location":"sequence/","title":"Magic Castle Sequence Diagrams","text":"The following sequence diagrams illustrate the inner working of Magic Castle once terraform apply is called. Some details were left out of the diagrams, but every diagram is followed by references to the code files that were used to build it.
"},{"location":"sequence/#1-cluster-creation","title":"1. Cluster creation","text":""},{"location":"sequence/#notes","title":"Notes","text":" puppet-magic_castle.git does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.
"},{"location":"sequence/#references","title":"References","text":" magic_castle:/common/design/main.tfmagic_castle:/openstack/network-1.tfmagic_castle:/openstack/network-2.tfmagic_castle:/common/configuration/main.tfmagic_castle:/openstack/infrastructure.tfmagic_castle:/common/provision/main.tfmagic_castle:/dns/cloudflare/main.tfmagic_castle:/dns/acme/main.tf
"},{"location":"sequence/#2-configuration-with-cloud-init","title":"2. Configuration with cloud-init","text":""},{"location":"sequence/#notes_1","title":"Notes","text":" config_git_url repo does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.- While the diagram represents each step as completed sequentially, each node provisioning is independent. The only step that requires synchronisation between nodes and the management node is the puppet certificate generation.
"},{"location":"sequence/#references_1","title":"References","text":" magic_castle:/common/configuration/puppet.yaml
"},{"location":"sequence/#3-configuration-with-puppet","title":"3. Configuration with Puppet","text":""},{"location":"sequence/#references_2","title":"References","text":" puppet-magic_castle:/manifests/site.pppuppet-magic_castle:/profile/manifests/base.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/freeipa.pp
"},{"location":"sequence/#4-configuration-with-consul-and-consul-template","title":"4. Configuration with Consul and Consul Template","text":""},{"location":"sequence/#references_3","title":"References","text":" puppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/cvmfs.pppuppet-magic_castle:/profile/manifests/slurm.pp
"},{"location":"terraform_cloud/","title":"Terraform Cloud","text":"This document explains how to use Magic Castle with Terraform Cloud.
"},{"location":"terraform_cloud/#what-is-terraform-cloud","title":"What is Terraform Cloud?","text":"Terraform Cloud is HashiCorp\u2019s managed service that allows to provision infrastructure using a web browser or a REST API instead of the command-line. This also means that the provisioned infrastructure parameters can be modified by a team and the state is stored in the cloud instead of a local machine.
When provisioning in commercial cloud, Terraform Cloud can also provide a cost estimate of the resources.
"},{"location":"terraform_cloud/#getting-started-with-terraform-cloud","title":"Getting started with Terraform Cloud","text":" - Create a Terraform Cloud account
- Create an organization, join one or choose one available to you
"},{"location":"terraform_cloud/#managing-a-magic-castle-cluster-with-terraform-cloud","title":"Managing a Magic Castle cluster with Terraform Cloud","text":""},{"location":"terraform_cloud/#creating-the-workspace","title":"Creating the workspace","text":" - Create a git repository in GitHub, GitLab, or any of the version control system provider supported by Terraform Cloud
- In this git repository, add a copy of the Magic Castle example
main.tf available for the cloud of your choice - Log in Terraform Cloud account
- Create a new workspace
- Choose Type: \"Version control workflow\"
- Connect to VCS: choose the version control provider that hosts your repository
- Choose the repository that contains your
main.tf - Configure settings: tweak the name and description to your liking
- Click on \"Create workspace\"
You will be redirected automatically to your new workspace.
"},{"location":"terraform_cloud/#providing-cloud-provider-credentials-to-terraform-cloud","title":"Providing cloud provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your cloud provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each cloud provider and how to retrieve the values of the variable from the provider.
If you plan on using these environment variables with multiple workspaces, it is recommended to create a credential variable set in Terraform Cloud.
"},{"location":"terraform_cloud/#aws","title":"AWS","text":"You need to define these environment variables: - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY (sensitive)
The value of these variables can either correspond to the value of access key created on the AWS Security Credentials - Access keys page, or you can add user dedicated to Terraform Cloud in AWS IAM Users, and use its access key.
"},{"location":"terraform_cloud/#azure","title":"Azure","text":"You need to define these environment variables: - ARM_CLIENT_ID - ARM_CLIENT_SECRET (sensitive) - ARM_SUBSCRIPTION_ID - ARM_TENANT_ID
Refer to Terraform Azure Provider - Creating a Service Principal to know how to create a Service Principal and retrieve the values for these environment variables.
"},{"location":"terraform_cloud/#google-cloud","title":"Google Cloud","text":"You need to define this environment variable: - GOOGLE_CLOUD_KEYFILE_JSON (sensitive)
The value of the variable will be the content of a Google Cloud service account JSON key file expressed a single line string. Example:
{\"type\": \"service_account\",\"project_id\": \"project-id-1234\",\"private_key_id\": \"abcd1234\",...}\n
You can use jq to format the string from the JSON file provided by Google:
jq . -c project-name-123456-abcdefjg.json\n
"},{"location":"terraform_cloud/#openstack-ovh","title":"OpenStack / OVH","text":"You need to define these environment variables: - OS_AUTH_URL - OS_PROJECT_ID - OS_REGION_NAME - OS_INTERFACE - OS_IDENTITY_API_VERSION - OS_USER_DOMAIN_NAME - OS_USERNAME - OS_PASSWORD (sensitive)
Apart from OS_PASSWORD, the values for these variables are available in OpenStack RC file provided for your project.
If you prefer to use OpenStack application credentials, you need to define at least these variables: - OS_AUTH_TYPE\u00a0 - OS_AUTH_URL - OS_APPLICATION_CREDENTIAL_ID - OS_APPLICATION_CREDENTIAL_SECRET
and potentially these too: - OS_IDENTITY_API_VERSION\u00a0 - OS_REGION_NAME - OS_INTERFACE
The values for these variables are available in OpenStack RC file provided when creating the application credentials.
"},{"location":"terraform_cloud/#providing-dns-provider-credentials-to-terraform-cloud","title":"Providing DNS provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your DNS provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each DNS provider and how to retrieve the values of the variable from the provider.
"},{"location":"terraform_cloud/#cloudflare","title":"CloudFlare","text":"Refer to DNS - CloudFlare section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#google-cloud-dns","title":"Google Cloud DNS","text":"Refer to DNS - Google Cloud section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#managing-magic-castle-variables-with-terraform-cloud-ui","title":"Managing Magic Castle variables with Terraform Cloud UI","text":"It is possible to use Terraform Cloud web interface to define variable values in your main.tf. For example, you could want to define a guest password without writing it directly in main.tf to avoid displaying publicly.
To manage a variable with Terraform Cloud: 1. edit your main.tf to define the variables you want to manage. In the following example, we want to manage the number of nodes and the guest password.
Add the variables at the beginning of the `main.tf`:\n ```hcl\n variable \"nb_nodes\" {}\n variable \"password\" {}\n ```\n\nThen replace the static value by the variable in our `main.tf`,\n\ncompute node count\n ```hcl\n node = { type = \"p2-3gb\", tags = [\"node\"], count = var.nb_nodes }\n ```\nguest password\n ```hcl\n guest_passwd = var.password\n ```\n
- Commit and push this changes to your git repository.
- In Terraform Cloud workspace associated with that repository, go in \"Variables.
- Under \"Terraform Variables\", click the \"Add variable\" button and create a variable for each one defined previously in the\u00a0
main.tf. Check \"Sensitive\" if the variable content should not never be shown in the UI or the API.
You may edit the variables at any point of your cluster lifetime.
"},{"location":"terraform_cloud/#applying-changes","title":"Applying changes","text":"To create your cluster, apply changes made to your main.tf or the variables, you will need to queue a plan. When you push to the default branch of the linked git repository, a plan will be automatically created. You can also create a plan manually. To do so, click on the \"Queue plan manually\" button inside your workspace, then \"Queue plan\".
Once the plan has been successfully created, you can apply it using the \"Runs\" section. Click on the latest queued plan, then on the \"Apply plan\" button at the bottom of the plan page.
"},{"location":"terraform_cloud/#auto-apply","title":"Auto apply","text":"It is possible to apply automatically a successful plan. Go in the \"Settings\" section, and under \"Apply method\" select \"Auto apply\". Any following successful plan will then be automatically applied.
"},{"location":"terraform_cloud/#magic-castle-terraform-cloud-and-the-cli","title":"Magic Castle, Terraform Cloud and the CLI","text":"Terraform cloud only allows to apply or destroy the plan as stated in the main.tf, but sometimes it can be useful to run some other terraform commands that are only available through the command-line interface, for example terraform taint.
It is possible to import the terraform state of a cluster on your local computer and then use the CLI on it.
-
Log in Terraform cloud:
terraform login\n
-
Create a folder where the terraform state will be stored:
mkdir my-cluster-1\n
-
Create a file named cloud.tf with the following content in your cluster folder:
terraform {\n cloud {\n organization = \"REPLACE-BY-YOUR-TF-CLOUD-ORG\"\n workspaces {\n name = \"REPLACE-BY-THE-NAME-OF-YOUR-WORKSPACE\"\n }\n }\n}\n
replace the values of organization and name with the appropriate value for your cluster. -
Initialize the folder and retrieve the state:
terraform init\n
To confirm the workspace has been properly imported locally, you can list the resources using:
terraform state list\n
"},{"location":"terraform_cloud/#enable-magic-castle-autoscaling","title":"Enable Magic Castle Autoscaling","text":"Magic Castle in combination with Terraform Cloud (TFE) can be configured to give Slurm the ability to create and destroy instances based on the job queue content.
To enable this feature: 1. Create a TFE API Token and save it somewhere safe.
1.1. If you subscribe to Terraform Cloud Team & Governance plan, you can generate\na [Team API Token](https://www.terraform.io/cloud-docs/users-teams-organizations/api-tokens#team-api-tokens).\nThe team associated with this token requires no access to organization and can be secret.\nIt does not have to include any member. Team API token is preferable as its permissions can be\nrestricted to the minimum required for autoscale purpose.\n
-
Create a workspace in TFE
2.1. Make sure the repo is private as it will contain the API token.
2.2. If you generated a Team API Token in 1, provide access to the workspace to the team:
- Workspace Settings -> Team Access -> Add team and permissions
- Select the team
- Click on \"Customize permissions for this team\"
- Under \"Runs\" select \"Apply\"
- Under \"Variables\" select \"Read and write\"
- Leave the rest as is and click on \"Assign custom permissions\"
2.3 In Configure settings, under Advanced options, for Apply method, select Auto apply.
-
Create the environment variables of the cloud provider credentials in TFE
- Create a variable named
pool in TFE. Set value to [] and check HCL. - Add a file named
data.yaml in your git repo with the following content: yaml\u00a0 --- profile::slurm::controller::tfe_token: <TFE API token> profile::slurm::controller::tfe_workspace: <TFE workspace id> Complete the file by replacing <TFE API TOKEN> with the token generated at step 1 and <TFE workspace id> (i.e.: ws-...) by the id of the workspace created at step 2. It is recommended to encrypt the TFE API token before committing data.yaml in git. Refer to section 4.13.1 of README.md to know how to encrypt the token. - Add
data.yaml in git and push. -
Modify main.tf:
- If not already present, add the following definition of the pool variable at the beginning of your
main.tf.
variable \"pool\" { description = \"Slurm pool of compute nodes\" }\n
- Add instances to
instances with the tags pool and node. These are the nodes that Slurm will able to create and destroy. - If not already present, add the following line after the instances definition to pass the list of compute nodes from Terraform cloud workspace variable to the provider module:
pool = var.pool\n
- On the right-hand-side of
public_keys =, replace [file(\"~/.ssh/id_rsa.pub\")] by a list of SSH public keys that will have admin access to the cluster. - After the line
public_keys = ..., add hieradata = file(\"data.yaml\"). - After the line
hieradata = ..., add generate_ssh_key = true. This will provide Terraform Cloud SSH admin access to the cluster and it will be used to upload configuration files. - Stage changes, commit and push to git repo.
-
Go to your workspace in TFE, click on Actions -> Start a new run -> Plan and apply -> Start run. Then, click on \"Confirm & Apply\" and \"Confirm Plan\".
- Compute nodes defined in step 8 can be modified at any point in the cluster lifetime and more pool compute nodes can be added or removed if needed.
"},{"location":"terraform_cloud/#considerations-for-autoscaling","title":"Considerations for autoscaling","text":"To reduce the time required for compute nodes to become available in Slurm, consider creating a compute node image.
JupyterHub will time out by default after 300 seconds if a node is not spawned yet. Since it may take longer than this to spawn a node, even with an image created, consider increasing the timeout by adding the following to your YAML configuration file:
jupyterhub::jupyterhub_config_hash:\n SlurmFormSpawner:\n start_timeout: 900\n
Slurm 23 adds the possibility for sinfo to report nodes that are not yet spawned. This is useful if you want JupyterHub to be aware of those nodes, for example if you want to allow to use GPU nodes without keeping them online at all time. To use that version of Slurm, add the following to your YAML configuration file:
profile::slurm::base::slurm_version: '23.02'\n
"},{"location":"terraform_cloud/#troubleshoot-autoscaling-with-terraform-cloud","title":"Troubleshoot autoscaling with Terraform Cloud","text":"If after enabling autoscaling with Terraform Cloud for your Magic Castle cluster, the number of nodes does not increase when submitting jobs, verify the following points:
- Go to the Terraform Cloud workspace webpage, and look for errors in the runs. If the runs were only triggered by changes to the git repo, it means scaling signals from the cluster do not reach the Terraform cloud workspace or no signals were sent at all.
- Make sure the Terraform Cloud workspace id matches with the value of
profile::slurm::controller::tfe_workspace in data.yaml. - Execute
squeue on the cluster, and verify the reasons why jobs are still in the queue. If under the column (Reason), there is the keyword ReqNodeNotAvail, it implies Slurm tried to boot the listed nodes, but they would not show up before the timeout, therefore Slurm marked them as down. It can happen if your cloud provider is slow to build the instances, or following a configuration problem like in 2. When Slurm marks a node as down, a trace is left in slurmctld's log - using zgrep on the slurm controller node (typically mgmt1): sudo zgrep \"marking down\" /var/log/slurm/slurmctld.log*\n
To tell Slurm these nodes are available again, enter the following command: sudo /opt/software/slurm/bin/scontrol update nodename=node[Y-Z] state=IDLE\n
Replace node[Y-Z] by the hostname range listed next to ReqNodeNotAvail in squeue. - Under
mgmt1:/var/log/slurm, look for errors in the file slurm_resume.log.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Magic Castle Documentation","text":""},{"location":"#1-setup","title":"1. Setup","text":"To use Magic Castle you will need:
- Terraform (>= 1.4.0)
- Authenticated access to a cloud
- Ability to communicate with the cloud provider API from your computer
- A project with operational limits meeting the requirements described in Quotas subsection.
- ssh-agent running and tracking your SSH key
"},{"location":"#11-terraform","title":"1.1 Terraform","text":"To install Terraform, follow the tutorial or go directly on Terraform download page.
You can verify Terraform was properly installed by looking at the version in a terminal:
terraform version\n
"},{"location":"#12-authentication","title":"1.2 Authentication","text":""},{"location":"#121-amazon-web-services-aws","title":"1.2.1 Amazon Web Services (AWS)","text":" - Go to AWS - My Security Credentials
- Create a new access key.
- In a terminal, export
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, environment variables, representing your AWS Access Key and AWS Secret Key: export AWS_ACCESS_KEY_ID=\"an-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"a-secret-key\"\n
Reference: AWS Provider - Environment Variables
"},{"location":"#122-google-cloud","title":"1.2.2 Google Cloud","text":" - Install the Google Cloud SDK
- In a terminal, enter :
gcloud auth application-default login
"},{"location":"#123-microsoft-azure","title":"1.2.3 Microsoft Azure","text":" - Install Azure CLI
- In a terminal, enter :
az login
Reference : Azure Provider: Authenticating using the Azure CLI
"},{"location":"#124-openstack-ovh","title":"1.2.4 OpenStack / OVH","text":" -
Download your OpenStack Open RC file. It is project-specific and contains the credentials used by Terraform to communicate with OpenStack API. To download, using OpenStack web page go to: Project \u2192 API Access, then click on Download OpenStack RC File then right-click on OpenStack RC File (Identity API v3), Save Link as..., and save the file.
-
In a terminal located in the same folder as your OpenStack RC file, source the OpenStack RC file:
source *-openrc.sh\n
This command will ask for a password, enter your OpenStack password.
"},{"location":"#13-cloud-api","title":"1.3 Cloud API","text":"Once you are authenticated with your cloud provider, you should be able to communicate with its API. This section lists for each provider some instructions to test this.
"},{"location":"#131-aws","title":"1.3.1 AWS","text":" - In a dedicated temporary folder, create a file named
test_aws.tf with the following content: provider \"aws\" {\n region = \"us-east-1\"\n}\n\ndata \"aws_ec2_instance_type\" \"example\" {\n instance_type = \"t2.micro\"\n}\n
- In a terminal, move to where the file is located, then:
terraform init\n
- Finally, test terraform communication with AWS:
terraform plan\n
If everything is configured properly, terraform will output: No changes. Your infrastructure matches the configuration.\n
Otherwise, it will output: Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.\n
- You can delete the temporary folder and its content.
"},{"location":"#132-google-cloud","title":"1.3.2 Google Cloud","text":"In a terminal, enter:
gcloud projects list\n
It should output a table with 3 columns PROJECT_ID NAME PROJECT_NUMBER\n
Take note of the project_id of the Google Cloud project you want to use, you will need it later.
"},{"location":"#133-microsoft-azure","title":"1.3.3 Microsoft Azure","text":"In a terminal, enter:
az account show\n
It should output a JSON dictionary similar to this: {\n \"environmentName\": \"AzureCloud\",\n \"homeTenantId\": \"98467e3b-33c2-4a34-928b-ed254db26890\",\n \"id\": \"4dda857e-1d61-457f-b0f0-e8c784d1fb20\",\n \"isDefault\": true,\n \"managedByTenants\": [],\n \"name\": \"Pay-As-You-Go\",\n \"state\": \"Enabled\",\n \"tenantId\": \"495fc59f-96d9-4c3f-9c78-7a7b5f33d962\",\n \"user\": {\n \"name\": \"user@example.com\",\n \"type\": \"user\"\n }\n}\n
"},{"location":"#134-openstack-ovh","title":"1.3.4 OpenStack / OVH","text":" - In a dedicated temporary folder, create a file named
test_os.tf with the following content: terraform {\n required_providers {\n openstack = {\n source = \"terraform-provider-openstack/openstack\"\n }\n }\n}\ndata \"openstack_identity_auth_scope_v3\" \"scope\" {\n name = \"my_scope\"\n}\n
- In a terminal, move to where the file is located, then:
terraform init\n
- Finally, test terraform communication with OpenStack:
terraform plan\n
If everything is configured properly, terraform will output: No changes. Your infrastructure matches the configuration.\n
Otherwise, it will output: Error: Error creating OpenStack identity client:\n
if the OpenStack cloud API cannot be reached. - You can delete the temporary folder and its content.
"},{"location":"#14-quotas","title":"1.4 Quotas","text":""},{"location":"#141-aws","title":"1.4.1 AWS","text":"The default quotas set by Amazon are sufficient to build the Magic Castle AWS examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Amazon EC2 service quotas.
"},{"location":"#142-google-cloud","title":"1.4.2 Google Cloud","text":"The default quotas set by Google Cloud are sufficient to build the Magic Castle GCP examples. To increase the limits, or request access to special resources like GPUs, refer to Google Compute Engine Resource quotas.
"},{"location":"#143-microsoft-azure","title":"1.4.3 Microsoft Azure","text":"The default quotas set by Microsoft Azure are sufficient to build the Magic Castle Azure examples. To increase the limits, or request access to special resources like GPUs or high performance network interface, refer to Azure subscription and service limits, quotas, and constraints.
"},{"location":"#144-openstack","title":"1.4.4 OpenStack","text":"Minimum project requirements:
- 1 floating IP
- 3 security groups
- 1 network (see note 1)
- 1 subnet (see note 1)
- 1 router (see note 1)
- 3 volumes
- 3 instances
- 8 VCPUs
- 7 neutron ports
- 12 GB of RAM
- 8 security rules
- 80 GB of volume storage
Note 1: Magic Castle supposes the OpenStack project comes with a network, a subnet and a router already initialized. If any of these components is missing, you will need to create them manually before launching terraform.
- Create and manage networks, JUSUF user documentation
- Create and manage network - UI, OpenStack Documentation
- Create and manage network - CLI, OpenStack Documentation
"},{"location":"#145-ovh","title":"1.4.5 OVH","text":"The default quotas set by OVH are sufficient to build the Magic Castle OVH examples. To increase the limits, or request access to special resources like GPUs, refer to OVHcloud - Increasing Public Cloud quotas.
"},{"location":"#15-ssh-agent","title":"1.5 ssh-agent","text":"To transfer configuration files, Terraform will connect to your cluster using SSH. To avoid providing your private key to Terraform directly, you will have to add it to the authentication agent, ssh-agent.
To learn how to start ssh-agent and add keys, refer to ssh-agent - How to configure, forwarding, protocol.
Note 1: If you own more than one key pair, make sure the private key added to ssh-agent corresponds to the public key that will be granted access to your cluster (refer to section 4.9 public_keys).
Note 2: If you have no wish to use ssh-agent, you can configure Magic Castle to generate a key pair specific to your cluster. The public key will be written in the sudoer authorized_keys and Terraform will be able to connect the cluster using the corresponding private key. For more information, refer to section 4.15 generate_ssh_key.
"},{"location":"#2-cloud-cluster-architecture-overview","title":"2. Cloud Cluster Architecture Overview","text":""},{"location":"#3-initialization","title":"3. Initialization","text":""},{"location":"#31-main-file","title":"3.1 Main File","text":" - Go to https://github.com/ComputeCanada/magic_castle/releases.
- Download the latest release of Magic Castle for your cloud provider.
- Open a Terminal.
- Uncompress the release:
tar xvf magic_castle*.tar.gz - Rename the release folder after your favourite superhero:
mv magic_castle* hulk - Move inside the folder:
cd hulk
The file main.tf contains Terraform modules and outputs. Modules are files that define a set of resources that will be configured based on the inputs provided in the module block. Outputs are used to tell Terraform which variables of our module we would like to be shown on the screen once the resources have been instantiated.
This file will be our main canvas to design our new clusters. As long as the module block parameters suffice to our need, we will be able to limit our configuration to this sole file. Further customization will be addressed during the second part of the workshop.
"},{"location":"#32-terraform","title":"3.2 Terraform","text":"Terraform fetches the plugins required to interact with the cloud provider defined by our main.tf once when we initialize. To initialize, enter the following command:
terraform init\n
The initialization is specific to the folder where you are currently located. The initialization process looks at all .tf files and fetches the plugins required to build the resources defined in these files. If you replace some or all .tf files inside a folder that has already been initialized, just call the command again to make sure you have all plugins.
The initialization process creates a .terraform folder at the root of your current folder. You do not need to look at its content for now.
"},{"location":"#321-terraform-modules-upgrade","title":"3.2.1 Terraform Modules Upgrade","text":"Once Terraform folder has been initialized, it is possible to fetch the newest version of the modules used by calling:
terraform init -upgrade\n
"},{"location":"#4-configuration","title":"4. Configuration","text":"In the main.tf file, there is a module named after your cloud provider, i.e.: module \"openstack\". This module corresponds to the high-level infrastructure of your cluster.
The following sections describes each variable that can be used to customize the deployed infrastructure and its configuration. Optional variables can be absent from the example module. The order of the variables does not matter, but the following sections are ordered as the variables appear in the examples.
"},{"location":"#41-source","title":"4.1 source","text":"The first line of the module block indicates to Terraform where it can find the files that define the resources that will compose your cluster. In the releases, this variable is a relative path to the cloud provider folder (i.e.: ./aws).
Requirement: Must be a path to a local folder containing the Magic Castle Terraform files for the cloud provider of your choice. It can also be a git repository. Refer to Terraform documentation on module source for more information.
Post build modification effect: terraform init will have to be called again and the next terraform apply might propose changes if the infrastructure describe by the new module is different.
"},{"location":"#42-config_git_url","title":"4.2 config_git_url","text":"Magic Castle configuration management is handled by Puppet. The Puppet configuration files are stored in a git repository. This is typically ComputeCanada/puppet-magic_castle repository on GitHub.
Leave this variable to its current value to deploy a vanilla Magic Castle cluster.
If you wish to customize the instances' role assignment, add services, or develop new features for Magic Castle, fork the ComputeCanada/puppet-magic_castle and point this variable to your fork's URL. For more information on Magic Castle puppet configuration customization, refer to MC developer documentation.
Requirement: Must be a valid HTTPS URL to a git repository describing a Puppet environment compatible with Magic Castle. If the repo is private, generate an access token with a permission to read the repo content, and provide the token in the config_git_url like this:
config_git_url = \"https://oauth2:${oauth-key-goes-here}@domain.com/username/repo.git\"\n
This works for GitHub and GitLab (including community edition). Post build modification effect: no effect. To change the Puppet configuration source, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#43-config_version","title":"4.3 config_version","text":"Since Magic Cluster configuration is managed with git, it is possible to specify which version of the configuration you wish to use. Typically, it will match the version number of the release you have downloaded (i.e: 9.3).
Requirement: Must refer to a git commit, tag or branch existing in the git repository pointed by config_git_url.
Post build modification effect: none. To change the Puppet configuration version, destroy the cluster or change it manually on the Puppet server.
"},{"location":"#44-cluster_name","title":"4.4 cluster_name","text":"Defines the ClusterName variable in slurm.conf and the name of the cluster in the Slurm accounting database (see slurm.conf documentation).
Requirement: Must be lowercase alphanumeric characters and start with a letter. It can include dashes. cluster_name must be 40 characters or less.
Post build modification effect: destroy and re-create all instances at next terraform apply.
"},{"location":"#45-domain","title":"4.5 domain","text":"Defines
- the Kerberos realm name when initializing FreeIPA.
- the internal domain name and the
resolv.conf search domain as int.{cluster_name}.{domain}
Optional modules following the current module in the example main.tf can be used to register DNS records in relation to your cluster if the DNS zone of this domain is administered by one of the supported providers. Refer to section 6. DNS Configuration and SSL Certificates for more details.
Requirements:
Post build modification effect: destroy and re-create all instances at next terraform apply.
"},{"location":"#46-image","title":"4.6 image","text":"Defines the name of the image that will be used as the base image for the cluster nodes.
You can use a custom image if you wish, but configuration management should be mainly done through Puppet. Image customization is mostly envisioned as a way to accelerate the configuration process by applying the security patches and OS updates in advance.
To specify a different image for an instance type, use the image instance attribute
Requirements: the operating system on the image must be from the RedHat family. This includes CentOS (7, 8), Rocky Linux (8), and AlmaLinux (8).
Post build modification effect: none. If this variable is modified, existing instances will ignore the change and future instances will use the new value.
"},{"location":"#461-aws","title":"4.6.1 AWS","text":"The image field needs to correspond to the Amazon Machine Image (AMI) ID. AMI IDs are specific to regions and architectures. Make sure to use the right ID for the region and CPU architecture you are using (i.e: x86_64).
To find out which AMI ID you need to use, refer to - AlmaLinux OS Amazon Web Services AMIs - CentOS list of official images available on the AWS Marketplace - Rocky Linux
Note: Before you can use the AMI, you will need to accept the usage terms and subscribe to the image on AWS Marketplace. On your first deployment, you will be presented an error similar to this one:
\u2502 Error: Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=cvugziknvmxgqna9noibqnnsy\n\u2502 status code: 401, request id: 1f04a85a-f16a-41c6-82b5-342dc3dd6a3d\n\u2502\n\u2502 on aws/infrastructure.tf line 67, in resource \"aws_instance\" \"instances\":\n\u2502 67: resource \"aws_instance\" \"instances\" {\n
To accept the terms and fix the error, visit the link provided in the error output, then click on the Click to Subscribe yellow button."},{"location":"#462-microsoft-azure","title":"4.6.2 Microsoft Azure","text":"The image field for Azure can either be a string or a map.
A string image specification will correspond to the image id. Image ids can be retrieved using the following command-line:
az image builder list\n
A map image specification needs to contain the following fields publisher, offer sku, and optionally version. The map is used to specify images found in Azure Marketplace. Here is an example:
{\n publisher = \"OpenLogic\",\n offer = \"CentOS-CI\",\n sku = \"7-CI\"\n}\n
"},{"location":"#463-openstack","title":"4.6.3 OpenStack","text":"The image name can be a regular expression. If more than one image is returned by the query to OpenStack, the most recent is selected.
"},{"location":"#47-instances","title":"4.7 instances","text":"The instances variable is a map that defines the virtual machines that will form the cluster. The map' keys define the hostnames and the values are the attributes of the virtual machines.
Each instance is identified by a unique hostname. An instance's hostname is written as the key followed by its index (1-based). The following map:
instances = {\n mgmt = { type = \"p2-4gb\", tags = [...] },\n login = { type = \"p2-4gb\", count = 1, tags = [...] },\n node = { type = \"c2-15gb-31\", count = 2, tags = [...] },\n gpu-node = { type = \"gpu2.large\", count = 3, tags = [...] },\n}\n
will spawn instances with the following hostnames: mgmt1\nlogin1\nnode1\nnode2\ngpu-node1\ngpu-node2\ngpu-node3\n
Hostnames must follow a set of rules, from hostname man page:
Valid characters for hostnames are ASCII letters from a to z, the digits from 0 to 9, and the hyphen (-). A hostname may not start with a hyphen.
Two attributes are expected to be defined for each instance: 1. type: name for varying combinations of CPU, memory, GPU, etc. (i.e: t2.medium); 2. tags: list of labels that defines the role of the instance.
"},{"location":"#471-tags","title":"4.7.1 tags","text":"Tags are used in the Terraform code to identify if devices (volume, network) need to be attached to an instance, while in Puppet code tags are used to identify roles of the instances.
Terraform tags:
login: identify instances accessible with SSH from Internet and pointed by the domain name A recordspool: identify instances created only when their hostname appears in the var.pool list.proxy: identify instances accessible with HTTP/HTTPS and pointed by the vhost A recordspublic: identify instances that need to have a public ip address reachable from Internetpuppet: identify instances configured as Puppet serversspot: identify instances that are to be spawned as spot/preemptible instances. This tag is supported in AWS, Azure and GCP. It is ignored by OpenStack and OVH.efa: attach an Elastic Fabric Adapter network interface to the instance. This tag is supported in AWS.ssl: identify instances that receive a copy of the SSL wildcard certificate for the domain
Puppet tags expected by the puppet-magic_castle environment.
login: identify a login instance (minimum: 2 CPUs, 2GB RAM)mgmt: identify a management instance i.e: FreeIPA server, Slurm controller, Slurm DB (minimum: 2 CPUs, 6GB RAM)nfs: identify the instance that acts as an NFS server.node: identify a compute node instance (minimum: 1 CPUs, 2GB RAM)pool: when combined with node, it identifies compute nodes that Slurm can resume/suspend to meet workload demand.proxy: identify the instance that executes the Caddy reverse proxy and JupyterHub.
In the Magic Castle Puppet environment, an instance cannot be tagged as mgmt and proxy.
You are free to define your own additional tags.
"},{"location":"#472-optional-attributes","title":"4.7.2 Optional attributes","text":"Optional attributes can be defined:
count: number of virtual machines with this combination of hostname prefix, type and tags to create (default: 1).image: specification of the image to use for this instance type. (default: global image value). Refer to section 10.12 - Create a compute node image to learn how this attribute can be leveraged to accelerate compute node configuration.-
disk_type: type of the instance's root disk (default: see the next table).
Provider disk_type disk_size (GiB) Azure Premium_LRS 30 AWS gp2 10 GCP pd-ssd 20 OpenStack null 10 OVH null 10 -
disk_size: size in gibibytes (GiB) of the instance's root disk containing the operating system and service software (default: see the previous table).
mig: map of NVIDIA Multi-Instance GPU (MIG) short profile names and count used to partition the instances' GPU, example for an A100: mig = { \"1g.5gb\" = 2, \"2g.10gb\" = 1, \"3g.20gb\" = 1 }\n
This is only functional with MIG supported GPUs, and with x86-64 processors (see NVIDIA/mig-parted issue #30).
For some cloud providers, it possible to define additional attributes. The following sections present the available attributes per provider.
"},{"location":"#aws","title":"AWS","text":"For instances with the spot tags, these attributes can also be set:
wait_for_fulfillment (default: true)spot_type (default: permanent)instance_interruption_behavior (default: stop)spot_price (default: not set)block_duration_minutes (default: not set) [note 1] For more information on these attributes, refer to aws_spot_instance_request argument reference
Note 1: block_duration_minutes is not available to new AWS accounts or accounts without billing history - AWS EC2 Spot Instance requests. When not available, its usage can trigger quota errors like this:
Error requesting spot instances: MaxSpotInstanceCountExceeded: Max spot instance count exceeded\n
"},{"location":"#azure","title":"Azure","text":"For instances with the spot tags, these attributes can also be set:
max_bid_price (default: not set)eviction_policy (default: Deallocate) For more information on these attributes, refer to azurerm_linux_virtual_machine argument reference
"},{"location":"#gcp","title":"GCP","text":" gpu_type: name of the GPU model to attach to the instance. Refer to Google Cloud documentation for the list of available models per regiongpu_count: number of GPUs of the gpu_type model to attach to the instance
"},{"location":"#473-post-build-modification-effect","title":"4.7.3 Post build modification effect","text":"Modifying any part of the map after the cluster is built will only affect the type of instances associated with what was modified at the next terraform apply.
"},{"location":"#48-volumes","title":"4.8 volumes","text":"The volumes variable is a map that defines the block devices that should be attached to instances that have the corresponding key in their list of tags. To each instance with the tag, unique block devices are attached, no multi-instance attachment is supported.
Each volume in map is defined a key corresponding to its and a map of attributes:
size: size of the block device in GB.type (optional): type of volume to use. Default value per provider:- Azure:
Premium_LRS - AWS:
gp2 - GCP:
pd-ssd - OpenStack:
null - OVH:
null
Volumes with a tag that have no corresponding instance will not be created.
In the following example:
instances = {\u00a0\n server = { type = \"p4-6gb\", tags = [\"nfs\"] }\n}\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 100 }\n scratch = { size = 100 }\n }\n mds = {\n oss1 = { size = 500 }\n oss2 = { size = 500 }\n }\n}\n
The instance server1 will have three volumes attached to it. The volumes tagged mds are not created since no instances have the corresponding tag.
To define an infrastructure with no volumes, set the volumes variable to an empty map:
volumes = {}\n
Post build modification effect: destruction of the corresponding volumes and attachments, and creation of new empty volumes and attachments. If an no instance with a corresponding tag exist following modifications, the volumes will be deleted.
"},{"location":"#49-public_keys","title":"4.9 public_keys","text":"List of SSH public keys that will have access to your cluster sudoer account.
Note 1: You need to add the private key associated with one of the public keys to your local authentication agent (i.e: ssh-add) for Terraform to be able to copy Puppet configuration files with scp on the cluster. Otherwise, Magic Castle can create a key pair for unique to this cluster, see section 4.15 - generate_ssh_key (optional).
Post build modification effect: trigger scp of hieradata files at next terraform apply. The sudoer account authorized_keys file will be updated by each instance's Puppet agent following the copy of the hieradata files.
"},{"location":"#410-nb_users-optional","title":"4.10 nb_users (optional)","text":"default value: 0
Defines how many guest user accounts will be created in FreeIPA. Each user account shares the same randomly generated password. The usernames are defined as userX where X is a number between 1 and the value of nb_users (zero-padded, i.e.: user01 if X < 100, user1 if X < 10).
If an NFS NFS home volume is defined, each user will have a home folder on a shared NFS storage hosted on the NFS server node.
User accounts do not have sudoer privileges. If you wish to use sudo, you will have to login using the sudoer account and the SSH keys listed in public_keys.
If you would like to add a user account after the cluster is built, refer to section 10.3 and 10.4.
Requirement: Must be an integer, minimum value is 0.
Post build modification effect: trigger scp of hieradata files at next terraform apply. If nb_users is increased, new guest accounts will be created during the following Puppet run on mgmt1. If nb_users is decreased, it will have no effect: the guest accounts already created will be left intact.
"},{"location":"#411-guest_passwd-optional","title":"4.11 guest_passwd (optional)","text":"default value: 4 random words separated by dots
Defines the password for the guest user accounts instead of using a randomly generated one.
Requirement: Minimum length 8 characters.
The password can be provided in a PKCS7 encrypted form. Refer to sub-section 4.14 eyaml_key for instructions on how to encrypt the password.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Password of all guest accounts will be changed to match the new password value.
"},{"location":"#412-sudoer_username-optional","title":"4.12 sudoer_username (optional)","text":"default value: centos
Defines the username of the account with sudo privileges. The account ssh authorized keys are configured with the SSH public keys with public_keys.
Post build modification effect: none. To change sudoer username, destroy the cluster or redefine the value of profile::base::sudoer_username in hieradata.
"},{"location":"#413-hieradata-optional","title":"4.13 hieradata (optional)","text":"default value: empty string
Defines custom variable values that are injected in the Puppet hieradata file. Useful to override common configuration of Puppet classes.
List of useful examples:
- Receive logs of Puppet runs with changes to your email, add the following line to the string:
profile::base::admin_email: \"me@example.org\"\n
- Define ip addresses that can never be banned by fail2ban:
profile::fail2ban::ignore_ip: ['132.203.0.0/16', '8.8.8.8']\n
- Remove one-time password field from JupyterHub login page:
jupyterhub::enable_otp_auth: false\n
- Setup AlertManager to receive Prometheus alerts on Slack:
prometheus::alertmanager::route:\n group_by:\n - 'alertname'\n - 'cluster'\n - 'service'\n group_wait: '5s'\n group_interval: '5m'\n repeat_interval: '3h'\n receiver: 'slack'\n\nprometheus::alertmanager::receivers:\n - name: 'slack'\n slack_configs:\n - api_url: 'https://hooks.slack.com/services/ABCDEFG123456'\n channel: \"#channel\"\n send_resolved: true\n username: 'username'\n
Refer to the following Puppet modules' documentation to know more about the key-values that can be defined:
- puppet-magic_castle
- puppet-jupyterhub
- puppet-prometheus
The file created from this string can be found on puppet as
/etc/puppetlabs/data/user_data.yaml\n
Requirement: The string needs to respect the YAML syntax.
Post build modification effect: trigger scp of hieradata files at next terraform apply. Each instance's Puppet agent will be reloaded following the copy of the hieradata files.
"},{"location":"#414-eyaml_private-optional","title":"4.14 eyaml_private (optional)","text":"default value: empty string
Defines the private RSA key required to decrypt the values encrypted with hiera-eyaml PKCS7. This key will be copied on the Puppet server.
Post build modification effect: trigger scp of private key file at next terraform apply.
"},{"location":"#4141-generate-eyaml-encryption-keys","title":"4.14.1 Generate eyaml encryption keys","text":"If you plan to track the cluster configuration files in git (i.e:main.tf, user_data.yaml), it would be a good idea to encrypt the sensitive property values.
Magic Castle uses hiera-eyaml to provide a per-value encryption of sensitive properties to be used by Puppet.
The private key and its corresponding public key wrapped in a X509 certificate can be generated with openssl:
openssl req -x509 -nodes -newkey rsa:2048 -keyout private_key.pkcs7.pem -out public_key.pkcs7.pem -batch\n
or with eyaml:
eyaml createkeys --pkcs7-public-key=public_key.pkcs7.pem --pkcs7-private-key=private_key.pkcs7.pem\n
"},{"location":"#4142-encrypting-sensitive-properties","title":"4.14.2 Encrypting sensitive properties","text":"To encrypt a sensitive property with openssl:
echo -n 'your-secret' | openssl smime -encrypt -aes-256-cbc -outform der public_key.pkcs7.pem | openssl base64 -A | xargs printf \"ENC[PKCS7,%s]\\n\"\n
To encrypt a sensitive property with eyaml:
eyaml encrypt -s 'your-secret' --pkcs7-public-key=public_key.pkcs7.pem -o string\n
"},{"location":"#4143-terraform-cloud","title":"4.14.3 Terraform cloud","text":"To provide the value of this variable via Terraform Cloud, encode the private key content with base64:
openssl base64 -A -in private_key.pkcs7.pem\n
Define a variable in your main.tf:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n}\n
Then make sure to decode it before passing it to the cloud provider module:
variable \"tfc_eyaml_key\" {}\nmodule \"openstack\" {\n ...\n eyaml_key = base64decode(var.tfc_eyaml_key)\n ...\n}\n
"},{"location":"#415-firewall_rules-optional","title":"4.15 firewall_rules (optional)","text":"default value:
{\n ssh = { \"from_port\" = 22, \"to_port\" = 22, tag = \"login\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n http = { \"from_port\" = 80, \"to_port\" = 80, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n https = { \"from_port\" = 443, \"to_port\" = 443, tag = \"proxy\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n globus = { \"from_port\" = 2811, \"to_port\" = 2811, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"54.237.254.192/29\" },\n myproxy = { \"from_port\" = 7512, \"to_port\" = 7512, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" },\n gridftp = { \"from_port\" = 50000, \"to_port\" = 51000, tag = \"dtn\", \"protocol\" = \"tcp\", \"cidr\" = \"0.0.0.0/0\" }\n}\n
Defines a map of firewall rules that control external traffic to the public nodes. Each rule is defined as a map of key-value pairs and has to be assigned a unique name:
from_port (req.): the lower part of the allowed port range, valid integer value needs to be between 1 and 65535.to_port (req.): the higher part of the allowed port range, valid integer value needs to be between 1 and 65535.tag (req.): instances with this tag will be assigned this firewall rule.ethertype (opt. default: \"IPv4\"): the layer 3 protocol type (\"IPv4\" or \"IPv6\").protocol (opt. default: \"tcp\"): the layer 4 protocol type.cidr (opt. default: \"0.0.0.0/0\"): the remote CIDR, the value needs to be a valid CIDR (i.e. 192.168.0.0/16).
If you would like Magic Castle to be able to transfer files and update the state of the cluster in Puppet, make sure there exists at least one effective firewall rule where from_port <= 22 <= to_port and for which the external IP address of the machine that executes Terraform is in the CIDR range (i.e: cidr = \"0.0.0.0/0\" being the most permissive). This corresponds to the ssh rule in the default firewall rule map. This guarantees that Terraform will be able to use SSH to connect to the cluster from anywhere. For more information about this requirement, refer to Magic Castle's bastion tag computation code.
Post build modification effect: modify the cloud provider firewall rules at next terraform apply.
"},{"location":"#416-generate_ssh_key-optional","title":"4.16 generate_ssh_key (optional)","text":"default_value: false
If true, Terraform will generate an ssh key pair that would then be used when copying file with Terraform file-provisioner. The public key will be added to the sudoer account authorized keys.
This parameter is useful when Terraform does not have access to one of the private key associated with the public keys provided in public_keys.
Post build modification effect:
false -> true: will cause Terraform failure. Terraform will try to use the newly created private SSH key to connect to the cluster, while the corresponding public SSH key is yet registered with the sudoer account.true -> false: will trigger a scp of terraform_data.yaml at next terraform apply. The Terraform public SSH key will be removed from the sudoer account authorized_keys file at next Puppet agent run.
"},{"location":"#417-software_stack-optional","title":"4.17 software_stack (optional)","text":"default_value: \"alliance\"
Defines the scientific software environment that users have access when they login. Possible values are:
- default -
\"alliance\" / \"computecanada\": Digital Alliance Research Alliance of Canada scientific software environment (previously Compute Canada environment) \"eessi\": European Environment for Scientific Software Installation (EESSI)null / \"\": no scientific software environment
Post build modification effect: trigger scp of hieradata files at next terraform apply.
"},{"location":"#418-pool-optional","title":"4.18 pool (optional)","text":"default_value: []
Defines a list of hostnames with the tag \"pool\" that have to be online. This variable is typically managed by the workload scheduler through Terraform API. For more information, refer to Enable Magic Castle Autoscaling
Post build modification effect: pool tagged hosts with name present in the list will be instantiated, others will stay uninstantiated or will be destroyed if previously instantiated.
"},{"location":"#419-skip_upgrade-optional","title":"4.19 skip_upgrade (optional)","text":"default_value = false
If true, the base image packages will not be upgraded during the first boot. By default, all packages are upgraded.
Post build modification effect: No effect on currently built instances. Ones created after the modification will take into consideration the new value of the parameter to determine whether they should upgrade the base image packages or not.
"},{"location":"#420-puppetfile-optional","title":"4.20 puppetfile (optional)","text":"default_value = \"\"
Defines a complement of modules to install with librarian puppet when initializing the Puppet environment on the first boot of the Puppet server. If the provided string include the forge\u00a0setting, the provided content will replace entirely the Magic Castle environment's Puppetfile.
Post build modification effect: None. To modify the Puppetfile after the cluster is initialized, log on the Puppet server and modify /etc/puppetlabs/code/environments/production/Puppetfile.
"},{"location":"#5-cloud-specific-configuration","title":"5. Cloud Specific Configuration","text":""},{"location":"#51-amazon-web-services","title":"5.1 Amazon Web Services","text":""},{"location":"#511-region","title":"5.1.1 region","text":"Defines the label of the AWS EC2 region where the cluster will be created (i.e.: us-east-2).
Requirement: Must be in the list of available EC2 regions.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#512-availability_zone-optional","title":"5.1.2 availability_zone (optional)","text":"default value: None
Defines the label of the data center inside the AWS region where the cluster will be created (i.e.: us-east-2a). If left blank, it chosen at random amongst the availability zones of the selected region.
Requirement: Must be in a valid availability zone for the selected region. Refer to AWS documentation to find out how list the availability zones.
"},{"location":"#52-microsoft-azure","title":"5.2 Microsoft Azure","text":""},{"location":"#521-location","title":"5.2.1 location","text":"Defines the label of the Azure location where the cluster will be created (i.e.: eastus).
Requirement: Must be a valid Azure location. To get the list of available location, you can use Azure CLI : az account list-locations -o table.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#522-azure_resource_group-optional","title":"5.2.2 azure_resource_group (optional)","text":"default value: None
Defines the name of an already created resource group to use. Terraform will no longer attempt to manage a resource group for Magic Castle if this variable is defined and will instead create all resources within the provided resource group. Define this if you wish to use an already created resource group or you do not have a subscription-level access to create and destroy resource groups.
Post build modification effect: rebuild of all instances at next terraform apply.
"},{"location":"#523-plan-optional","title":"5.2.3 plan (optional)","text":"default value:
{\n name = null\n product = null\n publisher = null\n}\n
Purchase plan information for Azure Marketplace image. Certain images from Azure Marketplace requires a terms acceptance or a fee to be used. When using this kind of image, you must supply the plan details.
For example, to use the official AlmaLinux image, you have to first add it to your account. Then to use it with Magic Castle, you must supply the following plan information:
plan = {\n name = \"8_7\"\n product = \"almalinux\"\n publisher = \"almalinux\"\n}\n
"},{"location":"#53-google-cloud","title":"5.3 Google Cloud","text":""},{"location":"#531-project","title":"5.3.1 project","text":"Defines the label of the unique identifier associated with the Google Cloud project in which the resources will be created. It needs to corresponds to GCP project ID, which is composed of the project name and a randomly assigned number.
Requirement: Must be a valid Google Cloud project ID.
Post build modification effect: rebuild of all resources at next terraform apply.
"},{"location":"#532-region","title":"5.3.2 region","text":"Defines the name of the specific geographical location where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud region. Refer to Google Cloud documentation for the list of available regions and their characteristics.
"},{"location":"#533-zone-optional","title":"5.3.3 zone (optional)","text":"default value: None
Defines the name of the zone within the region where the cluster resources will be hosted.
Requirement: Must be a valid Google Cloud zone. Refer to Google Cloud documentation for the list of available zones and their characteristics.
"},{"location":"#54-openstack-and-ovh","title":"5.4 OpenStack and OVH","text":""},{"location":"#541-os_floating_ips-optional","title":"5.4.1 os_floating_ips (optional)","text":"default value: {}
Defines a map as an association of instance names (key) to pre-allocated floating ip addresses (value). Example:
os_floating_ips = {\n login1 = \"132.213.13.59\"\n login2 = \"132.213.13.25\"\n }\n
- instances tagged as public that have an entry in this map will be assigned the corresponding ip address;
- instances tagged as public that do not have an entry in this map will be assigned a floating ip managed by Terraform.
- instances not tagged as public that have an entry in this map will not be assigned a floating ip.
This variable can be useful if you manage your DNS manually and you would like the keep the same domain name for your cluster at each build.
Post build modification effect: change the floating ips assigned to the public instances.
"},{"location":"#542-os_ext_network-optional","title":"5.4.2 os_ext_network (optional)","text":"default value: None
Defines the name of the external network that provides the floating ips. Define this only if your OpenStack cloud provides multiple external networks, otherwise, Terraform can find it automatically.
Post build modification effect: change the floating ips assigned to the public nodes.
"},{"location":"#544-subnet_id-optional","title":"5.4.4 subnet_id (optional)","text":"default value: None
Defines the ID of the internal IPV4 subnet to which the instances are connected. Define this if you have or intend to have more than one subnets defined in your OpenStack project. Otherwise, Terraform can find it automatically. Can be used to force a v4 subnet when both v4 and v6 exist.
Post build modification effect: rebuild of all instances at next terraform apply.
"},{"location":"#6-dns-configuration-and-wildcard-ssl-certificate","title":"6. DNS Configuration and Wildcard SSL Certificate","text":"Some functionalities in Magic Castle require the registration of DNS records under the cluster name in the selected domain. This includes web services like JupyterHub, Mokey and FreeIPA web portal.
If your domain DNS records are managed by one of the supported providers, follow the instructions in the corresponding sections to have the cluster's DNS records created and tracked by Magic Castle.
If your DNS provider is not supported, you can manually create the records. Refer to the subsection 6.3 for more details.
Optionally, Magic Castle can issue with Let's encrypt DNS-01 challenge a wildcard certificate of the form *.${cluster_name}.${domain_name}. Refer to the subsection 6.4 for more information. If no wildcard certificate is issued, the reverse proxy server Caddy will automatically issue one certificate per virtual host.
"},{"location":"#61-cloudflare","title":"6.1 Cloudflare","text":" - Uncomment the
dns module for Cloudflare in your main.tf. - Uncomment the
output \"hostnames\" block. - Download and install the Cloudflare Terraform module:
terraform init. - Export the environment variables
CLOUDFLARE_EMAIL and CLOUDFLARE_API_KEY, where CLOUDFLARE_EMAIL is your Cloudflare account email address and CLOUDFLARE_API_KEY is your account Global API Key available in your Cloudflare profile.
"},{"location":"#612-cloudflare-api-token","title":"6.1.2 Cloudflare API Token","text":"If you prefer using an API token instead of the global API key, you will need to configure a token with the following four permissions with the Cloudflare API Token interface.
Section Subsection Permission Zone DNS Edit Instead of step 5, export only CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_API_TOKEN, and CLOUDFLARE_DNS_API_TOKEN equal to the API token generated previously.
"},{"location":"#62-google-cloud","title":"6.2 Google Cloud","text":"requirement: Install the Google Cloud SDK
- Login to your Google account with gcloud CLI :
gcloud auth application-default login - Uncomment the
dns module for Google Cloud in your main.tf. - Uncomment the
output \"hostnames\" block. - In
main.tf's dns module, configure the variables project and zone_name with their respective values as defined by your Google Cloud project. - Download and install the Google Cloud Terraform module:
terraform init.
"},{"location":"#63-unsupported-providers","title":"6.3 Unsupported providers","text":"If your DNS provider is not currently supported by Magic Castle, you can create the DNS records manually.
Magic Castle provides a module that creates a text file with the DNS records that can then be imported manually in your DNS zone. To use this module, add the following snippet to your main.tf:
module \"dns\" {\n source = \"./dns/txt\"\n name = module.openstack.cluster_name\n domain = module.openstack.domain\n public_instances = module.openstack.public_instances\n}\n
Find and replace openstack in the previous snippet by your cloud provider of choice if not OpenStack (i.e: aws, gcp, etc.).
The file will be created after the terraform apply in the same folder as your main.tf and will be named as ${name}.${domain}.txt.
"},{"location":"#64-wildcard-ssl-certificate","title":"6.4 Wildcard SSL Certificate","text":"Requirement: A private key associated with one of the public keys needs to be tracked (i.e: ssh-add) by the local authentication agent (i.e: ssh-agent). This module uses the ssh-agent tracked SSH keys to authenticate and to copy SSL certificate files to the proxy nodes after their creation.
To issue a wildcard SSL certificate with Magic Castle, modify the dns module call in your main.tf like this:
- add the input:
issue_wildcard_cert = true - add the input:
email = \"replace.by@your.email\"
"},{"location":"#641-acme-account-private-key","title":"6.4.1 ACME Account Private Key","text":"To create the wildcard SSL certificate associated with the domain name, Magic Castle creates a private key and register a new ACME account with this key. This account registration process is done for each new cluster. However, ACME limits the number of new accounts that can be created to a maximum of 10 per IP Address per 3 hours.
If you plan to create more than 10 clusters per 3 hours, we recommend registering an ACME account first and then provide its private key in PEM format to Magic Castle DNS module, using the acme_key_pem variable.
"},{"location":"#how-to-generate-an-acme-account-private-key","title":"How to Generate an ACME Account Private Key","text":"In a separate folder, create a file with the following content
terraform {\n required_version = \">= 1.2.1\"\n required_providers {\n acme = {\n source = \"vancluever/acme\"\n }\n tls = {\n source = \"hashicorp/tls\"\n }\n }\n}\n\nvariable \"email\" {}\n\nprovider \"acme\" {\n server_url = \"https://acme-v02.api.letsencrypt.org/directory\"\n}\nresource \"tls_private_key\" \"private_key\" {\n algorithm = \"RSA\"\n}\nresource \"acme_registration\" \"reg\" {\n account_key_pem = tls_private_key.private_key.private_key_pem\n email_address = var.email\n}\nresource \"local_file\" \"acme_key_pem\" {\n content = tls_private_key.private_key.private_key_pem\n filename = \"acme_key.pem\"\n}\n
In the same folder, enter the following commands and follow the instructions:
terraform init\nterraform apply\n
Once done, copy the file named acme_key.pem somewhere safe, and where you will be able to refer to later on. Then, when the time comes to create a new cluster, add the following variable to the DNS module in your main.tf:
acme_key_pem = file(\"path/to/your/acme_key.pem\")\n
"},{"location":"#642-issuing-the-wildcard-ssl-certificate-manually","title":"6.4.2 Issuing the wildcard SSL certificate manually","text":"Magic Castle generates with Let's Encrypt a wildcard certificate for *.cluster_name.domain. You can use certbot DNS-01 challenge plugin to generate the wildcard certificate.
You will then need to copy the certificate files in the proper location on each proxy node. The reverse proxy configuration expects the following files to exist:
/etc/letsencrypt/live/${domain_name}/fullchain.pem/etc/letsencrypt/live/${domain_name}/privkey.pem/etc/letsencrypt/live/${domain_name}/chain.pem
Refer to the reverse proxy configuration for more details.
"},{"location":"#65-sshfp-records-and-dnssec","title":"6.5 SSHFP records and DNSSEC","text":"Magic Castle DNS module creates SSHFP records for all instances with a public ip address. These records can be used by SSH clients to verify the SSH host keys of the server. If DNSSEC is enabled for the domain and the SSH client is correctly configured, no host key confirmation will be prompted when connecting to the server.
For more information on how to activate DNSSEC, refer to your DNS provider documentation:
- CloudFlare - Enable DNSSEC
- Google Cloud - Manage DNSSEC configuration
To setup an SSH client to use SSHFP records, add
VerifyHostKeyDNS yes\n
to its configuration file (i.e.: ~/.ssh/config)."},{"location":"#7-planning","title":"7. Planning","text":"Once your initial cluster configuration is done, you can initiate a planning phase where you will ask Terraform to communicate with your cloud provider and verify that your cluster can be built as it is described by the main.tf configuration file.
Terraform should now be able to communicate with your cloud provider. To test your configuration file, enter the following command
terraform plan\n
This command will validate the syntax of your configuration file and communicate with the provider, but it will not create new resources. It is only a dry-run. If Terraform does not report any error, you can move to the next step. Otherwise, read the errors and fix your configuration file accordingly.
"},{"location":"#8-deployment","title":"8. Deployment","text":"To create the resources defined by your main, enter the following command
terraform apply\n
The command will produce the same output as the plan command, but after the output it will ask for a confirmation to perform the proposed actions. Enter yes.
Terraform will then proceed to create the resources defined by the configuration file. It should take a few minutes. Once the creation process is completed, Terraform will output the guest account usernames and password, the sudoer username and the floating ip of the login node.
Warning: although the instance creation process is finished once Terraform outputs the connection information, you will not be able to connect and use the cluster immediately. The instance creation is only the first phase of the cluster-building process. The configuration: the creation of the user accounts, installation of FreeIPA, Slurm, configuration of JupyterHub, etc.; takes around 15 minutes after the instances are created.
Once it is booted, you can follow an instance configuration process by looking at:
/var/log/cloud-init-output.logjournalctl -u puppet
If unexpected problems occur during configuration, you can provide these logs to the authors of Magic Castle to help you debug.
"},{"location":"#81-deployment-customization","title":"8.1 Deployment Customization","text":"You can modify the main.tf at any point of your cluster's life and apply the modifications while it is running.
Warning: Depending on the variables you modify, Terraform might destroy some or all resources, and create new ones. The effects of modifying each variable are detailed in the subsections of Configuration.
For example, to increase the number of computes nodes by one. Open main.tf, add 1 to node's count , save the document and call
terraform apply\n
Terraform will analyze the difference between the current state and the future state, and plan the creation of a single new instance. If you accept the action plan, the instance will be created, provisioned and eventually automatically add to the Slurm cluster configuration.
You could do the opposite and reduce the number of compute nodes to 0.
"},{"location":"#9-destruction","title":"9. Destruction","text":"Once you're done working with your cluster and you would like to recover the resources, in the same folder as main.tf, enter:
terraform destroy -refresh=false\n
The -refresh=false\u00a0flag is to avoid an issue where one or many of the data sources return no results and stall the cluster destruction with a message like the following:
Error: Your query returned no results. Please change your search criteria and try again.\n
This type of error happens when for example the specified image no longer exists (see issue #40). As for apply, Terraform will output a plan that you will have to confirm by entering yes.
Warning: once the cluster is destroyed, nothing will be left, even the shared storage will be erased.
"},{"location":"#91-instance-destruction","title":"9.1 Instance Destruction","text":"It is possible to destroy only the instances and keep the rest of the infrastructure like the floating ip, the volumes, the generated SSH host key, etc. To do so, set the count value of the instance type you wish to destroy to 0.
"},{"location":"#92-reset","title":"9.2 Reset","text":"On some occasions, it is desirable to rebuild some of the instances from scratch. Using terraform taint, you can designate resources that will be rebuilt at next application of the plan.
To rebuild the first login node :
terraform taint 'module.openstack.openstack_compute_instance_v2.instances[\"login1\"]'\nterraform apply\n
"},{"location":"#10-customize-cluster-software-configuration","title":"10. Customize Cluster Software Configuration","text":"Once the cluster is online and configured, you can modify its configuration as you see fit. We list here how to do most commonly asked for customizations.
Some customizations are done from the Puppet server instance (puppet). To connect to the puppet server, follow these steps:
- Make sure your SSH key is loaded in your ssh-agent.
- SSH in your cluster with forwarding of the authentication agent connection enabled:
ssh -A centos@cluster_ip. Replace centos by the value of sudoer_username if it is different. - SSH in the Puppet server instance:
ssh puppet
Note on Google Cloud: In GCP, OS Login lets you use Compute Engine IAM roles to manage SSH access to Linux instances. This feature is incompatible with Magic Castle. Therefore, it is turned off in the instances metadata (enable-oslogin=\"FALSE\"). The only account with sudoer rights that can log in the cluster is configured by the variable sudoer_username (default: centos).
"},{"location":"#101-disable-puppet","title":"10.1 Disable Puppet","text":"If you plan to modify configuration files manually, you will need to disable Puppet. Otherwise, you might find out that your modifications have disappeared in a 30-minute window.
Puppet executes a run every 30 minutes and at reboot. To disable puppet:
sudo puppet agent --disable \"<MESSAGE>\"\n
"},{"location":"#102-replace-the-guest-account-password","title":"10.2 Replace the Guest Account Password","text":"Refer to section 4.11.
"},{"location":"#103-add-ldap-users","title":"10.3 Add LDAP Users","text":"Users can be added to Magic Castle LDAP database (FreeIPA) with either one of the following methods: hieradata, command-line, and Mokey web-portal. Each method is presented in the following subsections.
New LDAP users are automatically assigned a home folder on NFS.
Magic Castle determines if an LDAP user should be member of a Slurm account based on its POSIX groups. When a user is added to a POSIX group, a daemon try to match the group name to the following regular expression:
(ctb|def|rpp|rrg)-[a-z0-9_-]*\n
If there is a match, the user will be added to a Slurm account with the same name, and will gain access to the corresponding project folder under /project.
Note: The regular expression represents how Compute Canada names its resources allocation. The regular expression can be redefined, see profile::accounts:::project_regex
"},{"location":"#1031-hieradata","title":"10.3.1 hieradata","text":"Using the hieradata variable in the main.tf, it is possible to define LDAP users.
Examples of LDAP user definition with hieradata are provided in puppet-magic_castle documentation.
"},{"location":"#1032-command-line","title":"10.3.2 Command-Line","text":"To add a user account after the cluster is built, log in mgmt1 and call:
kinit admin\nIPA_GUEST_PASSWD=<new_user_passwd> /sbin/ipa_create_user.py <username> [--group <group_name>]\nkdestroy\n
"},{"location":"#1033-mokey","title":"10.3.3 Mokey","text":"If user sign-up with Mokey is enabled, users can create their own account at
https://mokey.yourcluster.domain.tld/auth/signup\n
It is possible that an administrator is required to enable the account with Mokey. You can access the administrative panel of FreeIPA at :
https://ipa.yourcluster.domain.tld/\n
The FreeIPA administrator credentials can be retrieved from an encrypted file on the Puppet server. Refer to section 10.14 to know how.
"},{"location":"#104-increase-the-number-of-guest-accounts","title":"10.4 Increase the Number of Guest Accounts","text":"To increase the number of guest accounts after creating the cluster with Terraform, simply increase the value of nb_users, then call :
terraform apply\n
Each instance's Puppet agent will be reloaded following the copy of the hieradata files, and the new accounts will be created.
"},{"location":"#105-restrict-ssh-access","title":"10.5 Restrict SSH Access","text":"By default, instances tagged login have their port 22 opened to entire world. If you know the range of ip addresses that will connect to your cluster, we strongly recommend that you limit the access to port 22 to this range.
To limit the access to port 22, refer to section 4.14 firewall_rules, and replace the cidr of the ssh rule to match the range of ip addresses that have be the allowed to connect to the cluster. If there are more than one range, create multiple rules with distinct names.
"},{"location":"#106-add-packages-to-jupyter-default-python-kernel","title":"10.6 Add Packages to Jupyter Default Python Kernel","text":"The default Python kernel corresponds to the Python installed in /opt/ipython-kernel. Each compute node has its own copy of the environment. To add packages to this environment, add the following lines to hieradata in main.tf:
jupyterhub::kernel::venv::packages:\n - package_A\n - package_B\n - package_C\n
and replace package_* by the packages you need to install. Then call:
terraform apply\n
"},{"location":"#107-activate-globus-endpoint","title":"10.7 Activate Globus Endpoint","text":"No longer supported
"},{"location":"#108-recovering-from-puppet-rebuild","title":"10.8 Recovering from puppet rebuild","text":"The modifications of some of the parameters in the main.tf file can trigger the rebuild of the puppet instance. This instance hosts the Puppet Server on which depends the Puppet agent of the other instances. When puppet is rebuilt, the other Puppet agents cease to recognize Puppet Server identity since the Puppet Server identity and certificates have been regenerated.
To fix the Puppet agents, you will need to apply the following commands on each instance other than puppet once puppet is rebuilt:
sudo systemctl stop puppet\nsudo rm -rf /etc/puppetlabs/puppet/ssl/\nsudo systemctl start puppet\n
Then, on puppet, you will need to sign the new certificate requests made by the instances. First, you can list the requests:
sudo /opt/puppetlabs/bin/puppetserver ca list\n
Then, if every instance is listed, you can sign all requests:
sudo /opt/puppetlabs/bin/puppetserver ca sign --all\n
If you prefer, you can sign individual request by specifying their name:
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname NAME[,NAME]\n
"},{"location":"#109-dealing-with-banned-ip-addresses-fail2ban","title":"10.9 Dealing with banned ip addresses (fail2ban)","text":"Login nodes run fail2ban, an intrusion prevention software that protects login nodes from brute-force attacks. fail2ban is configured to ban ip addresses that attempted to login 20 times and failed in a window of 60 minutes. The ban time is 24 hours.
In the context of a workshop with SSH novices, the 20-attempt rule might be triggered, resulting in participants banned and puzzled, which is a bad start for a workshop. There are solutions to mitigate this problem.
"},{"location":"#1091-define-a-list-of-ip-addresses-that-can-never-be-banned","title":"10.9.1 Define a list of ip addresses that can never be banned","text":"fail2ban keeps a list of ip addresses that are allowed to fail to login without risking jail time. To add an ip address to that list, add the following lines to the variable hieradata\u00a0in main.tf:
profile::fail2ban::ignoreip:\n - x.x.x.x\n - y.y.y.y\n
where x.x.x.x and y.y.y.y are ip addresses you want to add to the ignore list. The ip addresses can be written using CIDR notations. The ignore ip list on Magic Castle already includes 127.0.0.1/8 and the cluster subnet CIDR. Once the line is added, call:
terraform apply\n
"},{"location":"#1092-remove-fail2ban-ssh-route-jail","title":"10.9.2 Remove fail2ban ssh-route jail","text":"fail2ban rule that banned ip addresses that failed to connect with SSH can be disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::jails: ['ssh-ban-root']\n
This will keep the jail that automatically ban any ip that tries to login as root, and remove the ssh failed password jail. Once the line is added, call:
terraform apply\n
"},{"location":"#1093-unban-ip-addresses","title":"10.9.3 Unban ip addresses","text":"fail2ban ban ip addresses by adding rules to iptables. To remove these rules, you need to tell fail2ban to unban the ips.
To list the ip addresses that are banned, execute the following command:
sudo fail2ban-client status ssh-route\n
To unban ip addresses, enter the following command followed by the ip addresses you want to unban:
sudo fail2ban-client set ssh-route unbanip\n
"},{"location":"#1094-disable-fail2ban","title":"10.9.4 Disable fail2ban","text":"While this is not recommended, fail2ban can be completely disabled. To do so, add the following line to the variable hieradata\u00a0in main.tf:
fail2ban::service_ensure: 'stopped'\n
then call :
terraform apply\n
"},{"location":"#1010-generate-a-new-ssl-certificate","title":"10.10 Generate a new SSL certificate","text":"The SSL certificate configured by the dns module is valid for 90 days. If you plan to use your cluster for more than 90 days, you will need to generate a new SSL certificate before the one installed on the cluster expires.
To generate a new certificate, use the following command on your computer:
terraform taint 'module.dns.module.acme.acme_certificate.certificate'\n
Then apply the modification:
terraform apply\n
The apply generates a new certificate, uploads it on the nodes that need it and reloads the reverse proxy if it is configured.
"},{"location":"#1011-set-selinux-in-permissive-mode","title":"10.11 Set SELinux in permissive mode","text":"SELinux can be set in permissive mode to debug new workflows that would be prevented by SELinux from working properly. To do so, add the following line to the variable hieradata\u00a0in main.tf:
selinux::mode: 'permissive'\n
"},{"location":"#1012-create-a-compute-node-image","title":"10.12 Create a compute node image","text":"When scaling the compute node pool, either manually by changing the count or automatically with Slurm autoscale, it can become beneficial to reduce the time spent configuring the machine when it boots for the first time, hence reducing the time requires before it becomes available in Slurm. One way to achieve this is to clone the root disk of a fully configured compute node and use it as the base image of future compute nodes.
This process has three steps:
- Prepare the volume for image cloning
- Create the image
- Configure Magic Castle Terraform code to use the new image
The following subsection explains how to accomplish each step.
Warning: While it will work in most cases, avoid re-using the compute node image of a previous deployment. The preparation steps cleans most of the deployment specific configuration and secrets, but there is no guarantee that the configuration will be entirely compatible with a different deployment.
"},{"location":"#10121-prepare-the-volume-for-cloning","title":"10.12.1 Prepare the volume for cloning","text":"The environment puppet-magic_castle installs a script that prepares the volume for cloning named prepare4image.sh.
To make sure a node is ready for cloning, open its puppet agent log and validate the catalog was successfully applied at least once:
journalctl -u puppet | grep \"Applied catalog\"\n
To prepare the volume for cloning, execute the following line while connected to the compute node:
sudo /usr/sbin/prepare4image.sh\n
Be aware that, since it is preferable for the instance to be powered off when cloning its volume, the script halts the machine once it is completed. Therefore, after executing prepare4image.sh, you will be disconnected from the instance.
The script prepare4image.sh executes the following steps in order:
- Stop and disable puppet agent
- Stop and disable slurm compute node daemon (
slurmd) - Stop and disable consul agent daemon
- Stop and disable consul-template daemon
- Unenroll the host from the IPA server
- Remove puppet agent configuration files in
/etc - Remove consul agent identification files
- Unmount NFS directories
- Remove NFS directories
/etc/fstab - Stop syslog
- Clear
/var/log/message content - Remove cloud-init's logs and artifacts so it can re-run
- Power off the machine
"},{"location":"#10122-create-the-image","title":"10.12.2 Create the image","text":"Once the instance is powered off, access your cloud provider dashboard, find the instance and follow the provider's instructions to create the image.
- AWS
- Azure
- GCP
- OpenStack
- OVH
Note down the name/id of the image you created, it will be needed during the next step.
"},{"location":"#10123-configure-magic-castle-terraform-code-to-use-the-new-image","title":"10.12.3 Configure Magic Castle Terraform code to use the new image","text":"Edit your main.tf and add image = \"name-or-id-of-your-image\" to the dictionary defining the instance. The instance previously powered off will be powered on and future non-instantiated machines will use the image at the next execution of terraform apply.
If the cluster is composed of heterogeneous compute nodes, it is possible to create an image for each type of compute nodes. Here is an example with Google Cloud
instances = {\n mgmt = { type = \"n2-standard-2\", tags = [\"puppet\", \"mgmt\", \"nfs\"], count = 1 }\n login = { type = \"n2-standard-2\", tags = [\"login\", \"public\", \"proxy\"], count = 1 }\n node = {\n type = \"n2-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n image = \"rocky-mc-cpu-node\"\n }\n gpu = {\n type = \"n1-standard-2\"\n tags = [\"node\", \"pool\"]\n count = 10\n gpu_type = \"nvidia-tesla-t4\"\n gpu_count = 1\n image = \"rocky-mc-gpu-node\"\n }\n}\n
"},{"location":"#1013-read-and-edit-secret-values-generated-at-boot","title":"10.13 Read and edit secret values generated at boot","text":"During the cloud-init initialization phase, bootstrap.sh script is executed. This script generates a set of encrypted secret values that are required by the Magic Castle Puppet environment:
profile::consul::acl_api_tokenprofile::freeipa::mokey::passwordprofile::freeipa::server::admin_passwordprofile::freeipa::server::ds_passwordprofile::slurm::accounting::passwordprofile::slurm::base::munge_key
To read or change the value of one of these keys, use eyaml edit command on the puppet host, like this:
sudo /opt/puppetlabs/puppet/bin/eyaml edit \\\n --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/boot_private_key.pkcs7.pem \\\n --pkcs7-public-key /etc/puppetlabs/puppet/eyaml/boot_public_key.pkcs7.pem \\\n /etc/puppetlabs/code/environments/production/data/bootstrap.yaml\n
It is also possible to redefine the values of these keys by adding the key-value pair to the hieradata configuration file. Refer to section 4.13 hieradata. User defined values take precedence over boot generated values in the Magic Castle Puppet data hierarchy.
"},{"location":"#1014-expand-a-volume","title":"10.14 Expand a volume","text":"Volumes defined in the volumes map can be expanded at will. After their creation, you can increase their size in the main.tf then call terraform apply and the associated block device will be expanded.
To benefit from the new storage, the following commands need to be ran as root on the instance to which the expanded volume is attached.
- Identify the physical volume path
pvscan\n
- Expand the physical volume
pvresize /dev/vdxyz # replace vdxyz by the volume identify at step 1\n
- Identify the volume group path
lvdisplay\n
- Expand the volume group using step volume group path identified
lvextend -l '+100%FREE' -r /dev/project_vg/project\n
- Resize the XFS filesystem:
xfs_growfs /dev/project_vg/project\n
"},{"location":"#11-customize-magic-castle-terraform-files","title":"11. Customize Magic Castle Terraform Files","text":"You can modify the Terraform module files in the folder named after your cloud provider (e.g: gcp, openstack, aws, etc.)
"},{"location":"design/","title":"Design","text":""},{"location":"design/#magic-castle-terraform-structure","title":"Magic Castle Terraform Structure","text":"Figure 1 (below) illustrates how Magic Castle is structured to provide a unified interface between multiple cloud providers. Each blue block is a file or a module, while white blocks are variables or resources. Arrows indicate variables or resources that contribute to the definition of the linked variables or resources. The figure can be read as a flow-chart from top to bottom. Some resources and variables have been left out of the chart to avoid cluttering it further.
Figure 1. Magic Castle Terraform Project Structure
main.tf: User provides the instances and volumes structure they wants as _map_s. instances = {\n mgmt = { type = \"p4-7.5gb\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"p2-3.75gb\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"p2-3.75gb\", tags = [\"node\"], count = 2 }\n}\n\nvolumes = {\n nfs = {\n home = { size = 100 }\n project = { size = 500 }\n scratch = { size = 500 }\n }\n}\n
-
common/design:
- the
instances map is expanded to form a new map where each entry represents a single host. instances = {\n mgmt1 = {\n type = \"p2-3.75gb\"\n tags = [\"puppet\", \"mgmt\", \"nfs\"]\n }\n login1 = {\n type = \"p2-3.75gb\"\n tags = [\"login\", \"public\", \"proxy\"]\n }\n node1 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n node2 = {\n type = \"p2-3.75gb\"\n tags = [\"node\"]\n }\n}\n
- the
volumes map is expanded to form a new map where each entry represent a single volume volumes = {\n mgmt1-nfs-home = { size = 100 }\n mgmt1-nfs-project = { size = 100 }\n mgmt1-nfs-scratch = { size = 500 }\n}\n
-
network.tf: the instances map from common/design is used to generate a network interface (nic) for each host, and a public ip address for each host with the public tag.
resource \"provider_network_interface\" \"nic\" {\n for_each = module.design.instances\n ...\n}\n
-
common/configuration: for each host in instances, a cloud-init yaml config that includes puppetservers is generated. These configs are outputted to a user_data map where the keys are the hostnames.
user_data = {\n for key, values in var.instances :\n key => templatefile(\"${path.module}/puppet.yaml\", { ... })\n}\n
-
infrastructure.tf: for each host in instances, an instance resource as defined by the selected cloud provider is generated. Each instance is initially configured by its user_data cloud-init yaml config.
resource \"provider_instance\" \"instances\" {\n for_each = module.design.instance\n user_data = module.instance_config.user_data[each.key]\n ...\n}\n
-
infrastructure.tf: for each volume in volumes, a block device as defined by the selected cloud provider is generated and attached it to its matching instance using an attachment resource.
resource \"provider_volume\" \"volumes\" {\n for_each = module.design.volumes\n size = each.value.size\n ...\n}\nresource \"provider_attachment\" \"attachments\" {\n for_each = module.design.volumes\n instance_id = provider_instance.instances[each.value.instance].id\n volume_id = provider_volume.volumes[each.key].id\n ...\n}\n
-
infrastructure.tf: the created instances' information are consolidated in a map named inventory.
inventory = {\n mgmt1 = {\n public_ip = \"\"\n local_ip = \"10.0.0.1\"\n id = \"abc1213-123-1231\"\n tags = [\"mgmt\", \"puppet\", \"nfs\"]\n }\n ...\n}\n
-
common/provision: the information from created instances is consolidated and written in a yaml file namedterraform_data.yaml that is uploaded on the Puppet server as part of the hieradata.
resource \"null_resource\" \"deploy_hieradata\" {\n ...\n provisioner \"file\" {\n content = var.terraform_data\n destination = \"terraform_data.yaml\"\n }\n ...\n}\n
-
outputs.tf: the information of all instances that have a public address are output as a map named public_instances.
"},{"location":"design/#resource-per-provider","title":"Resource per provider","text":"In the previous section, we have used generic resource name when writing HCL code that defines these resources. The following table indicate what resource is used for each provider based on its role in the cluster.
Resource AWS Azure Google Cloud Platform OpenStack OVH network aws_vpc azurerm_virtual_network google_compute_network prebuilt openstack_networking_network_v2 subnet aws_subnet azurerm_subnet google_compute_subnetwork prebuilt openstack_networking_subnet_v2 router aws_route not used google_compute_router built-in not used nat aws_internet_gateway not used google_compute_router_nat built-in not used firewall aws_security_group azurerm_network_security_group google_compute_firewall openstack_compute_secgroup_v2 openstack_compute_secgroup_v2 nic aws_network_interface azurerm_network_interface google_compute_address openstack_networking_port_v2 openstack_networking_port_v2 public ip aws_eip azurerm_public_ip google_compute_address openstack_networking_floatingip_v2 openstack_networking_network_v2 instance aws_instance azurerm_linux_virtual_machine google_compute_instance openstack_compute_instance_v2 openstack_compute_instance_v2 volume aws_ebs_volume azurerm_managed_disk google_compute_disk openstack_blockstorage_volume_v3 openstack_blockstorage_volume_v3 attachment aws_volume_attachment azurerm_virtual_machine_data_disk_attachment google_compute_attached_disk openstack_compute_volume_attach_v2 openstack_compute_volume_attach_v2"},{"location":"design/#using-reference-design-to-extend-for-a-new-cloud-provider","title":"Using reference design to extend for a new cloud provider","text":"Magic Castle currently supports five cloud providers, but its design makes it easy to add new providers. This section presents a step-by-step guide to add a new cloud provider support to Magic Castle.
-
Identify the resources. Using the Resource per provider table, read the cloud provider Terraform documentation, and identify the name for each resource in the table.
-
Check minimum requirements. Once all resources have been identified, you should be able to determine if the cloud provider can be used to deploy Magic Castle. If you found a name for each resource listed in table, the cloud provider can be supported. If some resources are missing, you will need to read the provider's documentation to determine if the absence of the resource can be compensated for somehow.
-
Initialize the provider folder. Create a folder named after the provider. In this folder, create two symlinks, one pointing to common/variables.tf and the other to common/outputs.tf. These files define the interface common to all providers supported by Magic Castle.
-
Define cloud provider specifics variables. Create a file named after your provider provider_name.tf\u00a0and define variables that are required by the provider but not common to all providers, for example the availability zone or the region. In this file, define two local variables named cloud_provider and cloud_region.
-
Initialize the infrastructure. Create a file named infrastructure.tf. In this file:
- define the provider block if it requires input parameters, i.e: var.region
provider \"provider_name\" {\n region = var.region\n}\n
- include the design module
module \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\n
-
Create the networking infrastructure. Create a file named network.tf and define the network, subnet, router, nat, firewall, nic and public ip resources using the module.design.instances map.
-
Create the volumes. In infrastructure.tf, define the volumes resource using module.design.volumes.
-
Consolidate the volume device information. In infrastructure.tf, define a local variable named volume_devices implementing the following logic in HCL. Replace the line starting by /dev/disk/by-id with the proper logic that would match the volume resource to its device path from within the instance to which it is attached.
volume_devices = {\n for ki, vi in var.volumes :\n ki => {\n for kj, vj in vi :\n kj => [for key, volume in module.design.volumes :\n \"/dev/disk/by-id/*${substr(provider_volume.volumes[\"${volume[\"instance\"]}-${ki}-${kj}\"].id, 0, 20)}\"\n if key == \"${volume[\"instance\"]}-${ki}-${kj}\"\n ]\n }\n}\n
-
Consolidate the instances' information. In infrastructure.tf, define a local variable named inventory that will be a map containing the following keys for each instance: public_ip, local_ip, prefix, tags, and specs (#cpu, #gpus, ram).
-
Create the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n generate_ssh_key = var.generate_ssh_key\n public_keys = var.public_keys\n volume_devices = local.volume_devices\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\n
-
Create the instances. In infrastructure.tf, define the instances resource using module.design.instances_to_build for the instance attributes and module.configuration.user_data for the initial configuration.
-
Attach the volumes. In infrastructure.tf, define the attachments resource using module.design.volumes and refer to the attribute each.value.instance to retrieve the instance's id to which the volume needs to be attached.
-
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=cloud_provider_instance_resource.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\n
-
Include the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\n
"},{"location":"design/#an-example","title":"An example","text":" -
Identify the resources. For Digital Ocean, Oracle Cloud and Alibaba Cloud, we get the following resource mapping: | Resource | Digital Ocean | Oracle Cloud | Alibaba Cloud | | ----------- | :-------------------- | :-------------------- | :-------------------- | | network | digitalocean_vpc | oci_core_vcn | alicloud_vpc | | subnet | built in vpc | oci_subnet | alicloud_vswitch | | router | n/a | oci_core_route_table | built in vpc | | nat | n/a | oci_core_internet_gateway | alicloud_nat_gateway | | firewall | digitalocean_firewall | oci_core_security_list | alicloud_security_group | | nic | n/a | built in instance | alicloud_network_interface | | public ip | digitalocean_floating_ip | built in instance | alicloud_eip | | instance | digitalocean_droplet | oci_core_instance | alicloud_instance | | volume | digitalocean_volume | oci_core_volume | alicloud_disk | | attachment | digitalocean_volume_attachment | oci_core_volume_attachment | alicloud_disk_attachment |
-
Check minimum requirements. In the preceding table, we can see Digital Ocean does not have the ability to define a network interface. The documentation also leads us to conclude that it is not possible to define the private ip address of the instances before creating them. Because the Puppet server ip address is required before generating the cloud-init YAML config for all instances, including the Puppet server itself, this means it impossible to use Digital Ocean to spawn a Magic Castle cluster. Oracle Cloud presents the same issue, however, after reading the instance documentation, we find that it is possible to define a static ip address as a string in the instance attribute. It would therefore be possible to create a datastructure in Terraform that would associate each instance hostname with an ip address in the subnet CIDR. Alibaba cloud has an answer for each resource, so we will use this provider in the following steps.
-
Initialize the provider folder. In a terminal:
git clone https://github.com/ComputeCanada/magic_castle.git\ncd magic_castle\nmkdir alicloud\ncd aliclcoud\nln -s ../common/{variables,outputs}.tf .\n
-
Define cloud provider specifics variables. Add the following to a new file alicloud.tf:
variable \"region\" { }\nlocals {\n cloud_provider = \"alicloud\"\n cloud_region = var.region\n}\n
-
Initialize the infrastructure. Add the following to a new file infrastructure.tf:
provider \"alicloud\" {\n region = var.region\n}\n\nmodule \"design\" {\n source = \"../common/design\"\n cluster_name = var.cluster_name\n domain = var.domain\n instances = var.instances\n pool = var.pool\n volumes = var.volumes\n}\n
-
Create the networking infrastructure. network.tf base template:
resource \"alicloud_vpc\" \"network\" { }\nresource \"alicloud_vswitch\" \"subnet\" { }\nresource \"alicloud_nat_gateway\" \"nat\" { }\nresource \"alicloud_security_group\" \"firewall\" { }\nresource \"alicloud_security_group_rule\" \"allow_in_services\" { }\nresource \"alicloud_security_group\" \"allow_any_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_ingress_inside_vpc\" { }\nresource \"alicloud_security_group_rule\" \"allow_egress_inside_vpc\" { }\nresource \"alicloud_network_interface\" \"nic\" { }\nresource \"alicloud_eip\" \"public_ip\" { }\nresource \"alicloud_eip_association\" \"eip_asso\" { }\n
-
Create the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk\" \"volumes\" {\n for_each = module.design.volumes\n}\n
-
Consolidate the volume devices' information. Add the following snippet to infrastructure.tf:
volume_devices = {\n for ki, vi in var.volumes :\n ki => {\n for kj, vj in vi :\n kj => [for key, volume in module.design.volumes :\n \"/dev/disk/by-id/virtio-${replace(alicloud_disk.volumes[\"${volume[\"instance\"]}-${ki}-${kj}\"].id, \"d-\", \"\")}\"\n if key == \"${volume[\"instance\"]}-${ki}-${kj}\"\n ]\n }\n}\n
-
Consolidate the instances' information. Add the following snippet to infrastructure.tf:
locals {\n inventory = { for x, values in module.design.instances :\n x => {\n public_ip = contains(values[\"tags\"], \"public\") ? alicloud_eip.public_ip[x].public_ip : \"\"\n local_ip = alicloud_network_interface.nic[x].private_ip\n tags = values[\"tags\"]\n id = alicloud_instance.instances[x].id\n specs = {\n cpus = ...\n gpus = ...\n ram = ...\n }\n }\n }\n}\n
-
Create the instance configurations. In infrastructure.tf, include the common/configuration module like this:
module \"configuration\" {\n source = \"../common/configuration\"\n inventory = local.inventory\n config_git_url = var.config_git_url\n config_version = var.config_version\n sudoer_username = var.sudoer_username\n generate_ssh_key = var.generate_ssh_key\n public_keys = var.public_keys\n volume_devices = local.volume_devices\n domain_name = module.design.domain_name\n cluster_name = var.cluster_name\n guest_passwd = var.guest_passwd\n nb_users = var.nb_users\n software_stack = var.software_stack\n cloud_provider = local.cloud_provider\n cloud_region = local.cloud_region\n}\n
-
Create the instances. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_instance\" \"instances\" {\n for_each = module.design.instances\n}\n
-
Attach the volumes. Add and complete the following snippet to infrastructure.tf:
resource \"alicloud_disk_attachment\" \"attachments\" {\n for_each = module.design.volumes\n}\n
-
Identify the public instances. In infrastructure.tf, define a local variable named public_instances that contains the attributes of instances that are publicly accessible from Internet and their ids.
locals {\n public_instances = { for host in keys(module.design.instances_to_build):\n host => merge(module.configuration.inventory[host], {id=alicloud_instance.instances[host].id})\n if contains(module.configuration.inventory[host].tags, \"public\")\n }\n}\n
-
Include the provision module to transmit Terraform data to the Puppet server. In infrastructure.tf, include the common/provision module like this
module \"provision\" {\n source = \"../common/provision\"\n bastions = local.public_instances\n puppetservers = module.configuration.puppetservers\n tf_ssh_key = module.configuration.ssh_key\n terraform_data = module.configuration.terraform_data\n terraform_facts = module.configuration.terraform_facts\n hieradata = var.hieradata\n sudoer_username = var.sudoer_username\n}\n
Once your new provider is written, you can write an example that will use the module to spawn a Magic Castle cluster with that provider.
module \"alicloud\" {\n source = \"./alicloud\"\n config_git_url = \"https://github.com/ComputeCanada/puppet-magic_castle.git\"\n config_version = \"main\"\n\n cluster_name = \"new\"\n domain = \"my.cloud\"\n image = \"centos_7_9_x64_20G_alibase_20210318.vhd\"\n nb_users = 10\n\n instances = {\n mgmt = { type = \"ecs.g6.large\", tags = [\"puppet\", \"mgmt\", \"nfs\"] }\n login = { type = \"ecs.g6.large\", tags = [\"login\", \"public\", \"proxy\"] }\n node = { type = \"ecs.g6.large\", tags = [\"node\"], count = 1 }\n }\n\n volumes = {\n nfs = {\n home = { size = 10 }\n project = { size = 50 }\n scratch = { size = 50 }\n }\n }\n\n public_keys = [file(\"~/.ssh/id_rsa.pub\")]\n\n # Alicloud specifics\n region = \"us-west-1\"\n}\n
"},{"location":"developers/","title":"Magic Castle Developer Documentation","text":""},{"location":"developers/#table-of-content","title":"Table of Content","text":" - Setup
- Where to start
- Puppet environment
- Troubleshooting
- Release
"},{"location":"developers/#1-setup","title":"1. Setup","text":"To develop for Magic Castle you will need: * Terraform (>= 1.4.0) * git * Access to a Cloud (e.g.: Compute Canada Arbutus) * Ability to communicate with the cloud provider API from your computer * A cloud project with enough room for the resource described in section Magic Caslte Doc 1.1. * [optional] Puppet Development Kit (PDK)
"},{"location":"developers/#2-where-to-start","title":"2. Where to start","text":"The Magic Castle project is defined by Terraform infrastructure-as-code component that is responsible of generating a cluster architecture in a cloud and a Puppet environment component that configures the cluster instances based on their role.
If you wish to add device, an instance, add a new networking interface or a filesystem, you will most likely need to develop some Terraform code. The project structure for Terraform code is described in the reference design document. The document also describes how one could work with current Magic Castle code to add support for another cloud provider.
If you wish to add a service to one of the Puppet environments, install a new software, modify an instance configuration or role, you will most likely need to develop some Puppet code. The following section provides more details on the Puppet environments available and how to develop them.
"},{"location":"developers/#3-puppet-environment","title":"3. Puppet environment","text":"Magic Castle Terraform code initialized every instances to be a Puppet agent and an instance with the tag puppet as the Puppet main server. On the Puppet main server, there is a folder containing the configuration code for the instances of the cluster, this folder is called a Puppet environment and it is pulled from GitHub during the initial configuration of the Puppet main server.
The source of that environment is provided to Terraform using the variable config_git_url.
A repository describing a Magic Castle Puppet environment must contain at the least the following files and folders:
config_git_repo\n\u2523 Puppetfile\n\u2523 environment.conf\n\u2523 hiera.yaml\n\u2517 data\n \u2517 common.yaml\n\u2517 manifests/\n \u2517 site.pp\n
Puppetfile specifies the Puppet modules that need to be installed in the environment.environment.conf overrides the primary server default settings for the environment.hiera.yaml configures an ordered list of YAML file data sources.data/common.yaml is common data source for the instances part of hierarchy defined by hiera.yaml.manifests/site.pp defines how each instance will be configured based on their hostname and/or tags.
An example of a bare-bone Magic Castle Puppet environment is available on GitHub: MagicCastle/puppet-environment, while the Puppet environment that replicates a Compute Canada HPC cluster is named ComputeCanada/puppet-magic_castle.
"},{"location":"developers/#terraform_datayaml-a-bridge-between-terraform-and-puppet","title":"terraform_data.yaml: a bridge between Terraform and Puppet","text":"To provide information on the deployed resources and the value of the input parameters, Magic Castle Terraform code uploads to the Puppet main server a file named terraform_data.yaml, in the folder /etc/puppetlabs/data/. There is also a symlink created in /etc/puppetlabs/code/environment/production/data/ to ease its usage inside the Puppet environment.
When included in the data hierarchy (hiera.yaml), terraform_data.yaml can provide information about the instances, the volumes and the variables set by the user through the main.tf file. The file has the following structure:
---\nterraform:\n data:\n cluster_name: \"\"\n domain_name: \"\"\n guest_passwd: \"\"\n nb_users: \"\"\n public_keys: []\n sudoer_username: \"\"\n instances:\n host1:\n hostkeys:\n rsa: \"\"\n ed25519: \"\"\n local_ip: \"x.x.x.x\"\n prefix: \"host\"\n public_ip: \"\"\n specs:\n \"cpus\": 0\n \"gpus\": 0\n \"ram\": 0\n tags:\n - \"tag_1\"\n - \"tag_2\"\n tag_ip:\n tag_1:\n - x.x.x.x\n tag_2:\n - x.x.x.x\n volumes:\n volume_tag1:\n volume_1:\n - \"/dev/disk/by-id/123-*\"\n volume_2:\n - \"/dev/disk/by-id/123-abc-*\"\n
The values provided by terraform_data.yaml can be accessed in Puppet by using the lookup() function. For example, to access an instance's list of tags:
lookup(\"terraform.instances.${::hostname}.tags\")\n
The data source can also be used to define a key in another data source YAML file by using the alias() function. For example, to define the number of guest accounts using the value of nb_users, we could add this to common.yaml profile::accounts::guests::nb_accounts: \"%{alias('terraform.data.nb_users')}\"\n
"},{"location":"developers/#configuring-instances-sitepp-and-classes","title":"Configuring instances: site.pp and classes","text":"The configuration of each instance is defined in manifests/site.pp file of the Puppet environment. In this file, it is possible to define a configuration based on an instance hostname
node \"mgmt1\" { }\n
or using the instance tags by defining the configuration for the default node : node default {\n $instance_tags = lookup(\"terraform.instances.${::hostname}.tags\")\n if 'tag_1' in $instances_tags { }\n}\n
It is possible to define Puppet resource directly in site.pp. However, above a certain level of complexity, which can be reach fairly quickly, it is preferable to define classes and include these classes in site.pp based on the node hostname or tags.
Classes can be defined in the Puppet environment under the following path: site/profile/manifests. These classes are named profile classes and the philosophy behind it is explained in Puppet documentation. Because these classes are defined in site/profile, their name has to start with the prefix profile::.
It is also possible to include classes defined externally and installed using the Puppetfile. These classes installed by r10k can be found in the modules folder of the Puppet environment.
"},{"location":"developers/#4-troubleshooting","title":"4. Troubleshooting","text":""},{"location":"developers/#41-cloud-init","title":"4.1 cloud-init","text":"To test new additions to puppet.yaml, it is possible to execute cloud-init phases manually. There are four steps that can be executed sequentially: init local, init modules config and modules final. Here are the corresponding commands to execute each step:
cloud-init init --local\ncloud-init init\ncloud-init modules --mode=config\ncloud-init modules --mode=final\n
It is also possible to clean a cloud-init execution and have it execute again at next reboot. To do so, enter the following command:
cloud-init clean\n
Add -r to the previous command to reboot the instance once cloud-init has finishing cleaning."},{"location":"developers/#42-selinux","title":"4.2 SELinux","text":"SELinux is enabled on every instances of a Magic Castle cluster. Some applications do not provide SELinux policies which can lead to their malfunctionning when SELinux is enabled. It is possible to track down the reasons why SELinux is preventing an application to work properly using the command-line tool ausearch.
If you suspect application app-a to be denied by SELinux to work properly, run the following command as root:
ausearch -c app-a --raw | grep denied\n
To see all requests denied by SELinux:
ausearch --raw | grep denied\n
Sometime, the denials are hidden from regular logging. To display all denials, run the following command as root:
semodule --disable_dontaudit --build\n
then re-execute the application that is not working properly. Once you have found the denials that are the cause of the problem, you can create a new policy to allow the requests that were previously denied with the following command:
ausearch -c app-a --raw | grep denied | audit2allow -a -M app-a\n
Finally, you can install the generated policy using the command provided by auditallow.
"},{"location":"developers/#building-the-policy-package-file-pp-from-the-enforcement-file-te","title":"Building the policy package file (.pp) from the enforcement file (.te)","text":"If you need to tweak an existing enforcement file and you want to recompile the policy package, you can with the following commands:
checkmodule -M -m -o my_policy.mod my_policy.te\nsemodule_package -o my_policy.pp -m my_policy.mod\n
"},{"location":"developers/#references","title":"References","text":" - https://wiki.gentoo.org/wiki/SELinux
- https://wiki.gentoo.org/wiki/SELinux/Tutorials/Where_to_find_SELinux_permission_denial_details
"},{"location":"developers/#5-release","title":"5. Release","text":"To build a release, use the script release.sh located at the root of Magic Castle git repo.
Usage: release.sh VERSION [provider ...]\n
The script creates a folder named releases where it was called. The VERSION argument is expected to correspond to git tag in the puppet-magic_castle repo. It could also be a branch name or a commit. If the provider optional argument is left blank, release files will be built for all providers currently supported by Magic Castle.
Examples:
"},{"location":"matrix/","title":"Comparison of Cloud HPC Cluster Projects","text":"Name Creator First public release date Software license AWS ParallelCluster AWS November 12, 2018 Apache License v2 Azure CycleCloud Microsoft October 17, 2018 MIT License Azure HPC On-Demand Platform Microsoft April 23, 2021 MIT License Cluster in the Cloud Matt Williams - University of Bristol March 27, 2019 MIT License ElastiCluster Riccardo Murri - University of Zurich July 17, 2013 GPLv3 Google HPC-Toolkit Google May 26, 2022 Apache License v2 Magic Castle F\u00e9lix-Antoine Fortin - Compute Canada August 26, 2019 MIT License On-Demand Data Centre Adaptive Computing - - Slurm on GCP SchedMD March 14, 2018 Apache License v2"},{"location":"matrix/#supported-cloud-providers","title":"Supported cloud providers","text":"Name Alibaba Cloud AWS Azure Google Cloud IBM Cloud OpenStack Oracle Cloud OVH\u00a0 AWS ParallelCluster no yes no no no no no no Azure CycleCloud no no yes no no no no no Azure HPC On-Demand Platform no no yes no no no no no Cluster-in-the-Cloud no yes no yes no no yes no ElastiCluster* no yes yes yes no yes no - Google HPC-Toolkit no no no yes no no no no Magic Castle* no yes yes yes no yes no yes On-Demand Data Centre yes yes yes yes no no yes no Slurm on GCP no no no yes no no no no * The documentation provides instructions on how to add support for other cloud providers.
"},{"location":"matrix/#supported-operating-systems","title":"Supported operating systems","text":"Name CentOS 7 CentOS 8 Rocky Linux 8 AlmaLinux 8 Debian 10 Ubuntu 18 Ubuntu 20 Windows 10 AWS ParallelCluster yes yes yes yes yes no yes no Azure CycleCloud yes yes yes yes yes no yes - Azure HPC On-Demand Platform yes no no yes no yes no yes Google HPC-Toolkit yes no no no no no no no Cluster in the Cloud no yes no no no no no no ElastiCluster yes yes yes yes no no no no Magic Castle yes yes yes yes no no no no On-Demand Data Centre - - - - - - - - Slurm on GCP yes no no no yes no yes no"},{"location":"matrix/#supported-job-schedulers","title":"Supported job schedulers","text":"Name AwsBatch Grid Engine HTCondor Moab Open PBS PBS Pro Slurm AWS ParallelCluster yes no no no no no yes Azure CycleCloud no yes yes no no yes yes Azure HPC On-Demand Platform no no no no yes no yes Google HPC-Toolkit no no no no no no yes Cluster in the Cloud no no no no no no yes ElastiCluster no yes no no no no yes Magic Castle no no no no no no yes On-Demand Data Centre no no no yes no no no Slurm on GCP no no no no no no yes"},{"location":"matrix/#technologies","title":"Technologies","text":"Name Infrastructure configuration Programming languages Configuration management Scientific software AWS ParallelCluster CLI generating YAML Python Chef Spack Azure CycleCloud WebUI or CLI + templates Python Chef Bring your own Azure HPC On-Demand Platform YAML files + shell scripts Shell, Terraform Ansible, Packer CVMFS Cluster in the Cloud CLI generating Terraform code Python, Terraform Ansible, Packer EESSI ElastiCluster CLI interpreting an INI file Python, Shell Ansible Bring your own Google HPC-Toolkit CLI generating Terraform code Go, Terraform Ansible, Packer Spack Magic Castle Terraform modules Terraform Puppet CC-CVMFS, EESSI On-Demand Data Centre - - - - Slurm GCP Terraform modules Terraform Ansible, Packer Spack"},{"location":"sequence/","title":"Magic Castle Sequence Diagrams","text":"The following sequence diagrams illustrate the inner working of Magic Castle once terraform apply is called. Some details were left out of the diagrams, but every diagram is followed by references to the code files that were used to build it.
"},{"location":"sequence/#1-cluster-creation","title":"1. Cluster creation","text":""},{"location":"sequence/#notes","title":"Notes","text":" puppet-magic_castle.git does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.
"},{"location":"sequence/#references","title":"References","text":" magic_castle:/common/design/main.tfmagic_castle:/openstack/network-1.tfmagic_castle:/openstack/network-2.tfmagic_castle:/common/configuration/main.tfmagic_castle:/openstack/infrastructure.tfmagic_castle:/common/provision/main.tfmagic_castle:/dns/cloudflare/main.tfmagic_castle:/dns/acme/main.tf
"},{"location":"sequence/#2-configuration-with-cloud-init","title":"2. Configuration with cloud-init","text":""},{"location":"sequence/#notes_1","title":"Notes","text":" config_git_url repo does not have to refer to ComputeCanada/puppet-magic_castle.git repo. Users can use their own fork. See the developer documentation for more details.- While the diagram represents each step as completed sequentially, each node provisioning is independent. The only step that requires synchronisation between nodes and the management node is the puppet certificate generation.
"},{"location":"sequence/#references_1","title":"References","text":" magic_castle:/common/configuration/puppet.yaml
"},{"location":"sequence/#3-configuration-with-puppet","title":"3. Configuration with Puppet","text":""},{"location":"sequence/#references_2","title":"References","text":" puppet-magic_castle:/manifests/site.pppuppet-magic_castle:/profile/manifests/base.pppuppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/freeipa.pp
"},{"location":"sequence/#4-configuration-with-consul-and-consul-template","title":"4. Configuration with Consul and Consul Template","text":""},{"location":"sequence/#references_3","title":"References","text":" puppet-magic_castle:/profile/manifests/consul.pppuppet-magic_castle:/profile/manifests/cvmfs.pppuppet-magic_castle:/profile/manifests/slurm.pp
"},{"location":"terraform_cloud/","title":"Terraform Cloud","text":"This document explains how to use Magic Castle with Terraform Cloud.
"},{"location":"terraform_cloud/#what-is-terraform-cloud","title":"What is Terraform Cloud?","text":"Terraform Cloud is HashiCorp\u2019s managed service that allows to provision infrastructure using a web browser or a REST API instead of the command-line. This also means that the provisioned infrastructure parameters can be modified by a team and the state is stored in the cloud instead of a local machine.
When provisioning in commercial cloud, Terraform Cloud can also provide a cost estimate of the resources.
"},{"location":"terraform_cloud/#getting-started-with-terraform-cloud","title":"Getting started with Terraform Cloud","text":" - Create a Terraform Cloud account
- Create an organization, join one or choose one available to you
"},{"location":"terraform_cloud/#managing-a-magic-castle-cluster-with-terraform-cloud","title":"Managing a Magic Castle cluster with Terraform Cloud","text":""},{"location":"terraform_cloud/#creating-the-workspace","title":"Creating the workspace","text":" - Create a git repository in GitHub, GitLab, or any of the version control system provider supported by Terraform Cloud
- In this git repository, add a copy of the Magic Castle example
main.tf available for the cloud of your choice - Log in Terraform Cloud account
- Create a new workspace
- Choose Type: \"Version control workflow\"
- Connect to VCS: choose the version control provider that hosts your repository
- Choose the repository that contains your
main.tf - Configure settings: tweak the name and description to your liking
- Click on \"Create workspace\"
You will be redirected automatically to your new workspace.
"},{"location":"terraform_cloud/#providing-cloud-provider-credentials-to-terraform-cloud","title":"Providing cloud provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your cloud provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each cloud provider and how to retrieve the values of the variable from the provider.
If you plan on using these environment variables with multiple workspaces, it is recommended to create a credential variable set in Terraform Cloud.
"},{"location":"terraform_cloud/#aws","title":"AWS","text":"You need to define these environment variables: - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY (sensitive)
The value of these variables can either correspond to the value of access key created on the AWS Security Credentials - Access keys page, or you can add user dedicated to Terraform Cloud in AWS IAM Users, and use its access key.
"},{"location":"terraform_cloud/#azure","title":"Azure","text":"You need to define these environment variables: - ARM_CLIENT_ID - ARM_CLIENT_SECRET (sensitive) - ARM_SUBSCRIPTION_ID - ARM_TENANT_ID
Refer to Terraform Azure Provider - Creating a Service Principal to know how to create a Service Principal and retrieve the values for these environment variables.
"},{"location":"terraform_cloud/#google-cloud","title":"Google Cloud","text":"You need to define this environment variable: - GOOGLE_CLOUD_KEYFILE_JSON (sensitive)
The value of the variable will be the content of a Google Cloud service account JSON key file expressed a single line string. Example:
{\"type\": \"service_account\",\"project_id\": \"project-id-1234\",\"private_key_id\": \"abcd1234\",...}\n
You can use jq to format the string from the JSON file provided by Google:
jq . -c project-name-123456-abcdefjg.json\n
"},{"location":"terraform_cloud/#openstack-ovh","title":"OpenStack / OVH","text":"You need to define these environment variables: - OS_AUTH_URL - OS_PROJECT_ID - OS_REGION_NAME - OS_INTERFACE - OS_IDENTITY_API_VERSION - OS_USER_DOMAIN_NAME - OS_USERNAME - OS_PASSWORD (sensitive)
Apart from OS_PASSWORD, the values for these variables are available in OpenStack RC file provided for your project.
If you prefer to use OpenStack application credentials, you need to define at least these variables: - OS_AUTH_TYPE\u00a0 - OS_AUTH_URL - OS_APPLICATION_CREDENTIAL_ID - OS_APPLICATION_CREDENTIAL_SECRET
and potentially these too: - OS_IDENTITY_API_VERSION\u00a0 - OS_REGION_NAME - OS_INTERFACE
The values for these variables are available in OpenStack RC file provided when creating the application credentials.
"},{"location":"terraform_cloud/#providing-dns-provider-credentials-to-terraform-cloud","title":"Providing DNS provider credentials to Terraform Cloud","text":"Terraform Cloud will invoke Terraform command-line in a remote virtual environment. For the CLI to be able to communicate with your DNS provider API, we need to define environment variables that Terraform will use to authenticate. The next sections explain which environment variables to define for each DNS provider and how to retrieve the values of the variable from the provider.
"},{"location":"terraform_cloud/#cloudflare","title":"CloudFlare","text":"Refer to DNS - CloudFlare section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#google-cloud-dns","title":"Google Cloud DNS","text":"Refer to DNS - Google Cloud section of Magic Castle main documentation to determine which environment variables needs to be set.
"},{"location":"terraform_cloud/#managing-magic-castle-variables-with-terraform-cloud-ui","title":"Managing Magic Castle variables with Terraform Cloud UI","text":"It is possible to use Terraform Cloud web interface to define variable values in your main.tf. For example, you could want to define a guest password without writing it directly in main.tf to avoid displaying publicly.
To manage a variable with Terraform Cloud: 1. edit your main.tf to define the variables you want to manage. In the following example, we want to manage the number of nodes and the guest password.
Add the variables at the beginning of the `main.tf`:\n ```hcl\n variable \"nb_nodes\" {}\n variable \"password\" {}\n ```\n\nThen replace the static value by the variable in our `main.tf`,\n\ncompute node count\n ```hcl\n node = { type = \"p2-3gb\", tags = [\"node\"], count = var.nb_nodes }\n ```\nguest password\n ```hcl\n guest_passwd = var.password\n ```\n
- Commit and push this changes to your git repository.
- In Terraform Cloud workspace associated with that repository, go in \"Variables.
- Under \"Terraform Variables\", click the \"Add variable\" button and create a variable for each one defined previously in the\u00a0
main.tf. Check \"Sensitive\" if the variable content should not never be shown in the UI or the API.
You may edit the variables at any point of your cluster lifetime.
"},{"location":"terraform_cloud/#applying-changes","title":"Applying changes","text":"To create your cluster, apply changes made to your main.tf or the variables, you will need to queue a plan. When you push to the default branch of the linked git repository, a plan will be automatically created. You can also create a plan manually. To do so, click on the \"Queue plan manually\" button inside your workspace, then \"Queue plan\".
Once the plan has been successfully created, you can apply it using the \"Runs\" section. Click on the latest queued plan, then on the \"Apply plan\" button at the bottom of the plan page.
"},{"location":"terraform_cloud/#auto-apply","title":"Auto apply","text":"It is possible to apply automatically a successful plan. Go in the \"Settings\" section, and under \"Apply method\" select \"Auto apply\". Any following successful plan will then be automatically applied.
"},{"location":"terraform_cloud/#magic-castle-terraform-cloud-and-the-cli","title":"Magic Castle, Terraform Cloud and the CLI","text":"Terraform cloud only allows to apply or destroy the plan as stated in the main.tf, but sometimes it can be useful to run some other terraform commands that are only available through the command-line interface, for example terraform taint.
It is possible to import the terraform state of a cluster on your local computer and then use the CLI on it.
-
Log in Terraform cloud:
terraform login\n
-
Create a folder where the terraform state will be stored:
mkdir my-cluster-1\n
-
Create a file named cloud.tf with the following content in your cluster folder:
terraform {\n cloud {\n organization = \"REPLACE-BY-YOUR-TF-CLOUD-ORG\"\n workspaces {\n name = \"REPLACE-BY-THE-NAME-OF-YOUR-WORKSPACE\"\n }\n }\n}\n
replace the values of organization and name with the appropriate value for your cluster. -
Initialize the folder and retrieve the state:
terraform init\n
To confirm the workspace has been properly imported locally, you can list the resources using:
terraform state list\n
"},{"location":"terraform_cloud/#enable-magic-castle-autoscaling","title":"Enable Magic Castle Autoscaling","text":"Magic Castle in combination with Terraform Cloud (TFE) can be configured to give Slurm the ability to create and destroy instances based on the job queue content.
To enable this feature: 1. Create a TFE API Token and save it somewhere safe.
1.1. If you subscribe to Terraform Cloud Team & Governance plan, you can generate\na [Team API Token](https://www.terraform.io/cloud-docs/users-teams-organizations/api-tokens#team-api-tokens).\nThe team associated with this token requires no access to organization and can be secret.\nIt does not have to include any member. Team API token is preferable as its permissions can be\nrestricted to the minimum required for autoscale purpose.\n
-
Create a workspace in TFE
2.1. Make sure the repo is private as it will contain the API token.
2.2. If you generated a Team API Token in 1, provide access to the workspace to the team:
- Workspace Settings -> Team Access -> Add team and permissions
- Select the team
- Click on \"Customize permissions for this team\"
- Under \"Runs\" select \"Apply\"
- Under \"Variables\" select \"Read and write\"
- Leave the rest as is and click on \"Assign custom permissions\"
2.3 In Configure settings, under Advanced options, for Apply method, select Auto apply.
-
Create the environment variables of the cloud provider credentials in TFE
- Create a variable named
pool in TFE. Set value to [] and check HCL. - Add a file named
data.yaml in your git repo with the following content: yaml\u00a0 --- profile::slurm::controller::tfe_token: <TFE API token> profile::slurm::controller::tfe_workspace: <TFE workspace id> Complete the file by replacing <TFE API TOKEN> with the token generated at step 1 and <TFE workspace id> (i.e.: ws-...) by the id of the workspace created at step 2. It is recommended to encrypt the TFE API token before committing data.yaml in git. Refer to section 4.14 of README.md to know how to encrypt the token. - Add
data.yaml in git and push. -
Modify main.tf:
- If not already present, add the following definition of the pool variable at the beginning of your
main.tf.
variable \"pool\" { description = \"Slurm pool of compute nodes\" }\n
- Add instances to
instances with the tags pool and node. These are the nodes that Slurm will able to create and destroy. - If not already present, add the following line after the instances definition to pass the list of compute nodes from Terraform cloud workspace variable to the provider module:
pool = var.pool\n
- On the right-hand-side of
public_keys =, replace [file(\"~/.ssh/id_rsa.pub\")] by a list of SSH public keys that will have admin access to the cluster. - After the line
public_keys = ..., add hieradata = file(\"data.yaml\"). - After the line
hieradata = ..., add generate_ssh_key = true. This will provide Terraform Cloud SSH admin access to the cluster and it will be used to upload configuration files. - Stage changes, commit and push to git repo.
-
Go to your workspace in TFE, click on Actions -> Start a new run -> Plan and apply -> Start run. Then, click on \"Confirm & Apply\" and \"Confirm Plan\".
- Compute nodes defined in step 8 can be modified at any point in the cluster lifetime and more pool compute nodes can be added or removed if needed.
"},{"location":"terraform_cloud/#considerations-for-autoscaling","title":"Considerations for autoscaling","text":"To reduce the time required for compute nodes to become available in Slurm, consider creating a compute node image.
JupyterHub will time out by default after 300 seconds if a node is not spawned yet. Since it may take longer than this to spawn a node, even with an image created, consider increasing the timeout by adding the following to your YAML configuration file:
jupyterhub::jupyterhub_config_hash:\n SlurmFormSpawner:\n start_timeout: 900\n
Slurm 23 adds the possibility for sinfo to report nodes that are not yet spawned. This is useful if you want JupyterHub to be aware of those nodes, for example if you want to allow to use GPU nodes without keeping them online at all time. To use that version of Slurm, add the following to your YAML configuration file:
profile::slurm::base::slurm_version: '23.02'\n
"},{"location":"terraform_cloud/#troubleshoot-autoscaling-with-terraform-cloud","title":"Troubleshoot autoscaling with Terraform Cloud","text":"If after enabling autoscaling with Terraform Cloud for your Magic Castle cluster, the number of nodes does not increase when submitting jobs, verify the following points:
- Go to the Terraform Cloud workspace webpage, and look for errors in the runs. If the runs were only triggered by changes to the git repo, it means scaling signals from the cluster do not reach the Terraform cloud workspace or no signals were sent at all.
- Make sure the Terraform Cloud workspace id matches with the value of
profile::slurm::controller::tfe_workspace in data.yaml. - Execute
squeue on the cluster, and verify the reasons why jobs are still in the queue. If under the column (Reason), there is the keyword ReqNodeNotAvail, it implies Slurm tried to boot the listed nodes, but they would not show up before the timeout, therefore Slurm marked them as down. It can happen if your cloud provider is slow to build the instances, or following a configuration problem like in 2. When Slurm marks a node as down, a trace is left in slurmctld's log - using zgrep on the slurm controller node (typically mgmt1): sudo zgrep \"marking down\" /var/log/slurm/slurmctld.log*\n
To tell Slurm these nodes are available again, enter the following command: sudo /opt/software/slurm/bin/scontrol update nodename=node[Y-Z] state=IDLE\n
Replace node[Y-Z] by the hostname range listed next to ReqNodeNotAvail in squeue. - Under
mgmt1:/var/log/slurm, look for errors in the file slurm_resume.log.
"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index d9049740..d5ee2bb7 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ
diff --git a/terraform_cloud/index.html b/terraform_cloud/index.html
index 478b41ad..e291cee5 100644
--- a/terraform_cloud/index.html
+++ b/terraform_cloud/index.html
@@ -1011,7 +1011,7 @@ Enable Magic Castle Autoscaling
Complete the file by replacing <TFE API TOKEN> with the token generated at step 1
and <TFE workspace id> (i.e.: ws-...) by the id of the workspace created at step 2.
It is recommended to encrypt the TFE API token before committing data.yaml in git. Refer
- to section 4.13.1 of README.md to
+ to section 4.14 of README.md to
know how to encrypt the token.
Add data.yaml in git and push.