Instance backups

[pcfriek1987]

As I'm still with this backup part in my head.

• Veeam enterprise is off the table because of the pricing.
• Backroll requires the server needing access to the storage, which with local storage is not going to work for me.
  And that where all the options possible (outside of starting to write your own, which well since I'm not a coder would also just be a whole lot of money).

While you can create disk snapshots my opinion is that they are not really backups.

While it seems the the internal names don't change after an instance has been created, the i- names, so in theory you can create a backup from a KVM and move it, how would you restore it? Since the names are in a database.

I'm sure I'm not the only one having this issue :) Even with a replicated ceph setup you should want backups.

So how do other people create backups on Cloudstack?

[rajujith]

@pcfriek1987 have you considered Dell NetWorker? although snapshots are not really backups would it help you if you store the snapshots in the remote zone as well ?

[JoaoJandre]

Are you interested in saving the VM's memory as well as the volume's data, or only the volume's data?

[pcfriek1987]

On a normal backup you wouldn't save the memory either, so the memory is of no concern to me.
For me it would be the data most, and a way to just restore a file if needed or the whole machine.

@rajujith what do you mean with remote zone? I only looked at Dell networker briefly, just now, do you have any experience with it?

[rajujith]

@pcfriek1987 Cloudstack now supports backing up volume snapshots to multiple zones: https://www.shapeblue.com/cloudstack-snapshot-copy/

I did a PoC with Dell Networker and it works reasonably well.

[JoaoJandre]

Great, CloudStack already has the built-in solution for VM's volume backups. If you want to backup your VMs volumes, you need to set snapshot.backup.to.secondary and create volume snapshots for the volumes you want backed up. For backups, VM snapshots will not do as they are not saved to a different storage area, but volume snapshots can be (depending on snapshot.backup.to.secondary). As you and @rajujith think that snapshots are not backups, let me explain:

According to IBM: "Backup and restore refers to technologies and practices for making periodic copies of data and applications to a separate, secondary device and then using those copies to recover the data and applications." ¹

So how do we apply this on CloudStack?

Your VM's volumes are always on the primary storage. When you create a snapshot, you copy all the volume's data to a secondary storage area (assuming snapshot.backup.to.secondary is true), which should be separated from your primary storage. Where it is located is up to you and your infrastructure, meaning you could create a secondary storage that is on another datacenter if you want, and store all your snapshots there. And when you need to, you can restore your volume using the snapshot via ACS.

Furthermore, depending on how you want to set it up, using secondary storage selectors (see #7659), you could have a single secondary storage that contains all of your volume snapshots (your backups) in a separate datacenter, and keep your templates/ISOs in a secondary storage that is inside the 'main' datacenter, so that the files you use frequently (templates/ISOs) are faster to access, while the backups (volume snapshots) are on another storage in case of a disaster on your 'main' datacenter.

Moreover, as @rajujith said, #7873 added the possibility of keeping redundant copies of your snapshots on various secondary storages, giving you even more safety.

As for the periodic part, you can create snapshot policies, that will take periodic snapshots of your volumes.

I hope you can see how volume snapshots are backups, at least using the snapshot.backup.to.secondary configuration.

Footnotes

1. https://www.ibm.com/topics/backup-and-restore ↩

[pcfriek1987]

Hi @JoaoJandre

I didn't know that was possible, and it seems quite interesting, will need to look at that, I thought you could have only one Secondary storage, which for me is an NFS mount on the management server.

But how does it work? I can see there is a piece of documentation (https://docs.cloudstack.apache.org/en/latest/adminguide/storage.html#direct-resources-to-a-specific-secondary-storage), but that is a bit too vague for me on how to set it as it just talks about this is the rule to set it, but not how they got there in the first place, is there an example of how to do this? For me it would be about number 3 (Direct snapshot of volumes with the KVM hypervisor to a specific secondary storage).

[BryanMLima]

Hello, pcfriek1987

I apologize for lack of clarity in the section Direct resources to a specific secondary storage, currently, you can only manage the secondary storage selectors through API, which is easier through CloudMonkey; it is planned to add the management of this feature to the UI in the future, simplifying its usage.

For now, you will need to use the APIs createSecondaryStorageSelector, listSecondaryStorageSelectors, removeSecondaryStorageSelector and updateSecondaryStorageSelector, which are available on 4.19.0.0 and newer versions of ACS.

So, for example, if you want to direct snapshots to a specific secondary storage, like you mentioned, you'll need to create the following rule:

if (snapshot.hypervisorType === 'KVM') {
   '7cdd9941-5bb5-4717-9afb-8cf803340253';
}

To create this rule through the API createSecondaryStorageSelector, you'll need to inform the required parameters using CloudMonkey:

Note that the heuristic rule was formatted to a one string JS script

(lab) 🐱 > create secondarystorageselector name="kvm-snapshots" description="Selector for directing snapshots to an especific secondary storage" zoneid=7cdd9941-5bb5-4717-9afb-8cf803340253 purpose=SNAPSHOT heuristicrule="if (snapshot.hypervisorType === 'KVM') { '7832f261-c602-4e8e-8580-2496ffbbc45d'; }"
{
  "heuristics": {
    "created": "2024-07-09T19:49:07+0000",
    "description": "Selector for directing snapshots to an especific secondary storage",
    "heuristicrule": "if (snapshot.hypervisorType === 'KVM') { '7832f261-c602-4e8e-8580-2496ffbbc45d'; }",
    "id": "47628716-4ac3-42f4-bbdb-c3afb9a09e24",
    "name": "kvm-snapshots",
    "purpose": "SNAPSHOT",
    "zoneid": "7cdd9941-5bb5-4717-9afb-8cf803340253"
  }
}

However, this only directs all snapshots for the hypervisor KVM to the specific, other types of secondary storage resources (such as ISOs, templates and volumes) can still be allocated to this storage.

Therefore, if you want to dedicate a secondary storage to only store snapshots, you'll need to create a selector for the remaining resource types filtering the desired secondary storage. Unfortunately, this feature expects that the JS output is a valid secondary storage UUID, therefore, you'll need to shuffle the UUIDs through the JS script. I will try to improve this part of the feature, to simplify the dedication of a resource to a specific secondary storage.

[pcfriek1987]

Hi @BryanMLima

So with the above I can dedicate a secondary storage for only the snapshots.

In this case I already have one now where everything goes to it, if I add a new secondary storage I can have that one for only the snapshots using the above right?

However, this only directs all snapshots for the hypervisor KVM to the specific, other types of secondary storage resources (such as ISOs, templates and volumes) can still be allocated to this storage.

Since this will be a bigger storage primarily for snapshots it will have enough space to also hold ISO's and templates until there is a better way to filter it, not sure what you mean with volumes? since i think snapshots includes both normal and volume snapshots right?

[BryanMLima]

In this case I already have one now where everything goes to it, if I add a new secondary storage I can have that one for only the snapshots using the above right?

@pcfriek1987, yes, you'll only need to change the UUID in the heuristic rule.

Since this will be a bigger storage primarily for snapshots it will have enough space to also hold ISO's and templates until there is a better way to filter it, not sure what you mean with volumes? since i think snapshots includes both normal and volume snapshots right?

Using the KVM as hypervisor, snapshots refers to volume snapshots, which are allocated to the secondary storages. In the other hand, VM snapshots are allocated in the primary storage in which the VM's root volume is allocated.

When I mentioned directing volumes to a specific secondary storage, I meant about the volumes that are uploaded locally or through a URL. For these volumes, we can use the heuristic rules.

As for your case specifically, I looked in to creating a pseudo-random heuristic rule for allocating the other resources. Below, it is presented a heuristic rule that will filter the secondary storage with UUID 7cdd9941-5bb5-4717-9afb-8cf803340253 from the available pools. Then, the script returns a pseudo-random secondary storage from the pool of secondary storages. I used the Math.random()¹ function native to JS, which is not the best approach, but it is what we have available at the moment. If you only have two secondary storages, this function is overkill, as you could just create a rule with the UUID of the other secondary storage. However, for an environment where there are multiple secondary storages, this rule will try to allocate randomly to the other pools.

function getRandomInt(max) {
  return Math.floor(Math.random() * max);
}

function removeSecondaryStorage(pool) {
  return pool.id != '7cdd9941-5bb5-4717-9afb-8cf803340253';
}

filteredSecondaryStorages = secondaryStorages.filter(removeSecondaryStorage);
randomIndex = getRandomInt(filteredSecondaryStorages.length);
filteredSecondaryStorages[randomIndex].id

With the rule above, you can create a selector for the resource types TEMPLATE, ISO and VOLUME, and they will filter the secondary storage with UUID specified in the function removeSecondaryStorage. Using the rule above alongside the one mentioned in the previous comment, the desired secondary storage will only store volume snapshots.

Footnotes

1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/random ↩