A couple of issues related to initial zvol replication

[surfchris]

I believe there are a couple of issues related to initial Pyznap replication of zvol. These issues could be related to each other...?

Issue no. 1: Zvol sparse-setting is not preserved during initial Pyznap send
ZFS property related to sparse/non-sparse zvol, ‘refreservation’, is somehow not transferred as part of initial Pyznap send.

Using ‘zfs send’ with ‘-R’, the source ‘refreservation’ is preserved. Without ‘-R’ destination local default value seems to be used rather than the value of the source zvol.

When replicating zvols on FreeNAS the ‘refreservation’ is always preserved, so I assume that is the right way to do it…

Simple non-sparse and sparse zvol (1Gb) can be created using:

zfs create -V 1G tank/nonsparse 
zfs create -s -V 1G tank/sparse

Issue no. 2: Initial Pyznap send of zvol only works recursively
When performing the initial replication to a new destination, pyznap doesn’t create a remote Zvol unless the Zvol is in a dataset that is recursively replicated.

Using Pyznap snap/send to replicate the zvol 'nonsparse':

[small/nonsparse]
frequent = 10
snap = yes
clean = yes
dest = ssh:34:[email protected]:BACKUP/test-pyznap/nonsparse

generates the error

root@pve:~# pyznap send
Sep 27 20:20:35 INFO: Starting pyznap...
Sep 27 20:20:35 INFO: Sending snapshots...
Sep 27 20:20:36 WARNING: lzop does not exist on [email protected], continuing without compression...
Sep 27 20:20:36 ERROR: Destination [email protected]:BACKUP/test-pyznap/nonsparse does not exist...
Sep 27 20:20:36 INFO: Finished successfully...

A destination zvol is created automatically on initial replication when using zfs send -R ...:

root@pve:~# zfs send -R small/nonsparse@first | ssh -p 34 [email protected] zfs receive BACKUP/test-pyznap/nonsparse

A workaround to both these issues is to run zfs send -R .... for the initial zvol replication, then use Pyznap.

[yboetz]

Both of these are the intended behavior. For issue 1, pyznap should always preserve local options, not take the source values. The idea is that you create the remote dataset manually and set different options. In my case for example I use gzip instead of lz4 for my backup datasets. So I wouldn't want that zfs overwrites the remote gzip option with the source dataset option lz4.

Issue 2 is exactly how it's supposed to work, same for datasets. If the dataset does not exist on the remote, it does not send. You should create the parent dataset manually. All dependent datasets are then automatically created, once pyznap is sure that the parent exists. Note that I cannot use the -R flag for zfs send, as this would replicate the whole filesystem. Recursive replication is handled by pyznap itself, without relying on -R.

[surfchris]

That makes 100% sense for datasets, and Pyznap seems to work perfectly for that. I just wonder if zvols should be treated a little differently.

Essentially what is happening when ‘refreservation’ is not transferred is that a non-sparse zvol on source become a sparse zvol on destination. And sparse zvols can become corrupt if written to when disk space is low.

[yboetz]

I'd rather not add more code for this special case. It shouldn't make that much difference if the zvol data is written to a sparse or nonsparse zvol. If you want to restore from your backup then you can simply create a new nonsparse zvol and transfer the data from your backup. And if you really want to make sure that your data ends up in a nonsparse zvol, you can easily create the zvol on your remote and then transfer the data.

[surfchris]

Well it's not as simple as that. 'refreservation' is a safety toggle that is put there for a reason: to ensure a zvol not to use more space than available. If the toggle is lost on a backup drive and you don't monitor disk space then there is a risk of corrupted zvols down the road.

The other issue is how to do the initial transfer of a single zvol to a destination. If you have a dataset with let’s say 100 of zvols and you want pyznap to replicate just one of those zvols, you can't. You either have to do all the zvols in the dataset or make an exclude rule for 99. A zvol is not like a dataset.

Nonetheless, I do have to say that pyznap is a wonderful tool and it's definitely a keeper for me. I just wanted to make you aware of these issues related to zvol. As I said earlier there is an easy workaround on these both these issues by using zfs send -R... for the initial zvol replication, and from there let Pyznap do the rest.

[yboetz]

Fair points. I never really used zvols extensively, so I never encountered this. As you mentioned, there are workarounds, either use zfs send -R initially or manually create the zvols on your dest, but I'll think about this a bit more. Not sure yet how I could implement this elegantly.

[surfchris]

I don't know how it should be implemented. I'm not even sure if there are additional zvol properties that are important and should be kept the same on source and destination. Using zfs send -R on the first zvol replication (only) will duplicate everything of course, but that is probably not the right way to implement it. Maybe someone else have some ideas. At least you are aware of the issue. The workaround works perfectly too.