falloc and multi threaded writes?

[ossimoi]

Was playing around with rclone and ZFS and stumbled upon this thread:

https://forum.rclone.org/t/multiple-write-streams-not-causing-fragmentation-on-zfs/44521/5

Had to try it out. Even 16 simultaneous write streams do not seem to cause any fragmentation on ZFS.

From rclone/rclone#3066 :

Reading the man page I think it would be safe to add FALLOC_FL_PUNCH_HOLE since rclone only ever preallocates space on a 0 sized file.

while transfering data to ZFS dataset with rclone, both du and du --apparent-size show the file size as currently transfered amount so the files are not sparse either.

Where does the magic happen here? What causes the writes to align sequentially?

From #10408 :

Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be
preallocated for a file in order to guarantee that later writes will not
run out of space. Instead, fallocate() space preallocation only checks
that sufficient space is currently available in the pool or the user's
project quota allocation, and then creates a sparse file of the requested
size.

[rincebrain]

Who said they're sequential?

Defining fragmentation (or "sequential" except for logical presentation of the data you gave it) on something where you never overwrite in place and allocations can happen from multiple disks becomes very complicated to do very quickly.

Preallocation is a noop on ZFS except for dropping any existing allocations in that range (if you do it with PUNCH_HOLE), and updating the metadata. Since how much space things take up is going to vary based on what you write, which top-level vdevs it gets written to, and how much contiguous free space there is, among other things, you would rapidly wind up with surprising outcomes like "preallocating" 500 GiB and then writing incompressible data that took up more than 500 GiB between metadata or other fun overheads, and getting ENOSPC early.

(Depending on your goal, you could conceivably use a reservation or refreservation, but that "just" subtracts enough to meet the difference between what you're using and it from the total free space available to anything, it doesn't carve out a specific chunk of the free space from it ahead of time.

Also, as a nit on myself, technically I suppose changing the logical file length would affect how the metadata encodes if you wrote, say, 4k 100G into a 0B file, versus 4k 100G into a 4T file, maybe, but that's pretty academic for most use cases...and if it's not, I have serious questions about your use case.)

[ossimoi]

Who said they're sequential?

I reprod the exact setup in the post I linked from rclone forums. Checking file segments with zdb, during transfer each file can have multiple segments, but after completion they always end up being in a single sequential segment.

Preallocation is a noop on ZFS except for dropping any existing allocations in that range (if you do it with PUNCH_HOLE), and updating the metadata.

rclone ORs KEEP_SIZE and PUNCH_HOLE for ZFS: https://github.com/rclone/rclone/blob/8f0e9f9f6bd78f6ac7488ac48bbe7e9a9ba418ab/lib/file/preallocate_unix.go#L19
Would writing 0's and punching afterwards achieve a proper preallocation which would allow each stream to write on preallocated space and so avoid fragmentation? IIRC this is not how CoW works, just trying to wrap my head around this.

[rincebrain]

My point is that ZFS literally gives not one damn about what order you write things in, it's still going to read-modify-write the same recordsize elements until it literally has no free space to get a contiguous allocation, and when gang blocks get involved, everything gets very sad indeed.