params.json

{"name":"WalB","tagline":"Block-level WAL for Efficient Backup and Replication","body":"## What is WalB?\r\n\r\nWalB is a Linux kernel module and its related userland tools\r\nfor incremental backup and asynchronous replication of storage devices.\r\n\r\nWalB was named after Block-level Write-Ahead Logging (WAL).\r\n\r\n## Characteristics\r\n\r\n- Block-level solution.\r\n- Small performance overhead.\r\n- Submitted order of write IOs will be preserved.\r\n\r\n## Functionalities\r\n\r\n- Online extraction of the logs, which you can replay on other block devices.\r\n- Online resize of a walb device.\r\n- Online WAL reset with adjusting to new log device size.\r\n- ~~Snapshot management~~. You cannot access to snapshot images directly.\r\n\r\n## Architecture\r\n\r\n- A walb device is wrapper of two underlying block devices, log and data.\r\n- A walb device behaves as a consistent block device.\r\n  - Write IOs will be stored in the log device as redo-logs and in the data device with no change.\r\n  - Read IOs will be just forward to the data device in almost cases.\r\n- Almost all space of a log device is a ring buffer to store logs for write IOs.\r\n- Stored logs can be retrieved from the log device using sequential read.\r\n\r\n## Pros and Cons\r\n\r\n### Pros\r\n\r\n- Negligible overhead for read IOs.\r\n- Small overhead for write IOs.\r\n  - Response time: almost the same as that of the underlying devices.\r\n  - Throughput: almost the same as that of the underlying devices, when the log and data devices are independent physically and the bandwidth of HBA is enough (2x + alpha).\r\n- You can use any filesystems on a walb device.\r\n- You can get portable and consistent diff files from logs.\r\nYou can store them for backup, compress and transfer them for remote replication, as you like.\r\n\r\n### Cons\r\n\r\n- WalB uses more than 2x bandwidth for writes.\r\n- A walb device requires additional storage space for logs.\r\n- You cannot access to snapshot images with a walb device only.\r\n\r\n## Compared with similar solutions\r\n\r\n### dm-snap\r\n\r\n- Snapshot management.\r\n  - dm-snap supports snapshot management using COW (copy-on-write).\r\n  - WalB does not support access to snapshot images.\r\n- Incremental backup\r\n  - dm-snap does not support incremental backup. Full-scan will be required.\r\n  - WalB supports incremental backup without full-scan of volumes.\r\n\r\n### dm-thin\r\n\r\n- Thin-provisioning.\r\n  - dm-thin supports thin-provisioning.\r\n  - WalB does not support thin-provisioning.\r\n- Snapshot management.\r\n  - dm-thin supports snapshot management using persistent address indexes.\r\n  - WalB does not support access to snapshot images\r\nbecause it does not maintain persistent indexes.\r\nWalb devices do not suffer from performance overhead to maintain persistent indexes.\r\n- Diff extraction for incremental backup and asynchronous replication.\r\n  - dm-thin supports retrieving changed blocks information to get diffs.\r\n  - WalB supports direct extraction of logs (can be diffs) with a sequential scan of log devices.\r\n- Fragmentation management.\r\n  - Using dm-thin devices, fragmentation will occur at block level.\r\nFragmentation causes performance degradation.\r\nA clever GC/reorganization algorithm will be required to solve the issue, for which\r\nextra IOs will be required of course.\r\n  - A walb device stores logs in its ring buffer and the oldest logs will be overwritten\r\nautomatically without extra IOs.\r\nIn addition, walb does not need persistent address indexes,\r\nwhich is address space conversion maps between\r\na walb device and its underlying data device,\r\nso block-level fragmentation does not occur.\r\nThat is, walb devices do not need explicit GC/reorganization at block level.\r\n\r\n### DRBD\r\n\r\n- Incremental backup.\r\n  - DRBD does not support incremental backup.\r\nIt does not have functionality to retrieve portable/consistent diff files.\r\nFull-scan of volumes will be required for backup.\r\n  - WalB supports incremental backup without full-scan.\r\n- Replication.\r\n  - DRBD supports both synchronous and asynchronous replication.\r\n  - WalB enables asynchronous replication using logs.\r\nWalB does not support synchronous replication.\r\n- Long-distance replication.\r\n  - A DRBD device uses limited-size socket buffer, so IO performance will decrease\r\nwhen the buffer is likely to be full in long-distance asynchronous replication.\r\nDRBD-proxy, for which another license is required, will solve the issue.\r\n  - A walb device can store much IOs to be sent to a remote site in its ring buffer.\r\nTherefore IO performance will not decrease due to long distance.\r\nHowever, read IOs for logs extraction may affect performance.\r\nCurrently walb does not have buffers for log extraction effectively,\r\nwhile underlying storage caches will help it.\r\n\r\n### Log-structured file systems\r\n\r\nEx. nilfs, btrfs, (or ZFS).\r\n\r\n- WalB is a block-level solution, not a file system one.\r\n- If you use whichever file system, you can backup your data incrementally\r\nand replicate them asynchronously using an underlying walb device.\r\n\r\n## Authors and Contributors\r\n\r\n[starpos](https://github.com/starpos/), [herumi](https://github.com/herumi/)","google":"","note":"Don't delete this file! It's used internally to help with page regeneration."}