Skip to content

Commit

Permalink
Initial rename/fork.
Browse files Browse the repository at this point in the history
  • Loading branch information
mmontagna committed Sep 16, 2019
1 parent b66375d commit 78e8e52
Show file tree
Hide file tree
Showing 1,214 changed files with 341,275 additions and 69 deletions.
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1 @@
include LICENSE README.md z3/z3.conf
include LICENSE README.md zfs3backup/zfs3backup.conf
61 changes: 30 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,28 @@
# z3 [![Build Status](https://travis-ci.org/PressLabs/z3.svg)](https://travis-ci.org/PressLabs/z3)
<!-- # zfs3backup [![Build Status](https://travis-ci.org/PressLabs/zfs3backup.svg)](https://travis-ci.org/PressLabs/zfs3backup)
-->

# Welcome to z3
This was forked from https://github.com/Presslabs/z3 which appears to be a dead project.

z3 is a ZFS to S3 backup tool. This is basically plumbing around `zfs send` and `zfs receive`
so you should have at least a basic understanding of what those commands do.
# Welcome to zfs3backup

z3 was developed by the awesome engineering team at [Presslabs](https://www.presslabs.com/),
a Managed WordPress Hosting provider.
zfs3backup is a ZFS to S3 backup tool. This is basically plumbing around `zfs send` and `zfs receive`
so you should have at least a basic understanding of what those commands do.

For more open-source projects, check [Presslabs Code](https://www.presslabs.org/).

## Usage
`z3 status` will show you the current state, what snapshots you have on S3 and on the local
`zfs3backup status` will show you the current state, what snapshots you have on S3 and on the local
zfs dataset.

`z3 backup` perform full or incremental backups of your dataset.
`zfs3backup backup` perform full or incremental backups of your dataset.

`z3 restore` restores your dataset to a certain snapshot.
`zfs3backup restore` restores your dataset to a certain snapshot.

See `zfs SUBCOMMAND --help` for more info.

### Installing
`pip install z3`
`pip install zfs3backup`

z3 is tested on python 2.7 with latest boto 2 and boto 2.2.2 (python-boto version on Ubuntu 12.04).
zfs3backup is tested on python 2.7 with latest boto 2 and boto 2.2.2 (python-boto version on Ubuntu 12.04).

#### Optional dependencies
```
Expand All @@ -37,11 +36,11 @@ apt-get install pigz
### Configuring
Most options can be configured as command line flags, environment variables or in a config file,
in that order of precedence.
The config file is read from `/etc/z3_backup/z3.conf` if it exists, some defaults are provided by the tool.
The config file is read from `/etc/zfs3backup_backup/zfs3backup.conf` if it exists, some defaults are provided by the tool.
BUCKET `S3_KEY_ID` and `S3_SECRET` can't be provided on the command line.
For a list of all options see `z3/sample.conf`.
For a list of all options see `zfs3backup/sample.conf`.

You'll usually want z3 to only backup certain snapshots (hourly/daily/weekly).
You'll usually want zfs3backup to only backup certain snapshots (hourly/daily/weekly).
To do that you can specify a `SNAPSHOT_PREFIX` (defaults to `zfs-auto-snap:daily`).

Defaults for `SNAPSHOT_PREFIX` and `COMPRESSOR` can be set per filesystem like so:
Expand All @@ -56,7 +55,7 @@ SNAPSHOT_PREFIX=weekly-non-spam

### Dataset Size, Concurrency and Memory Usage
Since the data is streamed from `zfs send` it gets read in to memory in chunks.
Z3 estimates a good chunk size for you: no smaller than 5MB and large enough
zfs3backup estimates a good chunk size for you: no smaller than 5MB and large enough
to produce at most 9999 chunks. These are S3 limitation for multipart uploads.
Here are some example chunk sizes for different datasets:
* 50 GiB: 5 MiB
Expand All @@ -71,55 +70,55 @@ Multiply that by `CONCURRENCY` to know how much memory your upload will use.
#### Status
```
# show global options
z3 --help
zfs3backup --help
# show status of backups for default dataset
z3 status
zfs3backup status
# show status for other dataset; only snapshots named daily-spam-*
z3 --dataset tank/spam --snapshot-prefix daily-spam- status
zfs3backup --dataset tank/spam --snapshot-prefix daily-spam- status
```

#### Backup
```
# show backup options
z3 backup --help
zfs3backup backup --help
# perform incremental backup the latest snapshot; use pigz4 compressor
z3 backup --compressor pigz4 --dry-run
zfs3backup backup --compressor pigz4 --dry-run
# inspect the commands that would be executed
z3 backup --compressor pigz4
zfs3backup backup --compressor pigz4
# perform full backup of a specific snapshot
z3 backup --full --snapshot the-part-after-the-at-sign --dry-run
zfs3backup backup --full --snapshot the-part-after-the-at-sign --dry-run
# inspect the commands that would be executed
z3 backup --full --snapshot the-part-after-the-at-sign
zfs3backup backup --full --snapshot the-part-after-the-at-sign
```

#### Restore
```
# see restore options
z3 restore --help
zfs3backup restore --help
# restore a dataset to a certain snapshot
z3 restore the-part-after-the-at-sign --dry-run
zfs3backup restore the-part-after-the-at-sign --dry-run
# inspect the commands that would be executed
z3 restore the-part-after-the-at-sign
zfs3backup restore the-part-after-the-at-sign
# force rollback of filesystem (zfs recv -F)
z3 restore the-part-after-the-at-sign --force
zfs3backup restore the-part-after-the-at-sign --force
```

### Other Commands
Other command line tools are provided.

`pput` reads a stream from standard in and uploads the data to S3.

`z3_ssh_sync` a convenience tool to allow you to push zfs snapshots to another host.
`zfs3backup_ssh_sync` a convenience tool to allow you to push zfs snapshots to another host.
If you need replication you should checkout zrep. This exists because we've already
got zrep between 2 nodes and needed a way to push backups to a 3rd machine.

`z3_get` called by `z3 restore` to download a backup.
`zfs3backup_get` called by `zfs3backup restore` to download a backup.

## Development Overview
### Running the tests
Expand Down Expand Up @@ -157,7 +156,7 @@ If backup/restore encounter unhealthy snapshots they abort execution.

### pput
pput is a simple tool with one job, read data from stdin and upload it to S3.
It's usually invoked by z3.
It's usually invoked by zfs3backup.

Consistency is important, it's better to fail hard when something goes wrong
than silently upload inconsistent or partial data.
Expand Down
6 changes: 3 additions & 3 deletions _tests/test_pput.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
import boto
import pytest

from z3.pput import (UploadSupervisor, UploadWorker, StreamHandler,
from zfs3backup.pput import (UploadSupervisor, UploadWorker, StreamHandler,
Result, WorkerCrashed, multipart_etag, parse_metadata,
retry, UploadException)
from z3.config import get_config
from zfs3backup.config import get_config


cfg = get_config()
Expand Down Expand Up @@ -185,7 +185,7 @@ def test_integration(sample_data):
stream_handler = StreamHandler(sample_data)
bucket = boto.connect_s3(
cfg['S3_KEY_ID'], cfg['S3_SECRET']).get_bucket(cfg['BUCKET'])
key_name = "z3_test_" + datetime.now().strftime("%Y%m%d_%H-%M-%S")
key_name = "zfs3backup_test_" + datetime.now().strftime("%Y%m%d_%H-%M-%S")
sup = UploadSupervisor(
stream_handler,
key_name,
Expand Down
16 changes: 8 additions & 8 deletions _tests/test_snap.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
import boto
import pytest

from z3.config import get_config
from z3.snap import (list_snapshots, S3SnapshotManager, ZFSSnapshotManager,
from zfs3backup.config import get_config
from zfs3backup.snap import (list_snapshots, S3SnapshotManager, ZFSSnapshotManager,
PairManager, CommandExecutor, IntegrityError, SoftError,
_humanize, handle_soft_errors)

Expand Down Expand Up @@ -403,7 +403,7 @@ def test_restore_full(s3_manager):
fake_cmd = FakeCommandExecutor()
pair_manager = PairManager(s3_manager, zfs_manager, command_executor=fake_cmd)
pair_manager.restore('pool/fs@snap_1_f')
expected = "z3_get {}pool/fs@snap_1_f | pigz -d | zfs recv pool/fs@snap_1_f".format(
expected = "zfs3backup_get {}pool/fs@snap_1_f | pigz -d | zfs recv pool/fs@snap_1_f".format(
FakeBucket.rand_prefix)
assert fake_cmd._called_commands == [expected]

Expand All @@ -417,9 +417,9 @@ def test_restore_incremental_empty_dataset(s3_manager):
pair_manager.restore('pool/fs@snap_3') # ask for an incremental snapshot
# all incremental snapshots until we hit a full snapshot are expected
expected = [
"z3_get {}pool/fs@snap_1_f | pigz -d | zfs recv pool/fs@snap_1_f",
"z3_get {}pool/fs@snap_2 | zfs recv pool/fs@snap_2",
"z3_get {}pool/fs@snap_3 | zfs recv pool/fs@snap_3",
"zfs3backup_get {}pool/fs@snap_1_f | pigz -d | zfs recv pool/fs@snap_1_f",
"zfs3backup_get {}pool/fs@snap_2 | zfs recv pool/fs@snap_2",
"zfs3backup_get {}pool/fs@snap_3 | zfs recv pool/fs@snap_3",
]
expected = [e.format(FakeBucket.rand_prefix) for e in expected]
assert fake_cmd._called_commands == expected
Expand All @@ -438,7 +438,7 @@ def test_restore_incremental(s3_manager):
pair_manager.restore('pool/fs@snap_3') # ask for an incremental snapshot
# all incremental snapshots until we hit a full snapshot are expected
expected = [
"z3_get {}pool/fs@snap_3 | zfs recv pool/fs@snap_3",
"zfs3backup_get {}pool/fs@snap_3 | zfs recv pool/fs@snap_3",
]
expected = [e.format(FakeBucket.rand_prefix) for e in expected]
assert fake_cmd._called_commands == expected
Expand Down Expand Up @@ -487,7 +487,7 @@ def test_restore_force(s3_manager):
pair_manager.restore('pool/fs@snap_3', force=True) # ask for an incremental snapshot
# all incremental snapshots until we hit a full snapshot are expected
expected = [
"z3_get {}pool/fs@snap_3 | zfs recv -F pool/fs@snap_3",
"zfs3backup_get {}pool/fs@snap_3 | zfs recv -F pool/fs@snap_3",
]
expected = [e.format(FakeBucket.rand_prefix) for e in expected]
assert fake_cmd._called_commands == expected
Expand Down
2 changes: 1 addition & 1 deletion _tests/test_ssh_sync.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# pylint: disable=redefined-outer-name,protected-access
import pytest

from z3.ssh_sync import snapshots_to_send, sync_snapshots
from zfs3backup.ssh_sync import snapshots_to_send, sync_snapshots


ALL = ['S_0', 'S_1', 'S_2', 'S_3', 'S_4']
Expand Down
18 changes: 9 additions & 9 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@


setup(
name="z3",
version="0.2.0",
name="zfs3backup",
version="0.0.0",
platforms='any',
packages=find_packages(),
include_package_data=True,
install_requires=["boto"],
author="PressLabs SRL",
author_email="contact@presslabs.com",
url="https://github.com/presslabs/z3",
author="Marco Montagna",
author_email="marcojoemontagna@gmail.com",
url="https://github.com/mmontagna/zfs3backup",
description="Backup ZFS snapshots to S3",
entry_points={
'console_scripts': [
'pput = z3.pput:main',
'z3 = z3.snap:main',
'z3_get = z3.get:main',
'z3_ssh_sync = z3.ssh_sync:main'
'pput = zfs3backup.pput:main',
'zfs3backup = zfs3backup.snap:main',
'zfs3backup_get = zfs3backup.get:main',
'zfs3backup_ssh_sync = zfs3backup.ssh_sync:main'
]
},
keywords='ZFS backup',
Expand Down
78 changes: 78 additions & 0 deletions venv/bin/activate
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="/home/marco/workspaces/z3/venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
36 changes: 36 additions & 0 deletions venv/bin/activate.csh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This file must be used with "source bin/activate.csh" *from csh*.
# You cannot run it directly.
# Created by Davide Di Blasi <[email protected]>.

alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; test "\!:*" != "nondestructive" && unalias deactivate && unalias pydoc'

# Unset irrelevant variables.
deactivate nondestructive

setenv VIRTUAL_ENV "/home/marco/workspaces/z3/venv"

set _OLD_VIRTUAL_PATH="$PATH"
setenv PATH "$VIRTUAL_ENV/bin:$PATH"



if ("" != "") then
set env_name = ""
else
set env_name = `basename "$VIRTUAL_ENV"`
endif

# Could be in a non-interactive environment,
# in which case, $prompt is undefined and we wouldn't
# care about the prompt anyway.
if ( $?prompt ) then
set _OLD_VIRTUAL_PROMPT="$prompt"
set prompt = "[$env_name] $prompt"
endif

unset env_name

alias pydoc python -m pydoc

rehash

Loading

0 comments on commit 78e8e52

Please sign in to comment.