Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backup functionality #161

Merged
merged 37 commits into from
Jan 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3696d13
Add preliminary backup command
eimrek Sep 26, 2023
ec151e2
Merge branch 'main' into backup-cmd
eimrek Oct 16, 2023
a701374
interface overhaul; still wip
eimrek Oct 25, 2023
15e9982
implement keep; delete old backups
eimrek Nov 1, 2023
b5a30f2
organized into a class & improved logging
eimrek Nov 1, 2023
15bdb18
plug-in func to backup_auto_folders (for aiida-core)
eimrek Nov 2, 2023
ccf56c6
omit symlink without error, if fs doesn't support it
eimrek Nov 7, 2023
f5441cc
add cli tests
eimrek Nov 8, 2023
9de905a
ci: make 'ssh localhost' work
eimrek Nov 8, 2023
a2031d9
cli: return 1 if failure
eimrek Nov 8, 2023
c030168
fix CI macos ssh localhost
eimrek Nov 28, 2023
bd11651
skip backup tests on windows
eimrek Nov 28, 2023
c5ecf2f
reorganize
eimrek Dec 6, 2023
5a20caa
add tests
eimrek Dec 7, 2023
8a2c33f
testing to 100%
eimrek Dec 7, 2023
5da58dc
remove module name in from logging output
eimrek Dec 7, 2023
b88d0ee
update docs
eimrek Dec 7, 2023
f5476a5
fix docs
eimrek Dec 7, 2023
fd2d038
tiny docstring update
eimrek Dec 7, 2023
a195ca5
re-add dest_trailing_slash to call_rsync, as it's used in aiida-core
eimrek Dec 7, 2023
3eb7d17
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
8fa8d6e
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
37af8c0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 11, 2024
d2b7f74
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
ecc32b4
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
c80a499
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
620693c
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
60cd429
Update disk_objectstore/cli.py
eimrek Jan 11, 2024
a699a79
Update docs/pages/backup.md
eimrek Jan 11, 2024
ae367a8
update dbackup docs
eimrek Jan 11, 2024
52b9bf2
Merge branch 'backup-cmd' of github.com:aiidateam/disk-objectstore in…
eimrek Jan 11, 2024
a33cc98
change cli docstring
eimrek Jan 11, 2024
94a37a5
remove verbosity check
eimrek Jan 11, 2024
ddd9150
Update disk_objectstore/backup_utils.py
eimrek Jan 11, 2024
54c1e23
move validate to constructor; use valueerror instead of backuperror
eimrek Jan 11, 2024
a79f73a
Merge branch 'backup-cmd' of github.com:aiidateam/disk-objectstore in…
eimrek Jan 11, 2024
ac10458
adapt tests
eimrek Jan 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cli: return 1 if failure
eimrek committed Nov 8, 2023
commit a2031d91e53468d9ef465f3112bd188bb73c7135
9 changes: 5 additions & 4 deletions disk_objectstore/cli.py
Original file line number Diff line number Diff line change
@@ -235,21 +235,21 @@
elif verbosity == "debug":
backup_utils.logger.setLevel(logging.DEBUG)
else:
click.echo("Unsupported verbosity.")
return
return 1

Check warning on line 239 in disk_objectstore/cli.py

Codecov / codecov/patch

disk_objectstore/cli.py#L238-L239

Added lines #L238 - L239 were not covered by tests

try:
backup_utils_instance = backup_utils.BackupUtilities(
dest, keep, rsync_exe, backup_utils.logger
)
except ValueError as e:
click.echo(f"Error: {e}")
return
return 1

Check warning on line 247 in disk_objectstore/cli.py

Codecov / codecov/patch

disk_objectstore/cli.py#L245-L247

Added lines #L245 - L247 were not covered by tests

success = backup_utils_instance.validate_inputs()
if not success:
click.echo("Input validation failed.")
return
return 1

Check warning on line 252 in disk_objectstore/cli.py

Codecov / codecov/patch

disk_objectstore/cli.py#L251-L252

Added lines #L251 - L252 were not covered by tests

with dostore.container as container:
success = backup_utils_instance.backup_auto_folders(
@@ -258,5 +258,6 @@
)
)
if not success:
click.echo("Error: backup failed.")
return
return 1

Check warning on line 262 in disk_objectstore/cli.py

Codecov / codecov/patch

disk_objectstore/cli.py#L261-L262

Added lines #L261 - L262 were not covered by tests
return 0

Unchanged files with check annotations Beta

if len(split_dest) == 2:
return split_dest[0], Path(split_dest[1])
# more than 1 colon:
raise ValueError("Invalid destination format: <remote>:<path>")

Check warning on line 30 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L30

Added line #L30 was not covered by tests
def is_exe_found(exe: str) -> bool:
self.logger.info(f"Checking if '{self.remote}' is accessible...")
success = self.run_cmd(["exit"])[0]
if not success:
self.logger.error(f"Remote '{self.remote}' is not accessible!")
return False

Check warning on line 81 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L80-L81

Added lines #L80 - L81 were not covered by tests
self.logger.info("Success! '%s' is accessible!", self.remote)
return True
True if validation passes, False otherwise.
"""
if self.keep < 0:
self.logger.error("keep variable can't be negative!")
return False

Check warning on line 97 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L96-L97

Added lines #L96 - L97 were not covered by tests
if self.remote:
if not self.check_if_remote_accessible():
return False

Check warning on line 101 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L101

Added line #L101 was not covered by tests
if not is_exe_found(self.rsync_exe):
self.logger.error(f"{self.rsync_exe} not accessible.")
return False

Check warning on line 105 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L104-L105

Added lines #L104 - L105 were not covered by tests
if additional_exes:
for add_exe in additional_exes:
if not is_exe_found(add_exe):
self.logger.error(f"{add_exe} not accessible.")
return False

Check warning on line 111 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L108-L111

Added lines #L108 - L111 were not covered by tests
path_exists = self.check_path_exists(self.path)
if not path_exists:
success = self.run_cmd(["mkdir", str(self.path)])[0]
if not success:
self.logger.error(f"Couldn't access/create '{str(self.path)}'!")
return False

Check warning on line 119 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L118-L119

Added lines #L118 - L119 were not covered by tests
return True
link_dest_str = str(link_dest.resolve())
else:
# for remote paths, we require absolute paths anyways
link_dest_str = str(link_dest)

Check warning on line 158 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L158

Added line #L158 was not covered by tests
all_args += [f"--link-dest={link_dest_str}"]
if src_trailing_slash:
dest_str = str(dest)
if dest_trailing_slash:
dest_str += "/"

Check warning on line 168 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L168

Added line #L168 was not covered by tests
if not self.remote:
all_args += [dest_str]
try:
res = subprocess.run(all_args, capture_output=True, text=True, check=True)
except subprocess.CalledProcessError as exc:
self.logger.error(f"{exc}")
return False

Check warning on line 179 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L177-L179

Added lines #L177 - L179 were not covered by tests
exit_code = res.returncode
self.logger.debug(
prev_backup_loose = prev_backup / loose_path_rel if prev_backup else None
success = self.call_rsync(loose_path, path, link_dest=prev_backup_loose)
if not success:
return False

Check warning on line 225 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L225

Added line #L225 was not covered by tests
self.logger.info(f"Transferred {str(loose_path)} to {str(path)}")
# step 2: back up sqlite db
f"Dumped the SQLite database to {str(sqlite_temp_loc)}"
)
else:
self.logger.error("'%s' was not created.", str(sqlite_temp_loc))
return False

Check warning on line 249 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L248-L249

Added lines #L248 - L249 were not covered by tests
# step 3: transfer the SQLITE database file
success = self.call_rsync(sqlite_temp_loc, path, link_dest=prev_backup)
if not success:
return False

Check warning on line 254 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L254

Added line #L254 was not covered by tests
self.logger.info(f"Transferred SQLite database to {str(path)}")
# step 4: transfer the packed files
packs_path_rel = packs_path.relative_to(container_root_path)
success = self.call_rsync(packs_path, path, link_dest=prev_backup)
if not success:
return False

Check warning on line 261 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L261

Added line #L261 was not covered by tests
self.logger.info(f"Transferred {str(packs_path)} to {str(path)}")
# step 5: transfer anything else in the container folder
],
)
if not success:
return False

Check warning on line 280 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L280

Added line #L280 was not covered by tests
return True
if success:
self.logger.info(f"Deleted old backup: {folder}")
else:
self.logger.warning("Warning: couldn't delete old backup: %s", folder)

Check warning on line 317 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L317

Added line #L317 was not covered by tests
def backup_auto_folders(
self,
try:
last_folder = self.get_last_backup_folder()
except subprocess.CalledProcessError:
self.logger.error("Couldn't determine last backup.")
return False

Check warning on line 344 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L342-L344

Added lines #L342 - L344 were not covered by tests
if last_folder:
self.logger.info(
last_folder,
)
if not success:
return False

Check warning on line 358 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L358

Added line #L358 was not covered by tests
# move live-backup -> backup_<timestamp>_<randstr>
timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
0
]
if not success:
return False

Check warning on line 369 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L369

Added line #L369 was not covered by tests
self.logger.info(
f"Backup moved from '{str(live_folder)}' to '{str(self.path / folder_name)}'."
["ln", "-sfn", str(folder_name), str(self.path / symlink_name)]
)[0]
if not success:
self.logger.warning(

Check warning on line 380 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L380

Added line #L380 was not covered by tests
f"Couldn't create symlink '{symlink_name}'. Perhaps the filesystem doesn't support it."
)
else:
try:
self.delete_old_backups()
except subprocess.CalledProcessError:
self.logger.error("Failed to delete old backups.")
return False

Check warning on line 390 in disk_objectstore/backup_utils.py

Codecov / codecov/patch

disk_objectstore/backup_utils.py#L388-L390

Added lines #L388 - L390 were not covered by tests
return True