Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: improve the backup section for v2.0 #5431

Merged
merged 2 commits into from
Mar 14, 2022
Merged

Docs: improve the backup section for v2.0 #5431

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1388bd3
Docs: improve the backup section for v2.0
sphuber Mar 12, 2022
e42ffe0
Merge branch 'develop' into fix/4853/docs-backup
chrisjsewell Mar 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 66 additions & 31 deletions docs/source/howto/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ Here are a few tips for tuning AiiDA performance:

.. dropdown:: Move the Postgresql database to a fast disk (SSD), ideally on a large partition.

1. Stop the AiiDA daemon and :ref:`back up your database <how-to:installation:backup:postgresql>`.
1. Stop the AiiDA daemon and :ref:`back up your database <how-to:installation:backup>`.

2. Find the data directory of your postgres installation (something like ``/var/lib/postgresql/9.6/main``, ``/scratch/postgres/9.6/main``, ...).

Expand Down Expand Up @@ -487,68 +487,103 @@ Updating from 0.x.* to 1.*

.. _how-to:installation:backup:

.. _how-to:installation:backup:software:

Backing up your installation
============================

A full backup of an AiiDA instance and AiiDA managed data requires a backup of:

* the AiiDA configuration folder, which is typically named ``.aiida`` and located in the home folder (see also :ref:`intro:install:setup`).
* the AiiDA configuration folder, which is named ``.aiida``.
The location of the folder is shown in the output of ``verdi status``.
This folder contains, among other things, the ``config.json`` configuration file and log files.

* files associated with nodes in the repository folder (one per profile). Typically located in the ``.aiida`` folder.
* the data stored for each profile.
Where the data is stored, depends on the storage backend used by each profile.

* queryable metadata in the PostgreSQL database (one per profile).
The panels below provide instructions for storage backends provided by ``aiida-core``.
To determine what storage backend a profile uses, call ``verdi profile show``.

.. tip:: Before creating a backup, it is recommended to run ``verdi storage maintain``.
This will optimize the storage which can significantly reduce the time required to create the backup.
For optimal results, run ``verdi storage maintain --full``.
Note that this requires the profile to not be in use by any other process.

.. todo::
.. tab-set::

.. _how-to:installation:backup:repository:
.. tab-item:: psql_dos

title: Repository backup
The ``psql_dos`` storage backend is the default backend for AiiDA.
It stores its data in a PostgreSQL database and a file repository on the local filesystem.
To fully backup the data stored for a profile, you should backup the associated database and file repository.

**PostgreSQL database**

.. _how-to:installation:backup:postgresql:
To export the entire database, we recommend to use the `pg_dump <https://www.postgresql.org/docs/12/app-pgdump.html>`_ utility:

Database backup
---------------
.. code-block:: console

PostgreSQL typically spreads database information over multiple files that, if backed up directly, are not guaranteed to restore the database.
We therefore strongly recommend to periodically dump the database contents to a file (which you can then back up using your method of choice).
pg_dump -h <database_hostname> -p <database_port> -d <database_name> -W > aiida_backup.psql

A few useful pointers:
The ``-W`` flag will ensure to prompt for the database password.
The parameters between brackets should be replaced with the values that have been configured for the profile.
You can retrieve these from the ``storage.config`` returned by the ``verdi profile show`` command.

* In order to avoid having to enter your database password each time you use the script, you can create a file ``.pgpass`` in your home directory containing your database credentials, as described `in the PostgreSQL documentation <https://www.postgresql.org/docs/12/libpq-pgpass.html>`_.
.. tip::

* In order to dump your database, use the `pg_dump utility from PostgreSQL <https://www.postgresql.org/docs/12/app-pgdump.html>`_. You can use as a starting example a bash script similar to :download:`this file <include/backup_postgresql.sh>`.
In order to avoid having to enter your database password each time you use the script, you can create a file ``.pgpass`` in your home directory containing your database credentials, as described `in the PostgreSQL documentation <https://www.postgresql.org/docs/12/libpq-pgpass.html>`_.

**File repository**

The file repository is a directory on the local file system.
The most efficient way to create a backup is to use the `rsync <https://en.wikipedia.org/wiki/Rsync>`_ utility.
The path of the repository is shown in the ``storage.config.repository_uri`` key returned by the ``verdi profile show`` command.
To create a backup, simply run:

.. code-block:: console

rsync -arvz <storage.config.repository_uri> /some/path/aiida_backup
chrisjsewell marked this conversation as resolved.
Show resolved Hide resolved

* You can setup the backup script to run daily using cron (see notes in the :ref:`previous section <how-to:installation:backup:repository>`).

.. _how-to:installation:backup:restore:

Restore backup
--------------
Restoring your installation
===========================

Restoring a backed up AiiDA installation requires:

In order to restore a backup, you will need to:
* restoring the backed up ``.aiida`` folder, with at the very least the ``config.json`` file it contains.
It should be placed in the path defined by the ``AIIDA_PATH`` environment variable.
To test the restoration worked, run ``verdi profile list`` to verify that all profiles are displayed.

1. Restore the repository folder that you backed up earlier in the same location as it used to be (you can check the location in the ``config.json`` file inside your ``.aiida`` folder, or simply using ``verdi profile show``).
* restoring the data of each backed up profile.
Like the backup procedure, this is dependent on the storage backend used by the profile.

2. Create an empty database following the instructions described in :ref:`database <intro:install:database>` skipping the ``verdi setup`` phase.
The database should have the same name and database username as the original one (i.e. if you are restoring on the original postgresql cluster, you may have to either rename or delete the original database).
The panels below provide instructions for storage backends provided by ``aiida-core``.
To determine what storage backend a profile uses, call ``verdi profile show``.

3. Change directory to the folder containing the database dump created with ``pg_dump``, and load it using the ``psql`` command.
.. tab-set::

.. tab-item:: psql_dos

To fully backup the data stored for a profile using the ``psql_dos`` backend, you should restore the associated database and file repository.

.. dropdown:: Example commands on Linux Ubuntu
**PostgreSQL database**

This is an example command, assuming that your dump is named ``aiidadb-backup.psql``:
To restore the PostgreSQL database from the ``.psql`` file that was backed up, first you should create an empty database following the instructions described in :ref:`database <intro:install:database>` skipping the ``verdi setup`` phase.
The backed up data can then be imported by calling:

.. code-block:: console

.. code-block:: bash
psql -h <database_hostname> -p <database_port> -d <database_name> -W < aiida_backup.psql

**File repository**

To restore the file repository, simply copy the directory that was backed up to the location indicated by the ``storage.config.repository_uri`` key returned by the ``verdi profile show`` command.
Like the backing up process, we recommend using ``rsync`` for this:

.. code-block:: console

psql -h localhost -U aiida -d aiidadb -f aiidadb-backup.psql
rsync -arvz /some/path/aiida_backup <storage.config.repository_uri>

After supplying your database password, the database should be restored.
Note that, if you installed the database on Ubuntu as a system service, you need to type ``sudo su - postgres`` to become the ``postgres`` UNIX user.

.. _how-to:installation:multi-user:

Expand Down
2 changes: 1 addition & 1 deletion docs/source/intro/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ If you use the same names as in the example commands above, then during the ``ve
.. admonition:: Don't forget to backup your database!
:class: tip title-icon-tip

See the :ref:`Database backup how-to <how-to:installation:backup:postgresql>`), and :ref:`how to move your database <how-to:installation:performance>`.
See the :ref:`Database backup how-to <how-to:installation:backup>`), and :ref:`how to move your database <how-to:installation:performance>`.

Database setup using 'peer' authentication
------------------------------------------
Expand Down