This workflow is only a template for creating new workflows.

We don't need to store that in this repository, it is fairly
simple to spin up a new workflow when needed.

References: BAR-296.

Signed-off-by: Israel Barth <[email protected]>

`main.yml` workflow has been touched by the PR for BAR-301, and it had
issues about outdated actions.

This commit fixes the following issues reported by `actionlint`:

```text
  .github/workflows/main.yml:19:15: the runner of "actions/checkout@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
     |
  19 |         uses: actions/checkout@v2
     |               ^~~~~~~~~~~~~~~~~~~
  .github/workflows/main.yml:21:15: the runner of "actions/setup-python@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
     |
  21 |         uses: actions/setup-python@v2
     |               ^~~~~~~~~~~~~~~~~~~~~~~
  .github/workflows/main.yml:41:15: the runner of "actions/checkout@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
     |
  41 |         uses: actions/checkout@v2
     |               ^~~~~~~~~~~~~~~~~~~
  .github/workflows/main.yml:43:15: the runner of "actions/setup-python@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
     |
  43 |         uses: actions/setup-python@v2
     |               ^~~~~~~~~~~~~~~~~~~~~~~
  .github/workflows/main.yml:47:15: the runner of "actions/cache@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
     |
  47 |         uses: actions/cache@v2
     |               ^~~~~~~~~~~~~~~~
```

References: BAR-301.

Signed-off-by: Israel Barth <[email protected]>

With the current docs for Barman 3.11 we are facing an issue: we use
different files to build the HTML manual and the Linux man pages. As
a consequence, we need to keep updating the same information across
diffrent sources for different docs.

This commit performs a POC for having a single source of truth when
building the Barman documentation. We use reStructuredText markup
language along with Sphinx to build any kind of documentation, i.e.:

* HTML docs (complete)
* PDF docs (do not include code API docs)
* Linux man pages (include only command and configuration reference)

The `index.rst` file defines the complete structure of the docs. Then,
through the variable `exclude_from_builder` in `conf.py` we are able
to define a blacklist of files for each different builder.

To build the documentation, we only need to run either of the
following commands, depending on the desired target:

* `tox -e docs -- html`: for HTML
* `tox -e docs -- latexpdf`: for PDF
* `tox -e docs --  man`: for Linux man pages

References: BAR-299.

Signed-off-by: Israel Barth <[email protected]>

This commit fixes the following issues reported by `shellcheck`:

```text
In .github/foundation/security/scan-hooks.sh line 10:
          source .venv/bin/activate
                 ^----------------^ SC1091 (info): Not following: .venv/bin/activate: openBinaryFile: does not exist (No such file or directory)

  In doc/build/build line 44:
  cp -va *.html *.pdf "${DISTDIR}"
         ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
                ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.

  In scripts/release.sh line 22:
  BASE="$(dirname $(cd $(dirname "$0"); pwd))"
                  ^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
                       ^-------------^ SC2046 (warning): Quote this to prevent word splitting.

  In scripts/release.sh line 27:
  if ! git tag -s -m "Release ${VERSION}" release/${VERSION}
                                                  ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  if ! git tag -s -m "Release ${VERSION}" release/"${VERSION}"

  In scripts/set-version.sh line 24:
  BASE="$(dirname $(cd $(dirname "$0"); pwd))"
                  ^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
                       ^-------------^ SC2046 (warning): Quote this to prevent word splitting.

  In scripts/set-version.sh line 63:
      if [ "$1" == false ]
                ^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.

  In scripts/set-version.sh line 70:
      echo $release_date
           ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
      echo "$release_date"

  In scripts/set-version.sh line 76:
  release_date=$(get_date $DATE)
                          ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  release_date=$(get_date "$DATE")

  In scripts/set-version.sh line 111:
  if branch=$(git symbolic-ref --short -q HEAD) && [ $branch = 'master' ]
                                                     ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  if branch=$(git symbolic-ref --short -q HEAD) && [ "$branch" = 'master' ]

  In scripts/set-version.sh line 138:
  if [ "$DOCKER" == true ]
                 ^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.

  In sphinx/generate_docs.sh line 20:
  BASEDIR=$(cd ${0%/*}; pwd )
            ^--------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
               ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  BASEDIR=$(cd "${0%/*}" || exit; pwd )

  In sphinx/generate_docs.sh line 78:
      printf "${RED}${1}${RSET}
  "
             ^-------------------^ SC2059 (info): Don't use variables in the printf format string. Use printf '..%s..' "$foo".

  In sphinx/generate_docs.sh line 89:
          --)
          ^-- SC2214 (warning): This case is not specified by getopts.

  In sphinx/generate_docs.sh line 104:
      BARMAN_DIR=$(cd "$BASEDIR/.."; pwd)
                   ^--------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
      BARMAN_DIR=$(cd "$BASEDIR/.." || exit; pwd)

  In sphinx/generate_docs.sh line 106:
      BARMAN_DIR=$(cd "$1"; pwd)
                   ^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
      BARMAN_DIR=$(cd "$1" || exit; pwd)

  In sphinx/generate_docs.sh line 113:
  cd "${BASEDIR}"
  ^-------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
  cd "${BASEDIR}" || exit

  In sphinx/generate_docs.sh line 120:
  ls "${BASEDIR}"/docs/*.rst | grep -v 'index.rst$' | xargs -trI X rm -f X
  ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

  In sphinx/generate_docs.sh line 127:
  make ${GEN_MODE}
       ^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  make "${GEN_MODE}"
```

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit fixes the following issue reported by `markdownlint`:

```text
INSTALL.md:1 MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "## Barman INSTALL instructions"]
```

Besides that, it configures `super-linter` to skip Markdown files under
`doc/` and `sphinx/` directories. Those directories contain the source
code for the current Barman documentation, which is going to be
replaced soon.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit introduces the changes proposed by `isort`, so we
please the issues reported by `super-linter`.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit fixes the following issues reported by `flake8`:

```text
barman/compression.py:66:18: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
barman/compression.py:424:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
barman/compression.py:429:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/test_compressor.py:457:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/test_compressor.py:459:20: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
tests/test_compressor.py:460:21: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
```

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit runs `black` so we stop getting failures reported by
`super-linter`.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

We used to have a link in the main toctree called "Barman API docs",
which pointed to the content generated by `autodoc` extension based
on the barman docstrings.

Through this commit we replace that toctree entry with an entry for
a new section called "Contributing to Barman". This sections intends
to provide complete information about how a developer can contribute
to the Barman project.

The old "Barman API docs" was moved under this new section.

References: BAR-320.

Signed-off-by: Israel Barth <[email protected]>

This commit adds a subsection named "Writing and building documentation"
under the section "Contributing to Barman".

Here we explain the style guidelines to be followed when writing
patches that include documentation. Also, it explains how to build the
documentation locally through `tox`.

References: BAR-324.

Signed-off-by: Israel Barth <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>

This commit leverages from the Sphinx capability of maintaining a
glossary.

We add a new `glossary.rst` file where the glossary terms are
maintained. For sake of example, we add a couple of terms in the
glossary (Barman and PGDG).

This is the basic workflow for using the glossary:

* To add a new term to the glossary, add a new entry to the
  `glossary.rst` file.
* To reference a term from the glossary, just use the syntax
  :term:`NAME_OF_TERM`, and Sphinx creates a link to the Glossary
  for you.

References: BAR-326.

Signed-off-by: Israel Barth <[email protected]>

This commit is:
1. Adding barman installation instructions and
the specifics for RHEL-based distros installation.
2. Adding RHEL and RPM terms to glossary

References: BAR-302

Signed-off-by: Andre <[email protected]>

This commit is:
1. Adding installation instructions for debian-based
distributions.
2. Adding DEB term to glossary

References: BAR-303

Signed-off-by: Andre <[email protected]>

add to DEB to glossary

Signed-off-by: Andre <[email protected]>

This commit applies a few small fixes to the glossary page:

* Fixes the section link name, which was broken;
* Adds punctuation to each item defined in the glossary.

Signed-off-by: Israel Barth <[email protected]>

This commit is adding documention on retention policies,

The docs were dividied in:

1. Overview - The content will be created in BAR-327
2. Minimum redundancy safety
3. Scope of retention policies
	3.1 Backup Redundancy
	3.2 Recovery Window
4. Use cases
	4.1 Point-In-Time Recovery
	4.2 Operational Efficiency and Space Management
	4.3 Long-Term Archival
5. How retention policies are enforced
6. Configurations and Syntax
7. Retention policy for block-level incremental backups

References: BAR-309

Signed-off-by: Andre <[email protected]>

This commit is adding a more robust introduction
on retention policies.

References: BAR-327

Signed-off-by: Andre <[email protected]>

This commit is:

1. Adding an introduction to the barman catalog section.
2. Adding "Purpose" section to catalog docs
3. Adding "Usage" section to catalog docs
4. Adding `list-backups` section documenting the CLI
command with example
5. Adding `show-backup` section documenting the CLI
command with example

References: BAR-307

Signed-off-by: Andre <[email protected]>

Adds a quickstart section to the new documentation, providing simple
setups for two common backup scenarios: backups with streaming
replication and backups with rsync and WAL archiving.

References: BAR-304

Signed-off-by: Gustavo William <[email protected]>

Add information about the `check` command to the new documentation.

References: BAR-305

Signed-off-by: Barbara Leidens <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>

This commit is:

1. Adding an introduction about recovery
2. Adding Usage section and the following subsection
	2.1 Remote Recovery
	2.2 Tablespace remapping
	2.3 PITR
	2.4 Fetching WALs from Barman
	2.5 Recovering Compressed backups
	2.6 Recovering block-lvl increm backups
3. Adding Limitations of .partion WAL files section

This commit is also adding the limitations on
`--no-get-wal` usage and staging directory to combine
backup in the beginning of the recovery process of
block-level incremental backups.

References: BAR-308 & BAR-206 & BAR-191

Signed-off-by: Andre <[email protected]>

This commit is ...
* Adding an introduction on the configuration reference
* Adding section `Usage` about the scope of the configuration
files and a simple example of these files. In the same section,
we have documented the Reserverd Words and Parameter Types.
* Adding a section `Options` with all options that can be
set in the configuration files. This section have the following
subsections:
	* General
	* Backups
	* Cloud Backups
	* Hook Scripts
	* Write-Ahead Logs (WAL)
	* Recovery
	* Retention Policies.
* Adding a section `Examples` with an example and the proper
explanation of that.

References: BAR-310

Signed-off-by: Andre <[email protected]>

The `barman-cloud-backup-keep` script has shortcuts for options
`--status` and `--release`. These were missing in the `barman`
CLI implementation, even though the error message suggested they
existed.

This aligns the shortcuts throughout the code so both implementations
have the same arguments.

References: BAR-356

Signed-off-by: Martín Marqués <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>

This commit adds documentation about the barman-cloud.

It was divided as follows:

* The main section is called `Barman for the cloud`. There is
an introduction on how Barman can work with cloud backups.
* Next section is called `barman-cli-cloud` that explains about
the package, followed by an `Installation` section.
* Lastly, the `Commands Reference` details each command
available from the package.

Each command was created in its own `.rst` file and they are
referenced in the `conf.py` file. This is meant to be used
to create MAN pages for each command.

This commit also adds the `--no-partial` to documentation
with the parameters description and alsoa brief explanation
in about its use.

References: BAR-312 & BAR-180

Signed-off-by: Andre <[email protected]>

This commit adds the license section of the documentation.

References: BAR-314

Signed-off-by: Andre <[email protected]>

Adds an "Introduction" subsection to the "Concepts" section of the new
documentation. It talks about general backup scenarios and advises on
disaster recovery planning.

References: BAR-338

Signed-off-by: Gustavo William <[email protected]>

Adds the "General backup concepts" subsection on the "Concepts" section.
This section primarily focuses on explaining the fundamental backup
concepts common across most relational databases and critical for
understanding backups and backup management tools.

References: BAR-339

Signed-off-by: Gustavo William <[email protected]>

This commit is adding the a reasoning note about why
we get `Check 'empty incoming directory' failed for server '<SERVER>'`
when WALs are present in the barman archive directory.

It also is formatting the documentation because of max sentence length,
some issues with the line breaks between bullet points and other house
keeping task so the actualy rendered doc looks nice and clean.

Closes GH-622 which exists in the barman repository (not this internal
one).

References: BAR-238

Signed-off-by: Andre <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>

This commit is adding details on the backup command docs.

* An Overview introduction on the command and some caveats
* Sections
	* Incremental Backups
	* Managing Bandwith Usage
	* Network Compression
	* Backup Compression
		* Compression algorithms
		* Compression workers
		* Compression level
		* Compression location
	* Immediate checkpoint
	* Streaming Backups - for pg_basebackup
		* Block-level incremental backups
	* Backup with Rsync through SSH - for rsync backups
		* File-level incremental backups
	* Cloud Snapshot Backups - for snapshot backups
		* Requirements and Configuration
		* Google Cloud Platform
		* Microsoft Azure
		* Amazon Web Services
		* Backup Process - snapshot backups process
		* Metadata - snapshots metadata
	* Hook scripts - using barman-cli-cloud as hook
	  scripts with barman.
	* Concurrent backup of a standby
* Added VLDB term to the glossary

References: BAR-306

Signed-off-by: Andre <[email protected]>

Adds the "Postgres backup concepts and terminology" subsectino to the
"Concepts" section. It explains some important backup concepts in the
context of Postgres.

References: BAR-340

Signed-off-by: Gustavo William <[email protected]>

Adds the "Architectural backup designs with Barman" section to the new
documentation.

References: BAR-330

Signed-off-by: Gustavo William <[email protected]>

This commit adds all commands available in barman,
the description and the details on options and fields
available for each command.

The commands section (commands.rst) starts with a
synopsis on the `barman` command, followed by the
global options, a shortcut table and a brief on exit
statuses. Lastly, there is a list of the sub-commands
with synopsis, description and parameters.

Each command has its own `.rst` file which use the
`.. include::` directive from sphinx to be able to
group the commands in the `commands.rst` file. This
was done to be able to produce single man pages for
each command.

In the `.conf` file, the commands are referenced to
tell sphinx to render man pages.

References: BAR-311

Signed-off-by: Andre <[email protected]>

This commit is detailing the configuration options

* custom_compression_filters
* custom_decompression_filters
* custom_compression_magic

This was brought up in the GH-270.

References: BAR-236

Signed-off-by: Andre <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>

Adds the "Barman concepts and terminology" subsection to the "Concepts"
section. Its purpose is to clarify some basic Barman concepts
as well as explain how Barman utilizes the features of Postgres that
was explained in the previous sections in order to manage backups
internally.

References: BAR-341

Signed-off-by: Gustavo William <[email protected]>

Adds the "Pre-requisites" section to the new docs. For now, it only
contains instructions on how to set up postgres and ssh connections.

References: BAR-355

Signed-off-by: Gustavo William <[email protected]>

This commit adds a `Configuration Models` section to
the configuration documentation to explain about models.

This section is divided in `Purpose`, `Application`, `Usage`,
`Benefits`.

This section focuses on how models work, why they are useful,
how to use it.

Lastly, there is an `Examples` section divided in
`Global Configuration`, `Server Configuration - Rsync`,
`Server Configuration - pg_basebackup`, `Model Configuration 1`
and `Model Configuration 2`.

This section have some examples to help the user understand
how configurations are set and how/why to work with it.

References: BAR-345 & BAR-344

Signed-off-by: Andre <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>

This commit adds the --reset argument to the config-swith
command from Barman that was missing.

References: BAR-376

Signed-off-by: Andre <[email protected]>

Adds the frequently asked questions section into the new documentation.

References: BAR-313

Signed-off-by: Gustavo William <[email protected]>

This commit adds the reference to the configuration.rst
to create MAN page 5 for configuration settings.

It also changes a little the configuration.rst formatting
for the MAN page. The example in the end is being omitted
for MAN page, only HTML rendering.

References: BAR-369

Signed-off-by: Andre <[email protected]>

Clarify an existing note about tablespaces remapping in Barman. We
should not distinguish between local and remote recovery. The same logic
is applied regardless of the destination.

References: BAR-351

Signed-off-by: Gustavo William <[email protected]>

Adds a brief explanation of the Postgres recovery process in the
"Postgres concepts and terminology" and a brief explanation of the
restore process that Barman performs in "Barman concepts in terminology" so
that the reader can distinguish between both terms.

References: BAR-370

Signed-off-by: Gustavo William <[email protected]>

Inspecting the code, it seems this directory is only used for duplicated WAL
files and unexpected files found in the WAL archive directory.

References: BAR-343

Signed-off-by: Gustavo William <[email protected]>

This commit goal is to change the cross-reference labels to
the recommended standard detailed in the contributing
section.

The existing labels were all replaced and some rewording was
needed to accommodate the changes.

There are a few labels that have no topic written yet so it
has a `-TBD` appended to it provisionally.

References: BAR-379

Signed-off-by: Andre <[email protected]>

There were some subsubsections which should actually be subsections.

References: BAR-386

Signed-off-by: Gustavo William <[email protected]>

This commit adds a new top section on `Hook Scripts`, a
sub-section to `Concepts` called `Hook Scripts` and
changes and add a few cross-reference labels.

It also removes Hook Scripts from the backup.rst and
adds the hook scripts section to the index file.

References: BAR-382

Signed-off-by: Andre <[email protected]>

This commit:

* Add cross-reference labels to support cross-reference
  between pages
* Add IAC to glossary
* Add section about snapshots in the recovery doc
* Add section about snapshots in the retetion policy doc

References: BAR-353

Signed-off-by: Andre <[email protected]>

This commit refactors the installation docs to accommodate
the barman-cli and barman-cli-cloud packages. there are
minor differences on how to install those packages depending
on the distribution used, so this is being explained in the
installation.rst file.

The barman_cloud installation section was also refactored to
reference the installation main section.

References: BAR-385

Signed-off-by: Andre <[email protected]>

In this commit we remove the python version in the tox section
for docs as it isn't really necessary.

References: BAR-388

Signed-off-by: Martín Marqués <[email protected]>

This commit provides instructions in the Barman docs for installing
the required postgres client package on the barman server in case
of needing the pg_receivewal tool for WAL streaming.

References: BAR-377

Signed-off-by: Martín Marqués <[email protected]>

This commit adds a section named `SLES-based distributions`
under installation.rst explaining how to install the packages
for the SLES-based distros.

It also adds SLES to the glossary.

References: BAR-390

Signed-off-by: Andre <[email protected]>

Remove SLES 12 from docs and add suggestion to the note.

References: BAR-390

Signed-off-by: Andre <[email protected]>

Add a delete annotation, which is applied to a backup at the
start of the delete process and only removed after it has finished.
The deletion process also ensures the `backup.info` file is the last
to be deleted. If a crash occurs, this file will remain, and the
system can resume the deletion upon recovery. In addition,
`cron_retention_policy` has been updated to check for backups marked
with this new annotation and, if found, the cron job will complete
the deletion process, preventing potential issues with orphaned files
and inaccurate backup size calculations.

References: BAR-244

Signed-off-by: Barbara Leidens <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>
Co-authored-by: Israel <[email protected]>

This commit:

* Add command references for barman-wal-restore and
  barman-wal-archive in separate files and include
  them under the Commands Reference under a new
  section named `barman-cli commands`.
* Change cross-reference label in the quickstart
* Add `WAL archiving via archive_command` to
  pre_requisites.rst
* Add documentation about barman-wal-restore to
`Fetching WALs from Barman` section under recovery.

This commit is also adding a tip note to the
`Fetching WALs from Barman` section about performance
gains when pg_wal and spool_dir are in the same
filesystem.

References: BAR-387 & BAR-205

Signed-off-by: Andre <[email protected]>

Postgres 17 has been released so we can now point to the
current postgres documentation for incremental backup information.

References: BAR-282

Signed-off-by: Martín Marqués <[email protected]>

In the documentation we made use of phrases like "positive
integer greater than zero", but the term positive interger
doesn't include the zero, so it's redundant the "greater
than zero" part.

References: BAR-346

Signed-off-by: Martín Marqués <[email protected]>

This commit adds a note in the documentations in the WAL streaming
section of the architectures recommending the users to always
stream from the primary node.

It also changes some of the references and splits the installation
of the postgres clients from the streaming setup.

References: BAR-368

Signed-off-by: Martín Marqués <[email protected]>

References: BAR-395

Signed-off-by: Martín Marqués <[email protected]>

When including files which contain reference anchors, sphinx will
complain that there are duplicate labels.

The workaround is to exclude those patterns. But in order to not
create a huge list of names in `exclude_patters` list, we changed
the extension of those included files from `.rst` to `inc..rst`,
and then just added `**/*.inc.rst` to the `exclude_patterns` list.

References: BAR-395

Signed-off-by: Martín Marqués <[email protected]>

References: BAR-395

Signed-off-by: Martín Marqués <[email protected]>

References: BAR-329

Signed-off-by: Martín Marqués <[email protected]>

Although the Material for Sphinx theme looked very nice, we had
an issue with it in regards to the TOC tree: either it renders
all tree nodes expanded, or it renders only the current section
nodes expanded.

We have been looking for alternative themes that could provide the
capability of expanding and contracting the tree nodes, and we found
these two:

* Read The Docs
* PyData

This commit delivers that feature by using PyData theme. We chose it
because the docs looked nicer on PyData than on Read The Docs theme.

It is worth noting that PyData renders the HTML docs a bit different
than Material and Read The Docs. Instead of merging the documents tree
with the page contents to create the left sidebar tree, PyData splits
the navigation through different parts:

* Top navbar: level-0 documents of the TOC tree.
* Left sidebar: level-1+ documents of the TOC tree.
* Right sidebar: active page content tree.

With that in mind, we had to apply some changes to the structure of
the docs:

* We had a single level-0 document defined in the root `index.rst`.
  Now we have these level-0 documents:
  * `user_guide/index.rst`: contains most of the previously existing
    documents, now as subdocuments under `user_guide/`.
  * `contributor_guide/index.rst`: the former `contributing.rst`, with
    a TOC tree. The subsections of the former `contributing.rst` were
    turned into separate subdocuments under `contributor_guide`. This
    was done because we had different logical pieces there, making sense
    to have them in separate subdocuments.
  * `release/index.rst`: the former `releases.rst`.
  * `faq/index.rst`: the former `faq.rst`.
  * `license/index.rst`: the former `license.rst`.

Besides that, these changes were applied to accomplish that goal:

* A new `index_pdf.rst` was created just so we have a different depth
  of the TOC tree between HTML and PDF outputs. As the `index.rst` is
  used by the main page of PyData, in that page we want only level-0
  heads. On the other hand, when building the index for the PDF, we want
  more levels for easy links to subsections.
* Image paths were fixed. They were using relative paths which no longer
  work, and that was preventing the PDF document to be built. Now we use
  absolute path when referencing images in the documentation.
* In Sphinx configuration:
  * Theme `sphinx_material` replaced with `pydata_sphinx_theme`
    * `html_theme_options` value changed to use options exposed by the
      new theme, in short:
      * Set up a couple icon buttons on the top nav.
      * Show 4 levels in the page content tree, and a edit page button
        in the right sidebar.
      * Hide the footer.
    * `html_side_bars`: to use the beauty left sidebar of PyData, with
      expandable/collapsable nodes.
  * `man_pages` now reference the new location of subdocuments, under
    the `user_guide` folder.
  * Autodoc of modules were moved from root to the `contributor_guide`
    folder.
    * Latex builder excludes was updated to reference the new path.
  * `latex_documents` was changed to reference the new `index_pdf.rst`
    file, which contains a different TOC tree depth than the default
    one used by HTML.

The `tox.ini` config was updated to reflect the usage of the new theme.

A side fix implemented by this commit: we fix the `exclude_patterns`
conf variable, so we don't exclude the `*.inc.rst` files when building
man pages. That is required so we are able to build both a complete man
page, and the split man pages.

References: BAR-394.

Signed-off-by: Israel Barth <[email protected]>

- Change message in license to reflect that we will acknowledge contribution
  in release notes.
- Unify the usage of Postgres instead of PostgreSQL. There are exceptions,
  like when refering to something which's name contains PostgreSQL, like
  "The PostgreSQL Administration Cookbook", or "The PostgreSQL Project".
- Add a paragraph at the beginning of the concepts section mentioning to the
  reader that if they are familiar with the topic of backups in Postgres,
  they should skip to the "Barman concepts and terminology" section.
- Change apt-get for apt.
- In the installation section list the packages that will be installed instead
  of using "others".
- Split the Postgres connection section from pre-requisites
  into "Postgres user" and "Postgres Connection".
- Add references to Postgres docs for configuring hba.conf,
  and to other parts of the barman docs on user setup.
- Add geo-redundancy section. The geo-redundancy section is empty, but this
  fixes the last two references that were broken in the docs.
- Move the backup from standby section before snapshot backups.
- Relabel master as primary.
- Mention `barman keep` in the retention policy section. I added a subsection
  on the usage of `barman keep` in the retention policy section. This was
  something that I found missing when reading the retention policy.
- Standardize lists: Some lists were out of the pattern, either using `-`
  instead of `*`, or ending items in `;` instead of `.`.
- Don't use Italics for emphasis.
- Standardize usage of monospace

References: BAR-397

Signed-off-by: Martín Marqués <[email protected]>
Signed-off-by: Israel Barth <[email protected]>
Co-authored-by: Andre Marchesini <[email protected]>
Co-authored-by: Gustavo William <[email protected]>

This commit removes this action that is not needed anymore. This
was actually used in the documentation workflow directly.

References: BAR-363

Signed-off-by: Andre <[email protected]>

This commit:

1. Modify .gitignore to allow build files for man pages
to be sent to the remote repository.
2. Add reference to man page files in the new_docs/conf.py
and setup.py.
3. Add man pages that were manually built.
4. Change MANIFEST.in, removing old docs and adding new
docs sample config files.

References: BAR-404

Signed-off-by: Andre <[email protected]>

Add man page files to the remote repository

When building packages, the foundation packaging workflow
uses these files in the deb/debian/rules and
rpm/barman.spectemplate (and rpm/barman.foundationspectemplate)
to properly set the manuals for the commands by packages
(barman, barman-cli and barman-cli-cloud). These directories
mentioned previously are in the barman-packaging repository.

References: BAR-404

Signed-off-by: Andre <[email protected]>

Move files from /doc to /new_docs

These files are used as doc files for rpm and deb packaging.

They are present in the deb/debian/barman.install and in the
rpm/barman.spectemplate (and rpm/barman.foundationspectemplate).

They are required to properly build packages for the barman
software.

References: BAR-404

Signed-off-by: Andre <[email protected]>

rebase

Signed-off-by: Andre <[email protected]>

This commit:

1. Adds `--md5` argument to the barman-wal-archive command.
	This is needed to keep compatibility between client
	and server version (PG server and Barman server). If
	the barman server is behind the PG server, this flag
	is needed so the client uses the `md5` hash algorithm
	to create the checksums. The barman server can only
	understand `md5` checksums at this point.

2. Changes  method name from `md5copyfileobj` to `hashCopyfileobj`
and refactor the code. A new paramter was added called `hash_algorithm`.
	This name is generic since we will use md5 or sha256
	depending on the case. The code was changed to accommodate
	the use of one of the two hash algorithms.

3. Adds an __init__ method to the ChecksumTarFile class so it is
initilized with hash_algorithm and HASHSUMS_FILE attributes for
`sha256`.
	This makes it easy to use the attributes in the class scope.
	The default value could have been set to `None` but it is
safer to keep a default value here since the methods using this
cannot receive None as an input.

4. Modifies the `addfile` method so it uses the new method
`hashCopyfileobj` with the new hash algorithm attribute as a parameter.

5. Modifies the `close` method so it uses generic names for variables.

6. Modifies the __init__ method from RemotePutWal class so it checks
for the new `--md5` flag, sets the correct algorithms and HASHSUMS file
and passes to the tar object.

7. Refactors `put_wal` command to keep compatibility between server and
client checksum calculations. The iteration on the tar file is in
streaming mode, so we cannot use random access. With this, we need to
iterate over the entire tar file, get the hash algorithm from the name
of the streamed HASHSUMS file and iterate over the dictionary to
calculate the checksums with the correct hash algorithm.

8. Refactored the `file_md5` method from utils so it has a generic name
to calculate with a hash algorithm. A new parameter was added
called `hash_algorithm`.

9. Add release_note for the feature - Add FIPS support

References: BAR-155

Signed-off-by: Andre <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>

This commit refactors all broken tests and adds
use cases for both algorithms and hashsum files.

References: BAR-155

Signed-off-by: Andre <[email protected]>

Removes duplicate information present on the new docs on the barman-cloud-backup
section.

References: BAR-422

Signed-off-by: Gustavo William <[email protected]>

This commit:

1. Replaces recover by restore for clarity. One good statement of
why we did this replacement is the following:

Barman restores a backup, preparing it for Postgres recovery.
When started, Postgres enters recovery mode, applying WALs
until reaching consistency or a specified target.

2. Replaces the static man page barman-recover for barman-restore.

References: BAR-337

Signed-off-by: Andre <[email protected]>
Co-authored-by: Israel Barth <[email protected]>

This commit replaces the recover word by restore to give
more clarity to the user when working with the restore
process from Barman.

Some of the important changes are:

1. Change the recover command to be `barman restore`
2. Change output messages when working with the command.

References: BAR-337

Signed-off-by: Andre <[email protected]>
Co-authored-by: Israel Barth <[email protected]>

This commit addresses all the broken tests created from
the changes made to the recover command.

References: BAR-337

Signed-off-by: Andre <[email protected]>

When backing up a standby server, Barman performs some checks to assert
that `conninfo` is really pointing to a standby (in recovery mode) and
that `primary_conninfo` is pointing to a primary (not in recovery).

The problem, as reported in the issues #704 and #744, is that when a
failover occurs, the `conninfo` will now be pointing to a primary
instead and the checks will start failing, requiring the user to change
Barman configs manually whenever a failover occurs.

This commit fixes the issue by making such checks non-critical, which
means they will still fail but Barman will keep operating regardless.
Essentially, Barman will ignore `primary_conninfo` if `conninfo` does
not point to a standby. Warnings about this misconfiguration will also
be emited whenever running any Barman command so the user can be aware.

References: BAR-348

Signed-off-by: Gustavo William <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>

Fixes some tests that were broken after lightening some checks
regarding `conninfo` and `primary_conninfo` in a previous commit.

References: BAR-348

Signed-off-by: Gustavo William <[email protected]>

This commit changes the model_name argument description from
the config-switch command to be clearer for a user that wants
to use the command. The model_name argument is an optional
argument but it's also a positional argument. As the help output
has positional argument and optional argument sections, the user
can get confused working with this command. The model_name
appears in the positional argument section of the help command.

References: BAR-378

Signed-off-by: Andre <[email protected]>

Execute isort on the whole projet cleaning up and sorting imports
in both tests and code.

References: bar-429

Signed-off-by: Giulio Calacoci <[email protected]>

Signed-off-by: Giulio Calacoci <[email protected]>

This commit creates a Dockerfile in the new_docs folder
that can be used as a base environment to build the docs.
A new script to build the docs is also created.

References: BAR-363

Signed-off-by: Andre <[email protected]>

This issue was identified while working on BAR-289. The `linter.yml`
had a failure because of wrong formatting in the configuration file
for Isort. Turns out it was defined using `toml` syntax instead of
`configparser` syntax.

This commit fixes that oversight.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This issue was identified while working on BAR-289. The `linter.yml`
is using a `.flake8` file which contains a set of configuration "A",
while `tox` is using a `tox.ini` file which contains another set of
configuration "B".

This commit fixes that oversight. Now both `super-linter` and `tox`
will use the very same `tox.ini`, so we have a single source of
truth for the Flake8 configuration.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit fixes the following issues reported by `shellcheck`:

```text
In .github/foundation/security/scan-hooks.sh line 10:
          source .venv/bin/activate
                 ^----------------^ SC1091 (info): Not following: .venv/bin/activate: openBinaryFile: does not exist (No such file or directory)

  In doc/build/build line 44:
  cp -va *.html *.pdf "${DISTDIR}"
         ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
                ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.

  In scripts/release.sh line 22:
  BASE="$(dirname $(cd $(dirname "$0"); pwd))"
                  ^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
                       ^-------------^ SC2046 (warning): Quote this to prevent word splitting.

  In scripts/release.sh line 27:
  if ! git tag -s -m "Release ${VERSION}" release/${VERSION}
                                                  ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  if ! git tag -s -m "Release ${VERSION}" release/"${VERSION}"

  In scripts/set-version.sh line 24:
  BASE="$(dirname $(cd $(dirname "$0"); pwd))"
                  ^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
                       ^-------------^ SC2046 (warning): Quote this to prevent word splitting.

  In scripts/set-version.sh line 63:
      if [ "$1" == false ]
                ^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.

  In scripts/set-version.sh line 70:
      echo $release_date
           ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
      echo "$release_date"

  In scripts/set-version.sh line 76:
  release_date=$(get_date $DATE)
                          ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  release_date=$(get_date "$DATE")

  In scripts/set-version.sh line 111:
  if branch=$(git symbolic-ref --short -q HEAD) && [ $branch = 'master' ]
                                                     ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  if branch=$(git symbolic-ref --short -q HEAD) && [ "$branch" = 'master' ]

  In scripts/set-version.sh line 138:
  if [ "$DOCKER" == true ]
                 ^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.

  In sphinx/generate_docs.sh line 20:
  BASEDIR=$(cd ${0%/*}; pwd )
            ^--------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
               ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  BASEDIR=$(cd "${0%/*}" || exit; pwd )

  In sphinx/generate_docs.sh line 78:
      printf "${RED}${1}${RSET}
  "
             ^-------------------^ SC2059 (info): Don't use variables in the printf format string. Use printf '..%s..' "$foo".

  In sphinx/generate_docs.sh line 89:
          --)
          ^-- SC2214 (warning): This case is not specified by getopts.

  In sphinx/generate_docs.sh line 104:
      BARMAN_DIR=$(cd "$BASEDIR/.."; pwd)
                   ^--------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
      BARMAN_DIR=$(cd "$BASEDIR/.." || exit; pwd)

  In sphinx/generate_docs.sh line 106:
      BARMAN_DIR=$(cd "$1"; pwd)
                   ^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
      BARMAN_DIR=$(cd "$1" || exit; pwd)

  In sphinx/generate_docs.sh line 113:
  cd "${BASEDIR}"
  ^-------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

  Did you mean:
  cd "${BASEDIR}" || exit

  In sphinx/generate_docs.sh line 120:
  ls "${BASEDIR}"/docs/*.rst | grep -v 'index.rst$' | xargs -trI X rm -f X
  ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

  In sphinx/generate_docs.sh line 127:
  make ${GEN_MODE}
       ^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.

  Did you mean:
  make "${GEN_MODE}"
```

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

This commit fixes the following issue reported by `markdownlint`:

```text
INSTALL.md:1 MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "## Barman INSTALL instructions"]
```

Besides that, it configures `super-linter` to skip Markdown files under
`doc/` and `sphinx/` directories. Those directories contain the source
code for the current Barman documentation, which is going to be
replaced soon.

References: BAR-315.

Signed-off-by: Israel Barth <[email protected]>

References: BAR-356

Signed-off-by: Martín Marqués <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>
Signed-off-by: Israel Barth <[email protected]>

References: BAR-244

Signed-off-by: Barbara Leidens <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>
Co-authored-by: Martín Marqués <[email protected]>
Co-authored-by: Israel <[email protected]>

Add a release note about the validation of the latest Barman features
in both PGE 17 and EPAS 17.

References: BAR-331

Signed-off-by: Gustavo William <[email protected]>

This commit:

1. Modify .gitignore to allow build files for man pages
to be sent to the remote repository.
2. Add reference to man page files in the new_docs/conf.py
and setup.py.
3. Add man pages that were manually built.
4. Change MANIFEST.in, removing old docs and adding new
docs sample config files.

References: BAR-404

Signed-off-by: Andre <[email protected]>

Add man page files to the remote repository

When building packages, the foundation packaging workflow
uses these files in the deb/debian/rules and
rpm/barman.spectemplate (and rpm/barman.foundationspectemplate)
to properly set the manuals for the commands by packages
(barman, barman-cli and barman-cli-cloud). These directories
mentioned previously are in the barman-packaging repository.

References: BAR-404

Signed-off-by: Andre <[email protected]>

Move files from /doc to /new_docs

These files are used as doc files for rpm and deb packaging.

They are present in the deb/debian/barman.install and in the
rpm/barman.spectemplate (and rpm/barman.foundationspectemplate).

They are required to properly build packages for the barman
software.

References: BAR-404

Signed-off-by: Andre <[email protected]>

rebase

Signed-off-by: Andre <[email protected]>

References: BAR-155

Signed-off-by: Andre <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>
Signed-off-by: Israel Barth <[email protected]>

References: BAR-348

Signed-off-by: Gustavo William <[email protected]>
Signed-off-by: Giulio Calacoci <[email protected]>
Signed-off-by: Israel Barth <[email protected]>

This workflow should no longer exist in the public repository, so
this commit removes the workflow.

References: BAR-434.

Signed-off-by: Israel Barth <[email protected]>

We removed the old `NEWS` file and replaced it with a new
`RELNOTES.md` file. The new file is better structured and
written in markdown language.

This new markdown file has been included in the new user
documentation as well, so they can access that info both
from GitHub and from the user docs.

References: BAR-290.

Signed-off-by: Giulio Calacoci <[email protected]>
Signed-off-by: Israel Barth <[email protected]>

In the release notes we have this message:

"`barman keep` command can only be applied to full backups when
`backup_method = postgres`. If a full backup has incremental backups
that depend on it, all of the incrementals are also kept by Barman."

In fact, the incremental backups are marked as valid not kept
(user can still delete them).

This commit amends the release note to be precise on this matter.

References: BAR-408

Signed-off-by: Andre <[email protected]>

This commit changes the documentation of the put-wal command that
was only mentioning the usage of MD5SUMS file for validation. In
BAR-155, we added support for FIPS compliance making sha256 the
default algorithm, so this needs to reflect in the whole doc.
Barman can still use md5, but it is not the default anymore.

This also adds the man pages where put-wal is mentioned.

References: BAR-437 & BAR-155

Signed-off-by: Andre <[email protected]>

This commit refactors the _backup_report method for both window and
redundancy retention policies.

The code now handles the case when retention policy status of the
root backup is KEEP_STANDALONE and the incrementals are in or out of
the retention policy rule. For KEEP_FULL, it remains the same logic.
For KEEP_STANDALONE, when the root backup is within the retention policy
rule, the incrementals are considered valid. When this root backup falls
out of the retention policy rule, it is considered obsolete and will be
removed by the maintenance cron command.

This commit also modifies the `_propagate_retention_status_to_children`
so it does not have any logic regarding the retention policy. Now it just
propagates the status without checking the keep flag. This logic was moved
to the _backup_report method, so we handle FULL and STANDALONE KEEP flags.
The reason to move this was:
1. The checks for retention policy status should be a "business" logic for
_backup_report and the _propagate_retention_status_to_children should only
handle the propagation of the status.
2. We have both STANDALONE and FULL logic in the same place.

References: BAR-366

Signed-off-by: Andre <[email protected]>

This commit refactors existing tests and adds new tests
to cover the new logic implemented for STANDALONE backups.

References: BAR-366

Signed-off-by: Andre <[email protected]>

This commit modifies the ` Retention policy for block-level
incremental backups` section in the retetion policies
documentation.

This modification is made regarding the change on the behavior of
the propagation of the KEEP flags from the root backup to its
incremental backups.

References: BAR-366

Signed-off-by: Andre <[email protected]>

This commit adds a release note about this modification.

References: BAR-366

Signed-off-by: Andre <[email protected]>

This commit adds a release note that should have been created
together with the BAR-389. BAR-389 is a PR about removing packaging
and smoke tests support for RHEL7, debian10 and ubuntu1804.

References: BAR-415 & BAR-389

Signed-off-by: Andre <[email protected]>

As it's been reported in the issue #1018, it seems that using the
`upload_fileobj` method to upload WAL files has more latency than using
`put_object`.

Various tests were run using a Python script similar to the one given in
the issue and the results were always similar to this:

The average latency for a file of 16 MB (default size of WAL files):
- using `upload_fileobj`: 4805.58967328ms
- using `put_object`: 3744.6347383500006ms

In this case, `put_object` is about 21% faster.

The average latency for file of 1GB (maximum size of relational files, by default):
- using `upload_fileobj`: 102020.6359049ms
- using `put_object`: 125556.6254884ms

In this case, `upload_fileobj` is about 18% faster.

So it really depends on the file size. I don't quite know the reason for
sure, but `upload_fileobj` uses multipart upload depending on the file
size while `put_object` always upload files in a single PUT request, so
that could add to the reason as multipart uploading requires some extra
steps.

AWS recommends considering multipart upload for files above 100MB. In
some testings I found out that the difference between both methods
really starts appearing for files around 150MB. But the results can vary
a lot from test to test, so I suppose it's better to rely on AWS numbers
as they probably have done better and more controlled tests than me in
order to write down that number.

That said, this commit slightly changes the upload method we call on the
boto3 library. We still use `upload_fileobj`, but now we pass a config
object containing a `multipart_threshold` value of 100MB, which
determines when multipart uploads should be triggered. When a file
smaller than that is uploaded, boto3 will actually trigger the
`put_object`  behind the scenes. Therefore, this is a simpler change as replacing
it for `put_object` directly would involve more code changes and the
methods don't have quite the same signature.

I also tested using `upload_fileobj` with different threshold values and
the results were pretty similar to the previous ones i.e. small files
being uploaded faster when multipart is not used and vice-versa, so it
must really use `put_object` behind the scenes.

Lastly, I also compared the changed code with the previous running on a
real Postgres cluster on an AWS instance, where network is more stable,
and the results were similar, showing a performance gain of around 15%
when avoiding multipart upload for 16MB WAL files.

Links:
- original issue: #1018
- AWS recommendation of when using multipart uploads: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
- boto3 code which I think is used by `upload_fileobj`: https://github.com/boto/s3transfer/blob/develop/s3transfer/__init__.py#L727-L755
- `upload_fileobj` command ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/upload_fileobj.html

References: BAR-374

Signed-off-by: Gustavo William <[email protected]>