Sync/release 3.12.0 #1034
Merged
Sync/release 3.12.0 #1034
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This workflow is only a template for creating new workflows. We don't need to store that in this repository, it is fairly simple to spin up a new workflow when needed. References: BAR-296. Signed-off-by: Israel Barth <[email protected]>
`main.yml` workflow has been touched by the PR for BAR-301, and it had
issues about outdated actions.
This commit fixes the following issues reported by `actionlint`:
```text
.github/workflows/main.yml:19:15: the runner of "actions/checkout@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
|
19 | uses: actions/checkout@v2
| ^~~~~~~~~~~~~~~~~~~
.github/workflows/main.yml:21:15: the runner of "actions/setup-python@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
|
21 | uses: actions/setup-python@v2
| ^~~~~~~~~~~~~~~~~~~~~~~
.github/workflows/main.yml:41:15: the runner of "actions/checkout@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
|
41 | uses: actions/checkout@v2
| ^~~~~~~~~~~~~~~~~~~
.github/workflows/main.yml:43:15: the runner of "actions/setup-python@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
|
43 | uses: actions/setup-python@v2
| ^~~~~~~~~~~~~~~~~~~~~~~
.github/workflows/main.yml:47:15: the runner of "actions/cache@v2" action is too old to run on GitHub Actions. update the action's version to fix this issue [action]
|
47 | uses: actions/cache@v2
| ^~~~~~~~~~~~~~~~
```
References: BAR-301.
Signed-off-by: Israel Barth <[email protected]>
With the current docs for Barman 3.11 we are facing an issue: we use different files to build the HTML manual and the Linux man pages. As a consequence, we need to keep updating the same information across diffrent sources for different docs. This commit performs a POC for having a single source of truth when building the Barman documentation. We use reStructuredText markup language along with Sphinx to build any kind of documentation, i.e.: * HTML docs (complete) * PDF docs (do not include code API docs) * Linux man pages (include only command and configuration reference) The `index.rst` file defines the complete structure of the docs. Then, through the variable `exclude_from_builder` in `conf.py` we are able to define a blacklist of files for each different builder. To build the documentation, we only need to run either of the following commands, depending on the desired target: * `tox -e docs -- html`: for HTML * `tox -e docs -- latexpdf`: for PDF * `tox -e docs -- man`: for Linux man pages References: BAR-299. Signed-off-by: Israel Barth <[email protected]>
This commit fixes the following issues reported by `shellcheck`:
```text
In .github/foundation/security/scan-hooks.sh line 10:
source .venv/bin/activate
^----------------^ SC1091 (info): Not following: .venv/bin/activate: openBinaryFile: does not exist (No such file or directory)
In doc/build/build line 44:
cp -va *.html *.pdf "${DISTDIR}"
^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
In scripts/release.sh line 22:
BASE="$(dirname $(cd $(dirname "$0"); pwd))"
^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
^-------------^ SC2046 (warning): Quote this to prevent word splitting.
In scripts/release.sh line 27:
if ! git tag -s -m "Release ${VERSION}" release/${VERSION}
^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
if ! git tag -s -m "Release ${VERSION}" release/"${VERSION}"
In scripts/set-version.sh line 24:
BASE="$(dirname $(cd $(dirname "$0"); pwd))"
^------------------------^ SC2046 (warning): Quote this to prevent word splitting.
^-------------^ SC2046 (warning): Quote this to prevent word splitting.
In scripts/set-version.sh line 63:
if [ "$1" == false ]
^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.
In scripts/set-version.sh line 70:
echo $release_date
^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
echo "$release_date"
In scripts/set-version.sh line 76:
release_date=$(get_date $DATE)
^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
release_date=$(get_date "$DATE")
In scripts/set-version.sh line 111:
if branch=$(git symbolic-ref --short -q HEAD) && [ $branch = 'master' ]
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
if branch=$(git symbolic-ref --short -q HEAD) && [ "$branch" = 'master' ]
In scripts/set-version.sh line 138:
if [ "$DOCKER" == true ]
^-- SC3014 (warning): In POSIX sh, == in place of = is undefined.
In sphinx/generate_docs.sh line 20:
BASEDIR=$(cd ${0%/*}; pwd )
^--------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
BASEDIR=$(cd "${0%/*}" || exit; pwd )
In sphinx/generate_docs.sh line 78:
printf "${RED}${1}${RSET}
"
^-------------------^ SC2059 (info): Don't use variables in the printf format string. Use printf '..%s..' "$foo".
In sphinx/generate_docs.sh line 89:
--)
^-- SC2214 (warning): This case is not specified by getopts.
In sphinx/generate_docs.sh line 104:
BARMAN_DIR=$(cd "$BASEDIR/.."; pwd)
^--------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
Did you mean:
BARMAN_DIR=$(cd "$BASEDIR/.." || exit; pwd)
In sphinx/generate_docs.sh line 106:
BARMAN_DIR=$(cd "$1"; pwd)
^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
Did you mean:
BARMAN_DIR=$(cd "$1" || exit; pwd)
In sphinx/generate_docs.sh line 113:
cd "${BASEDIR}"
^-------------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
Did you mean:
cd "${BASEDIR}" || exit
In sphinx/generate_docs.sh line 120:
ls "${BASEDIR}"/docs/*.rst | grep -v 'index.rst$' | xargs -trI X rm -f X
^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
In sphinx/generate_docs.sh line 127:
make ${GEN_MODE}
^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
make "${GEN_MODE}"
```
References: BAR-315.
Signed-off-by: Israel Barth <[email protected]>
This commit fixes the following issue reported by `markdownlint`: ```text INSTALL.md:1 MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "## Barman INSTALL instructions"] ``` Besides that, it configures `super-linter` to skip Markdown files under `doc/` and `sphinx/` directories. Those directories contain the source code for the current Barman documentation, which is going to be replaced soon. References: BAR-315. Signed-off-by: Israel Barth <[email protected]>
This commit introduces the changes proposed by `isort`, so we please the issues reported by `super-linter`. References: BAR-315. Signed-off-by: Israel Barth <[email protected]>
This commit fixes the following issues reported by `flake8`: ```text barman/compression.py:66:18: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` barman/compression.py:424:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` barman/compression.py:429:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` tests/test_compressor.py:457:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` tests/test_compressor.py:459:20: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` tests/test_compressor.py:460:21: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()` ``` References: BAR-315. Signed-off-by: Israel Barth <[email protected]>
This commit runs `black` so we stop getting failures reported by `super-linter`. References: BAR-315. Signed-off-by: Israel Barth <[email protected]>
We used to have a link in the main toctree called "Barman API docs", which pointed to the content generated by `autodoc` extension based on the barman docstrings. Through this commit we replace that toctree entry with an entry for a new section called "Contributing to Barman". This sections intends to provide complete information about how a developer can contribute to the Barman project. The old "Barman API docs" was moved under this new section. References: BAR-320. Signed-off-by: Israel Barth <[email protected]>
This commit adds a subsection named "Writing and building documentation" under the section "Contributing to Barman". Here we explain the style guidelines to be followed when writing patches that include documentation. Also, it explains how to build the documentation locally through `tox`. References: BAR-324. Signed-off-by: Israel Barth <[email protected]> Co-authored-by: Martín Marqués <[email protected]>
This commit leverages from the Sphinx capability of maintaining a glossary. We add a new `glossary.rst` file where the glossary terms are maintained. For sake of example, we add a couple of terms in the glossary (Barman and PGDG). This is the basic workflow for using the glossary: * To add a new term to the glossary, add a new entry to the `glossary.rst` file. * To reference a term from the glossary, just use the syntax :term:`NAME_OF_TERM`, and Sphinx creates a link to the Glossary for you. References: BAR-326. Signed-off-by: Israel Barth <[email protected]>
This commit is: 1. Adding barman installation instructions and the specifics for RHEL-based distros installation. 2. Adding RHEL and RPM terms to glossary References: BAR-302 Signed-off-by: Andre <[email protected]>
This commit is: 1. Adding installation instructions for debian-based distributions. 2. Adding DEB term to glossary References: BAR-303 Signed-off-by: Andre <[email protected]> add to DEB to glossary Signed-off-by: Andre <[email protected]>
This commit applies a few small fixes to the glossary page: * Fixes the section link name, which was broken; * Adds punctuation to each item defined in the glossary. Signed-off-by: Israel Barth <[email protected]>
This commit is adding documention on retention policies, The docs were dividied in: 1. Overview - The content will be created in BAR-327 2. Minimum redundancy safety 3. Scope of retention policies 3.1 Backup Redundancy 3.2 Recovery Window 4. Use cases 4.1 Point-In-Time Recovery 4.2 Operational Efficiency and Space Management 4.3 Long-Term Archival 5. How retention policies are enforced 6. Configurations and Syntax 7. Retention policy for block-level incremental backups References: BAR-309 Signed-off-by: Andre <[email protected]>
This commit is adding a more robust introduction on retention policies. References: BAR-327 Signed-off-by: Andre <[email protected]>
This commit is: 1. Adding an introduction to the barman catalog section. 2. Adding "Purpose" section to catalog docs 3. Adding "Usage" section to catalog docs 4. Adding `list-backups` section documenting the CLI command with example 5. Adding `show-backup` section documenting the CLI command with example References: BAR-307 Signed-off-by: Andre <[email protected]>
Adds a quickstart section to the new documentation, providing simple setups for two common backup scenarios: backups with streaming replication and backups with rsync and WAL archiving. References: BAR-304 Signed-off-by: Gustavo William <[email protected]>
Add information about the `check` command to the new documentation. References: BAR-305 Signed-off-by: Barbara Leidens <[email protected]> Co-authored-by: Martín Marqués <[email protected]>
This commit is: 1. Adding an introduction about recovery 2. Adding Usage section and the following subsection 2.1 Remote Recovery 2.2 Tablespace remapping 2.3 PITR 2.4 Fetching WALs from Barman 2.5 Recovering Compressed backups 2.6 Recovering block-lvl increm backups 3. Adding Limitations of .partion WAL files section This commit is also adding the limitations on `--no-get-wal` usage and staging directory to combine backup in the beginning of the recovery process of block-level incremental backups. References: BAR-308 & BAR-206 & BAR-191 Signed-off-by: Andre <[email protected]>
This commit is ... * Adding an introduction on the configuration reference * Adding section `Usage` about the scope of the configuration files and a simple example of these files. In the same section, we have documented the Reserverd Words and Parameter Types. * Adding a section `Options` with all options that can be set in the configuration files. This section have the following subsections: * General * Backups * Cloud Backups * Hook Scripts * Write-Ahead Logs (WAL) * Recovery * Retention Policies. * Adding a section `Examples` with an example and the proper explanation of that. References: BAR-310 Signed-off-by: Andre <[email protected]>
The `barman-cloud-backup-keep` script has shortcuts for options `--status` and `--release`. These were missing in the `barman` CLI implementation, even though the error message suggested they existed. This aligns the shortcuts throughout the code so both implementations have the same arguments. References: BAR-356 Signed-off-by: Martín Marqués <[email protected]> Signed-off-by: Giulio Calacoci <[email protected]>
This commit adds documentation about the barman-cloud. It was divided as follows: * The main section is called `Barman for the cloud`. There is an introduction on how Barman can work with cloud backups. * Next section is called `barman-cli-cloud` that explains about the package, followed by an `Installation` section. * Lastly, the `Commands Reference` details each command available from the package. Each command was created in its own `.rst` file and they are referenced in the `conf.py` file. This is meant to be used to create MAN pages for each command. This commit also adds the `--no-partial` to documentation with the parameters description and alsoa brief explanation in about its use. References: BAR-312 & BAR-180 Signed-off-by: Andre <[email protected]>
This commit adds the license section of the documentation. References: BAR-314 Signed-off-by: Andre <[email protected]>
Adds an "Introduction" subsection to the "Concepts" section of the new documentation. It talks about general backup scenarios and advises on disaster recovery planning. References: BAR-338 Signed-off-by: Gustavo William <[email protected]>
Adds the "General backup concepts" subsection on the "Concepts" section. This section primarily focuses on explaining the fundamental backup concepts common across most relational databases and critical for understanding backups and backup management tools. References: BAR-339 Signed-off-by: Gustavo William <[email protected]>
This commit is adding the a reasoning note about why we get `Check 'empty incoming directory' failed for server '<SERVER>'` when WALs are present in the barman archive directory. It also is formatting the documentation because of max sentence length, some issues with the line breaks between bullet points and other house keeping task so the actualy rendered doc looks nice and clean. Closes GH-622 which exists in the barman repository (not this internal one). References: BAR-238 Signed-off-by: Andre <[email protected]> Co-authored-by: Martín Marqués <[email protected]>
This commit is adding details on the backup command docs. * An Overview introduction on the command and some caveats * Sections * Incremental Backups * Managing Bandwith Usage * Network Compression * Backup Compression * Compression algorithms * Compression workers * Compression level * Compression location * Immediate checkpoint * Streaming Backups - for pg_basebackup * Block-level incremental backups * Backup with Rsync through SSH - for rsync backups * File-level incremental backups * Cloud Snapshot Backups - for snapshot backups * Requirements and Configuration * Google Cloud Platform * Microsoft Azure * Amazon Web Services * Backup Process - snapshot backups process * Metadata - snapshots metadata * Hook scripts - using barman-cli-cloud as hook scripts with barman. * Concurrent backup of a standby * Added VLDB term to the glossary References: BAR-306 Signed-off-by: Andre <[email protected]>
Adds the "Postgres backup concepts and terminology" subsectino to the "Concepts" section. It explains some important backup concepts in the context of Postgres. References: BAR-340 Signed-off-by: Gustavo William <[email protected]>
Adds the "Architectural backup designs with Barman" section to the new documentation. References: BAR-330 Signed-off-by: Gustavo William <[email protected]>
This commit adds the release notes for BAR-264, which introduced lz4 compression support for WAL files. References: BAR-264 Signed-off-by: Gustavo William <[email protected]>
This commit adds a way to create tags when taking snapshot backups with cloud scripts. A new flag called tags can be used to tag snapshots for AWS when creating backups. Usage, e.g., tag1,val1 tag2,val2 References: BAR-417 Signed-off-by: Andre <[email protected]> Co-authored-by: Israel <[email protected]>
This commit makes a small modification on a unit test to also test for tags being applied to the snapshot when created. References: BAR-417 Signed-off-by: Andre <[email protected]> Co-authored-by: Israel <[email protected]>
Add info to doc of the barman-cloud-backup command and the build for the man page. References: BAR-417 Signed-off-by: Andre <[email protected]>
This commit adds a release note for the ticket bar-417. References: BAR-417 Signed-off-by: Andre <[email protected]>
We copied the original piece of documentation from the old documentation and added a few other commands which are useful for debugging barman issue. References: BAR-406 Signed-off-by: Martín Marqués <[email protected]> Signed-off-by: Israel Barth <[email protected]>
Introduced support for xz compression on WAL files. It can be enabled by specifying `xz` in the `compression` server parameter. WALs will be compressed when entering the Barman's WAL archive. For the cloud, it can ne enabled by specifying `--xz` when running `barman-cloud-wal-archive`. References: BAR-265 Signed-off-by: Gustavo William <[email protected]>
This commit adds some tests to the xz WAL compression implementation introduced in a previous commit. It also fixes very small tests that were broken. References: BAR-265 Signed-off-by: Gustavo William <[email protected]>
This commit adds the release notes for BAR-265, which introduced xz compression support for WAL files. References: BAR-265 Signed-off-by: Gustavo William <[email protected]>
The `sphinx` directory used to contain some very old scripts used to build the docs through Sphinx in the past. This commit removes that directory as it has not been used for a long time. The `doc` directory contains the Barman documentation and man pages that used to be used to build the docs up to Barman 3.11.1. That used pandoc to achieve that goal. We have been refactoring the docs of Barman, and the new version uses Sphinx to build fresh new docs. With that in mind, we have renamed the `doc` directory as `old_docs`, just so we can have a quick copy of any of the old docs pages in the soon future, if needed. In any case, that `old_docs` directory will be eventually removed from this repository. Other files were also updated in the repo so reflect these changes in `sphinx` and `doc` directories: * `linter.yml`: updated the regex to filter out `old_docs` instead of `doc` and `sphinx`. * `INSTALL.md`: updated to simply redirect the user to the docs page. * `README.rst`: temporarily updated renaming `doc` as `old_docs`. Also, removed the man pages on web, which no longer exist as a separate section. * `recovery.rst`: updated the links to runbooks, to reference the ones from `old_docs`. Once we move them to the new docs, we will need to update the links again. References: BAR-413. Signed-off-by: Israel Barth <[email protected]>
Now that the new docs are more stable we are renaming the `new_docs` directory as `docs`. As a consequence of that change, we had to reflect it throughout the repo: * `.gitignore`: to ignore build files from `docs` instead of `new_docs`. * `MANIFEST.in`: to include files from `docs` instead of `new_docs`. * `README.rst`: to explain about `docs` instead of `old_docs`. * `docs/README.md`: to explain about `docs` instead of `new_docs`. * `docs/build/build`: to build from `docs` instead of `new_docs`. * `docs/conf.py`: to build from `docs` instead of `new_docs`. * `docs/contributor_guide/writing_docs.rst`: to explain using the `docs` path instead of `new_docs`. * `setup.py`: to include files from `docs` instead of `new_docs`. * `tox.ini`: to build from `docs` instead of `new_docs`. References: BAR-413. Signed-off-by: Israel Barth <[email protected]>
The release scripts will no longer be maintained in the public repository of Barman, but rather in an internal repository which contains all the logic for releasing a new version. References: BAR-454. Signed-off-by: Israel Barth <[email protected]>
Add the "WAL archiving" section to the new docs. For now it only contains content about WAL compression and synchronous WAL streaming. References: BAR-452 Signed-off-by: Gustavo William <[email protected]>
This commit modifies the `check_wal_streaming` method in the Server class and the get_wal_conninfo in the ServerConfig class. Previously, when `conninfo` pointed to a standby server, Barman would incorrectly use this connection to check the replication status, resulting in the following error: `replication slot (WAL streaming): FAILED (replication slot 'barman' doesn't exist. Please execute 'barman receive-wal --create-slot pg17')` The changes: 1. get_wal_conninfo() When having wal-specific conninfos, wal_conninfo is not directly dependent of wal_streaming_conninfo anymore. If wal_streaming_conninfo is present and in the absence of wal_conninfo, wal_conninfo will fall back to wal_streaming_conninfo. If wal_streaming_conninfo is not present, wal_conninfo can still be set and used, and if it is not present, it will fall back to conninfo. 2. check_wal_streaming If wal_conninfo is different than conninfo, it means we have wal-specific configuration and it must be use to check replication status. As this config is not dependent of wal_streaming_conninfo, it can be used in the conditional. References: BAR-409 Signed-off-by: Andre <[email protected]>
This commit fixes all broken tests that happened because of the change in the implementation. The main change is that now when having wal_conninfo set without the presence of wal_streaming_conninfo, conninfo is overriden. References: BAR-409 Signed-off-by: Andre <[email protected]>
This commit fixes the documentation where there are mentions to wal-specific configuration. Now wal_conninfo is not directly dependent of having the wal_streaming_conninfo configured, it can be independently set. Also, it must be set if conninfo is pointing to a standby References: BAR-409 Signed-off-by: Andre <[email protected]>
This commit adds a release note for the ticket BAR-409 which fixes a bug reported in the GH issue [#1024](#1024). References: BAR-409 Signed-off-by: Andre <[email protected]>
A new `--keep-compression` option has been added to both `barman-wal-restore` and `get-wal`. This option controls whether compressed WAL files should be decompressed on the Barman server before being fetched. When specified with `get-wal`, default decompression is skipped, and the output is the WAL file content in its original state. When specified with `barman-wal-restore`, the WAL file is fetched as-is and, if compressed, decompressed on the client side. References: BAR-435 Signed-off-by: Gustavo William <[email protected]>
This commit fixes some tests that were broken by the `--keep-compression` flag introduced in a preivous commmit. References: BAR-435 Signed-off-by: Gustavo William <[email protected]>
This commit adds tests to the new `--keep-compression` flag functionality introduced in a preivous commit. References: BAR-435 Signed-off-by: Gustavo William <[email protected]>
Adds a release note the BAR-435, which introduced the new `--keep-compression` flag in `barman-wal-restore` and `get-wal`. References: BAR-435 Signed-off-by: Gustavo William <[email protected]>
This commit updates the documentation to describe the new `--keep-compression` option added in previous commits. It also insert notes about the deprecation of some compression options on `get-wal` and `barman-wal-restore`. References: BAR-435 Signed-off-by: Gustavo William <[email protected]>
Replace existing images that show the architectural design for multiple suggested architectures. Change the section for Hybrid architecture to one where users can use rsync for backups to have incremental backups and parallel copy, together with WAL streaming for aditional performance of WAL copy and possible RPO zero. Add a fallover section for WAL archiving. References: BAR-367 Signed-off-by: Martín Marqués <[email protected]> Co-authored-by: Giulio Calacoci <[email protected]>
This commit adds a release note for the jira ticket BAR-337. References: BAR-337 Signed-off-by: Andre <[email protected]> Co-authored-by: Israel <[email protected]>
Signed-off-by: Andre <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.