Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve documentation for "how to release" #1359

Merged
merged 11 commits into from
Nov 25, 2024
Merged

Improve documentation for "how to release" #1359

Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9970015
initial update
kevinjqliu Nov 21, 2024
e6635fb
edits
kevinjqliu Nov 21, 2024
8cc9030
add gpg instructions
kevinjqliu Nov 21, 2024
35d8301
verify artifacts
kevinjqliu Nov 21, 2024
ab7a6a0
add twine not
kevinjqliu Nov 21, 2024
d59b950
grammar
kevinjqliu Nov 21, 2024
91cad4d
edits
kevinjqliu Nov 21, 2024
2e16cdd
remove old artifacts
kevinjqliu Nov 21, 2024
3e1016d
update doc workflow action
kevinjqliu Nov 22, 2024
88aeeca
and name
kevinjqliu Nov 22, 2024
647ab0e
add docs on patch vs major/minor release
kevinjqliu Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 143 additions & 61 deletions mkdocs/docs/how-to-release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,31 @@
- under the License.
-->

# How to release
# How to Release

The guide to release PyIceberg.
This guide outlines the process for releasing PyIceberg in accordance with the [Apache Release Process](https://infra.apache.org/release-publishing.html). The steps include:

The first step is to publish a release candidate (RC) and publish it to the public for testing and validation. Once the vote has passed on the RC, the RC turns into the new release.
1. Preparing for a release
2. Publishing a Release Candidate (RC)
3. Community Voting and Validation
4. Publishing the Final Release (if the vote passes)
5. Post-Release Step

## Preparing for a release
## Requirements

Before running the release candidate, we want to remove any APIs that were marked for removal under the @deprecated tag for this release.
* A GPG key must be registered and published in the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS). Follow [the instructions for setting up a GPG key and uploading it to the KEYS file](#set-up-gpg-key-and-upload-to-apache-iceberg-keys-file).
* SVN Access
* Permission to upload artifacts to the [Apache development distribution](https://dist.apache.org/repos/dist/dev/iceberg/) (requires Apache Commmitter access).
* Permission to upload artifacts to the [Apache release distribution](https://dist.apache.org/repos/dist/release/iceberg/) (requires Apache PMC access).
* PyPI Access
* The `twine` package must be installed for uploading releases to PyPi.
* A PyPI account with publishing permissions for the [pyiceberg project](https://pypi.org/project/pyiceberg/).

## Preparing for a Release

### Remove Deprecated APIs

Before running the release candidate, we want to remove any APIs that were marked for removal under the `@deprecated` tag for this release. See [#1269](https://github.com/apache/iceberg-python/pull/1269).

For example, the API with the following deprecation tag should be removed when preparing for the 0.2.0 release.

Expand All @@ -48,23 +64,22 @@ deprecation_message(
)
```

## Running a release candidate
### Update Library Version

Make sure that the version is correct in `pyproject.toml` and `pyiceberg/__init__.py`. Correct means that it reflects the version that you want to release.
Update the version in `pyproject.toml` and `pyiceberg/__init__.py` to match the release version. See [#1276](https://github.com/apache/iceberg-python/pull/1276).

### Setting the tag
## Publishing a Release Candidate (RC)

Make sure that you're on the right branch, and the latest branch:
### Create Tag

For a Major/Minor release, make sure that you're on `main`, for patch versions the branch corresponding to the version that you want to patch, i.e. `pyiceberg-0.6.x`.
Ensure you are on the correct branch:

```bash
git checkout <branch>
git fetch --all
git reset --hard apache/<branch>
```
* For a major/minor release, use the `main` branch
* For a patch release, use the branch corresponding to the patch version, i.e. `pyiceberg-0.6.x`.

Create a signed tag:

Set the tag on the last commit:
Replace `VERSION` and `RC` with the appropriate values for the release.

```bash
export RC=rc1
Expand All @@ -74,48 +89,49 @@ export VERSION_BRANCH=${VERSION_WITHOUT_RC//./-}
export GIT_TAG=pyiceberg-${VERSION}

git tag -s ${GIT_TAG} -m "PyIceberg ${VERSION}"
git push apache ${GIT_TAG}

export GIT_TAG_REF=$(git show-ref ${GIT_TAG})
export GIT_TAG_HASH=${GIT_TAG_REF:0:40}
export LAST_COMMIT_ID=$(git rev-list ${GIT_TAG} 2> /dev/null | head -n 1)
git push [email protected]:apache/iceberg-python.git ${GIT_TAG}
```

The `-s` option will sign the commit. If you don't have a key yet, you can find the instructions [here](http://www.apache.org/dev/openpgp.html#key-gen-generate-key). To install gpg on a M1 based Mac, a couple of additional steps are required: <https://gist.github.com/phortuin/cf24b1cca3258720c71ad42977e1ba57>.
If you have not published your GPG key in [KEYS](https://downloads.apache.org/iceberg/KEYS) yet, you must publish it before sending the vote email by doing:

```bash
svn co https://dist.apache.org/repos/dist/release/iceberg icebergsvn
cd icebergsvn
echo "" >> KEYS # append a newline
gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures
gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block
svn commit -m "add key for <YOUR NAME HERE>"
```
### Publish Release Candidate (RC)

### Upload to Apache SVN
#### Upload to Apache Dev SVN

Both the source distribution (`sdist`) and the binary distributions (`wheels`) need to be published for the RC. The wheels are convenient to avoid having people to install compilers locally. The downside is that each architecture requires its own wheel. [use `cibuildwheel`](https://github.com/pypa/cibuildwheel) runs in Github actions to create a wheel for each of the architectures.
##### Create Artifacts for SVN

Before committing the files to the Apache SVN artifact distribution SVN hashes need to be generated, and those need to be signed with gpg to make sure that they are authentic.
Run the [`Python release` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml).

Go to [Github Actions and run the `Python release` action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml). **Set the version to main, since we cannot modify the source**.
* Tag: Use the newly created tag.
* Version: Set the `version` to `main`, as the source cannot be modified.

![Github Actions Run Workflow for SVN Upload](assets/images/ghactions-run-workflow-svn-upload.png)

Download the zip, and sign the files:
This action will generate:

* Source distribution (`sdist`)
* Binary distributions (`wheels`) for each architectures. These are created using [`cibuildwheel`](https://github.com/pypa/cibuildwheel)

##### Download Artifacts, Sign, and Generate Checksums

Download the ZIP file containing the artifacts from the GitHub Actions run and unzip it.

Navigate to the release directory. Sign the files and generate checksums:

* `.asc` files: GPG-signed versions of each artifact to ensure authenticity.
* `.sha512` files: SHA-512 checksums for verifying file integrity.

```bash
cd release-main/

for name in $(ls pyiceberg-*.whl pyiceberg-*.tar.gz)
do
gpg --yes --armor --local-user [email protected] --output "${name}.asc" --detach-sig "${name}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol, thanks! :D

gpg --yes --armor --output "${name}.asc" --detach-sig "${name}"
shasum -a 512 "${name}" > "${name}.sha512"
done
```

Now we can upload the files from the same directory:
##### Upload Artifacts to Apache Dev SVN

Now, upload the files from the same directory:

```bash
export SVN_TMP_DIR=/tmp/iceberg-${VERSION_BRANCH}/
Expand All @@ -128,21 +144,51 @@ svn add $SVN_TMP_DIR_VERSIONED
svn ci -m "PyIceberg ${VERSION}" ${SVN_TMP_DIR_VERSIONED}
```

### Upload to PyPi
Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/dev/iceberg](https://dist.apache.org/repos/dist/dev/iceberg/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a section for deleting old RCs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

svn delete https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-<OLD_RC_VERSION> -m "Remove old RC artifacts"

I think this is the right command, will test during 0.8.1 release


#### Upload to PyPi

##### Create Artifacts for PyPi

Run the [`Python release` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-release.yml).

Go to Github Actions and run the `Python release` action again. This time, set the **version** of the release candidate as the input: e.g. `0.7.0rc1`. Download the zip and unzip it locally.
* Tag: Use the newly created tag.
* Version: Set the `version` to release candidate, e.g. `0.7.0rc1`.

![Github Actions Run Workflow for PyPi Upload](assets/images/ghactions-run-workflow-pypi-upload.png)

Next step is to upload them to pypi. Please keep in mind that this **won't** bump the version for everyone that hasn't pinned their version, since it is set to an RC [pre-release and those are ignored](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#pre-release-versioning).
##### Download Artifacts

Download the zip file from the Github Action run and unzip locally.

##### Upload Artifacts to PyPi

Upload release candidate to PyPi. This **won't** bump the version for everyone that hasn't pinned their version, since it is set to an RC [pre-release and those are ignored](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#pre-release-versioning).

<!-- prettier-ignore-start -->

!!! note
`twine` might require an PyPi API token.

<!-- prettier-ignore-end -->

```bash
twine upload release-0.7.0rc1/*
twine upload release-${VERSION}/*
```

Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history).

## Vote

### Generate Vote Email

Final step is to generate the email to the dev mail list:

```bash
export GIT_TAG_REF=$(git show-ref ${GIT_TAG})
export GIT_TAG_HASH=${GIT_TAG_REF:0:40}
export LAST_COMMIT_ID=$(git rev-list ${GIT_TAG} 2> /dev/null | head -n 1)

cat << EOF > release-announcement-email.txt
To: [email protected]
Subject: [VOTE] Release Apache PyIceberg $VERSION
Expand Down Expand Up @@ -185,12 +231,19 @@ Please vote in the next 72 hours.
[ ] +0
[ ] -1 Do not release this because...
EOF

cat release-announcement-email.txt
```

## Vote has passed
### Send Vote Email

Verify the content of `release-announcement-email.txt` and send it to `[email protected]` with the corresponding subject line.

## Vote has failed

If there are concerns with the RC, address the issues and generate another RC.

## Publish the Final Release (Vote has passed)

A minimum of 3 binding +1 votes is required to pass an RC.
Once the vote has been passed, you can close the vote thread by concluding it:

```text
Expand All @@ -205,36 +258,46 @@ The release candidate has been accepted as PyIceberg <VERSION>. Thanks everyone,
Kind regards,
```

### Copy the artifacts to the release dist
### Upload the accepted RC to Apache Release SVN
<!-- prettier-ignore-start -->

```bash
export RC=rc2
export VERSION=0.7.0${RC}
export VERSION_WITHOUT_RC=${VERSION/rc?/}
!!! note
Only a PMC member has the permission to upload an artifact to the SVN release dist.

<!-- prettier-ignore-end -->

```bash
export SVN_DEV_DIR_VERSIONED="https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-${VERSION}"
export SVN_RELEASE_DIR_VERSIONED="https://dist.apache.org/repos/dist/release/iceberg/pyiceberg-${VERSION_WITHOUT_RC}"

svn mv ${SVN_DEV_DIR_VERSIONED} ${SVN_RELEASE_DIR_VERSIONED} -m "PyIceberg: Add release ${VERSION_WITHOUT_RC}"
```

Verify the artifact is uploaded to [https://dist.apache.org/repos/dist/release/iceberg](https://dist.apache.org/repos/dist/release/iceberg/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a step here to only host the lastest release on the dist/release/. I recently cleaned it up, but it would be good to make this part of the release itself :)


### Upload the accepted release to PyPi

The latest version can be pushed to PyPi. Check out the Apache SVN and make sure to publish the right version with `twine`:

<!-- prettier-ignore-start -->

!!! note
Only a PMC member has the permission to upload an artifact to the SVN release dist.
`twine` might require an PyPi API token.

<!-- prettier-ignore-end -->

### Upload the accepted release to PyPi

The latest version can be pushed to PyPi. Check out the Apache SVN and make sure to publish the right version with `twine`:

```bash
svn checkout https://dist.apache.org/repos/dist/release/iceberg /tmp/iceberg-dist-release/
cd /tmp/iceberg-dist-release/pyiceberg-${VERSION_WITHOUT_RC}
twine upload pyiceberg-*.whl pyiceberg-*.tar.gz
```

Verify the artifact is uploaded to [PyPi](https://pypi.org/project/pyiceberg/#history).

## Post Release

### Send out Release Announcement Email

Send out an announcement on the dev mail list:

```text
Expand All @@ -253,19 +316,19 @@ This Python release can be downloaded from: https://pypi.org/project/pyiceberg/<
Thanks to everyone for contributing!
```

## Release the docs
### Release the docs

A committer triggers the [`Python Docs` Github Actions](https://github.com/apache/iceberg-python/actions/workflows/python-ci-docs.yml) through the UI by selecting the branch that just has been released. This will publish the new docs.
Run the [`Python Docs` Github Action](https://github.com/apache/iceberg-python/actions/workflows/python-ci-docs.yml).

## Update the Github template
### Update the Github template

Make sure to create a PR to update the [GitHub issues template](https://github.com/apache/iceberg-python/blob/main/.github/ISSUE_TEMPLATE/iceberg_bug_report.yml) with the latest version.

## Update the integration tests
### Update the integration tests

Ensure to update the `PYICEBERG_VERSION` in the [Dockerfile](https://github.com/apache/iceberg-python/blob/main/dev/Dockerfile).

## Create a Github Release Note
### Create a Github Release Note

Create a [new Release Note](https://github.com/apache/iceberg-python/releases/new) on the iceberg-python Github repository.

Expand All @@ -278,3 +341,22 @@ Then, select the previous release version as the **Previous tag** to use the dif
**Generate release notes**.

**Set as the latest release** and **Publish**.

## Misc

### Set up GPG key and Upload to Apache Iceberg KEYS file

To set up GPG key locally, see the instructions [here](http://www.apache.org/dev/openpgp.html#key-gen-generate-key).

To install gpg on a M1 based Mac, a couple of additional steps are required: <https://gist.github.com/phortuin/cf24b1cca3258720c71ad42977e1ba57>.

Then, published GPG key to the [Apache Iceberg KEYS file](https://downloads.apache.org/iceberg/KEYS):

```bash
svn co https://dist.apache.org/repos/dist/release/iceberg icebergsvn
cd icebergsvn
echo "" >> KEYS # append a newline
gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures
gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block
svn commit -m "add key for <YOUR NAME HERE>"
```