Skip to content

Release Procedures

Jump to bottom
Zach Burnett edited this page Feb 9, 2024 · 23 revisions

This guide assumes that you have cloned romancal, and added a remote named upstream pointing to the central repository:

git remote add upstream https://github.com/spacetelescope/romancal.git

Before starting the release process, ensure that the latest Jenkins regression test run for the main / master branch has passed successfully.

1. Update metadata in the main / master branch

  1. Edit the change log CHANGES.rst; in the entry corresponding to your intended release (a.b.c (unreleased)), change unreleased to the current date in YYYY-MM-DD format. Also add a new change log entry above, with the format a.b.d (unreleased).

    +0.16.1 (unreleased)
+===================
+
+
+
-0.16.0 (unreleased)
+0.16.0 (2020-05-04)
===================

  2. Update the Software vs DMS build version map section of README.md to show the latest release version, date, and the CRDS_CONTEXT this release is being currently tested with. This can be obtained from Nadia Dencheva or David Kauffman, and is usually the current operational context. For now, use the test server (https://roman-crds-test.stsci.edu):

export CRDS_SERVER_URL=https://roman-crds-test.stsci.edu
crds list --operational-context

  1. Update the release version and date in CITATION.cff, and add any new contributors.

  2. Make sure dependencies in pyproject.toml are updated. In particular, ensure that crds, stcal, roman_datamodels, and rad include their newest tested versions, and exclude any older versions with incompatible API.

Important

If stcal and stdatamodels have new updates that change the API used by romancal, make sure to use the relevant reference files on Artifactory when testing.

Caution

PyPI will reject any submission with Git dependencies in its metadata (i.e. git+https://github.com/spacetelescope/rad.git@main). Dependencies must be defined against a released version available on PyPI (i.e. rad>=0.17.0).

  1. Update the requirements-sdp.txt file, as per its commented instructions.

  2. Commit changes to your fork:

git add CHANGES.rst README.md 
git commit -m "Release 0.3.0 document updates."
git push <UserName> ReleaseDocs
  1. Create a Pull Request from the ReleaseDocs branch on your fork to spacetelescope/romancal.

If this release is a cherry-picked patch to an release branch, make the changes on the master branch first, then cherry-pick and modify as needed over to the release branch. Note that cherry-picked release branches may also need to change JenkinsfileRT, so artifact publication happens when then regression tests pass.

2. Create or update the release branch

If you're making a major or minor version release, then the release branch will not yet exist. If you're releasing a patch version, then a release branch will already exist. Select one of the next two sections accordingly.

New major / minor version

  1. Fetch and checkout the upstream main / master:

    git fetch --all --tags
git checkout -t upstream/master

  2. Inspect the log to ensure that no commits have snuck in since your change log update:

    git log

  3. Create a new release branch. The name of the release branch should share the major and minor version of your release version, but the patch version should be x. For example, when releasing 1.8.0, name the branch release/1.8.x.

git checkout -b release/a.b.x
  1. Push the branch to the upstream remote: 
    git push -u upstream HEAD

Patch release of an existing major / minor version

In the case of a patch release, the release branch will already exist.

  1. Checkout and freshen release branch (this assumes that your local branch is already tracking upstream/release/a.b.x):

    git checkout release/a.b.x
git pull

  2. Cherry-pick relevant commits from main / master that should be included in the patch release (including the new changelog commit):

    git cherry-pick ...

Tip

Alternatively, you can pull all changes from master into the release branch:

git fetch upstream
git pull upstream master
  1. Push updates to the upstream remote: 
    git push upstream HEAD

Important

If the release branch has diverged from master, you will have to run the regression tests on the release branch and have them pass to be able to do the automated release to DMS (see the DMS release section below for details). Additionally, you should tag main / master with a development tag (i.e. 1.8.9dev) so the locally-installed version shows correctly.

Tips for git cherry-pick

Say a bug needs to be fixed in 0.16.0, and is also present in master.

  1. Fix the bug on master through a PR. Merge it and get the hash of the commit. As an example, let's say the commit hash is f41120e.

  2. Cherry pick all commits that need to be included in the patch release.

    git cherry-pick -x f41120e

  3. If there are conflicts, resolve them before you continue with other commits. In general, start with the oldest commit so that conflicts are minimized.

Warning

Sometimes a commit is a merge commit with more than one parent. The above command will fail with an error message:

error: Commit f41120e is a merge but no -m option was given.
fatal: cherry-pick failed

This discussion might be helpful.

In this case, look at the parents of the commit and choose which one is the ancestor:

git cherry-pick -x -m 1 f41120e

(this tells git to use the first parent)

  1. Use git show to verify which changes are going into the release branch.

3. Review the release branch's latest automated test run

The creation or update of the release branch should trigger Github Actions CI; you can follow the tests here. All tests should pass; if not, fix the issue and push the changes to HEAD.

Important

Make sure the CI runs the tests with the latest public releases of astropy and asdf.

4. Tag the desired release commit with the release version

At this point, you should have the release branch checked out and ready to tag. Tag the release branch with the new release and push the tag to the forked repository.

  1. Create an annotated tag with a name that matches your intended release. It is useful to include the DMS build along with the build of the calibration pipeline:

    git tag -a a.b.c -m "Romancal DMS Build 23Q4_B11"

  2. Push the new tag to the upstream remote:

    git push upstream a.b.c

5. Maintain the stable branch (if necessary)

The stable branch points to the latest official release of romancal. If the current release has become the latest, then the next step is to rewrite the stable branch to point our new tag.

git checkout stable
git reset --hard a.b.c
git push upstream stable --force

6. Publish to test.pypi.org

This step is optional, but it is a good smoke check, especially if there have been any changes to the install procedure (i.e. setup.py, setup.cfg, pyproject.toml) since the last release, or any changes to README.md or anything else that gets included in long_description in setup.cfg.

This step requires permissions to write to test.pypi.org for romancal.

Before proceeding, you will need the twine and build packages:

pip install twine build

  1. Checkout the release tag, clean the directory, and make sure umask and permissions are set correctly:

    git checkout a.b.c
git clean -xdf
umask 0022
chmod -R a+Xr .

  2. Check the package setup and create the package sdist and wheel:

    python -m build .

  3. Upload the package to PyPi's test server (you need an account and be added as maintainer):

    twine check --strict dist/*
twine upload --repository testpypi dist/*

  4. Check that it looks good on the test server. Make sure it installs without errors in a new environment:

    pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple romancal[test]==a.b.c
pip install pytest-xdist

  5. Run the tests on the installed package. Change to a directory that does not contain the romancal source and confirm that tests pass on the installed package.

    pushd /
pytest -n auto --pyargs romancal
popd

  6. If the package looks good on test.pypi.org and installs OK and the tests pass, then proceed.

8. Make a GitHub release and publish the package to PyPI

Create a new GitHub release

  1. Visit the spacetelescope/romancal repository's releases page.

  2. Click Draft a new release.

  3. Select the existing tag that you just created and pushed, and title the release Romancal Build x.y rcN, where N is the current release candidate for DMS (we can always remove the rcN part from the title later if this turns out to be the final release).

  4. Publish the release.

Validate that the package did publish to PyPI

Publishing the GitHub release should trigger an automated workflow that should build the wheel and source distribution and publish the package to PyPI.

After this workflow completes, you can confirm that the new release appears on PyPI: https://pypi.org/project/romancal/#history

Additionally, you can test installing the new version with pip:

pip install romancal==a.b.c

9. Prepare main / master for further development

  1. Make sure CHANGES.rst has a section for the next release.

  2. If the commit with the release tag is not on main / master, you should tag the next commit on main / master with a development tag (i.e. if we just released version 1.2.1 on release/1.2.x, the development tag should be 1.2.2.dev on the main / master commit directly after the divergence). This allows setuptools-scm to show the correct version when installing the package locally (pip install .).

    git fetch upstream
git checkout upstream/master
git tag -a a.b.d.dev -m "development tag after divergence"
git push upstream a.b.d.dev

10. Deliver the release to DMS

TBD

Clone this wiki locally