Update release process #410

forsyth2 · 2023-03-03T21:18:51Z

Update release process. Resolves #396.

forsyth2 · 2023-03-03T21:21:12Z

@xylar @tomvothecoder Is this sufficient to address #396 or are further changes needed? Does build_workflow.yml also need to be updated, or only release_workflow.yml?

$ grep -in conda release_workflow.yml 
37:      # Using pip for Sphinx dependencies because it takes too long to reproduce a conda environment (~10 secs vs. 3-4 mins

$ grep -in conda build_workflow.yml 
54:      - name: Cache Conda
59:          path: ~/conda_pkgs_dir
60:          key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
61:            hashFiles('conda/dev.yml') }}
63:      - name: Build Conda Environment
64:        uses: conda-incubator/setup-miniconda@v2
67:          environment-file: conda/dev.yml
69:          auto-update-conda: true
74:        name: Show Conda Environment Info
76:          conda config --set anaconda_upload no
77:          conda info
78:          conda list
117:      # Using pip for Sphinx dependencies because it takes too long to reproduce a conda environment (~10 secs vs. 3-4 mins)

forsyth2 · 2023-03-03T21:21:35Z

docs/source/dev_guide/release.rst

@@ -81,7 +81,7 @@ Releasing Anaconda
 ------------------

 1. Be sure to have already completed :ref:`Releasing On GitHub <github-release>`. This triggers the CI/CD workflow that handles Anaconda releases.
-2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows <https://github.com/E3SM-Project/zppy/actions>`_.
+2. Wait for a bot PR to come up automatically on conda-forge after the GitHub release.
 3. Check the https://anaconda.org/e3sm/zppy page to view the newly updated package.


Is there a new page to point to?

The conda-forge bot PR can come as quickly as an hour later or as late as a day later (or in rare cases, longer). It depends on your specific needs; if you want a quicker release on conda-forge, you could simply bump up the version too (a new PR with version change and the new tar sha256). Also somewhere else down this page, it says something about anaconda.org/e3sm/zppy: Do you still intend to release on the e3sm channel? The conda-forge ones will be available at anaconda.org/conda-forge/zppy

(For "backward" compatibility — if you will — you could also simply copy and paste the built artifacts from the conda-forge channel to the e3sm channel; also, note that there is a "staging" process in conda-forge, where the bots upload the artifacts an intermediate channel called cf-staging before moving the artifacts to the main conda-forge channel; the whole process used to take on the order of an hour, but now significantly shorter)

Here's a link on the auto PR stuff: https://conda-forge.org/docs/maintainer/updating_pkgs.html#how-does-regro-cf-autotick-bot-create-automatic-version-updates

The conda-forge bot PR can come as quickly as an hour later or as late as a day later

The conda-forge ones will be available at anaconda.org/conda-forge/zppy

The most recent commit addresses these.

if you want a quicker release

I'm not sure how quickly @xylar will need releases. Usually when we're testing E3SM Unified, we want to get release candidates integrated pretty quickly so we can start testing them. I suppose a day is a reasonable amount of time?

Do you still intend to release on the e3sm channel?

I don't think so? I think one channel is probably going to be sufficient.

mahf708 · 2023-03-04T00:26:51Z

docs/source/dev_guide/release.rst

@@ -81,7 +81,7 @@ Releasing Anaconda
 ------------------

 1. Be sure to have already completed :ref:`Releasing On GitHub <github-release>`. This triggers the CI/CD workflow that handles Anaconda releases.
-2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows <https://github.com/E3SM-Project/zppy/actions>`_.
+2. Wait for a bot PR to come up automatically on conda-forge after the GitHub release.
 3. Check the https://anaconda.org/e3sm/zppy page to view the newly updated package.

   * Release candidates are assigned the ``e3sm_dev`` label


Do you still want to do this on the conda-forge side btw? There's a similar mechanism, but the label will likely have to be longer

Here's a link on this (if you're still interested in keeping up with this): https://conda-forge.org/docs/maintainer/knowledge_base.html#pre-release-builds

Do you still want to do this on the conda-forge side

I think this is probably a good idea, so we can differentiate between release candidates and actual releases. I'm sure that will be helpful to @xylar when assembling Unified as well.

Yes, we use the e3sm_dev label on conda-forge just like on the e3sm channel. I absolutely have to have a release candidate for testing E3SM-Unified because the conda-forge/label/e3sm_dev channel takes priority over the conda-forge channel, meaning that it won't find a release for any package that has ever had a release candidate on the e3sm_dev label. That's kind of inconvenient but I don't think there's a good way around it.

Here's a link on this (if you're still interested in keeping up with this): https://conda-forge.org/docs/maintainer/knowledge_base.html#pre-release-builds

I'm not quite sure I follow. Does the bot PR not work -- e.g., we have to make a manual PR with the dev or rc label?

You have to make the rc tag locally (I wouldn't make a release because an rc is not a release). Then, you make a PR to conda-forge from your fork of the feedstock. You need to start from the current dev branch and update the version number and the sha256 sum manually. You need to set the build number back to 0 if needed. You need to make the dev branch the target of the PR. That way, the package build on conda-forge will end up with the e3sm_dev label.

We can also work together on this for the first few times.

You have to make the rc tag locally

You can publish the tag without a release, or do you mean "here" by locally?

I think continuing with everything the same here is fine https://github.com/E3SM-Project/zppy/releases/tag/v2.2.0rc4 (i.e., publishing these tags as prereleases). The only minor difference is at conda-forge. Just let us know when there's a release/pre-release and we can work together. It's best learned as it happens

Yes, locally means here.

I really wouldn't do a release page for release candidates. It's confusing when folks are looking for actual releases.

I really wouldn't do a release page for release candidates. It's confusing when folks are looking for actual releases.

Yeah, that's a good point. A tag (with no release) is more than enough :)

mahf708 · 2023-03-04T00:40:47Z

docs/source/dev_guide/release.rst

@@ -81,7 +81,7 @@ Releasing Anaconda
 ------------------

 1. Be sure to have already completed :ref:`Releasing On GitHub <github-release>`. This triggers the CI/CD workflow that handles Anaconda releases.
-2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows <https://github.com/E3SM-Project/zppy/actions>`_.
+2. Wait for a bot PR to come up automatically on conda-forge after the GitHub release.


I would also note two important things here:

One need to still merge the PR on the conda-forge side (we could set up an automerge bot as well)

It is important to "rerender" the feedstock before merging (the bots usually do this automatically, but in case a person opens the PR, they should also rerender). Docs: https://conda-forge.org/docs/maintainer/updating_pkgs.html#rerendering-feedstocks

xylar · 2023-03-08T20:11:45Z

@xylar @tomvothecoder Is this sufficient to address #396 or are further changes needed? Does build_workflow.yml also need to be updated, or only release_workflow.yml?

I will need to defer to @tomvothecoder on this but I believe only elease_workflow.yml should need to be modified.

tomvothecoder · 2023-03-08T20:51:45Z

@xylar @tomvothecoder Is this sufficient to address #396 or are further changes needed? Does build_workflow.yml also need to be updated, or only release_workflow.yml?

I will need to defer to @tomvothecoder on this but I believe only elease_workflow.yml should need to be modified.

Only release_workflow.yml needs to modified since that's where the conda package is built and uploaded after a GH release.

forsyth2

@mahf708 @xylar Ok, I think I addressed all your comments. Let me know if this looks good to merge.

Just let us know when there's a release/pre-release and we can work together. It's best learned as it happens

That seems like a good idea, thanks! The next release candidate deadline for E3SM-Unified is June 1st, so I imagine around the last week of May the next release process will begin (with the production release occurring roughly a month later).

forsyth2 · 2023-03-22T00:51:12Z

docs/source/dev_guide/release.rst

+Releasing on GitHub: release candidates
+-------------------
+
+1. Create a tag for the release candidate at https://github.com/E3SM-Project/zppy/tags.


I don't see a way to create a tag without creating a release. https://stackoverflow.com/questions/64561146/how-do-i-create-a-tag-in-github-without-creating-a-release suggests creating a release and then deleting it. Is it something to do on the command-line side (https://git-scm.com/book/en/v2/Git-Basics-Tagging)?

I think using the prerelease feature may be good. The bots won't pick it up as long as it is named with rc, beta, alpha, etc.

Alternatively,

$ git tag -a v0.0.1 -m "test tag" $ git push origin v0.0.1

https://github.com/mahf708/test-repo-tags

We currently use the pre-release feature for release-candidates (https://github.com/E3SM-Project/zppy/releases/tag/v2.2.0rc4 for instance). I do like that as it allows me to write up a summary of what's changed between RCs. That said, Xylar does have a point about it confusing people; it's annoying to sort through multiple RCs to get to actual releases.

As a wild example, you can see how tensorflow does things: https://github.com/tensorflow/tensorflow/releases/tag/v2.12.0-rc1

Ah, I see. An empty prerelease page may help, but I am not sure. I will defer to Xylar :)

That structure is very similar to how I organize the zstash and zppy pre-release and release notes already, albeit with way more detail.

forsyth2 · 2023-03-22T00:51:55Z

docs/source/dev_guide/release.rst

+   * Start from the current dev branch and update the version number and the sha256 sum manually.
+   * Set the build number back to 0 if needed.
+   * Make the dev branch the target of the PR. Then, the package build on conda-forge will end up with the ``e3sm_dev`` label.


I think this section will definitely need clarification, but we can improve the wording when we work on the next release together.

Sounds good, and yes, you will want either me or @xylar to do the initial swipe at this lest we end up unintentionally producing artifacts (packages). The idea is that you want to "rerender" it locally and then push directly into a new branch. It's much easier to just let us do it though 😸

forsyth2 · 2023-03-22T00:52:53Z

docs/source/dev_guide/release.rst

@@ -63,7 +63,12 @@ Bumping the Version

 .. _github-release:

-Releasing on GitHub
+Releasing on GitHub: release candidates
+-------------------


Note to self: rst might require the entire section title to be underlined.

mahf708

The docs changes look good to me (I haven't reviewed the workflow changes carefully but I trust you!)

Thanks and looking forward to getting things rolling on conda-forge 😺

forsyth2 added DevOps CI/CD, configuration, etc. Documentation Files in `docs` modified labels Mar 3, 2023

forsyth2 self-assigned this Mar 3, 2023

forsyth2 commented Mar 3, 2023

View reviewed changes

mahf708 reviewed Mar 4, 2023

View reviewed changes

forsyth2 commented Mar 22, 2023

View reviewed changes

mahf708 approved these changes Mar 22, 2023

View reviewed changes

Update release process

c226a45

forsyth2 force-pushed the n396-release-process branch from eaea296 to c226a45 Compare March 22, 2023 21:32

forsyth2 merged commit a3244cc into main Mar 22, 2023

forsyth2 deleted the n396-release-process branch March 22, 2023 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update release process #410

Update release process #410

forsyth2 commented Mar 3, 2023

forsyth2 commented Mar 3, 2023

forsyth2 Mar 3, 2023

mahf708 Mar 4, 2023

mahf708 Mar 4, 2023

mahf708 Mar 4, 2023

forsyth2 Mar 8, 2023

mahf708 Mar 4, 2023

mahf708 Mar 4, 2023

forsyth2 Mar 8, 2023

xylar Mar 8, 2023

forsyth2 Mar 11, 2023

xylar Mar 11, 2023

mahf708 Mar 11, 2023

mahf708 Mar 11, 2023

xylar Mar 11, 2023

mahf708 Mar 11, 2023

mahf708 Mar 4, 2023

xylar commented Mar 8, 2023

tomvothecoder commented Mar 8, 2023

forsyth2 left a comment

forsyth2 Mar 22, 2023

mahf708 Mar 22, 2023

forsyth2 Mar 22, 2023

mahf708 Mar 22, 2023

mahf708 Mar 22, 2023

forsyth2 Mar 22, 2023

forsyth2 Mar 22, 2023

mahf708 Mar 22, 2023

forsyth2 Mar 22, 2023

mahf708 left a comment

Update release process #410

Update release process #410

Conversation

forsyth2 commented Mar 3, 2023

forsyth2 commented Mar 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xylar commented Mar 8, 2023

tomvothecoder commented Mar 8, 2023

forsyth2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahf708 left a comment

Choose a reason for hiding this comment