Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update HPC Tier Definitions #3138

Merged
merged 25 commits into from
Dec 9, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
83d76cd
Update the supported platform list
DavidHuber-NOAA Dec 3, 2024
1f1bcce
Reformat the table
DavidHuber-NOAA Dec 3, 2024
d9b7ca9
Reformat table again
DavidHuber-NOAA Dec 3, 2024
14f30df
More formatting changes
DavidHuber-NOAA Dec 3, 2024
385c039
More formatting and content
DavidHuber-NOAA Dec 3, 2024
4a9df1e
Rearrange
DavidHuber-NOAA Dec 3, 2024
18d92f1
More rearrangements. Update badges.
DavidHuber-NOAA Dec 3, 2024
ac1a69e
Add more badges
DavidHuber-NOAA Dec 3, 2024
c60aedf
Merge branch 'NOAA-EMC:develop' into doc/hpc_tiers
DavidHuber-NOAA Dec 3, 2024
ff54a88
Add workflows to get Hercules and WCOSS2 status
DavidHuber-NOAA Dec 3, 2024
4e8434f
Update README.md
DavidHuber-NOAA Dec 3, 2024
cdbd1ea
Typo
DavidHuber-NOAA Dec 3, 2024
dfc11b5
Add capabilities table
DavidHuber-NOAA Dec 3, 2024
1c660a1
Remove Orion badge
DavidHuber-NOAA Dec 3, 2024
f7b58e0
Merge branch 'doc/hpc_tiers' of github.com:davidhuber-noaa/global-wor…
DavidHuber-NOAA Dec 3, 2024
6ad77e0
Merge branch 'develop' into doc/hpc_tiers
DavidHuber-NOAA Dec 4, 2024
723f777
Try reformatting again
DavidHuber-NOAA Dec 4, 2024
341be87
Try reformatting again
DavidHuber-NOAA Dec 4, 2024
d55a75d
Reformat table again
DavidHuber-NOAA Dec 5, 2024
39ea5f1
Try a list table
DavidHuber-NOAA Dec 5, 2024
c80994c
Merge remote-tracking branch 'emc/develop' into doc/hpc_tiers
DavidHuber-NOAA Dec 5, 2024
7ba61d1
Narrow the table a bit
DavidHuber-NOAA Dec 5, 2024
8c662a5
Update Orion TC tracker support info
DavidHuber-NOAA Dec 6, 2024
6b6e4cb
Merge branch 'develop' into doc/hpc_tiers
aerorahul Dec 9, 2024
703a581
Merge branch 'develop' into doc/hpc_tiers
aerorahul Dec 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .github/workflows/hercules.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Hercules

on:
pull_request_target:
branches:
- develop
types: [closed]

jobs:

getlabels:
runs-on: ubuntu-22.04
outputs:
labels: ${{ steps.id.outputs.labels }}
steps:
- name: Get Label Steps
id: id
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.event.repository.name }}
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
run: |
LABELS1="$(gh api repos/$OWNER/$REPO_NAME/pulls/$PULL_REQUEST_NUMBER --jq '.labels.[].name')"
LABELS=$(echo "$LABELS1" | tr '\n' ' ')
echo "labels=$LABELS" >> $GITHUB_OUTPUT

passed:
if: contains( needs.getlabels.outputs.labels, 'CI-Hercules-Passed') && github.event.pull_request.merged
runs-on: ubuntu-22.04
needs:
- getlabels

steps:
- name: Passed
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: hercules.json
label: hercules
message: passing
color: green

failed:
if: contains( needs.getlabels.outputs.labels, 'CI-Hercules-Failed') && github.event.pull_request.merged
runs-on: ubuntu-latest
needs:
- getlabels

steps:
- name: Failed
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: hercules.json
label: hercules
message: failing
color: red


pending:
if: "!contains( needs.getlabels.outputs.labels, 'CI-Hercules-Passed') && !contains( needs.getlabels.outputs.labels, 'CI-Hercules-Failed')"
runs-on: ubuntu-latest
needs:
- getlabels

steps:
- name: Pending
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: hercules.json
label: hercules
message: pending
color: orange
81 changes: 81 additions & 0 deletions .github/workflows/wcoss2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: WCOSS2

on:
pull_request_target:
branches:
- develop
types: [closed]

jobs:

getlabels:
runs-on: ubuntu-22.04
outputs:
labels: ${{ steps.id.outputs.labels }}
steps:
- name: Get Label Steps
id: id
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO_NAME: ${{ github.event.repository.name }}
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }}
run: |
LABELS1="$(gh api repos/$OWNER/$REPO_NAME/pulls/$PULL_REQUEST_NUMBER --jq '.labels.[].name')"
LABELS=$(echo "$LABELS1" | tr '\n' ' ')
echo "labels=$LABELS" >> $GITHUB_OUTPUT

passed:
if: contains( needs.getlabels.outputs.labels, 'CI-Wcoss2-Passed') && github.event.pull_request.merged
runs-on: ubuntu-latest
needs:
- getlabels

steps:
- name: Passed
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: wcoss2.json
label: wcoss2
message: passing
color: green

failed:
if: contains( needs.getlabels.outputs.labels, 'CI-Wcoss2-Failed') && github.event.pull_request.merged
runs-on: ubuntu-latest
needs:
- getlabels

steps:
- name: Failed
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: wcoss2.json
label: wcoss2
message: failing
color: red


pending:
if: "!contains( needs.getlabels.outputs.labels, 'CI-Wcoss2-Passed') && !contains( needs.getlabels.outputs.labels, 'CI-Wcoss2-Failed')"
runs-on: ubuntu-latest
needs:
- getlabels

steps:
- name: Pending
uses: schneegans/[email protected]
with:
forceUpdate: true
auth: ${{ secrets.CLI_DYNAMIC_BADGES }}
gistID: e35aa2904a54deae6bbb1fdc2d960c71
filename: wcoss2.json
label: wcoss2
message: pending
color: orange
43 changes: 29 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,42 @@

![Custom badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/emcbot/e35aa2904a54deae6bbb1fdc2d960c71/raw/hera.json)
![Custom badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/emcbot/e35aa2904a54deae6bbb1fdc2d960c71/raw/orion.json)
![Custom badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/emcbot/e35aa2904a54deae6bbb1fdc2d960c71/raw/hercules.json)
![Custom badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/emcbot/e35aa2904a54deae6bbb1fdc2d960c71/raw/wcoss2.json)

# global-workflow
Global Workflow currently supporting the Global Forecast System (GFS) with the [UFS-weather-model](https://github.com/ufs-community/ufs-weather-model) and [GSI](https://github.com/NOAA-EMC/GSI)-based Data Assimilation System.
The Global Workflow supporting the Global Forecast System (GFS) and the Global Ensemble Forecasting System (GEFS) with the [UFS-weather-model](https://github.com/ufs-community/ufs-weather-model). Data assimilation, currently only available for the GFS, is provide by both the [GSI](https://github.com/NOAA-EMC/GSI)- and [GDASApp (JEDI)](https://github.com/NOAA-EMC/GDASApp)-based Data Assimilation systems.
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved

The `global-workflow` depends on the following prerequisities to be available on the system:
In progress [documentation](https://global-workflow.readthedocs.io/en/latest/) is available.

* Workflow Engine - [Rocoto](https://github.com/christopherwharrop/rocoto) and [ecFlow](https://github.com/ecmwf/ecflow) (for NWS Operations)
* Compiler - Intel Compiler Suite
* Software - NCEPLIBS (various), ESMF, HDF5, NetCDF, and a host of other softwares (see module files under /modulefiles for additional details)
# Prerequisites

The `global-workflow` current supports the following tier-1 machines:
The Global Workflow depends on the following prerequisities to be available on the system:

* NOAA RDHPCS - Hera
* MSU HPC - Orion
* MSU HPC - Hercules
* NOAA's operational HPC - WCOSS2
* Workflow Engines - [Rocoto](https://github.com/christopherwharrop/rocoto) and [ecFlow](https://github.com/ecmwf/ecflow) (for NWS Operations)
* Compiler - Intel classic compiler suite version 2021.x
* Software - NCEPLIBS (various), ESMF, HDF5, NetCDF, and a host of other softwares (see module files under /modulefiles for additional details).
- [Spack-stack](https://github.com/JCSDA/spack-stack) is recommended for the installation of dependent libraries.

Additionally, the following tier-2 machine is supported:
* SSEC at Univ. of Wisconsin - S4 (Note that S2S+ experiments are not fully supported)
## Supported platforms

Documentation (in progress) is available [here](https://global-workflow.readthedocs.io/en/latest/).
The Global Workflow currently supports the following machines at the indicated tier.

| HPC | Tier | Notes |
| --------------------------------------- |:----:|:--------------------------------------------------------------------------:|
| WCOSS2<br>NCO | 1 | GEFS testing is not regularly performed. |
| Hera<br>NOAA RDHPCS | 1 | |
| Hercules<br>MSU | 1 | Currently does not support the TC Tracker. |
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved
| Orion<br>MSU | 2 | The GSI runs very slowly on Orion. |
| Gaea C5/C6<br>RDHPCS | 3 | Currently non-operational following an OS upgrade.<br>Supported by EPIC. |
| AWS, GCP, Azure <br>NOAA Parallel Works | 3 | Supported by EPIC. |
| Jet<br>RDHPCS | 3 | Supported by NESDIS. |
| S4<br>SSEC | 3 | Currently non-operational following an OS upgrade.<br>Supported by NESDIS. |

<ins>**Tier Definitions**</ins>

1. Fully supported by the EMC global workflow team. CI testing is regularly performed on these systems, the majority of the global workflow features are supported, and the team will address any platform-specific features, bugs, upgrades, and requests for data.
2. Supported by the global workflow team on an ad-hoc basis. CI tests are supported on these systems, but not regularly performed.
3. No official support by the global workflow team, but may be supported by other entities (e.g. EPIC).

# Disclaimer

Expand Down