From e41d4e256d8568cd37989ab972311a66fe2825b3 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Wed, 8 May 2024 18:55:53 -0500 Subject: [PATCH 1/7] update ror-updates section for v2 --- README.md | 158 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 118 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index f9ab81321..8e198421e 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,10 @@ Test repository for developing deployment process for ROR record updates. 1. [Create JSON files for new and updated ROR records](#create-json-files-for-new-and-updated-ror-records) 2. [Validate files in ror-updates](#validate-files-in-ror-updates) 3. [Generate relationships](#generate-relationships) +4. [Remove relationships to inactive records](#remove-relationships) +5. [Update labels in relationships](#update-labels) +6. [Update addresses](#update-addresses) +7. [Update last modified dates](#update-last-mod) 4. [Validate files again in ror-updates](#validate-files-again-in-ror-updates) ## Release to Staging @@ -34,7 +38,7 @@ Test repository for developing deployment process for ROR record updates. ## Create JSON files for new and updated ROR records JSON files for new and updated ROR records are created by the ROR metadata curation lead and curation advisory board as part of the process managed in [ror-updates](https://github.com/ror-community/ror-updates). **Only changes requested and approved through this curation process are included in ROR data releases.** -Schema-valid ROR record JSON can be generated using the [Leo form app](https://leo.dev.ror.org/). Note that relationships should not be included in record files; these are created in the [Generate relationships](#generate-relationships) step of the deployment process. +Schema-valid ROR record JSON can be generated from a CSV using the [ROR API /bulkupdate endpoint](https://github.com/ror-community/ror-api?tab=readme-ov-file#createupdate-multiple-record-files-from-a-csv). Note that relationships should not be included in record files; these are created in the [Generate relationships](#generate-relationships) step of the deployment process. Once created, push JSON files to ror-updates in a new branch per below. @@ -81,15 +85,14 @@ JSON files for new and updated ROR records should be validated before generating ). 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/validate.yml (Actions > Create relationships in the ror-updates repository) -2. Click Run workflow at right, choose the branch you're working on and leave "Check box to validate relationships" _unchecked_. If you're working on the Main branch or the directory containing the files you'd like to validate has a diffferent name from the branch you're working on, enter name of that directory. +2. Click Run workflow at right, choose the schema version and branch you're working on, check the "Check box to skip Geonames validation" box and leave "Check box to validate relationships" and "_unchecked_. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. 3. Click the green Run workflow button. -4. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the validate-filtes box on the workflow run page in Github. +4. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` 5. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to your working branch and repeat steps 1-3 to re-run the Validate files workflow. ## Generate relationships -Relationships are not included in the intitial ROR record JSON files. Relationships are generated using a script [generaterelationships.py](https://github.com/ror-community/ror-api/blob/dev/rorapi/management/commands/generaterelationships.py) triggered by a Github action [Create relationships](https://github.com/ror-community/ror-updates/blob/staging/.github/workflows/generate_relationships.yml), which should be run AFTER all new and updated JSON records to be included in the release are uploaded to ror-updates. +Relationships are not included in the intitial ROR record JSON files. Relationships are generated using a script [generaterelationships.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/generate_relationships/generate_relationships.py) triggered by a Github action [Create relationships](https://github.com/ror-community/ror-updates/blob/staging/.github/workflows/generate_relationships.yml), which should be run AFTER all new and updated JSON records to be included in the release are uploaded to ror-updates. -*Note: Currently relationships can only be added using this process, not removed. Removing relationships from an existing record requires manually downloading and editing the record, and adding it to the release branch.* 1. Create relationships list as a CSV file using the template [[TEMPLATE] relationships.csv](https://docs.google.com/spreadsheets/d/17rA549Q6Vc-YyH8WUtXUOvsAROwCDmt1vy4Rjce-ELs) and name the file relationships.csv. **IMPORTANT! File must be named relationships.csv and fields used by the script must be formatted correctly**. Template fields used by the script are: @@ -97,9 +100,11 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh |-----------------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------------| | Record ID | ROR ID of record being added/updated, in URL form | https://ror.org/015m7w34 | | Related ID | ROR ID of the related record, in URL form | https://ror.org/02baj6743 | -| Relationship of Related ID to Record ID | One the following values: Parent, Child, Related | Parent | +| Relationship of Related ID to Record ID | One the following values: Parent, Child, Related | Parent, Delete (case insensitive) | | Name of org in Related ID | Name of the related organization, as it appears in the name field of the related ROR record | Indiana University – Purdue University Columbus | -| Current location of Related ID | Production or Release branch | Production | +| Current location of Related ID | Production or Release branch (not required for Delete relationship) | Production | + +*Note: Relationships can be deleted by specifying the 2 records in Record ID and Related ID and "Delete" in Relationship of Related ID to Record ID. All relationships between the 2 records will be deleted from both records. It does not matter which record is in Record ID and which is in Related ID, and neither record needs to be in the release. Records will be downloaded if needed.* 2. Place the relationships.csv inside the parent directory where the ROR record JSON files are located (ex: rc-v1.3-review - don't put it inside new or updates directories). 3. Commit and push the relationships.csv file to your working branch, ex: @@ -108,14 +113,48 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh git commit -m "adding relationships list to v1.3" git push origin rc-v1.3-review -3. Go to https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml (Actions > Create relationships in the ror-records repository) -4. Click Run workflow at right, choose the branch that you just pushed relationships.csv to, and click the green Run workflow button. -5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the generate-relationships box on the workflow run page in Github. -6. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. +3. Go to https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml (Actions > Create relationships in the ror-updates repository) +4. Click Run workflow at right, choose the schema version and branch that you just pushed relationships.csv to, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +6. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the rc-vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. + +## Remove relationships to inactive records +Active records cannot contain relationships to inactive or withdrawn records. On each release, all release records must be checked for status changes and, for any where status changed from active to inactive or withdrawn, relationships to those records must be removed from active records (both in the release and in production). + +Relationships to inactive records are removed using a script [remove_relationships.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/remove_relationships/remove_relationships.py) triggered by a Github action [Remove relationships to inactive records](https://github.com/ror-community/ror-updates/actions/workflows/remove_relationships.yml), which must be run AFTER the Generate relationships action. + + +1. Go to https://github.com/ror-community/ror-updates/actions/workflows/remove_relationships.yml (Actions > Remove relationships to inactive records in the ror-updates repository) +4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. +5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` + +## Update labels in relationships +When the ROR display name changes in a release record, relationships labels must be updated in any related records. Records are checked for ROR display name changes and relationship labels in related records are updated using a script [update_related.py](https://github.com/ror-community/curation_ops/blob/main/update_related_records/update_related.py) triggered by a Github action [Update labels in relationships](https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml), which must be run AFTER the Remove relationships to inactive records action. + +1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml (Actions > Update labels in related records in the ror-updates repository) +4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` + +## Update locations +All release records must have their locations.geonames_details checked against the Geonames API and updated with the latest Geonames data if any differences exist. Locations are updated using a script [update_addresses.py](https://github.com/ror-community/curation_ops/blob/main/update_address_only/update_addresses.py) triggered by a Github action [Update addresses](https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml). This action should be run after all changes to release files are complete, except last modified date updates. + +1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml (Actions > Update addresses in the ror-updates repository) +4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +5. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` + +## Update last modified dates +All release records must have their last modified date updated to match the (planned) date of the release. Ideally, this date should match the data dump file date. Release file generation is often completed 1 or more days before the release is actually deployed. In that case, the planned release date should be used (not the current date). + +Last modified dates are updated using a script [update_last_mod.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/utilities/update_last_mod/update_last_mod.py) triggered by a Github action [Update last modified date](https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml). This action should be run after all other changes to release files are complete. + +1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml(Actions > Update last modified date in the ror-updates repository) +4. Click Run workflow at right, choose the rc-vX.X branch, enter the planned release date in the "Date value to populate last_modified field with (YYYY-MM-DD)" field and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +5. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` ## Validate files again in ror-updates -After relationships are added, record files should be validated again to ensure that the changes applied by the script are still schema-valid. -Follow the steps above to run validation, but make sure to tick the "Check box to validate relationships" checkbox. +After all record updates above are complete, record files should be validated again to ensure that the changes applied by the script are still schema-valid. During this validation run, relationships and locations should also be validated. + +Follow the steps above to run validation, but make sure to tick the "Check box to validate relationships" checkbox and leave the "Check box to skip Geonames validation" box unchecked. # Release to Staging @@ -177,17 +216,19 @@ Before finalizing a release candidate, JSON files for new and updated ROR record ), which should be run after all new and updated JSON records to be included in the release are uploaded to the vX.X branch. 1. Go to https://github.com/ror-community/ror-records/actions/workflows/validate.yml (Actions > Create relationships in the ror-records repository) -2. Click Run workflow at right, choose the current vX.X branch, tick "Check box to validate relationshpis", and click the green Run workflow button. -3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the validate-filtes box on the workflow run page in Github. +2. Click Run workflow at right, choose the schema version and current vX.X branch, tick "Check box to validate relationships", leave "Check box to skip Geonames validation" unchecked, and click the green Run workflow button. +3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` 4. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to the vX.X branch and repeat steps 1-3 to re-run the Validate files workflow. **IMPORTANT! Validate files workflow must succeed before proceeding to deployment** -## Deploy to Staging -Deploying to staging.ror.org/search and api.staging.ror.org requires making a Github pull request and merging it. Each of these actions triggers different automated workflows: +## Deploy v2 to Staging +Deploying v2 to staging.ror.org/search and api.staging.ror.org requires making a Github pull request and merging it. Each of these actions triggers different automated workflows: - **Open pull request against Staging branch:** Check user permissions and validate files -- **Merge pull request to Staging branch:** Check user permissions, deploy release candidate to Staging API +- **Merge pull request to Staging branch:** Check user permissions, deploy release candidate to Staging API (v2 only) + +v1 must be deployed separately, after data dump is created. *Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* @@ -199,57 +240,96 @@ Deploying to staging.ror.org/search and api.staging.ror.org requires making a Gi 6. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request 7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. 8. Once the Staging pull request workflow has completed successfully, click Merge pull request -9. A Github action [Deploy to staging](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the ROR Elasticsearch instance. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org +9. A Github action [Deploy to staging v2 only](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the v2 ROR Elasticsearch index. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org/v2 ### Multiple staging releases -If records needed to be added or changed after an initial Staging release, add the new/updated records to the existing release branch per [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch) and repeat the steps to [Generate relationships](#generate-relationships), [Validate files](#validate-files) and [Deploy to Staging](#deploy-to-staging). A few points to note with multiple Staging releases: +If records needed to be added or changed after an initial Staging release, add the new/updated records to the existing release branch per [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch) and repeat the steps to [Generate relationships](#generate-relationships), [Validate files](#validate-files) and [Deploy to v2 Staging](#deploy-to-v2-staging). A few points to note with multiple Staging releases: - Do not remove records from the release branch that have already been deployed to Staging. Overwrite any records already deployed to Staging that require changes and leave the rest as they are. When ready to deploy to poduction, the release branch should contain all new/updated records to be included in the production release. - Include relationships for any records already in merged to the vX.X directory on Staging in the release branch relationships.csv (not just the current deployment) - Deleting record files from the release branch after they have been deployed to Staging will not remove them from the Staging index. At the moment, this will need to be done manually by a developer; in the future, we will add a mechanism to remove records from the Staging index that have been deleted from an release branch. This does not affect production, as the production index is completely separate. -## Test Staging release -*Note: There is currently no staging UI search app (https://staging.ror.org connects to production search app using api.ror.org). UI tests should be run against https://dev.ror.org.* +## Test v2 Staging release ### New records Choose several new records from the Staging release and, for each record: 1. Check that the record can be retrieved from the Staging API - curl https://api.staging.ror.org/organizations/[RORID] + curl https://api.staging.ror.org/v2/organizations/[RORID] 2. Check that the record can be searched by name in the Staging API (make sure to [escape spaces and reserved characters](https://ror.readme.io/docs/rest-api#reserved-characters)) - curl https://api.staging.ror.org/organizations?query=[STAGING%20RECORD%20NAME] + curl https://api.staging.ror.org/v2/organizations?query=[STAGING%20RECORD%20NAME] + +3. Check that the record can be searched by name in the Staging UI and the results are as expected. + + https://staging.ror.org/search > Enter name in search box ### Updated records Choose several updated records from the Staging release and, for each record: -1. Check that the record can be retrieved from the Staging API - curl https://api.staging.ror.org/organizations/[RORID] +1. Check that the record can be retrieved from the v2 Staging API -2. Check that the record can be searched by name in the Staging API (make sure to [escape spaces and reserved characters](https://ror.readme.io/docs/rest-api#reserved-characters)) + curl https://api.staging.ror.org/v2/organizations/[RORID] - curl https://api.staging.ror.org/organizations?query=[RECORD%20NAME] +2. Check that the record can be searched by name in the v2 Staging API (make sure to [escape spaces and reserved characters](https://ror.readme.io/docs/rest-api#reserved-characters)) -3. Retrieve the record from the Staging API and the Production API and compare changes to verify that the expected changes where made. + curl https://api.staging.ror.org/v2/organizations?query=[RECORD%20NAME] - curl https://api.staging.ror.org/organizations/[ROR ID] > staging_[RORID].json - curl https://api.ror.org/organizations/[ROR ID] > prod_[RORID].json +3. Retrieve the record from the v2 Staging API and the v2 Production API and compare changes to verify that the expected changes where made. + + curl https://api.staging.ror.org/v2/organizations/[ROR ID] > staging_[RORID].json + curl https://api.ror.org/v2/organizations/[ROR ID] > prod_[RORID].json diff staging_[RORID].json prod_[RORID].json +4. Check that the record can be searched by name in the Staging UI and the results are as expected. + + https://staging.ror.org/search > Enter name in search box ### Unchanged records Choose several records from Production that were not included in the release and for each record: -1. Retrieve the record from the Staging API and the Production API and compare changes to verify that the records are identical. +1. Retrieve the record from the v2 Staging API and the v2 Production API and compare changes to verify that the records are identical. curl https://api.staging.ror.org/organizations/[ROR ID] > staging_[RORID].json - curl https://api.ror.org/organizations/[ROR ID] > prod_[RORID].json + curl https://api.ror.org/v2/organizations/[ROR ID] > prod_[RORID].json diff staging_[RORID].json prod_[RORID].json ```diff``` result should be nothing (no response). -# Deploy to Production +## Create data dump for internal use +A data dump must be created in order to index v1. To create a data dump with records in both v2 and v1: + +1. In the ror-records repository, go to [Actions > Create data dump](https://github.com/ror-community/ror-records/actions/workflows/generate_dump.yml) +2. Click Run Workflow, select the schema version that the release was curated in (should be v2), enter the name of the release directory (ex, v1.46) and enter the name of the last production data dump from ror-data that the new dump should be built from, ex 2021-09-23-ror-data (without the .zip file extension) +3. Click the Run workflow button +4. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump zip should now be available in ror-data. + +## Deploy v1 to Staging +Deploying v2 to staging.ror.org/search and api.staging.ror.org requires making a Github pull request and merging it. Each of these actions triggers different automated workflows: + +- **Open pull request against Staging branch:** Check user permissions and validate files +- **Merge pull request to Staging branch:** Check user permissions, deploy release candidate to Staging API (v2 only) + +v1 must be deployed separately, after data dump is created. + +*Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* + +1. Go to https://github.com/ror-community/ror-records/pulls (Pull requests tab in ror-records repository) +2. Click New pull request at right +3. Click the Base dropdown at left and choose Staging. Important! Do not make a pull request against the default Main branch. +4. Click the Compare dropdown and choose the vX.X branch that you have been working with in the previous steps. +5. Click Create pull request and enter ```Merge vX.X to staging``` in the Title field. Leave the Comments field blank. +6. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request +7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. +8. Once the Staging pull request workflow has completed successfully, click Merge pull request +9. A Github action [Deploy to staging](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the ROR Elasticsearch instance. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org + +## Test v1 Staging release + +# Release to production + +## Deploy v2 to Production Deploying to ror.org/search and api.ror.org requires making a Github pull request and merging it, then tagging and publishing a new release. Each of these actions triggers different automated workflows: - **Open pull request against Main branch:** Check user permissions and validate files(?) @@ -274,7 +354,7 @@ Deploying to ror.org/search and api.ror.org requires making a Github pull reques 14. Click Publish release 9. A Github action Deploy to production will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production ROR Elasticsearch instance and generates a new data dump file in TBD. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org -## Test production release +## Test v2 Production release Choose several new, updated and unchanged records and, for each record: @@ -294,11 +374,9 @@ Choose several new, updated and unchanged records and, for each record: https://ror.org/search > Enter name in search box -## Create data dump for internal use -1. In the ror-records repository, go to [Actions > Create data dump](https://github.com/ror-community/ror-records/actions/workflows/generate_dump.yml) -2. Click Run Workflow and enter the name of the release directory (ex, v1.0) and the name of the last production data dump from ror-data that the new dump should be built from, ex 2021-09-23-ror-data (without the .zip file extension) -3. Click the Run workflow button -4. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump should now be available in ror-data. +## Deploy v1 to Production + +## Test v1 Production release # Publish public data dump From 6ca2a64d6e67697a80f95c3c1449fe94785b3b24 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Wed, 8 May 2024 19:48:43 -0500 Subject: [PATCH 2/7] update staging deployment sections --- README.md | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 8e198421e..18c4604de 100644 --- a/README.md +++ b/README.md @@ -306,26 +306,15 @@ A data dump must be created in order to index v1. To create a data dump with rec 4. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump zip should now be available in ror-data. ## Deploy v1 to Staging -Deploying v2 to staging.ror.org/search and api.staging.ror.org requires making a Github pull request and merging it. Each of these actions triggers different automated workflows: +v1 deployment to api.staging.ror.org is done via a full reindex from a data dump. This means that a data dump must be generated in ror-data before v1 can be indexed. -- **Open pull request against Staging branch:** Check user permissions and validate files -- **Merge pull request to Staging branch:** Check user permissions, deploy release candidate to Staging API (v2 only) - -v1 must be deployed separately, after data dump is created. - -*Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* - -1. Go to https://github.com/ror-community/ror-records/pulls (Pull requests tab in ror-records repository) -2. Click New pull request at right -3. Click the Base dropdown at left and choose Staging. Important! Do not make a pull request against the default Main branch. -4. Click the Compare dropdown and choose the vX.X branch that you have been working with in the previous steps. -5. Click Create pull request and enter ```Merge vX.X to staging``` in the Title field. Leave the Comments field blank. -6. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request -7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. -8. Once the Staging pull request workflow has completed successfully, click Merge pull request -9. A Github action [Deploy to staging](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the ROR Elasticsearch instance. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org +1. Go to https://github.com/ror-community/ror-records/actions/workflows/staging_index_dump.yml (Actions > STAGING index full data dump in the ror-records repository) +2. Click Run workflow at right, set the branch to Staging, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. +3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.staging.ror.org/v1/organizations to see whether the dump was indexed. ## Test v1 Staging release +Perform API tests above, using v1 base URL https://api.staging.ror.org/v1/organizations . v1 is not available in the UI. # Release to production From f7c9bb0062fab3f9bd73b947e7e2f74fd94dcbf1 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Thu, 9 May 2024 09:50:06 -0500 Subject: [PATCH 3/7] update production release steps and start updating lists --- README.md | 100 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 55 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 18c4604de..2b2053c1f 100644 --- a/README.md +++ b/README.md @@ -7,13 +7,13 @@ Test repository for developing deployment process for ROR record updates. 1. [Create JSON files for new and updated ROR records](#create-json-files-for-new-and-updated-ror-records) 2. [Validate files in ror-updates](#validate-files-in-ror-updates) 3. [Generate relationships](#generate-relationships) -4. [Remove relationships to inactive records](#remove-relationships) -5. [Update labels in relationships](#update-labels) -6. [Update addresses](#update-addresses) -7. [Update last modified dates](#update-last-mod) -4. [Validate files again in ror-updates](#validate-files-again-in-ror-updates) +4. [Remove relationships to inactive records](#remove-relationships-to-inactive-records) +5. [Update labels in relationships](#update-labels-in-relationships) +6. [Update locations](#update-locations) +7. [Update last modified dates](#update-last-modified-dates) +8. [Validate files again in ror-updates](#validate-files-again-in-ror-updates) -## Release to Staging +## Release v2 to Staging 1. [Create new release branch](#create-new-release-branch-in-ror-records) 2. [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch-in-ror-records) 3. [Validate files in ror-records](#validate-files-in-ror-records) @@ -68,17 +68,17 @@ These steps assume that you have already [installed and configured git](https:// mkdir rc-v1.3-review/new mkdir rc-v1.3-review/updates -2. Place the JSON files for new and updated records inside the directories you just created. -3. Add and commit the files +3. Place the JSON files for new and updated records inside the directories you just created. +4. Add and commit the files git add rc-v1.3-review/ git commit -m "add new and updated ROR records to rc-v1.3-review" -4. Push the files and new branch to the remote ror-records repository +5. Push the files and new branch to the remote ror-records repository git push origin rc-v1.3-review -5. Repeat steps 1-3 if additional files need to be added to the release candidate. +6. Repeat steps 1-3 if additional files need to be added to the release candidate. ## Validate files in ror-updates JSON files for new and updated ROR records should be validated before generating relationshps to check that they comply with the ROR schema and contained properly formatted JSON. Validation is performed by a script [run_validations.py](https://github.com/ror-community/validation-suite/blob/main/run_validations.py) triggered by a Github action [Validate files](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/validate.yml @@ -113,10 +113,10 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh git commit -m "adding relationships list to v1.3" git push origin rc-v1.3-review -3. Go to https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml (Actions > Create relationships in the ror-updates repository) -4. Click Run workflow at right, choose the schema version and branch that you just pushed relationships.csv to, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` -6. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the rc-vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. +4. Go to https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml (Actions > Create relationships in the ror-updates repository) +5. Click Run workflow at right, choose the schema version and branch that you just pushed relationships.csv to, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +6. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +7. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the rc-vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. ## Remove relationships to inactive records Active records cannot contain relationships to inactive or withdrawn records. On each release, all release records must be checked for status changes and, for any where status changed from active to inactive or withdrawn, relationships to those records must be removed from active records (both in the release and in production). @@ -125,22 +125,22 @@ Relationships to inactive records are removed using a script [remove_relationshi 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/remove_relationships.yml (Actions > Remove relationships to inactive records in the ror-updates repository) -4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. -5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. +3. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` ## Update labels in relationships When the ROR display name changes in a release record, relationships labels must be updated in any related records. Records are checked for ROR display name changes and relationship labels in related records are updated using a script [update_related.py](https://github.com/ror-community/curation_ops/blob/main/update_related_records/update_related.py) triggered by a Github action [Update labels in relationships](https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml), which must be run AFTER the Remove relationships to inactive records action. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml (Actions > Update labels in related records in the ror-updates repository) -4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -5. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +3. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` ## Update locations All release records must have their locations.geonames_details checked against the Geonames API and updated with the latest Geonames data if any differences exist. Locations are updated using a script [update_addresses.py](https://github.com/ror-community/curation_ops/blob/main/update_address_only/update_addresses.py) triggered by a Github action [Update addresses](https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml). This action should be run after all changes to release files are complete, except last modified date updates. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml (Actions > Update addresses in the ror-updates repository) -4. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -5. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` +2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +3. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` ## Update last modified dates All release records must have their last modified date updated to match the (planned) date of the release. Ideally, this date should match the data dump file date. Release file generation is often completed 1 or more days before the release is actually deployed. In that case, the planned release date should be used (not the current date). @@ -148,8 +148,8 @@ All release records must have their last modified date updated to match the (pla Last modified dates are updated using a script [update_last_mod.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/utilities/update_last_mod/update_last_mod.py) triggered by a Github action [Update last modified date](https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml). This action should be run after all other changes to release files are complete. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml(Actions > Update last modified date in the ror-updates repository) -4. Click Run workflow at right, choose the rc-vX.X branch, enter the planned release date in the "Date value to populate last_modified field with (YYYY-MM-DD)" field and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -5. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` +2. Click Run workflow at right, choose the rc-vX.X branch, enter the planned release date in the "Date value to populate last_modified field with (YYYY-MM-DD)" field and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +3. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` ## Validate files again in ror-updates After all record updates above are complete, record files should be validated again to ensure that the changes applied by the script are still schema-valid. During this validation run, relationships and locations should also be validated. @@ -192,7 +192,7 @@ These steps assume that you have already [installed and configured git](https:// 3. Click the Add file button at right and choose Upload files 4. Add your folder of files as prompted 5. Under Commit changes, type a commit message in the first text box, leave the radio button set to "Commit directly to the vX.X branch" and click Commit changes. -5. Repeat steps 1-4 if additional files need to be added to the release candidate. +6. Repeat steps 1-4 if additional files need to be added to the release candidate. ### Git CLI 1. Create in new directory in the root of the ror-records repository the **exact same name** as the release branch (ex, v1.3). @@ -319,11 +319,11 @@ Perform API tests above, using v1 base URL https://api.staging.ror.org/v1/organi # Release to production ## Deploy v2 to Production -Deploying to ror.org/search and api.ror.org requires making a Github pull request and merging it, then tagging and publishing a new release. Each of these actions triggers different automated workflows: +Deploying v2 to ror.org/search and api.ror.org requires making a Github pull request and merging it, then tagging and publishing a new release. Each of these actions triggers different automated workflows: -- **Open pull request against Main branch:** Check user permissions and validate files(?) -- **Merge pull request to Main branch:** Check user permissions and push individual JSON records included in the release to a new directory with the release tag as the directory name in [ror-updates](https://github.com/ror-community/ror-updates). This is for curator convenience; this copy of the release records is not used elsewhere in the deployment process. -- **Publish release:** Check user permissions, deploy release to production API, create new data dump zip file and place in ror-data repository. +- **Open pull request against Main branch:** Check user permissions +- **Merge pull request to Main branch:** No action (approval required to merge, though) +- **Publish release:** Check user permissions, copy files to S3, index v2 release to production API *Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* @@ -333,15 +333,16 @@ Deploying to ror.org/search and api.ror.org requires making a Github pull reques 4. Click the Compare dropdown and choose Staging. 5. Click Create pull request, enter ```Merge vX.X to production``` in the Title field. Leave the Comments field blank. 6. Double-check that the Base dropdown is set to Main and that the list of updated files appears to be correct, then click Create pull request -7. A Github action Production pull request will be triggered, which does TBD. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. -8. Once the Production pull request workflow has completed successfully, click Merge pull request. -9. Go to https://github.com/ror-community/ror-records/releases (Release tab in ror-records repository) -10. Click Draft new release at right -11. Click the Choose a tag dropdown and enter the version number for the release, ex v1.3. This should be the same number as the release branch and the directory that the release files are located in. Click Create new tag X.X on publish. -12. In the Release name field, enter ```vX.X``` (replace X.X with the release tag number) -13. In the Dsecribe this release field, enter ```Includes updates listed in https://github.com/ror-community/ror-records/milestone/X``` (link to correct milestone for this release) -14. Click Publish release -9. A Github action Deploy to production will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production ROR Elasticsearch instance and generates a new data dump file in TBD. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org +7. A Github action Production pull request will be triggered, which checks that the user has permission to open a pull request. +8. Once the Production pull request workflow has completed successfully, request approval from another ROR team member in Slack. At least one approval is required before merging the pull request. +9. After the pull request has been approved, click Merge pull request. +10. Go to https://github.com/ror-community/ror-records/releases (Release tab in ror-records repository) +11. Click Draft new release at right +12. Click the Choose a tag dropdown and enter the version number for the release, ex v1.3. This should be the same number as the release branch and the directory that the release files are located in. Click Create new tag X.X on publish. +13. In the Release name field, enter ```vX.X``` (replace X.X with the release tag number) +14. In the Dsecribe this release field, enter ```Includes updates listed in https://github.com/ror-community/ror-records/milestone/X``` (link to correct milestone for this release) +15. Click Publish release +16. A Github action [Deploy to prod v2 only](https://github.com/ror-community/ror-records/actions/workflows/main_release.yml) will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production v2 ROR Elasticsearch index. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org ## Test v2 Production release @@ -349,7 +350,7 @@ Choose several new, updated and unchanged records and, for each record: 1. Check that the record can be retrieved from the Production API and the results are as expected. - curl https://api.ror.org/organizations/[RORID] + curl https://api.ror.org/v2/organizations/[RORID] 2. Check that the record can be retrieved from the Production API and the results are as expected. @@ -357,7 +358,7 @@ Choose several new, updated and unchanged records and, for each record: 3. Check that the record can be searched by name in the Production API (make sure to [escape spaces and reserved characters](https://ror.readme.io/docs/rest-api#reserved-characters)) and the results are as expected. - curl https://api.ror.org/organizations?query=[STAGING%20RECORD%20NAME] + curl https://api.ror.org/v2/organizations?query=[STAGING%20RECORD%20NAME] 4. Check that the record can be searched by name in the Production UI and the results are as expected. @@ -365,7 +366,15 @@ Choose several new, updated and unchanged records and, for each record: ## Deploy v1 to Production +v1 deployment to api.ror.org is done via a full reindex from a data dump. This means that a data dump must be generated in ror-data before v1 can be indexed. + +1. Go to https://github.com/ror-community/ror-records/actions/workflows/prod_index_dump.yml (Actions > PROD index full data dump in the ror-records repository) +2. Click Run workflow at right, set the branch to Main, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. +3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.ror.org/v1/organizations to see whether the dump was indexed. + ## Test v1 Production release +Perform API tests above, using v1 base URL https://api.ror.org/v1/organizations . v1 is not available in the UI. # Publish public data dump @@ -379,8 +388,6 @@ Choose several new, updated and unchanged records and, for each record: - **Records updated**: 257 ``` -Added and updated lines are not required. These will not be included in Zenodo description text if they don't exist in the release notes. - ## Publish public data dump to Zenodo **IMPORTANT! Data dump must be created in ror-data AND release must be created in ror-updates before publishing the dump to Zenodo!** The script that publishes the dump to Zenodo uses the zip file from ror-data and information from the Github release notes to create a new version in Zenodo. @@ -403,17 +410,20 @@ Announce the production release on the following channels: 2. Close the issues in the milestone 3. Move the issues in the milestone to the "Done" column on the project board -## Manual deployment +## Manual deployment v2 only + +**This process can only be used for v2. To deploy (or re-deploy) v1, follow the deploy v1 steps above to deploy from a data dump.** -Occasionally, a 504 Gateway Timeout happens while indexing files into AWS ElasticSearch. This will cause the automatic Deploy to Staging (on PR merge) or Deploy to Prod (on release) actions to fail at the Index file step. +Occasionally, a 504 Gateway Timeout happens while indexing files into AWS ElasticSearch during automated v2 staging or production deployment. This will cause the automatic Deploy to Staging (on PR merge) or Deploy to Prod (on release) actions to fail at the Index file step. In this case, you can re-run the deployment action manually: 1. In the ror-records repository, go to [Actions > Manual deploy to Staging](https://github.com/ror-community/ror-records/actions/workflows/staging_manual_index.yml) or [Actions > Manual deploy to Prod](https://github.com/ror-community/ror-records/actions/workflows/prod_manual_index.yml), depending on which environment you need to deploy to. 2. Click **Run Workflow**, then click **Run worfklow from** and choose the main branch when deploying to production or the staging branch when deploying to staging. -3. In the **Name of the release tag that you would like to deploy**, enter the release name (ex, v1.17) -3. Click the Run workflow button -4. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. +3. In the **Schema version** dropdown, choose v2 (this indicates the version that release files are fomatted in) +4. In the **Name of the release tag that you would like to deploy**, enter the release name (ex, v1.17) +5. Click the Run workflow button +6. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. From 4c812300549a0d1fc0d9e90a98346393278080ed Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Thu, 9 May 2024 09:59:49 -0500 Subject: [PATCH 4/7] update anchor links --- README.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 2b2053c1f..9fbcd1acd 100644 --- a/README.md +++ b/README.md @@ -17,21 +17,27 @@ Test repository for developing deployment process for ROR record updates. 1. [Create new release branch](#create-new-release-branch-in-ror-records) 2. [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch-in-ror-records) 3. [Validate files in ror-records](#validate-files-in-ror-records) -4. [Deploy to Staging](#deploy-to-staging) -5. [Test Staging release](#test-staging-release) +4. [Deploy v2 to Staging](#deploy-v2-to-staging) +5. [Test v2 Staging release](#test-v2-staging-release) +6. [Create data dump](#create-data-dump) +7. [Test data dump](#test-data-dump) +7. [Deploy v1 to Staging](#deploy-v1-to-staging) +8. [Test v1 Staging release](#test-v1-staging-release) ## Release to production -1. [Deploy to Production](#deploy-to-production) -2. [Test production release](#test-production-release) -3. [Create data dump for internal use](#create-data-dump-for-internal-use) +1. [Deploy v2 to Production](#deploy-v2-to-production) +2. [Test v2 Production release](#test-v2-production-release) +3. [Deploy v1 to Production](#deploy-v1-to-production) +4. [Test v1 Production release](#test-v1-production-release) ## Publish public data dump 1. [Generate release in ror-updates Github](#generate-release-in-ror-updates-github) 2. [Publish public data dump to Zenodo](#publish-public-data-dump-to-zenodo) 3. [Announce public data dump](#announce-public-data-dump) +4, [Clean up ror-updates Github](#clean-up-ror-updates-github) ## Misc -1. [Manual deployment](#manual-deployment) +1. [Manual deployment v2 only](#manual-deployment-v2-only) # Prepare JSON files in ror-updates @@ -184,7 +190,7 @@ These steps assume that you have already [installed and configured git](https:// git checkout -b v1.3 -# Push new/updated ROR JSON files to release branch in ror-records +## Push new/updated ROR JSON files to release branch in ror-records ### Github UI 1. On your computer, place all the JSON files for the new and updated ROR records into a folder with the **exact same name** as the release branch (ex, v1.3). @@ -297,7 +303,7 @@ Choose several records from Production that were not included in the release and ```diff``` result should be nothing (no response). -## Create data dump for internal use +## Create data dump A data dump must be created in order to index v1. To create a data dump with records in both v2 and v1: 1. In the ror-records repository, go to [Actions > Create data dump](https://github.com/ror-community/ror-records/actions/workflows/generate_dump.yml) @@ -305,6 +311,9 @@ A data dump must be created in order to index v1. To create a data dump with rec 3. Click the Run workflow button 4. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump zip should now be available in ror-data. +## Test data dump +TODO: add info about running data dump tests + ## Deploy v1 to Staging v1 deployment to api.staging.ror.org is done via a full reindex from a data dump. This means that a data dump must be generated in ror-data before v1 can be indexed. @@ -405,11 +414,12 @@ Announce the production release on the following channels: - PID Forum - API Users' Group -# Clean up ror-updates Github +## Clean up ror-updates Github 1. Close the release milestone 2. Close the issues in the milestone 3. Move the issues in the milestone to the "Done" column on the project board +# Misc ## Manual deployment v2 only **This process can only be used for v2. To deploy (or re-deploy) v1, follow the deploy v1 steps above to deploy from a data dump.** From fec46f7f9c690a01a6e18bfadb0761eab3aa2991 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Tue, 14 May 2024 16:14:11 -0500 Subject: [PATCH 5/7] udpate readme formatting --- README.md | 61 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 34 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 9fbcd1acd..c69e3f3f0 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Test repository for developing deployment process for ROR record updates. 1. [Generate release in ror-updates Github](#generate-release-in-ror-updates-github) 2. [Publish public data dump to Zenodo](#publish-public-data-dump-to-zenodo) 3. [Announce public data dump](#announce-public-data-dump) -4, [Clean up ror-updates Github](#clean-up-ror-updates-github) +4. [Clean up ror-updates Github](#clean-up-ror-updates-github) ## Misc 1. [Manual deployment v2 only](#manual-deployment-v2-only) @@ -49,7 +49,7 @@ Schema-valid ROR record JSON can be generated from a CSV using the [ROR API /bul Once created, push JSON files to ror-updates in a new branch per below. ### Create new release branch in ror-updates -These steps assume that you have already [installed and configured git](https://git-scm.com/downloads) on your computer, and that you have cloned the [ror-records](https://github.com/ror-community/ror-updates) repository locally. +These steps assume that you have already [installed and configured git](https://git-scm.com/downloads) on your computer, and that you have cloned the [ror-updates](https://github.com/ror-community/ror-updates) repository locally. 1. In your local ror-updates directory, checkout the Main branch @@ -69,7 +69,8 @@ These steps assume that you have already [installed and configured git](https:// mkdir rc-v1.3-review -2. Create subdirectories ```new``` and ```updates``` inside the directory you just created. Note: Creating both directories is not required in order for validation and relationship generation actions to work. If there are only new records or only updates, you only need to create one directory. If there's no ```updates``` directory but the relatinships.csv file contains relationships to production records, the relationship generation action will create an ```updates``` directory and download production records into it. +2. Create subdirectories ```new``` and ```updates``` inside the directory you just created. +*Note: Creating both directories is not required in order for validation and relationship generation actions to work. If there are only new records or only updates, you only need to create one directory. If there's no ```updates``` directory but the relatinships.csv file contains relationships to production records, the relationship generation action will create an ```updates``` directory and download production records into it.* mkdir rc-v1.3-review/new mkdir rc-v1.3-review/updates @@ -87,17 +88,21 @@ These steps assume that you have already [installed and configured git](https:// 6. Repeat steps 1-3 if additional files need to be added to the release candidate. ## Validate files in ror-updates -JSON files for new and updated ROR records should be validated before generating relationshps to check that they comply with the ROR schema and contained properly formatted JSON. Validation is performed by a script [run_validations.py](https://github.com/ror-community/validation-suite/blob/main/run_validations.py) triggered by a Github action [Validate files](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/validate.yml +JSON files for new and updated ROR records should be validated before generating relationshps to check that they comply with the ROR schema and contained properly formatted JSON. Validation is performed by a script [run_validations.py](https://github.com/ror-community/validation-suite/blob/main/run_validations.py) triggered by a Github action [Validate files](https://github.com/ror-community/ror-updates/actions/workflows/validate.yml ). 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/validate.yml (Actions > Create relationships in the ror-updates repository) -2. Click Run workflow at right, choose the schema version and branch you're working on, check the "Check box to skip Geonames validation" box and leave "Check box to validate relationships" and "_unchecked_. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. -3. Click the green Run workflow button. -4. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` -5. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to your working branch and repeat steps 1-3 to re-run the Validate files workflow. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch rc-vX.X + - **Schema version:** v2 + - **Check box to skip Geonames validation:** Checked + - **Check box to validate relationships:** Unchecked + - **Name of parent directory containing records to validate:** Blank, unless files are located in a directory with a different name from the release branch name +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. +4. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to your working branch and repeat steps 1-3 to re-run the Validate files workflow. ## Generate relationships -Relationships are not included in the intitial ROR record JSON files. Relationships are generated using a script [generaterelationships.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/generate_relationships/generate_relationships.py) triggered by a Github action [Create relationships](https://github.com/ror-community/ror-updates/blob/staging/.github/workflows/generate_relationships.yml), which should be run AFTER all new and updated JSON records to be included in the release are uploaded to ror-updates. +Relationships are not included in the intitial ROR record JSON files. Relationships are generated using a script [generaterelationships.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/generate_relationships/generate_relationships.py) triggered by a Github action [Create relationships](https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml), which should be run AFTER all new and updated JSON records to be included in the release are uploaded to ror-updates. 1. Create relationships list as a CSV file using the template [[TEMPLATE] relationships.csv](https://docs.google.com/spreadsheets/d/17rA549Q6Vc-YyH8WUtXUOvsAROwCDmt1vy4Rjce-ELs) and name the file relationships.csv. **IMPORTANT! File must be named relationships.csv and fields used by the script must be formatted correctly**. Template fields used by the script are: @@ -106,7 +111,7 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh |-----------------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------------| | Record ID | ROR ID of record being added/updated, in URL form | https://ror.org/015m7w34 | | Related ID | ROR ID of the related record, in URL form | https://ror.org/02baj6743 | -| Relationship of Related ID to Record ID | One the following values: Parent, Child, Related | Parent, Delete (case insensitive) | +| Relationship of Related ID to Record ID | One the following values: Parent, Child, Related, Delete (case insensitive) | Parent | | Name of org in Related ID | Name of the related organization, as it appears in the name field of the related ROR record | Indiana University – Purdue University Columbus | | Current location of Related ID | Production or Release branch (not required for Delete relationship) | Production | @@ -120,8 +125,12 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh git push origin rc-v1.3-review 4. Go to https://github.com/ror-community/ror-updates/actions/workflows/generate_relationships.yml (Actions > Create relationships in the ror-updates repository) -5. Click Run workflow at right, choose the schema version and branch that you just pushed relationships.csv to, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -6. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +5. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch rc-vX.X + - **Schema version:** v2 + - **Name of parent directory containing records to generate relationships for:** Blank, unless files are located in a directory with a different name from the release branch name + +6. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 7. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the rc-vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. ## Remove relationships to inactive records @@ -132,21 +141,21 @@ Relationships to inactive records are removed using a script [remove_relationshi 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/remove_relationships.yml (Actions > Remove relationships to inactive records in the ror-updates repository) 2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to validate has a different name from the branch you're working on, enter name of that directory. -3. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Update labels in relationships -When the ROR display name changes in a release record, relationships labels must be updated in any related records. Records are checked for ROR display name changes and relationship labels in related records are updated using a script [update_related.py](https://github.com/ror-community/curation_ops/blob/main/update_related_records/update_related.py) triggered by a Github action [Update labels in relationships](https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml), which must be run AFTER the Remove relationships to inactive records action. +When the ROR display name changes in a release record, relationship labels must be updated in any related records. Records are checked for ROR display name changes and relationship labels in related records are updated using a script [update_related.py](https://github.com/ror-community/curation_ops/blob/main/update_related_records/update_related.py) triggered by a Github action [Update labels in relationships](https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml), which must be run AFTER the Remove relationships to inactive records action. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml (Actions > Update labels in related records in the ror-updates repository) 2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -3. It will take a few minutes for the workflow to run. If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Update locations All release records must have their locations.geonames_details checked against the Geonames API and updated with the latest Geonames data if any differences exist. Locations are updated using a script [update_addresses.py](https://github.com/ror-community/curation_ops/blob/main/update_address_only/update_addresses.py) triggered by a Github action [Update addresses](https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml). This action should be run after all changes to release files are complete, except last modified date updates. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml (Actions > Update addresses in the ror-updates repository) 2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -3. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Update last modified dates All release records must have their last modified date updated to match the (planned) date of the release. Ideally, this date should match the data dump file date. Release file generation is often completed 1 or more days before the release is actually deployed. In that case, the planned release date should be used (not the current date). @@ -155,7 +164,7 @@ Last modified dates are updated using a script [update_last_mod.py](https://gith 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml(Actions > Update last modified date in the ror-updates repository) 2. Click Run workflow at right, choose the rc-vX.X branch, enter the planned release date in the "Date value to populate last_modified field with (YYYY-MM-DD)" field and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. -3. It will take a few minutes for the workflow to run (can be longer for large releases or if the Geonames API is disagreeable). If sucessful, the workflow will update ROR record JSON files in the branch that you specified, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Validate files again in ror-updates After all record updates above are complete, record files should be validated again to ensure that the changes applied by the script are still schema-valid. During this validation run, relationships and locations should also be validated. @@ -223,7 +232,7 @@ Before finalizing a release candidate, JSON files for new and updated ROR record 1. Go to https://github.com/ror-community/ror-records/actions/workflows/validate.yml (Actions > Create relationships in the ror-records repository) 2. Click Run workflow at right, choose the schema version and current vX.X branch, tick "Check box to validate relationships", leave "Check box to skip Geonames validation" unchecked, and click the green Run workflow button. -3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to the vX.X branch and repeat steps 1-3 to re-run the Validate files workflow. **IMPORTANT! Validate files workflow must succeed before proceeding to deployment** @@ -244,9 +253,9 @@ v1 must be deployed separately, after data dump is created. 4. Click the Compare dropdown and choose the vX.X branch that you have been working with in the previous steps. 5. Click Create pull request and enter ```Merge vX.X to staging``` in the Title field. Leave the Comments field blank. 6. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request -7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. +7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 8. Once the Staging pull request workflow has completed successfully, click Merge pull request -9. A Github action [Deploy to staging v2 only](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the v2 ROR Elasticsearch index. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org/v2 +9. A Github action [Deploy to staging v2 only](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org/v2 ### Multiple staging releases If records needed to be added or changed after an initial Staging release, add the new/updated records to the existing release branch per [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch) and repeat the steps to [Generate relationships](#generate-relationships), [Validate files](#validate-files) and [Deploy to v2 Staging](#deploy-to-v2-staging). A few points to note with multiple Staging releases: @@ -308,8 +317,7 @@ A data dump must be created in order to index v1. To create a data dump with rec 1. In the ror-records repository, go to [Actions > Create data dump](https://github.com/ror-community/ror-records/actions/workflows/generate_dump.yml) 2. Click Run Workflow, select the schema version that the release was curated in (should be v2), enter the name of the release directory (ex, v1.46) and enter the name of the last production data dump from ror-data that the new dump should be built from, ex 2021-09-23-ror-data (without the .zip file extension) -3. Click the Run workflow button -4. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump zip should now be available in ror-data. +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Test data dump TODO: add info about running data dump tests @@ -319,7 +327,7 @@ v1 deployment to api.staging.ror.org is done via a full reindex from a data dump 1. Go to https://github.com/ror-community/ror-records/actions/workflows/staging_index_dump.yml (Actions > STAGING index full data dump in the ror-records repository) 2. Click Run workflow at right, set the branch to Staging, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. -3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.staging.ror.org/v1/organizations to see whether the dump was indexed. ## Test v1 Staging release @@ -351,7 +359,7 @@ Deploying v2 to ror.org/search and api.ror.org requires making a Github pull req 13. In the Release name field, enter ```vX.X``` (replace X.X with the release tag number) 14. In the Dsecribe this release field, enter ```Includes updates listed in https://github.com/ror-community/ror-records/milestone/X``` (link to correct milestone for this release) 15. Click Publish release -16. A Github action [Deploy to prod v2 only](https://github.com/ror-community/ror-records/actions/workflows/main_release.yml) will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production v2 ROR Elasticsearch index. If sucessful, a green checkbox will be shown in the pull request details, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org +16. A Github action [Deploy to prod v2 only](https://github.com/ror-community/ror-records/actions/workflows/main_release.yml) will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org/v2/organizations ## Test v2 Production release @@ -379,7 +387,7 @@ v1 deployment to api.ror.org is done via a full reindex from a data dump. This m 1. Go to https://github.com/ror-community/ror-records/actions/workflows/prod_index_dump.yml (Actions > PROD index full data dump in the ror-records repository) 2. Click Run workflow at right, set the branch to Main, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. -3. It will take a few minutes for the workflow to run. If sucessful, a green checkbox will be shown on the workflow runs list in Github, and a success messages will be posted to the #ror-curation-releases Slack channel. If the workflow run is unsuccessful, an red X will be shown on the workflow runs list in Github and an error message will be posted to Slack. To see the error details, click the action run URL in the Slack message, ex `Please check: https://github.com/ror-community/dev-ror-records/actions/runs/8978212832 ` +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.ror.org/v1/organizations to see whether the dump was indexed. ## Test v1 Production release @@ -403,8 +411,7 @@ The script that publishes the dump to Zenodo uses the zip file from ror-data and 1. In the ror-records repository, go to [Actions > Publish data dump to Zenodo](https://github.com/ror-community/ror-records/actions/workflows/publish_dump_zenodo.yml) 2. Click Run Workflow and enter the release name (ex, v1.17). Leave Zenodo environment and Parent Zenodo Record ID set to the defaults. -3. Click the Run workflow button -4. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. The new data dump should now be available in https://zenodo.org/communities/ror-data. The DOI of the new upload will be shown in the "execute script" step of the Action run output in Github. +3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel.communities/ror-data. The DOI of the new upload will be shown in the "execute script" step of the Action run output in Github. ## Announce production release Announce the production release on the following channels: From 47de678fbe0de497dd29bdeb2dc0256b9a03b097 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Tue, 14 May 2024 16:41:02 -0500 Subject: [PATCH 6/7] udpate readme formatting --- README.md | 77 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 61 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index c69e3f3f0..c5138e378 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,7 @@ JSON files for new and updated ROR records should be validated before generating - **Schema version:** v2 - **Check box to skip Geonames validation:** Checked - **Check box to validate relationships:** Unchecked - - **Name of parent directory containing records to validate:** Blank, unless files are located in a directory with a different name from the release branch name + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to your working branch and repeat steps 1-3 to re-run the Validate files workflow. @@ -128,7 +128,7 @@ Relationships are not included in the intitial ROR record JSON files. Relationsh 5. Click Run workflow at right and set the following options: - **Use workflow from:** Branch rc-vX.X - **Schema version:** v2 - - **Name of parent directory containing records to generate relationships for:** Blank, unless files are located in a directory with a different name from the release branch name + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name 6. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 7. If this workflow fails, there's likely a mistake in the relationships.csv or one or more of the ROR record JSON files that needs to be corrected. In that case, make the needed corrections, commit and push the files to the rc-vX.X branch and repeat steps 3-5 to re-run the Create relationships workflow. @@ -147,14 +147,22 @@ Relationships to inactive records are removed using a script [remove_relationshi When the ROR display name changes in a release record, relationship labels must be updated in any related records. Records are checked for ROR display name changes and relationship labels in related records are updated using a script [update_related.py](https://github.com/ror-community/curation_ops/blob/main/update_related_records/update_related.py) triggered by a Github action [Update labels in relationships](https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml), which must be run AFTER the Remove relationships to inactive records action. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_labels.yml (Actions > Update labels in related records in the ror-updates repository) -2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch rc-vX.X + - **Schema version:** v2 + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Update locations All release records must have their locations.geonames_details checked against the Geonames API and updated with the latest Geonames data if any differences exist. Locations are updated using a script [update_addresses.py](https://github.com/ror-community/curation_ops/blob/main/update_address_only/update_addresses.py) triggered by a Github action [Update addresses](https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml). This action should be run after all changes to release files are complete, except last modified date updates. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_addresses.yml (Actions > Update addresses in the ror-updates repository) -2. Click Run workflow at right, choose the schema version and the rc-vX.X branch, and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch rc-vX.X + - **Schema version:** v2 + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Update last modified dates @@ -163,13 +171,22 @@ All release records must have their last modified date updated to match the (pla Last modified dates are updated using a script [update_last_mod.py](https://github.com/ror-community/curation_ops/blob/v2-crosswalk/utilities/update_last_mod/update_last_mod.py) triggered by a Github action [Update last modified date](https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml). This action should be run after all other changes to release files are complete. 1. Go to https://github.com/ror-community/ror-updates/actions/workflows/update_last_mod.yml(Actions > Update last modified date in the ror-updates repository) -2. Click Run workflow at right, choose the rc-vX.X branch, enter the planned release date in the "Date value to populate last_modified field with (YYYY-MM-DD)" field and click the green Run workflow button. If the directory containing the files you'd like to update has a different name from the branch you're working on, enter name of that directory. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch rc-vX.X + - **Date value to populate last_modified field with:** Planned release date (YYYY-MM-DD) + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Validate files again in ror-updates After all record updates above are complete, record files should be validated again to ensure that the changes applied by the script are still schema-valid. During this validation run, relationships and locations should also be validated. -Follow the steps above to run validation, but make sure to tick the "Check box to validate relationships" checkbox and leave the "Check box to skip Geonames validation" box unchecked. +Follow the steps above to run validation, but with Geonames and relationship validation: + - **Use workflow from:** Branch rc-vX.X + - **Schema version:** v2 + - **Check box to skip Geonames validation:** Unchecked + - **Check box to validate relationships:** Checked + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name # Release to Staging @@ -231,7 +248,13 @@ Before finalizing a release candidate, JSON files for new and updated ROR record ), which should be run after all new and updated JSON records to be included in the release are uploaded to the vX.X branch. 1. Go to https://github.com/ror-community/ror-records/actions/workflows/validate.yml (Actions > Create relationships in the ror-records repository) -2. Click Run workflow at right, choose the schema version and current vX.X branch, tick "Check box to validate relationships", leave "Check box to skip Geonames validation" unchecked, and click the green Run workflow button. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch vX.X + - **Schema version:** v2 + - **Check box to skip Geonames validation:** Unchecked + - **Check box to validate relationships:** Checked + - **Name of parent directory:** Blank, unless files are located in a directory with a different name from the release branch name + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, there's an issue with the data in one of more ROR record JSON files that needs to be corrected. In that case, check the error details, make the needed corrections, commit and push the files to the vX.X branch and repeat steps 1-3 to re-run the Validate files workflow. @@ -316,7 +339,13 @@ Choose several records from Production that were not included in the release and A data dump must be created in order to index v1. To create a data dump with records in both v2 and v1: 1. In the ror-records repository, go to [Actions > Create data dump](https://github.com/ror-community/ror-records/actions/workflows/generate_dump.yml) -2. Click Run Workflow, select the schema version that the release was curated in (should be v2), enter the name of the release directory (ex, v1.46) and enter the name of the last production data dump from ror-data that the new dump should be built from, ex 2021-09-23-ror-data (without the .zip file extension) +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch Staging (Main is also OK - doesn't matter) + - **Schema version:** v2 + - **Name of the directory that the new release is located in:** vX.X + - **Check box to validate relationships:** Checked + - **Name of the existing release zip file to base this data dump from:** Name of the last production data dump from ror-data that the new dump should be built from, ex 2021-09-23-ror-data (without the .zip file extension) + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. ## Test data dump @@ -326,7 +355,12 @@ TODO: add info about running data dump tests v1 deployment to api.staging.ror.org is done via a full reindex from a data dump. This means that a data dump must be generated in ror-data before v1 can be indexed. 1. Go to https://github.com/ror-community/ror-records/actions/workflows/staging_index_dump.yml (Actions > STAGING index full data dump in the ror-records repository) -2. Click Run workflow at right, set the branch to Staging, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch Staging (will fail if any other branch selected) + - **Name of existing release dump file to index:** Name of the dump file to index, ex 2021-09-23-ror-data (without the .zip file extension). Must exist in ror-data. + - **Schema version to index:** v1 + - **ROR data env:** prod (uses dump from ror-data) + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.staging.ror.org/v1/organizations to see whether the dump was indexed. @@ -386,7 +420,12 @@ Choose several new, updated and unchanged records and, for each record: v1 deployment to api.ror.org is done via a full reindex from a data dump. This means that a data dump must be generated in ror-data before v1 can be indexed. 1. Go to https://github.com/ror-community/ror-records/actions/workflows/prod_index_dump.yml (Actions > PROD index full data dump in the ror-records repository) -2. Click Run workflow at right, set the branch to Main, enter the name of the dump file to index from (must exist in ror-data repo), set "Schema version to index" to v2, set "ROR data env" to prod, and click the green Run workflow button. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch Main (will fail if any other branch selected) + - **Name of existing release dump file to index:** Name of the dump file to index, ex 2021-09-23-ror-data (without the .zip file extension). Must exist in ror-data. + - **Schema version to index:** v1 + - **ROR data env:** prod (uses dump from ror-data) + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. 4. If this workflow fails, most likely there was a timeout in the API server. In this case, the file was likely successfully indexed; the server connection just timed out before indexing completed. Check https://api.ror.org/v1/organizations to see whether the dump was indexed. @@ -410,7 +449,12 @@ Perform API tests above, using v1 base URL https://api.ror.org/v1/organizations The script that publishes the dump to Zenodo uses the zip file from ror-data and information from the Github release notes to create a new version in Zenodo. 1. In the ror-records repository, go to [Actions > Publish data dump to Zenodo](https://github.com/ror-community/ror-records/actions/workflows/publish_dump_zenodo.yml) -2. Click Run Workflow and enter the release name (ex, v1.17). Leave Zenodo environment and Parent Zenodo Record ID set to the defaults. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Branch Main + - **Release version:** vX.X + - **Zenodo environment:** Prod + - **Parent Zenodo record ID:** 6347575 + 3. Click the green Run workflow button. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel.communities/ror-data. The DOI of the new upload will be shown in the "execute script" step of the Action run output in Github. ## Announce production release @@ -436,11 +480,12 @@ Occasionally, a 504 Gateway Timeout happens while indexing files into AWS Elasti In this case, you can re-run the deployment action manually: 1. In the ror-records repository, go to [Actions > Manual deploy to Staging](https://github.com/ror-community/ror-records/actions/workflows/staging_manual_index.yml) or [Actions > Manual deploy to Prod](https://github.com/ror-community/ror-records/actions/workflows/prod_manual_index.yml), depending on which environment you need to deploy to. -2. Click **Run Workflow**, then click **Run worfklow from** and choose the main branch when deploying to production or the staging branch when deploying to staging. -3. In the **Schema version** dropdown, choose v2 (this indicates the version that release files are fomatted in) -4. In the **Name of the release tag that you would like to deploy**, enter the release name (ex, v1.17) -5. Click the Run workflow button -6. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. +2. Click Run workflow at right and set the following options: + - **Use workflow from:** Main branch when deploying to production or the staging branch when deploying to staging. + - **Schema version to index:** v2 + - **Name of the release tag that you would like to deploy (prod) OR Name of the directory you would like to deploy (staging):** vX.X + +5. Click the Run workflow button. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. From 43bfe6841faa58203a5553e342f5347019042a15 Mon Sep 17 00:00:00 2001 From: lizkrznarich Date: Tue, 14 May 2024 17:04:44 -0500 Subject: [PATCH 7/7] udpate readme formatting --- README.md | 53 ++++++++++++++++++++++++++++------------------------- 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index c5138e378..ced1fdfca 100644 --- a/README.md +++ b/README.md @@ -182,6 +182,7 @@ Last modified dates are updated using a script [update_last_mod.py](https://gith After all record updates above are complete, record files should be validated again to ensure that the changes applied by the script are still schema-valid. During this validation run, relationships and locations should also be validated. Follow the steps above to run validation, but with Geonames and relationship validation: +f - **Use workflow from:** Branch rc-vX.X - **Schema version:** v2 - **Check box to skip Geonames validation:** Unchecked @@ -271,18 +272,19 @@ v1 must be deployed separately, after data dump is created. *Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* 1. Go to https://github.com/ror-community/ror-records/pulls (Pull requests tab in ror-records repository) -2. Click New pull request at right -3. Click the Base dropdown at left and choose Staging. Important! Do not make a pull request against the default Main branch. -4. Click the Compare dropdown and choose the vX.X branch that you have been working with in the previous steps. -5. Click Create pull request and enter ```Merge vX.X to staging``` in the Title field. Leave the Comments field blank. -6. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request -7. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. -8. Once the Staging pull request workflow has completed successfully, click Merge pull request -9. A Github action [Deploy to staging v2 only](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org/v2 +2. Click New pull request at right and set the following options: + - Base: Staging (Important! Do not make a pull request against the default Main branch) + - Compare: vX.X branch +3. Click Create pull request and enter ```Merge vX.X to staging``` in the Title field. Leave the Comments field blank. +4. Double-check that the Base dropdown is set to Staging and that the list of updated files appears to be correct, then click Create pull request +5. A Github action [Staging pull request](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/staging_pull_request.yml) will be triggered which (1) verifies that the user is allowed to perform a release to staging and (2) runs the file validation script again. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. +6. Once the Staging pull request workflow has completed successfully, click Merge pull request +7. A Github action [Deploy to staging v2 only](https://github.com/ror-community/ror-records/blob/staging/.github/workflows/merge.yml) will be triggered, which pushes the new and updated JSON files from the vX.X directory to AWS S3 and indexes the data into the v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://staging.ror.org/search and https://api.staging.ror.org/v2 ### Multiple staging releases -If records needed to be added or changed after an initial Staging release, add the new/updated records to the existing release branch per [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch) and repeat the steps to [Generate relationships](#generate-relationships), [Validate files](#validate-files) and [Deploy to v2 Staging](#deploy-to-v2-staging). A few points to note with multiple Staging releases: +If records needed to be added or changed after an initial Staging release, add the new/updated records to the existing release branch per [Push new/updated ROR JSON files to release branch](#push-newupdated-ror-json-files-to-release-branch) and repeat the steps to [Deploy to v2 Staging](#deploy-to-v2-staging). A few points to note with multiple Staging releases: +- Relationship generation/removal, label updates, addresses updates and last modified date udpates may need to be re-run in ror-updates, depending on changes. - Do not remove records from the release branch that have already been deployed to Staging. Overwrite any records already deployed to Staging that require changes and leave the rest as they are. When ready to deploy to poduction, the release branch should contain all new/updated records to be included in the production release. - Include relationships for any records already in merged to the vX.X directory on Staging in the release branch relationships.csv (not just the current deployment) - Deleting record files from the release branch after they have been deployed to Staging will not remove them from the Staging index. At the moment, this will need to be done manually by a developer; in the future, we will add a mechanism to remove records from the Staging index that have been deleted from an release branch. This does not affect production, as the production index is completely separate. @@ -379,21 +381,22 @@ Deploying v2 to ror.org/search and api.ror.org requires making a Github pull req *Note that only specific Github users (ROR staff) are allowed to open/merge pull requests and create releases.* 1. Go to https://github.com/ror-community/ror-records/pulls (Pull requests tab in ror-records repository) -2. Click New pull request at right -3. Click the Base dropdown at left and choose Main. -4. Click the Compare dropdown and choose Staging. -5. Click Create pull request, enter ```Merge vX.X to production``` in the Title field. Leave the Comments field blank. -6. Double-check that the Base dropdown is set to Main and that the list of updated files appears to be correct, then click Create pull request -7. A Github action Production pull request will be triggered, which checks that the user has permission to open a pull request. -8. Once the Production pull request workflow has completed successfully, request approval from another ROR team member in Slack. At least one approval is required before merging the pull request. -9. After the pull request has been approved, click Merge pull request. -10. Go to https://github.com/ror-community/ror-records/releases (Release tab in ror-records repository) -11. Click Draft new release at right -12. Click the Choose a tag dropdown and enter the version number for the release, ex v1.3. This should be the same number as the release branch and the directory that the release files are located in. Click Create new tag X.X on publish. -13. In the Release name field, enter ```vX.X``` (replace X.X with the release tag number) -14. In the Dsecribe this release field, enter ```Includes updates listed in https://github.com/ror-community/ror-records/milestone/X``` (link to correct milestone for this release) -15. Click Publish release -16. A Github action [Deploy to prod v2 only](https://github.com/ror-community/ror-records/actions/workflows/main_release.yml) will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org/v2/organizations +2. Click New pull request at right and set the following options: + - Base: Main + - Compare: Staging + +3. Click Create pull request, enter ```Merge vX.X to production``` in the Title field. Leave the Comments field blank. +4. Double-check that the Base dropdown is set to Main and that the list of updated files appears to be correct, then click Create pull request +5. A Github action Production pull request will be triggered, which checks that the user has permission to open a pull request. +6. Once the Production pull request workflow has completed successfully, request approval from another ROR team member in Slack. At least one approval is required before merging the pull request. +7. After the pull request has been approved, click Merge pull request. +8. Go to https://github.com/ror-community/ror-records/releases (Release tab in ror-records repository) +9. Click Draft new release at right +10. Click the Choose a tag dropdown and enter the version number for the release, ex v1.3. This should be the same number as the release branch and the directory that the release files are located in. Click Create new tag X.X on publish. +11. In the Release name field, enter ```vX.X``` (replace X.X with the release tag number) +12. In the Dsecribe this release field, enter ```Includes updates listed in https://github.com/ror-community/ror-records/milestone/X``` (link to correct milestone for this release) +13. Click Publish release +14. A Github action [Deploy to prod v2 only](https://github.com/ror-community/ror-records/actions/workflows/main_release.yml) will be triggered, which pushes the new and updated JSON files to AWS S3, indexes the data into the production v2 ROR Elasticsearch index. Success/error status will be shown in Github actions and posted to the #ror-curation-releases Slack channel. The new data should now be available in https://ror.org/search and https://api.ror.org/v2/organizations ## Test v2 Production release @@ -485,7 +488,7 @@ In this case, you can re-run the deployment action manually: - **Schema version to index:** v2 - **Name of the release tag that you would like to deploy (prod) OR Name of the directory you would like to deploy (staging):** vX.X -5. Click the Run workflow button. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel. +3. Click the Run workflow button. If sucessful, a green checkbox will be shown in the Actions history, and a success messages will be posted to the #ror-curation-releases Slack channel.