Fix: Optimistic lock contention on HCA replicas (#6648)

[nadove-ucsc]

Connected issues: #6648

Checklist

Author

• PR is a draft
• Target branch is develop
• Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
• On ZenHub, PR is connected to all issues it (partially) resolves
• PR description links to connected issues
• PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
• PR title references all connected issues
• For each connected issue, there is at least one commit whose title references that issue

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

• Added p tag to titles of partial commits
• This PR is labeled partial _{or completely resolves all connected issues}
• This PR partially resolves each of the connected issues _{or does not have the partial label}

Author (chains)

• This PR is blocked by previous PR in the chain _{or is not chained to another PR}
• The blocking PR is labeled base _{or this PR is not chained to another PR}
• This PR is labeled chained _{or is not chained to another PR}

Author (reindex, API changes)

• Added r tag to commit title _{or the changes introduced by this PR will not require reindexing of any deployment}
• This PR is labeled reindex:dev _{or the changes introduced by it will not require reindexing of dev}
• This PR is labeled reindex:anvildev _{or the changes introduced by it will not require reindexing of anvildev}
• This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
• This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
• This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod _{or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod}
• This PR and its connected issues are labeled API _{or this PR does not modify a REST API}
• Added a (A) tag to commit title for backwards (in)compatible changes _{or this PR does not modify a REST API}
• Updated REST API version number in app.py _{or this PR does not modify a REST API}

Author (upgrading deployments)

• Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (hotfixes)

• Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
• Reverted the temporary hotfixes for any connected issues _{or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR}

Author (before every review)

• Rebased PR branch on develop, squashed old fixups
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

Peer reviewer (after approval)

• PR is not a draft
• Ticket is in Review requested column
• PR is awaiting requested review from system administrator
• PR is assigned to only the system administrator

System administrator (after approval)

• Actually approved the PR
• Labeled connected issues as demo or no demo
• Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
• Decided if PR can be labeled no sandbox
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Moved connected issues to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Checked reindex:… labels and r commit title tag
• Checked that demo expectations are clear _{or all connected issues are labeled no demo}
• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator _{or this PR is not labeled deploy:gitlab}

System administrator

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Added sandbox label _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab dev _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab anvildev _{or PR is labeled no sandbox}
• Build passes in sandbox deployment _{or PR is labeled no sandbox}
• Build passes in anvilbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in sandbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in anvilbox deployment _{or PR is labeled no sandbox}
• Deleted unreferenced indices in sandbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in dev}
• Deleted unreferenced indices in anvilbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev}
• Started reindex in sandbox _{or this PR is not labeled reindex:dev}
• Started reindex in anvilbox _{or this PR is not labeled reindex:anvildev}
• Checked for failures in sandbox _{or this PR is not labeled reindex:dev}
• Checked for failures in anvilbox _{or this PR is not labeled reindex:anvildev}
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but only included p if the PR is also labeled partial}
• Moved connected issues to Merged lower column in ZenHub
• Moved blocked issues to Triage _{or no issues are blocked on the connected issues}
• Pushed merge commit to GitHub

Operator (chain shortening)

• Changed the target branch of the blocked PR to develop _{or this PR is not labeled base}
• Removed the chained label from the blocked PR _{or this PR is not labeled base}
• Removed the blocking relationship from the blocked PR _{or this PR is not labeled base}
• Removed the base label from this PR _{or this PR is not labeled base}

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev

Operator (reindex)

• Deindexed all unreferenced catalogs in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed all unreferenced catalogs in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Deindexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Indexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Indexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Started reindex in dev _{or this PR does not require reindexing dev}
• Started reindex in anvildev _{or this PR does not require reindexing anvildev}
• Checked for, triaged and possibly requeued messages in both fail queues in dev _{or this PR does not require reindexing dev}
• Checked for, triaged and possibly requeued messages in both fail queues in anvildev _{or this PR does not require reindexing anvildev}
• Emptied fail queues in dev _{or this PR does not require reindexing dev}
• Emptied fail queues in anvildev _{or this PR does not require reindexing anvildev}

Operator

• Propagated the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[coveralls]

coverage: 85.613%. remained the same
when pulling 87b9140 on issues/nadove-ucsc/6648-optimistic-lock-contention-hca-replicas
into 17dadfa on develop.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.60%. Comparing base (56c9ed8) to head (d0b59af).

Additional details and impacted files

@@                               Coverage Diff                                @@
##           issues/nadove-ucsc/6691-leftovers-orphan-prs    #6693      +/-   ##
================================================================================
+ Coverage                                         85.58%   85.60%   +0.01%     
================================================================================
  Files                                               155      155              
  Lines                                             20916    20940      +24     
================================================================================
+ Hits                                              17901    17925      +24     
  Misses                                             3015     3015              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.