Skip to content

Commit

Permalink
backmerge master to feature/APPEALS-19847 (#19358)
Browse files Browse the repository at this point in the history
* hid pact and mst check behind toggles

* added caching to BGS contention call

* updated feature toggles to include current user

* adding logging for BGS contention calls

* Added toggle to index.html.erb

* Progress on conference types radio field

* added 24 hr caching and commented out toggle for bgs call

* adding conference type selection logic

* APPEALS-25141: created migrations to add pexip bool column to users virtual hearings and conference links

* APPEALS-25141: set default value to true on users

* Work on radio field selection

* Fixed errors with conference selection radio field

* laurenoelle/APPEALS-19871 initial files

* APPEALS-17497: MST & PACT Special issues identification

* Clean up code after new import file

* optimized participant contentions method to return contentions to cut down on BGS calls

* added feature toggle guard clause for bgs calls for ratings

* hid pact and mst check behind toggles

* added caching to BGS contention call

* updated feature toggles to include current user

* adding logging for BGS contention calls

* added 24 hr caching and commented out toggle for bgs call

* Update copy file and state

* hid blue water and burn pit behind mst/pact toggle

* APPEALS-25112 Added styling for radio field component

* Fixing error with .pick

* Fixing .pick

* APPEALS-25130 updated user.rb

* APPEALS-25141: rolling back changes, creating new migrations, deleting the old, to meeting type varchar default on users only

* added toggle for mst on special issues list

* APPEALS-25141: removed version 5.2 from migrations

* routed judge/attorney decisions to special issues pending on mst pact toggle

* updated back button routing and special issues showing on decisions for ama appeals

* hid mst pact special issues banner behind feature toggles

* Account for failures in ready for resync

* HunJerBAH APPEALS-26881: Reject Unknown Attributes in RatingDecisions Hash (#19072)

* added handling for unknown attributes being added into rating decision hash

* updated paren grouping

* Update open_hearing_tasks_without_active_descendants_checker_spec.rb

* Adding a rails initializer for monitoring deprecation warnings

* Create jest test

* allow slack notifications in uat

* Test passing

* Fixed syntax errors from merged code

* Radio buttons fixed to only display on Hearings Admin page

* Wrap li elements in ul elements

* Test that radio button values change when selected

* Update to testing logic with radio buttons

* Add ro-colocated examples

* Modifying jest tests

* APPEALS-25141: refactoring migrations to include up/down methods, adding default for virt hearings and conf links

* Fix ro-colocated request body

* aaron/APPEALS-25478 (#19070)

* APPEALS-25478 Added unit tests to BatchProcessRescueJobSpec to increase code coverage to 100%.

* APPEALS-25478 Added unit tests to BatchProcessRescueJobSpec to increase code coverage to 100%.

* APPEALS-25478 Added Senty Alert when a record is declared stuck.

* APPEALS-25478 Refactered and Updated PriorityEndProductSyncQueueSpec file to have 100% complete code coverage.

* APPEALS-25478 Refactor and Linting of batch_process_priority_ep_sync_spec.rb.

* APPEALS-25478 Added to batch_process_priority_ep_sync_job_spec.rb to achieve 100% code coverage.

* APPEALS-25478 Removed unintentional SImple Cov line in RSPEC.

* APPEALS-25478 Updated batch_process_spec.rb to bring code coverage up to 100%.

* APPEALS-25478 Updated ENV variable names to ensure uniqueness.

* APPEALS-25478 Updated Error messages to include unique identifiers in batch_process_priority_ep_sync.rb.

* APPEALS-25478 Added RSPEC for the CaseflowStuckRecord model.

* APPEALS-25478 Updated RSPEC for PopulateEndProductSyncQueueJob so that it has 100% code coverage.

* APPEALS-25478 Removed table lock from find_records method.

* APPEALS-25478 Removed accidental changes to issue_spec.rb

* APPEALS-25478 Updated method naming conventions to be more explicit and readable.

* APPEALS-25478 Fixed failing RSPEC priority_end_product_sync_queue_spec.rb.

* APPEALS-25478 Added additional tests to batch_process_priority_ep_sync_spec.rb.

* APPEALS-25478 Updated Code Comments.

* APPEALS-25478 Fixed Typo on code comment.

* add exception handling to audit remove script

* Change address_line_1 to nullable: true

* Add migration

* Allow audit table entry for address_line_1 to have null val

* test update to yarn dependency cache keys

* Take setup-node for a spin

* Revert "Take setup-node for a spin"

This reverts commit 337ea0a.

* Add a spec

* Fix whitespace

* Remove flipper tables

* Styling comments updated after review

* unskip test (#19076)

Co-authored-by: Craig Reese <[email protected]>

* Revert 'Merge pull request #19038 from department-of-veterans-affairs/b_reed/APPEALS-25130-v3'

This reverts commit ab8ce3c, reversing
changes made to 6fc2c0f.

* hotfix/APPEALS-27124 Updated MailRequestValidator and VbmsDistributionDestination spec for derived destinations

* updated deserialize method to check on rating decision model for attributes in hash

* Fixed linting issues and updated jest snapshot

* Add class checks for ro-colocated?

* Craig/test yarn cache (#19096)

modify .github/workflows/workflow.yml

* exclude db/scripts in codeclimate (#19103)

* Revert "APPEALS-17497: MST & PACT Special issues identification"

* Use case

* Updated language following feedback from UX team

* remove z-index

* add z-index

* Update webdrivers

* Revert "Update webdrivers"

This reverts commit 7cd71cd.

* APPEALS-17497 MST & PACT Special issues identification changes

* Adding Subscriber to monitor deprecation warnings

* Add spec file for deprecation warnings

* Test wait

* Reverse test update

* APPEALS-27327: MST and PACT BGS Fix  (#19116)

* added grouping for feature toggles

* moved special issue deserialization for ratings into contestable issue model and added datadog incrementing for reporting

* added wildcard to datadog counters and removed redundant logging in bgs_service

* APPEALS-25141: updated user_spec.rb

* updated data dog service logging

* Delete deprecation_alerts.rb

* test adding assertion (#19127)

* test adding assertion

* added assertion to more tests

* APPEALS-26633 Added method to update meeting_type in Users Controller, and updated Organizations User model with new method to change meeting_type

* Add in FT

* APPEALS-26633 Progress on linking backend and frontend radio option changes and added modifyConferenceTypes

* Apply suggestions from code review

Co-authored-by: Jeremy Croteau <[email protected]>

* Update deprecation_warning_subscriber_spec.rb

* add concurrency to workflow.yml (#19136)

* add concurrency to workflow.yml

* Fix typo

* APPEALS-26633 Framing for logic to link onChange function

* eli/APPEALS-27096 (#19140)

* add external-db-create check to avoid error when running command

* priority_ep_sync_batch_process_job

* populate_end_product_sync_queue_job

* priority_ep_sync_batch_process

* config/environments/development ENVs

* config/environment/test ENVs

* scheduled_jobs => BatchProcessPriorityEpSync name change

* factories/vbms_ext_claim => duplicate claim_id fix

* batch_process_priority_ep_sync_job_spec

* priority_end_product_sync_queue_spec

* priority_ep_sync_batch_process_spec

* batch_process_spec

* rename batch process job spec and batch process model spec

* caseflow_stuck_records_spec

* populate_end_product_sync_queue_job_spec

* priority_ep_sync_batch_process_job_spec

* batch_process_rescue_job_spec

* APPEALS-27096 Updated Schema file.

* APPEALS-27096 Added Batch Process & PEPSQ ENV variables to demo.

* APPEALS-27096 Added migration to add columns to caseflow_stuck_records and updated schema file.

---------

Co-authored-by: Jeffrey Aaron Willis <[email protected]>

* 🐛 Pre-emptive bugfix - TypeError Exception: allocator undefined for Thread::Backtrace::Location

* 🔊 Log additional metadata to Sentry on deprecation warnings

* aaron/APPEALS-27497 (#19139)

* APPEALS-27947 Added Audit Removal scripts for table, function, and trigger.

* APPEALS-27497 Moved Audit files to appropriate locations.  Updated Makefile to reflect changes to audit file locations.

* APPEALS-27497 Fixed Indentation in Makefile.

* Updated for AC as of August 8

* commented out bgs call and adding logging for production

* Fixed spacing

* added data dog logging for fetch rating profile and ratings in range

* Adding modifyConferenceType to patch to backend

* updated contentions response to handle blank? responses in addition to nil?

* 🚨 Lint: Remove trailing whites space

c

* 🥅 Add blanket rescue to DeprecationWarningSubscriber

(Puts on tin foil hat) Prevent potential exceptions from SlackService from impacting user experience

* ✏️ Fix typo in test description

* 🧪 Fix test so that it will pass in non-'test' environments as well

* put in explicit nil return statement for contention BGS call testing.

* aaron/APPEALS-27780 (#19160)

* APPEALS-27780 Added Priority End Product Sync Queue ID to Priority End Product Sync Queue Audit table.  Updated trigger function to account for this.

* APPEALS-27780 Fixed drop_priority_end_product_sync_queue_audit_trigger.rb

* APPEALS-27780 Changed Job Duration Constants for Priority Ep Sync Batch Process Job and Populate End Product Sync Queue Job to minutes from hours.

* 🩹 Address deprecation on affected usages of 'order' method

DEPRECATION WARNING: Dangerous query method (method whose arguments are used as raw SQL) called with non-attribute argument(s)

* Eli/APPEALS-27996 (#19170)

* APPEALS-27996 => updated command 'make audit' to build tables together, then functions together, then triggers together

* APPEALS-27996 => removed whitespace at the top of create_caseflow_stuck_records migration file

* APPEALS-27996 => deleted folder 'external_modals', added folder 'external_models' and added vbms_ext_claim.rb to it

* APPEALS-27996 => added index creation to create_vbms_ext_claim_table.sql

* Update deprecation_warning_subscriber.rb

* Revert "Update deprecation_warning_subscriber.rb"

This reverts commit 50bf09f.

* use env variable for efolder express url

* add efolder url env variable to demo environmentt

* Add meeting_type to user to capture new meeting type from frontend, debugging in byebug

* Hotfix/appeals 27980 NightlySyncsJob (#19173)

* add error handling to sync_vacols_cases

* add test for sync_bgs_attorneys method

* add slack notification on job error, add tests

* codeclimate

* codeclimate

* Fixed bug so frontend hearings change is persisted to backend and updates user meeting type

* Revert "APPEALS-17497 MST & PACT Special issues identification"

* ✨ Raise error on disallowed deprecations in dev/test envs

* Fixed linting issues and added rspec test to check the new meeting type

* Add information about patch response test

* Fix code climates issues

* ✨ Replace 'DeprecationWarningSubscriber' with 'DeprecationWarnings' handlers

This will allow us to better tailor handling of deprecation warnings to each environment

* 💡 Update comment

* 🚨 Lint

* 🔥 Remove copy-pasta from pecs

* 🚨 Lint II

* 🚨 Lint III

* Jest testing fixed to check for changing values by radio button selection

* Pamatya/APPEALS-24131: Fix flaky/skipped tests in models/idt/token_spec.rb (#19208)

* removing the xit and adding new test

* Added a test for an expired key for the Idt::Token active method

* APPEALS-24131: removed unused let statement

* Removed the sleep statement from the test.

* Included a docket_spec test fix as well.

---------

Co-authored-by: = <[email protected]>

* Remove skipped and unneeded feature test (#19214)

This will be explained more in the accompanying PR but this test has
been skipped for a couple years and does not need to be in the code
base anymore as it does not add any value.

* APPEALS-24145 Fix test scenario "Assigning it to complete the claims establishment" (#19209)

* deleted a flakey hang and added an expect for the url

* Added a few more expects

* small formating changes

* removed no longer needed comment

* removed unneccesarcy expect

* Revert "laurenoelle/APPEALS-19871 initial files"

* Hearing Admin selection changed to Hearings Management

* aaron/APPEALS-28367 (#19210)

* APPEALS-28367 Added Slack Notification upon job failure on priority_ep_sync_batch_process_job.rb.  Updated RSPEC.

* APPEALS-28367 Added slack  alerts for job failures in populate end product sync queue job.  Refactored error logging and sentry alerts in populate end product sync queue job & priority ep sync batch process job.  Updated RSPEC.

* APPEALS-28367 Added Slack Alert to BatchProcessRescueJob upon failure to reprocess.  Updated RSPEC.

* APPEALS-28367 Added more negative testing for slack alerts in populate_end_product_sync_queue_job.rb.

* APPEALS-28367 Refactored batch_process.rb by adding private attr_accessors for counter instance variables.

* APPEALS-28367 Removed extranneous blank line from end_product_establishment factory.

* APPEALS-28367 Added Slack Alert negative testing to RSPEC for batch_process_rescue_job & priority_ep_sync_batch_process_job_spec.

* APPEALS-28367 Fixed linting issue for ready_to_batch scope in priority_end_product_sync_queue.rb.

* APPEALS-28367 Refactored RSPEC.

* APPEALS-28367 Removed Instance variables from RSPECs.  Updated ENV variables in test.rb

* APPEALS-28367 Updated parameter name in no-op method in batch_process.rb to resolve code climate issue.  Updated RSPEC to reflect this change.

* APPEALS-28367 Added Reek statements to address various code climate alerts.

* APPEALS-28367 Removed SLEEP DURATION Stubbing in caseflow_stuck_records to address code climate alert.

* fixing flaky test (#19231)

* fixing flaky test

* removing commented code

* fix flakey tests in login spec (#19233)

* Revert "lthompson/APPEALS-26633"

* APPEALS-28989 Added Ensure block that will always update last_synced_at date/time regardless of error to ensure that SyncReviewsJob does not immediately re-queue failing end product establishment and allows for other end product estaablishments to attempt sync with BGS first.  Updated RSPEC to reflect this change.

* Add jest tests for `CancelIntakeModal` (#19238)

* added the code needed for the test to pass (#19243)

* TYLERB/APPEALS-29085: Fix flakyness in the pre_docket_spec.rb file (#19247)

* Added a few more expect statements to the predocket spec to attempt to reduce flakyness.

* Changed the visit case details to the reload_case_details method.

* Changed all of the visit appeal uuid/external id links to the case details page to use the reload_case_detail_page to reduce flakyness.

* nkirby/APPEALS-28544-v2 (#19225)

* condense process_batch examples

* condense create_batch examples

* condense find_records_to_batch examples

* tag 21351 tests

* remove pr tags

* Appeals 28984 - fix code climate issues for feature/APPEALS-21351-merged (#19228)

* init commit - testing code climate fix

* removed method from initialize and add attt_reader

* revert changes of initalizer but added attr_reader :file_number

* removed @file_number and changed request_issue_1

* removed veteran_file_number and just added veteran

* added veteran_file_number back and kept the vet

* removed veteran,left file_number, n removed id

* changes to error string and removed sleep comment

* fixed too many lines, double quotes, and space

* long lines

* remove line to long pepsq line13 had to add lambda

* replacedw/ doublequotes & moved for alphabet order

* removed extra line

* removed sleep comment

* removed sleep comment from caseflowstuckrecordspec

* removed sleep comment populate_end_prod_sync_qu

* removed line to long and ( ) around methods

* ignore line length warn

* re-enable line length

* removed un used code

* disabled FeatureEnvy Smell

* removed comment

* Disabled InstanceVariableAssumption

* Update batch_process_rescue_job.rb

* APPEALS-28984 - uncommented claimant_participant_id & disabled FeatureEnvy for line 185

* moved comments

* add missing comma

* Add three phases to sync jobs

* Switch to standard error

* Remove old FTs

* fixed linting on sync and added test for RedisMutex lock error (#19237)

* fixed linting on sync and added test for RedisMutex lock error

* added epe id to display in error message

* fixed linting

* rubocop long line fixes

* fixed new line causing test failure

* fixed new lines

* fixed rubocop long lines

* APPEALS-28960 Updated formatting of log message and content to include "#sync!".

---------

Co-authored-by: TuckerRose <[email protected]>
Co-authored-by: Jeffrey Aaron Willis <[email protected]>

* Update MAC_M1.md

* APPEALS-29860 Fixed Indentation linting issue on sync! method. (#19268)

* Update log statement

* Fix test

* Better uniq

* clear user session after each test (#19254)

* added comment to makefile (#19305)

* added comment to makefile

* additional comment

---------

Co-authored-by: Jonathan Tsang <[email protected]>

* Remove coverage step from workflow for now

* Add back in returns

---------

Co-authored-by: HunJerBAH <[email protected]>
Co-authored-by: Matt Roth <[email protected]>
Co-authored-by: breedbah <[email protected]>
Co-authored-by: raymond-hughes <[email protected]>
Co-authored-by: 631862 <[email protected]>
Co-authored-by: Matthew Thornton <[email protected]>
Co-authored-by: Ariana Konhilas <[email protected]>
Co-authored-by: Lauren Berry <[email protected]>
Co-authored-by: youfoundmanesh <[email protected]>
Co-authored-by: jshields13 <[email protected]>
Co-authored-by: breedbah <[email protected]>
Co-authored-by: mchbidwell <[email protected]>
Co-authored-by: HunJerBAH <[email protected]>
Co-authored-by: AnandEdara <[email protected]>
Co-authored-by: kristeja <[email protected]>
Co-authored-by: kristeja <[email protected]>
Co-authored-by: (Jeffrey) Aaron Willis <[email protected]>
Co-authored-by: Craig Reese <[email protected]>
Co-authored-by: Matthew Thornton <[email protected]>
Co-authored-by: Clay Sheppard <[email protected]>
Co-authored-by: Craig Reese <[email protected]>
Co-authored-by: Jeff Marks <[email protected]>
Co-authored-by: Jeffrey Aaron Willis <[email protected]>
Co-authored-by: Jeremy Croteau <[email protected]>
Co-authored-by: Eli Brown <[email protected]>
Co-authored-by: Jeremy Croteau <[email protected]>
Co-authored-by: Lauren Berry <[email protected]>
Co-authored-by: Raymond Hughes <[email protected]>
Co-authored-by: mikefinneran <[email protected]>
Co-authored-by: sbashamoni <[email protected]>
Co-authored-by: Prajwal Amatya <[email protected]>
Co-authored-by: = <[email protected]>
Co-authored-by: Brandon Lee Dorner <[email protected]>
Co-authored-by: Sean Craig <[email protected]>
Co-authored-by: Tyler Broyles <[email protected]>
Co-authored-by: nkirby-va <[email protected]>
Co-authored-by: Enrilo Ugalde <[email protected]>
Co-authored-by: Will Love <[email protected]>
Co-authored-by: TuckerRose <[email protected]>
Co-authored-by: Jonathan Tsang <[email protected]>
  • Loading branch information
1 parent 5fdecdc commit 378b044
Show file tree
Hide file tree
Showing 104 changed files with 4,663 additions and 333 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ jobs:
run: |
./ci-bin/capture-log "DB=etl bundle exec rake db:create db:schema:load db:migrate"
./ci-bin/capture-log "bundle exec rake db:create db:schema:load db:migrate"
./ci-bin/capture-log "make -f Makefile.example external-db-create"
# added line to create external table(s) that are needed for tests

# We don't want to seed DBs here because DatabaseCleaner just truncates it anyway. The setup_vacols
# rake task needs to be run because it adds data to two tables that are ignored by DBCleaner
Expand Down
5 changes: 3 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ gem "paranoia", "~> 2.2"
gem "pdf-forms"
# Used in Caseflow Dispatch
gem "pdfjs_viewer-rails", git: "https://github.com/senny/pdfjs_viewer-rails.git", ref: "a4249eacbf70175db63b57e9f364d0a9a79e2b43"
#Used to build out PDF files on the backend
#https://github.com/pdfkit/pdfkit
# Used to build out PDF files on the backend
# https://github.com/pdfkit/pdfkit
gem "pdfkit"
gem "pg", platforms: :ruby
# Application server: Puma
Expand All @@ -61,6 +61,7 @@ gem "rails", "5.2.4.6"
gem "rainbow"
# React
gem "react_on_rails", "11.3.0"
gem "redis-mutex"
gem "redis-namespace"
gem "redis-rails", "~> 5.0.2"
gem "request_store"
Expand Down
5 changes: 5 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,10 @@ GEM
redis-activesupport (5.0.4)
activesupport (>= 3, < 6)
redis-store (>= 1.3, < 2)
redis-classy (2.4.1)
redis-namespace (~> 1.0)
redis-mutex (4.0.2)
redis-classy (~> 2.0)
redis-namespace (1.6.0)
redis (>= 3.0.4)
redis-rack (2.0.4)
Expand Down Expand Up @@ -783,6 +787,7 @@ DEPENDENCIES
rainbow
rb-readline
react_on_rails (= 11.3.0)
redis-mutex
redis-namespace
redis-rails (~> 5.0.2)
request_store
Expand Down
22 changes: 13 additions & 9 deletions MAC_M1.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ OpenSSL
---
1. Download [email protected] and openssl@3 from this [link](https://boozallen.sharepoint.com/teams/VABID/appeals/Documents/Forms/AllItems.aspx?id=%2Fteams%2FVABID%2Fappeals%2FDocuments%2FDevelopment%2FDeveloper%20Setup%20Resources%2FM1%20Mac%20Developer%20Setup&viewid=8a8eaf3e%2D2c12%2D4c87%2Db95f%2D4eab3428febd)
2. Open “Finder” and find the two folders under “Downloads”
3. Extract the `.tar.gz` files
3. Extract the `.tar.gz` or `.zip` archives
4. In each of the extracted folders:
1. Navigate to the `/usr/local/homebrew/Cellar` subfolder
2. Copy the openssl folder to your local machine's `/usr/local/homebrew/Cellar` folder
Expand Down Expand Up @@ -168,22 +168,26 @@ Run dev setup scripts in Caseflow repo
1. Open a **Rosetta** terminal and navigate to /usr/local, run the command ```sudo spctl --global-disable```
2. In the **Rosetta** terminal, install pyenv and the required python2 version:
1. `brew install pyenv`
2. `pyenv install 2.7.18`
3. In the caseflow directory, run `pyenv local 2.7.18` to set the version
2. `pyenv rehash`
3. `pyenv install 2.7.18`
4. In the caseflow directory, run `pyenv local 2.7.18` to set the version
3. In the **Rosetta** terminal navigate to caseflow folder:
1. set ```RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/homebrew/Cellar/[email protected]"```
2. run `rbenv install 2.7.3`
3. run `gem install pg:1.1.4 -- --with-pg-config=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config`
4. Install [email protected] by doing the following (these steps assume that vi/vim is the default editor):
1. set ```export RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/homebrew/Cellar/[email protected]"```
2. run `rbenv install $(cat .ruby-version)`
3. run `rbenv rehash`
4. run `gem install bundler -v $(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)`
5. run `gem install pg:1.1.4 -- --with-pg-config=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config`
6. Install [email protected] by doing the following (these steps assume that vi/vim is the default editor):
1. run `brew edit [email protected]`
2. go to line 21 in the editor by typing `:21`
Note: the line being removed is `disable! date: "2023-06-19", because: "depends on Python 2 to build"`
3. delete the line by pressing `d` twice
4. save and quit by typing `:x`
5. Configure build opts for gem `therubyracer`:
5. run `HOMEBREW_NO_INSTALL_FROM_API=1 brew install [email protected]`
7. Configure build opts for gem `therubyracer`:
1. `bundle config build.libv8 --with-system-v8`
2. `bundle config build.therubyracer --with-v8-dir=$(brew --prefix [email protected])`
6. run ```./scripts/dev_env_setup_step2.sh```
8. run ```./scripts/dev_env_setup_step2.sh```
If you get a permission error while running gem install or bundle install, **do not run using sudo.**
Set the permissions back to you for every directory under /.rbenv
* Enter command: `sudo chown -R <your name under /Users> /Users/<your name>/.rbenv`
Expand Down
31 changes: 30 additions & 1 deletion Makefile.example
Original file line number Diff line number Diff line change
Expand Up @@ -156,20 +156,44 @@ audit: ## Create caseflow_audit schema, tables, and triggers in postgres
bundle exec rails r db/scripts/audit/tables/create_vbms_distributions_audit.rb
bundle exec rails r db/scripts/audit/tables/create_vbms_distribution_destinations_audit.rb
bundle exec rails r db/scripts/audit/tables/create_vbms_uploaded_documents_audit.rb
bundle exec rails r db/scripts/audit/tables/create_priority_end_product_sync_queue_audit.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_appeal_states_audit_table_function.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_vbms_communication_packages_audit_table_function.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_vbms_distributions_audit_table_function.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_vbms_distribution_destinations_audit_table_function.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_vbms_uploaded_documents_audit_table_function.rb
bundle exec rails r db/scripts/audit/functions/add_row_to_priority_end_product_sync_queue_audit_table_function.rb
bundle exec rails r db/scripts/audit/triggers/create_appeal_states_audit_trigger.rb
bundle exec rails r db/scripts/audit/triggers/create_vbms_communication_packages_audit_trigger.rb
bundle exec rails r db/scripts/audit/triggers/create_vbms_distributions_audit_trigger.rb
bundle exec rails r db/scripts/audit/triggers/create_vbms_distribution_destinations_audit_trigger.rb
bundle exec rails r db/scripts/audit/triggers/create_vbms_uploaded_documents_audit_trigger.rb
bundle exec rails r db/scripts/audit/triggers/create_priority_end_product_sync_queue_audit_trigger.rb

audit-remove: ## Remove caseflow_audit schema, tables and triggers in postgres
bundle exec rails r db/scripts/audit/remove_caseflow_audit_schema.rb

# The external-db make commands create/remove replicas (for local environment only) of external db tables that exist in Prod
# These tables should not be included as part of migrations
external-db-create: ## Creates external_vbms_ext_claim table
bundle exec rails r db/scripts/external/create_vbms_ext_claim_table.rb

external-db-remove: ## Remove external_vbms_ext_claim table
bundle exec rails r db/scripts/external/remove_vbms_ext_claim_table.rb

# This needs to be manually run after make reset/migrate in order for local tests involving external tables to pass.
# Otherwise the caseflow_certification_test schema will not create these tables and will error out.
external-db-create-test: ## Creates table in caseflow_certification_test DB for local RSPEC tests
bundle exec rails r -e test db/scripts/external/create_vbms_ext_claim_table.rb

remove-vbms-ext-claim-seeds: ## Drops audit tables, removes all PriorityEndProductSyncQueue, BatchProcess, and seed-vbms-ext-claim records, then rebuilds audit tables
make audit-remove
make external-db-create
bundle exec rails r db/scripts/external/remove_vbms_ext_claim_seeds.rb
make audit

reseed-vbms-ext-claim: remove-vbms-ext-claim-seeds seed-vbms-ext-claim ## Re-seeds database with records created from seed-vbms-ext-claim

c: ## Start rails console
bundle exec rails console

Expand All @@ -188,7 +212,7 @@ db-migrate: ## Migrate main Caseflow db
db-rollback: ## Rollback main Caseflow db
bundle exec rake db:rollback

migrate: etl-migrate etl-test-prepare db-migrate ## Migrate all Rails databases
migrate: external-db-remove etl-migrate etl-test-prepare db-migrate ## Migrate all non-external Rails databases

rollback: etl-rollback db-rollback ## Rollback all Rails databases

Expand All @@ -199,9 +223,14 @@ reset: reset-dbs seed-dbs enable-feature-flags ## Resets databases and enable fe

reset-dbs: ## Resets Caseflow and ETL database schemas
make audit-remove
make external-db-remove
DB=etl bundle exec rake db:drop db:create db:schema:load
bundle exec rake db:drop db:create db:schema:load
make audit
make external-db-create

seed-vbms-ext-claim: ## Seed only vbms_ext_claim
bundle exec rake db:seed:vbms_ext_claim

seed-dbs: ## Seed all databases
bundle exec rake local:vacols:seed
Expand Down
10 changes: 7 additions & 3 deletions app/controllers/idt/api/v2/distributions_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,18 @@ def format_response(response)
response_body = response.raw_body

begin
parsed_response = JSON.parse(response_body)
parsed_response = if [ActiveSupport::HashWithIndifferentAccess, Hash].include?(response_body.class)
response_body
else
JSON.parse(response_body)
end

# Convert keys from camelCase to snake_case
parsed_response.deep_transform_keys do |key|
key.to_s.underscore.gsub(/e(\d)/, 'e_\1')
end
rescue JSON::ParseError => error
log_error(error + " Distribution ID: #{params[:distribution_id]}")
rescue StandardError => error
log_error(error)

response_body
end
Expand Down
35 changes: 26 additions & 9 deletions app/jobs/ama_notification_efolder_sync_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,15 @@ class AmaNotificationEfolderSyncJob < CaseflowJob
def perform
RequestStore[:current_user] = User.system_user

all_active_ama_appeals = appeals_recently_outcoded + appeals_never_synced + ready_for_resync

sync_notification_reports(all_active_ama_appeals.first(BATCH_LIMIT.to_i))
all_active_ama_appeals = if FeatureToggle.enabled?(:phase_1_notification_sync_job_rollout)
appeals_never_synced
elsif FeatureToggle.enabled?(:phase_2_notification_sync_job_rollout)
appeals_never_synced + ready_for_resync
else
appeals_recently_outcoded + appeals_never_synced + ready_for_resync
end

sync_notification_reports(all_active_ama_appeals.uniq(&:id).first(BATCH_LIMIT.to_i))
end

private
Expand Down Expand Up @@ -98,18 +104,21 @@ def ready_for_resync
# Return: Array of active appeals
def get_appeals_from_prev_synced_ids(appeal_ids)
appeal_ids.in_groups_of(1000, false).flat_map do |ids|
Appeal.active.find_by_sql(
Appeal.find_by_sql(
<<-SQL
SELECT appeals.*
FROM appeals
SELECT appeals.* FROM appeals
JOIN tasks t ON appeals.id = t.appeal_id
AND t.appeal_type = 'Appeal'
JOIN (#{appeals_on_latest_notifications(ids)}) AS notifs ON
notifs.appeals_id = appeals."uuid"::text AND notifs.appeals_type = 'Appeal'
JOIN (#{appeals_on_latest_doc_uploads(ids)}) AS vbms_uploads ON
vbms_uploads.appeal_id = appeals.id AND vbms_uploads.appeal_type = 'Appeal'
WHERE
WHERE (
notifs.notified_at > vbms_uploads.attempted_at
OR
notifs.created_at > vbms_uploads.attempted_at
)
AND t.TYPE = 'RootTask' AND t.status NOT IN ('completed', 'cancelled')
GROUP BY appeals.id
SQL
)
Expand All @@ -120,8 +129,16 @@ def appeals_on_latest_notifications(appeal_ids)
<<-SQL
SELECT n1.* FROM appeals a
JOIN notifications n1 on n1.appeals_id = a."uuid"::text AND n1.appeals_type = 'Appeal'
LEFT OUTER JOIN notifications n2 ON (n2.appeals_id = a."uuid"::text AND n1.appeals_type = 'Appeal' AND
(n1.notified_at < n2.notified_at OR (n1.notified_at = n2.notified_at AND n1.id < n2.id)))
AND (n1.email_notification_status IS NULL OR
n1.email_notification_status NOT IN ('No Participant Id Found', 'No Claimant Found', 'No External Id'))
AND (n1.sms_notification_status IS NULL OR
n1.sms_notification_status NOT IN ('No Participant Id Found', 'No Claimant Found', 'No External Id'))
LEFT OUTER JOIN notifications n2 ON (n2.appeals_id = a."uuid"::text AND n1.appeals_type = 'Appeal'
AND (n2.email_notification_status IS NULL OR
n2.email_notification_status NOT IN ('No Participant Id Found', 'No Claimant Found', 'No External Id'))
AND (n2.sms_notification_status IS NULL OR
n2.sms_notification_status NOT IN ('No Participant Id Found', 'No Claimant Found', 'No External Id'))
AND (n1.notified_at < n2.notified_at OR (n1.notified_at = n2.notified_at AND n1.id < n2.id)))
WHERE n2.id IS NULL
AND n1.id IS NOT NULL
AND (n1.email_notification_status <> 'Failure Due to Deceased'
Expand Down
28 changes: 28 additions & 0 deletions app/jobs/batch_processes/batch_process_rescue_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# frozen_string_literal: true

# This job will search for and reprocess unfinished Batch Processes nightly.
# Search Criteria is for Batch Processes that are in an unfinished state ('PRE_PROCESSING', 'PROCESSING') &
# have a created_at date/time that is greater than the ERROR_DELAY defined within batch_process.rb
class BatchProcessRescueJob < CaseflowJob
queue_with_priority :low_priority

# :reek:FeatureEnvy
def perform
batches = BatchProcess.needs_reprocessing
if batches.any?
batches.each do |batch|
begin
batch.process_batch!
rescue StandardError => error
log_error(error, extra: { active_job_id: job_id.to_s, job_time: Time.zone.now.to_s })
slack_msg = "Error running #{self.class.name}. Error: #{error.message}. Active Job ID: #{job_id}."
slack_msg += " See Sentry event #{Raven.last_event_id}." if Raven.last_event_id.present?
slack_service.send_notification("[ERROR] #{slack_msg}", self.class.to_s)
next
end
end
else
Rails.logger.info("No Unfinished Batches Could Be Identified. Time: #{Time.zone.now}.")
end
end
end
82 changes: 82 additions & 0 deletions app/jobs/batch_processes/priority_ep_sync_batch_process_job.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# frozen_string_literal: true

class PriorityEpSyncBatchProcessJob < CaseflowJob
queue_with_priority :low_priority

# Using macro-style definition. The locking scope will be TheClass#method and only one method can run at any
# given time.
include RedisMutex::Macro

# Default options for RedisMutex#with_lock
# :block => 1 # Specify in seconds how long you want to wait for the lock to be released.
# # Specify 0 if you need non-blocking sematics and return false immediately. (default: 1)
# :sleep => 0.1 # Specify in seconds how long the polling interval should be when :block is given.
# # It is NOT recommended to go below 0.01. (default: 0.1)
# :expire => 10 # Specify in seconds when the lock should be considered stale when something went wrong
# # with the one who held the lock and failed to unlock. (default: 10)
#
# RedisMutex.with_lock("PriorityEpSyncBatchProcessJob", block: 60, expire: 100)
# Key => "PriorityEpSyncBatchProcessJob"

JOB_DURATION ||= ENV["BATCH_PROCESS_JOB_DURATION"].to_i.minutes
SLEEP_DURATION ||= ENV["BATCH_PROCESS_SLEEP_DURATION"].to_i

# Attempts to create & process batches for 50 minutes
# There will be a 5 second rest between each iteration
# Job will end if there are no records are left to batch

# rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/CyclomaticComplexity
def perform
setup_job
loop do
break if job_running_past_expected_end_time? || should_stop_job

begin
batch = nil
RedisMutex.with_lock("PriorityEpSyncBatchProcessJob", block: 60, expire: 100) do
batch = ActiveRecord::Base.transaction do
records_to_batch = PriorityEpSyncBatchProcess.find_records_to_batch
next if records_to_batch.empty?

PriorityEpSyncBatchProcess.create_batch!(records_to_batch)
end
end

batch ? batch.process_batch! : stop_job(log_no_records_found: true)

sleep(SLEEP_DURATION)
rescue StandardError => error
log_error(error, extra: { job_id: job_id.to_s, job_time: Time.zone.now.to_s })
slack_msg = "Error running #{self.class.name}. Error: #{error.message}. Active Job ID: #{job_id}."
slack_msg += " See Sentry event #{Raven.last_event_id}." if Raven.last_event_id.present?
slack_service.send_notification("[ERROR] #{slack_msg}", self.class.to_s)
stop_job
end
end
end
# rubocop:enable Metrics/MethodLength, Metrics/AbcSize, Metrics/CyclomaticComplexity

private

attr_accessor :job_expected_end_time, :should_stop_job

def setup_job
RequestStore.store[:current_user] = User.system_user
@should_stop_job = false
@job_expected_end_time = Time.zone.now + JOB_DURATION
end

def job_running_past_expected_end_time?
Time.zone.now > job_expected_end_time
end

# :reek:BooleanParameter
# :reek:ControlParameter
def stop_job(log_no_records_found: false)
self.should_stop_job = true
if log_no_records_found
Rails.logger.info("#{self.class} Cannot Find Any Records to Batch."\
" Job will be enqueued again at the top of the hour. Active Job ID: #{job_id}. Time: #{Time.zone.now}")
end
end
end
Loading

0 comments on commit 378b044

Please sign in to comment.