Addressing issue #2987 - Adding missing indexes for jobs view and correcting small bug

[jonathanpmoraes]

Closes: #2987

Solution

Lack of indexes to jobs tables were causing a slowdown on the queries related to jobs. This pull request includes changes to improve database handling and performance in the Marquez application. The most important changes include modifying error handling during database migration, adding indexes to various tables to optimize queries, and creating a new view for jobs.

Note: All database schema changes require discussion. Issue link: #2987

Database optimization:

• api/src/main/resources/marquez/db/migration/V75_1__AddJobTableIndexes.sql: Added multiple indexes to the jobs, job_facets, runs, jobs_tag_mapping, tags, namespaces, and job_versions tables to improve query performance for jobs.

View creation:

• api/src/main/resources/marquez/db/migration/V75_2__CreateJobsView.sql: Created or replaced the jobs_view to include various job details, optimizing the way job data is queried and displayed.

Error handling:

• api/src/main/java/marquez/MarquezApp.java: Modified the run method to throw a FlywayException with a detailed message instead of logging and calling onFatalError when a database migration error occurs. When this error occurs, the application will now display the error message instead of a generic "Stopping app" message.

One-line summary: Added missing indexes for jobs view that were taking a long time to be displayed.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[netlify]

❌ Deploy Preview for peppy-sprite-186812 failed.

Name	Link
🔨 Latest commit	5cbd9ec
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/679515be70cda40008cb0366

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 81.19%. Comparing base (c44d5be) to head (4b9ca0f).

Files with missing lines	Patch %	Lines
api/src/main/java/marquez/MarquezApp.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3024      +/-   ##
============================================
+ Coverage     81.02%   81.19%   +0.17%     
- Complexity      241     1506    +1265     
============================================
  Files            43      268     +225     
  Lines           996     7355    +6359     
  Branches         40      325     +285     
============================================
+ Hits            807     5972    +5165     
- Misses          160     1225    +1065     
- Partials         29      158     +129     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.