fix(mysql): index gap lock deadlock

[david-leifker]

Effects MySQL

Fix a condition where multiple transactions concurrently inserting new rows within the same index gap would result in a deadlock. Further details on the condition can be found here.

The solution involves changing the isolation level to READ_COMMITTED and relying on the read locks which were added using "select for update"

Added a smoke-test case with interleaved URNs with transaction concurrency.

Database transaction handling for all databases

• Extend "for update" to the default aspects logic
• Default aspects, if determined to exist later in the transaction will drop without exception
• Validation exceptions conditional depending on the transaction's batch origin.
  • Client's synchronous request will return a validation exception and abort the transaction to the client per previous behavior
  • If the batch is a batch of MCPs read by the mce-consumer, a validation exception will not impact the entire batch to prevent non-related MCPs from being impacted by 1 validation error. This is disabled by default: MCP_CONSUMER_BATCH_ENABLED is not enabled by default.

Feature

Added header support to CREATE/CREATE_ENTITY change types to allow non-exception behavior if the write should be silently dropped if the CREATE/CREATE_ENTITY check fails. Clients can determine if the case is an error or simply ignored.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub