[RFC-15][HUDI-1325] Merge updates of unsynced instants to metadata table

[rmpifer]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

There can be the possibility that the dataset timeline and the metadata table timeline become out of sync. When trying to read from the metadata table while the timeline is out of sync you would get incorrect values for getAllFilesInPartition and getAllPartitionPaths.

This change provides a way to overcome this scenario by reading unsynced timeline instants and merging it with existing metadata table records to get the most up to date state of the file system

JIRA: https://issues.apache.org/jira/browse/HUDI-1325

Brief change log

• The logic of converting timeline metadata to metadata table records was directly tied to the commit phase in FSBackedMetadataWriter. Refactored this logic to a utility class HoodieTableMetadataTimelineUtil
• Created a scanner HoodieMetadataMergedInstantRecordScanner which handles conversion of timeline instants to metadata records and merges results
• Added third step in FSBackedTableMetadata.getMergedRecordByKey which uses the new scanner mentioned to fetch the HoodieRecord associated with the desired key from the unsynced timeline instants and merge it with the record from the metadata table
• When converting rollback operation to metadata table records there was logic that re-read from the metadata table to ensure any files being deleted as part of roll back existed.

// Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
// metadata table. Hence, the deleted filed need to be checked against the metadata.

This doesn't make sense since all instants are processed in serial order so there would never be the case where a rollback was being written before an instant earlier on the timeline was already synced. Removed this logic because it created circular dependency when implementing timeline merging

• Changed the validate metadata step in tests to use the metadata reader FSBackedTableMetadata. By default when metadata writer FSBackedTableMetadataWriter is initialized it syncs all instants to the metadata table. By using the reader we can simulate metadata table being out of sync.
• Modified initMetaClient in test base class to allow table type to be passed in since table type is always set as COPY_ON_WRITE if using this method to initialize the meta client

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end.
• Added HoodieClientWriteTest to verify the change.
• Manually verified the change by running a job locally.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[vinothchandar]

@rmpifer I was looking for this in apache/hudi :). and totally missed that it's here.

Can we retarget this to apache/hudi/rfc-15?

[vinothchandar]

Looks like a good amount of code was reusuable. I am wondering if we can do this in a separate TimelineMergedTableMetadata implementation, that wraps HoodieBackedTableMetadata?

[codecov-io]

Codecov Report

Merging #6 (75a3352) into rfc-15 (7f84b12) will decrease coverage by 0.06%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             rfc-15       #6      +/-   ##
============================================
- Coverage     43.79%   43.73%   -0.07%     
  Complexity     3379     3379              
============================================
  Files           573      575       +2     
  Lines         24400    24438      +38     
  Branches       2445     2449       +4     
============================================
  Hits          10687    10687              
- Misses        12692    12730      +38     
  Partials       1021     1021              

Flag	Coverage Δ	Complexity Δ
hudicli	27.48% <0.00%> (+0.35%)	0.00 <0.00> (ø)
hudiclient	24.74% <0.00%> (+0.41%)	0.00 <0.00> (ø)
hudicommon	51.64% <0.00%> (-0.95%)	0.00 <0.00> (ø)
hudihadoopmr	33.05% <ø> (ø)	0.00 <ø> (ø)
hudispark	67.19% <ø> (ø)	0.00 <ø> (ø)
huditimelineservice	64.43% <ø> (ø)	0.00 <ø> (ø)
hudiutilities	69.38% <ø> (ø)	0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ	Complexity Δ
...hudi/metadata/HoodieBackedTableMetadataWriter.java	4.01% <0.00%> (+1.34%)	4.00 <0.00> (ø)
...pache/hudi/metadata/HoodieBackedTableMetadata.java	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...data/HoodieMetadataMergedInstantRecordScanner.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)
...hudi/metadata/HoodieTableMetadataTimelineUtil.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)