Fix crc mismatch during deepstore upload retry task

[tibrewalpratik17]

After #14406, we are able to successfully take deepstore backup but now we see that there are lot of UpsertCompactionTask failures with the following error:

java.lang.IllegalStateException: Crc mismatched between ZK and deepstore copy of segment: rta_cadence_visibility_gs_production__0__641__20240923T2154Z. Expected crc from ZK: 2934158065, crc from deepstore: 2050894616
	at org.apache.pinot.plugin.minion.tasks.upsertcompaction.UpsertCompactionTaskExecutor.convert(UpsertCompactionTaskExecutor.java:69)
	at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:132)
	at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:60)
	at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:157)
	at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:118)
	at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent

It seems that the deepstore-upload retry task can take a backup from any arbitrary replica and not particularly the one with which the CRC matches in ZK. This patch is to fix the issue where we allow deepstore upload only from the replica which matches the ZK metadata's crc values. If we don't find one, then we end up taking the deepstore backup anyways from one random replica.

For divergent CRC in replicas, the reason can be particularly using text-indexes. We have been discussing this in multiple issues: #13491, #11004

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 7.14286% with 26 lines in your changes missing coverage. Please review.

Project coverage is 63.91%. Comparing base (59551e4) to head (271c305).
Report is 1381 commits behind head on master.

Files with missing lines	Patch %	Lines
...che/pinot/server/api/resources/TablesResource.java	0.00%	13 Missing ⚠️
...t/controller/util/ServerSegmentMetadataReader.java	0.00%	8 Missing ⚠️
.../core/realtime/PinotLLCRealtimeSegmentManager.java	28.57%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14506      +/-   ##
============================================
+ Coverage     61.75%   63.91%   +2.15%     
- Complexity      207     1570    +1363     
============================================
  Files          2436     2673     +237     
  Lines        133233   146814   +13581     
  Branches      20636    22513    +1877     
============================================
+ Hits          82274    93831   +11557     
- Misses        44911    46051    +1140     
- Partials       6048     6932     +884     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.82% <7.14%> (+2.11%)	⬆️
java-21	63.78% <7.14%> (+2.16%)	⬆️
skip-bytebuffers-false	63.85% <7.14%> (+2.10%)	⬆️
skip-bytebuffers-true	63.76% <7.14%> (+36.03%)	⬆️
temurin	63.91% <7.14%> (+2.15%)	⬆️
unittests	63.90% <7.14%> (+2.15%)	⬆️
unittests1	55.56% <ø> (+8.67%)	⬆️
unittests2	34.62% <7.14%> (+6.89%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[Jackie-Jiang]

This doesn't solve the problem when the committing server is not up. A more robust way is to allow server to change the crc when updating the download URL. Having inconsistent crc in ZK metadata and deep store segment is very risky

[Jackie-Jiang]

Ideally we need to make text index deterministic. Having indeterministic crc can cause lots of problems. Re-downloading all segments during server startup is not acceptable

[tibrewalpratik17]

A more robust way is to allow server to change the crc when updating the download URL.

So currently server uploads the file with a UUID suffix inside the segmentUploadDir and it is controller which moves the zip file to the final deepstore location and then controller updates the download URL.

If we move the logic of updating ZK to server that might become more risky as controller <> deepstore may fail and we might be left with a non-empty downloadURL pointing to an empty path (may cause FileNotFound exception in other places).

Ideally we need to make text index deterministic. Having indeterministic crc can cause lots of problems. Re-downloading all segments during server startup is not acceptable

Agreed! But this seems a larger scope problem to what we are doing here. Right now given we know replicas have divergent crc, deepstore-upload retry backing up segments from random replica seems to be problematic which we are trying to solve.

[Jackie-Jiang]

A more robust way is to allow server to change the crc when updating the download URL.

So currently server uploads the file with a UUID suffix inside the segmentUploadDir and it is controller which moves the zip file to the final deepstore location and then controller updates the download URL.

If we move the logic of updating ZK to server that might become more risky as controller <> deepstore may fail and we might be left with a non-empty downloadURL pointing to an empty path (may cause FileNotFound exception in other places).

Let me clarify: we keep the existing update logic, but also update crc when modifying download url. Both crc and download url update should be posted atomically as one ZK record change.

[tibrewalpratik17]

Let me clarify: we keep the existing update logic, but also update crc when modifying download url. Both crc and download url update should be posted atomically as one ZK record change.

Makes sense! Updated the PR accordingly.