feat(consensus): sequence transaction from foreign LocalPrepare/Accept

[sdbondi]

Description

• sequence transaction from foreign LocalPrepare/Accept
• park and request applicable missing transactions from foreign proposals
• minor swarm improvements/fixes
• evidence uses QC that justifies the block containing the transaction atom (not the parent of that block)

Motivation and Context

On receipt of a valid foreign proposal, a node may not be aware of all shard group applicable transactions in the following cases:

• they were not propagated due to some communication failure
• the shard group is only involved in outputs and were not initially part of the transaction

The node determines involvement in the transactions from a proposal by inspecting the evidence, which must contain at least one valid substate address applicable to the shard group. The transaction is requested as required and the foreign proposal message is unparked once all transactions are received.

How Has This Been Tested?

New consensus test for output-only involvement of a shard. Manually, swarm with 20 vns

Manually multi SG testing reveals that reliability is still poor, with SGs often getting stuck on LocalPrepared. This seems to be an issue with the broadcasting of proposals and is being investigated.

What process can a PR reviewer use to test or verify this change?

Multishard testing with swarm

Breaking Changes

• None
• Requires data directory to be deleted
• Other - Please specify

[github-actions]

Test Results (CI)

550 tests  +1   550 ✅ +1   1h 40m 35s ⏱️ - 5m 55s
 58 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit 61e2216. ± Comparison against base commit f5f3860.

This pull request removes 6 and adds 7 tests. Note that renamed tests count towards both.

consensus_tests ‑ consensus::foreign_shard_decides_to_abort
consensus_tests ‑ substate_store::it_allows_locks_within_one_transaction
tari_dan_storage ‑ consensus_models::command::export_bindings_evidence
tari_dan_storage ‑ consensus_models::command::export_bindings_shardevidence
tari_dan_storage ‑ consensus_models::foreign_proposal::export_bindings_foreignproposal
tari_dan_storage ‑ consensus_models::foreign_proposal::export_bindings_foreignproposalstate

consensus_tests ‑ consensus::foreign_shard_group_decides_to_abort
consensus_tests ‑ consensus::multishard_local_inputs_and_outputs_foreign_outputs
consensus_tests ‑ substate_store::it_allows_requesting_the_same_lock_within_one_transaction
tari_dan_storage ‑ consensus_models::evidence::export_bindings_evidence
tari_dan_storage ‑ consensus_models::evidence::export_bindings_shardevidence
tari_dan_storage ‑ consensus_models::evidence::tests::it_merges_two_evidences_together
tari_dan_storage ‑ consensus_models::foreign_proposal::export_bindings_foreignproposalatom

♻️ This comment has been updated with latest results.