split storage files

[jchris]

Split the storage streams:

• Prolly (deterministic)
• Index
• Clock
• ipld hashmap

[jchris]

• Bulk operations change
  • Clock points to CARs
  • Values are CIDs

[jchris]

Pub-Sub System for UCAN Invocation

1. Cryptographically Secure Update Function: Develop a secure update function that allows all actors to perform only their portion of the work.

   • Filter Function: Implement a filter function on the stream.
   • Batch Updates: Allow for batch operations by the client.
     • Enable diff data from the client for the operation. This diff should then be sent to a remote and used to update the latest reference alongside any other updates. The system should be capable of rejecting these updates from the queue if necessary.
     • Include read blocks in the new proof. This could lead to a constant factor increase in car files but would also allow for the deletion of old car files.
     • The system should be able to handle multiple writes in a mutex without needing outside IO, just working on the cars in the queue. If the leaf has any of the same orphan blocks, the system should be capable of merging cleanly, even rejecting stale updates like in Fauna.
     • Implement a method to call a conflict without decoding blocks, and create a list of all blocks that occur more than once.
   • Hosting: Consider the UCAN protocol in terms of hosting - if different parties are competing to host it.
     • Use DynamoDB for batch tracking.

2. State Updates Include Previous State: Ensure that state updates also contain the previous state.

   • Mutex Batch Update: Implement a Mutex Batch Update feature where previous blocks are passed as source input for the next Mutex Batch Update.
   • Epochs: Integrate epochs into the w3clock.

[jchris]

related fireproof-storage/fireproof#17