-
Notifications
You must be signed in to change notification settings - Fork 528
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1152 from Deitos-Network/master
Deitos Network - Milestone Delivery 2
- Loading branch information
Showing
1 changed file
with
80 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Milestone Delivery :mailbox: | ||
|
|
||
| **The delivery is according to the official [milestone delivery guidelines](https://github.com/w3f/Grants-Program/blob/master/docs/Support%20Docs/milestone-deliverables-guidelines.md).** | ||
|
|
||
| * **Application Document:** [Deitos Network](https://github.com/w3f/Grants-Program/blob/master/applications/Deitos_Network.md) | ||
| * **Milestone Number:** 2 | ||
|
|
||
| ## Milestone 2: Integration and Storage Management | ||
|
|
||
| Following the successful completion of the first milestone, which established the foundational settings for the Deitos protocol and the Infrastructure Provider (IP) stack, the second milestone shifts focus towards the integration of the protocol with storage management processes. This phase is pivotal in enabling users to upload their files to the Hadoop Distributed File System (HDFS) in accordance with their agreements. Additionally, the Deitos Protocol has been enhanced through the inclusion of several key components, further augmenting its functionality and robustness. These enhancements include: | ||
|
|
||
| - **Data Integrity Protocol:** A mechanism designed to ensure the integrity and authenticity of the files stored within the infrastructure provider. | ||
|
|
||
| - **File Verification Process:** This process involves the systematic verification of files post-upload to confirm their integrity and compliance with the terms of the user agreement. | ||
|
|
||
| - **Deitos Verifier:** A dedicated component responsible for executing the file verification process. The Deitos Verifier operates as an off-chain worker that retrieves and validates file hashes, comparing them against the hashes recorded on the blockchain at the time of upload. | ||
|
|
||
| - **CLI for Upload Process:** To streamline the file upload process, a specific Command Line Interface (CLI) tool has been developed. This tool simplifies interactions with the Deitos protocol for users, enabling them to efficiently upload files to HDFS under the terms of their agreements. The CLI tool embodies the user-friendly approach of the Deitos Protocol, making it accessible for users to securely manage their file uploads. | ||
|
|
||
| The integration of these components into the Deitos Protocol stack represents a significant advancement in the project's development. By focusing on data integrity, secure file verification, and user-friendly upload processes, the second milestone reinforces the protocol's capacity to provide a reliable and efficient decentralized storage solution. | ||
|
|
||
| ## Repository links | ||
|
|
||
| - https://github.com/Deitos-Network/deitos-node/tree/w3f-grant-m2-delivery | ||
| - https://github.com/Deitos-Network/deitos-ip/tree/w3f-grant-m2-delivery | ||
| - https://github.com/Deitos-Network/deitos-cli/tree/w3f-grant-m2-delivery | ||
| - https://github.com/Deitos-Network/deitos-gate/tree/w3f-grant-m2-delivery | ||
| - https://github.com/Deitos-Network/deitos-verifier/tree/w3f-grant-m2-delivery | ||
|
|
||
| ## Instructions and Considerations for Testing | ||
|
|
||
| To conduct the tests, only two repositories are required: the [Deitos IP](https://github.com/Deitos-Network/deitos-ip/tree/w3f-grant-m2-delivery) for infrastructure management and the [Deitos CLI](https://github.com/Deitos-Network/deitos-cli/tree/w3f-grant-m2-delivery) for file uploads. These components are essential for interacting with the Deitos protocol and executing the file upload process. All other necessary components are included and integrated within the IP's Docker container, streamlining the setup and testing process. | ||
|
|
||
| ### Starting the Docker Image | ||
|
|
||
| Before initiating the Docker image, please ensure that your system has at least 100 GB of free space. This precaution is to prevent the build process from failing due to insufficient disk space. Adequate free space is crucial for a smooth and uninterrupted build process, especially considering the size and complexity of the Deitos IP stack. | ||
|
|
||
| ### Slow Download Rates from Apache Repositories | ||
|
|
||
| We have observed that some Apache repositories are experiencing slow download rates, which can extend the time required to build the entire IP stack. To mitigate this issue and enhance the build efficiency, we have streamlined the IP stack by removing unnecessary components. One such removal is the Deitos client stack, which is not required for the current testing phase. This decision allows for a more focused and expedient testing process, ensuring that the deliverables can be fully tested with only the IP stack, without the need for additional components that may complicate the setup or extend the build time unnecessarily. | ||
|
|
||
| By following these instructions and considerations, testers can efficiently set up and conduct their tests, focusing on the core functionalities of the Deitos protocol and its integration with storage management processes. | ||
|
|
||
|
|
||
|
|
||
| ## Deliverables of Milestone #1 | ||
|
|
||
|
|
||
| | Number | Deliverable | Specification | | ||
| | -----: | ----------- | ------------- | | ||
| | **0a.** | License | The project will utilize the GNU General Public License v3 (GPLv3) license. | | ||
| | **0b.** | Documentation | Building upon the documentation provided in the first milestone, this milestone will introduce a new set of user guidelines. As the grant approaches its conclusion and all implementation details are settled, we will provide thorough protocol documentation. | | ||
| | **0c.** | Testing and Testing Guide | 1) All the different pallets and functionalities will contain their own unit testing contained in the `test.rs` files including the inline documentation for each test purpose. 2) This milestone will deliver the necessary tools to establish a local testing environment, allowing for comprehensive testing of all functionalities. | | ||
| | **0d.** | Docker | As we plan to deliver the software to infrastructures providers (IP) in docker images, all the protocol services will be configured and delivered in the IP Docker image. | | ||
| | **0e.** | Articles | For the grant conclusion we will create two Medium articles: The first one will be a project introduction targeting a more general and wide audience. Some of the content will include: 1) Deitos Network introduction and network offering, 2) Roles and protocol functioning, 3) Design decisions. On the second article, we will delve into more technical information where the development aspect of this grant will be discussed. The audience for this second article will be more technical and the following items will be discussed: 1) Architecture in depths, 2) Substrate pallets descriptions, 3) Internal functioning of each module. | | ||
| | 1. | Proxy Development | 1) Complete development and deployment of the described proxy ensuring interaction between infrastructure providers, consumers and the substrate node. 2) Mechanism to reserve resources on the infrastructure provider for a consumer upon agreement commitment. 3) A system focused on storage where user segmentation is achieved through dynamic users generated on Hadoop. 4) Authentication derived from a signed transaction initiated by the pallet-deitos pallet. 5) Development of a module to validate consumer signatures and commit actions upon successful verification, ensuring no traditional passwords are stored in the system. | | ||
| | 2. | File Uploads (Client Interface) | 1) Delivery of a client interface to facilitate file content splitting and hash calculation. 2) Creation of a generic algorithm to uniformly split files and calculate segment hashes. 3) Mechanism for producing and publishing signed transactions reflecting the computed results.| | ||
| | 3. | File Upload Verification (Provider Side) | 1) Using the previously generic algorithm to uniformly split files and calculate segment hashes for each file or part upon receiving the consumer's signed transaction. Files are marked as 'verified' post successful hash validation. 3) Constant monitoring of blocks to detect unverified files, triggering an OCW for hash verification based on consumer transactions. | | ||
| | 4. | Data Integrity Protocol | 1) Comprehensive development and deployment of the Data Integrity Protocol. 2) Utilizing BABE-generated randomness to select files/parts, directing infrastructure providers to create and validate respective hashes. 3) In case of hash mismatches during the data integrity protocol, a system to penalize the provider by reducing their staked amount. | | ||
|
|
||
| ## Deliverable Links | ||
| | Number | Description | Link | ||
| | -----: | ----------- | ----------- | ||
| | **0a.** | Deitos Node License | https://github.com/Deitos-Network/deitos-node/blob/w3f-grant-m2-delivery/LICENSE | ||
| | **0a.** | Deitos IP stack License | https://github.com/Deitos-Network/deitos-ip/blob/w3f-grant-m2-delivery/LICENSE | ||
| | **0a.** | Deitos CLI | https://github.com/Deitos-Network/deitos-cli/tree/w3f-grant-m2-delivery/LICENSE | ||
| | **0a.** | Deitos Gate License | https://github.com/Deitos-Network/deitos-gate/tree/w3f-grant-m2-delivery/LICENSE | ||
| | **0a.** | Deitos Verifier License | https://github.com/Deitos-Network/deitos-verifier/blob/w3f-grant-m2-delivery/LICENSE | ||
| | **0b.** | `pallet-deitos-fs` README Documentation | https://github.com/Deitos-Network/deitos-node/blob/w3f-grant-m2-delivery/pallets/deitos-fs/README.md | ||
| | **0b.** | `pallet-deitos-fs` Inline Documentation | https://github.com/Deitos-Network/deitos-node/blob/w3f-grant-m2-delivery/pallets/deitos-fs/src/lib.rs | ||
| | **0b.** | Deitos IP Documentation | https://github.com/Deitos-Network/deitos-ip/blob/w3f-grant-m2-delivery/README.md | ||
| | **0c.** | `pallet-deitos-fs` Unit Testing | https://github.com/Deitos-Network/deitos-node/tree/w3f-grant-m2-delivery/pallets/deitos-fs/src/tests | ||
| | **0c.** | `pallet-deitos-fs` User Guide | https://github.com/Deitos-Network/deitos-node/blob/w3f-grant-m2-delivery/docs/pallet-deitos-user-guide.md (section 5) | ||
| | **0c.** | Deitos IP Docker User Guide | https://github.com/Deitos-Network/deitos-ip/blob/w3f-grant-m2-delivery/README.md | ||
| | **0d.** | Deitos IP | https://github.com/Deitos-Network/deitos-ip/blob/w3f-grant-m2-delivery | ||
| | **0e.** | Article #1 | https://forum.polkadot.network/t/deitos-network-forging-a-new-path-in-big-data-and-ai-with-blockchain-technology/4961 | ||
| | **0e.** | Article #2 | https://medium.com/@deitos.network/deitos-network-web3-foundation-grant-f267cb10e7a8 | ||
| | **2.** | File Uploads (Client Interface) | https://github.com/Deitos-Network/deitos-cli/tree/w3f-grant-m2-delivery | ||
| | **3.** | File Upload Verification (Provider Side) | https://github.com/Deitos-Network/deitos-verifier/tree/w3f-grant-m2-delivery | ||
| | **4.** | Data Integrity Protocol | https://github.com/Deitos-Network/deitos-node/tree/w3f-grant-m2-delivery/pallets/deitos-fs/lib.rs |