[litep2p-milestone]: Improve litep2p robustness

[lexnv]

The Vision

Improve the robustness and fault-tolerance of the litep2p crate and bring the crate to a higher level of maturity.

The goal is to stabilize the litep2p and use it as the default backend of Substrate-based chains.

The Plan

To achieve this, we'll focus our efforts on:

• Feature Completeness
  • Implement missing features (validate ping payloads, generate unique random ping payloads, ensure kad records TTL)
• Robustness
  • Ensure core code logic does not panic upon unexpected/malicious user behaviors, panics, unwraps, undocumented expects, array indexing without length validation etc
  • Fix incorrect functionality and divergences from the libp2p specification
  • Repair bugs and limit the surface of race conditions
• Maintenance: keep the code simple
  • Avoid code duplicates (transport layer)
  • Coherent APIs and errors
  • Adopt rust clippy and best practices
  • Improve and add documentation
• Performance Improvements
  • Avoid unnecessary clones
  • Improve the query engine
  • Analyze memory and CPU consumption
• Testing
  • increase code coverage by unit-testing sensitive components (multiselect / noise, kad etc)
  • improve our integration testing scenarios
  • add fuzzing
  • create a plan for continuous testing and triaging in our test stack and keep track of triage reports

Feature Completeness

Preview Give feedback

1. Implement connection limits #17
   2 of 2
   easy enhancement +2
   
2. kad: Revisit FIND_NODE and increase test coverage #100
   enhancement +1
   
3. kad: Implement ADD_PROVIDER command #166
   enhancement +1
4. kad: Implement GET_PROVIDERS command #167
   enhancement +1
5. transport: Change AddressStore to keep track of more dial addresses #180
   1 of 1
   enhancement +1
   
6. transport/manager: Add ability to accept/reject incoming connections before negotiation #186
   enhancement +1
   
7. request-response: Provide more details into rejected requests for a high request failure #188
   1 of 1
   enhancement +1
   
8. identify: Update local node listen addresses #191
   1 of 1
   enhancement +1
   
9. errors: Add DialError error and ListDialFailures event for better error reporting #206
   +0
   
10. transport: Implement TransportService::local_peer_id() #224
    +0
    
11. kad: Expose all peer records of GET_VALUE query #96
    +0
    

Robustness

Preview Give feedback

1. crypto/noise: Make code fault tolerant #124
   +0
   
2. ping: Make ping fault tolerant wrt outbound substreams races #133
   +0
3. kad: Ensure Kademlia Record has TTL #129
   2 of 2
   +0
   
4. protocol/notif: Fix panic on missing peer state #143
   bug +1
   
5. transport-manager: Unknown connection opened #172
   2 of 2
   bug +1
   
6. kad: Fix substream opening and dialing race #222
   bug +1
   
7. query-executor: Save the task waker on empty futures #219
   +0
   
8. crypto/noise: Generate keypair only for Curve25519 #214
   enhancement +1
   
9. transport: Fix double lock and state overwrite on disconnected peers #179
   +0
   
10. kad: Do not update memory store on incoming GetRecordSuccess #190
    +0
    
11. transport: Reject secondary connections with different connection IDs #176
    +0
    
12. Fix TCP, WebSocket, QUIC leaking connection IDs in reject() #198
    bug +1
    
13. kad: Update connection status of an existing bucket entry #181
    enhancement +1
    
14. manager: Fix connection limits tracking of rejected connections #286
    bug +1
    
15. transport: Fix waking up on filtered events from poll_next #287
    bug +1
    
16. transports: Fix missing Poll::Ready(None) event from listenener #285
    bug enhancement +2
    
17. manager: Avoid overflow on stream implementation for TransportContext #283
    bug +1
    
18. manager: Log when polling returns Ready(None) #284
    enhancement +1
    
19. transport_service: Improve connection stability by downgrading connections on substream inactivity #260
    enhancement +1
    
20. tcp/websocket/quic: Fix cancel memory leak #272
    bug +1
    
21. kad: Fix not retrieving local records #221
    bug +1
    
22. transport: Abort canceled dial attempts for TCP, WebSocket and Quic #255
    enhancement +1
    
23. kad/executor: Add timeout for writting frames #277
    enhancement +1
    
24. kad: Avoid cloning the KademliaMessage and use reference for RoutingTable::closest #233
    enhancement +1
    
25. crypto/noise: Verify crypto/noise signature payload #278
    enhancement +1
    
26. req-resp: Fix panic on connection closed for substream open failure #291
    +0
    
27. transport_service/logs: Provide less details for trace logs #292
    +0
    
28. req-resp: Fix memory leak of pending substreams #297
    bug +1
    
29. notification: Fix memory leak of pending substreams #296
    bug +1
    
30. mdns/fix: Failed to register opened substream #301
    bug +1
    
31. identify: Replace FuturesUnordered with FuturesStream #302
    enhancement +1
    

Maintenance

Preview Give feedback

1. transport: Extract common transport code between TCP and WebSocket #70
   2 of 2
   enhancement +1
   
2. Possible use of libp2p_identity::PeerId as PeerId type #73
   +0
3. kad: Remove unused serde cfg #262
   +0
   
4. transport_service: Improve logs and move code from tokio::select macro #254
   enhancement +1
   

Testing

Preview Give feedback

1. mutlistream_select/tests: Fuzz Message encode / decode #127
   +0
2. mutlistream_select/tests: Add tests for LengthDelimited #126
   +0
3. crypto/noise/tests: Increase test coverage #125
   +0
4. minor: fix tests without websocket feature #215
   bug easy +2
   
5. kad: Refactor GetRecord query and add tests #97
   +0
   
6. ci: Run CI on master and PRs #87
   +0
   
7. ci: Enable dependabot #89
   +0
   
8. ci: Ensure cargo-machete is working with rust version from CI #303
   enhancement +1
   
9. chore: Update hickory-resolver to version 0.24.2 #304
   enhancement +1
   

Performance

Preview Give feedback

1. crypto/noise: Investigate the performance impact of creating a new RNG and DH in snow #123
   +0
2. substream: Use write_all instead of manually writing bytes #217
   enhancement +1
   
3. find_node: Optimize parallelism factor for slow to respond peers #220
   +0
   

Outstanding

Preview Give feedback

1. Any plan to support WASM target? #99
   +0
2. ping: Report back ping failures #132
   +0
3. ping: Verify received "pong" payload #134
   +0
4. identify: Open inbound substream for unknown peer #103
   bug +1
5. multistream: Lazy negotiated protocol cannot be closed before receiving messages #60
   bug easy +2
   
6. Research the possibility of embedding Peerset into NotificationProtocol #34
   enhancement medium +2
7. Start using V1Lazy when applicable #1
   enhancement +1
8. notification: Invalid handshake state #299
   bug +1
9. error: Merge NegociatedError with SubstreamError #128
   +0
10. clippy: Revisit allowed clippy rules #85
    +0