Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: 00029 refactor demo to lmax disruptor and 00055 persistence design doc #41

Merged
merged 1 commit into from
Aug 12, 2024

Conversation

mattp-swirldslabs
Copy link
Contributor

@mattp-swirldslabs mattp-swirldslabs commented Jul 9, 2024

Description:

  • Integration with LMAX Disruptor RingBuffer
  • Changed toy types to align with names and relationships like
    hedera-protobufs
  • Boosted unit test coverage to 100% for most packages
  • Adjusted Docker container to run as the hedera user rather than root
  • Changed the consumer bidirection stream to server streaming
  • Added exception handling when reading and writing to storage
  • Refactored getBlock to singleBlock to align with hedera-protobufs rpc service definition and types
  • Separated ReadBlock and WriteBlock
  • Added RemoveBlock to handle removal of partially written Blocks when an exception is thrown
  • Adjusted producer and consumer scripts to work with new types
  • Added metrics to report: live block items counted, blocks persisted, single blocks retrieved, and current subscribers
  • Block Persistence Design Doc #55

Related issue(s):

Fixes #29, #55

Notes for reviewer:

Checklist

  • Documented (Code comments, README, etc.)
  • Tested (unit, integration, etc.)

@mattp-swirldslabs mattp-swirldslabs self-assigned this Jul 9, 2024
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch from 552c9b9 to 582cfae Compare July 9, 2024 21:12
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch 3 times, most recently from 6e392a8 to e1c8b21 Compare July 17, 2024 20:10
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch 3 times, most recently from fe67e8e to b9ca958 Compare July 29, 2024 21:28
@AlfredoG87 AlfredoG87 changed the title 00029 refactor demo disruptor feature: 00029 refactor demo disruptor Jul 29, 2024
@AlfredoG87 AlfredoG87 changed the title feature: 00029 refactor demo disruptor refactor: 00029 refactor demo disruptor Jul 29, 2024
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch 2 times, most recently from 1860e91 to 67a942e Compare July 30, 2024 17:12
@mattp-swirldslabs mattp-swirldslabs modified the milestone: 0.1.0 Jul 30, 2024
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch 4 times, most recently from 9d1a28b to 3dd03b6 Compare August 1, 2024 21:08
@mattp-swirldslabs mattp-swirldslabs marked this pull request as ready for review August 5, 2024 14:50
@mattp-swirldslabs mattp-swirldslabs requested a review from a team as a code owner August 5, 2024 14:50
@mattp-swirldslabs mattp-swirldslabs changed the title refactor: 00029 refactor demo disruptor refactor: 00029 refactor demo to lmax disruptor and 00055 persistence design doc Aug 5, 2024
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch from 714cad6 to 217188f Compare August 5, 2024 21:14
Copy link
Contributor

@AlfredoG87 AlfredoG87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, only thing that I would add is a Diagram on the design doc, so is easier to understand the ring buffer and the interactions between the classes.

Leaved some nits.

@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch 2 times, most recently from 8f52ae3 to 2111430 Compare August 7, 2024 21:15
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch from 2111430 to be78459 Compare August 8, 2024 14:49
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch from be78459 to 4890c77 Compare August 8, 2024 17:42
Copy link

codecov bot commented Aug 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (1a98d20) to head (be5d345).

Additional details and impacted files
@@              Coverage Diff               @@
##               main       #41       +/-   ##
==============================================
+ Coverage     70.00%   100.00%   +30.00%     
- Complexity       33       110       +77     
==============================================
  Files            10        19        +9     
  Lines           170       420      +250     
  Branches          6        23       +17     
==============================================
+ Hits            119       420      +301     
+ Misses           47         0       -47     
+ Partials          4         0        -4     
Files Coverage Δ
...va/com/hedera/block/server/BlockStreamService.java 100.00% <100.00%> (+45.16%) ⬆️
...ava/com/hedera/block/server/ServiceStatusImpl.java 100.00% <100.00%> (ø)
...erver/consumer/ConsumerStreamResponseObserver.java 100.00% <100.00%> (ø)
...java/com/hedera/block/server/data/ObjectEvent.java 100.00% <100.00%> (ø)
...ock/server/mediator/LiveStreamMediatorBuilder.java 100.00% <100.00%> (ø)
.../block/server/mediator/LiveStreamMediatorImpl.java 100.00% <100.00%> (+5.88%) ⬆️
...om/hedera/block/server/metrics/MetricsService.java 100.00% <100.00%> (ø)
.../hedera/block/server/persistence/storage/Util.java 100.00% <100.00%> (ø)
...ver/persistence/storage/read/BlockAsDirReader.java 100.00% <100.00%> (ø)
...sistence/storage/read/BlockAsDirReaderBuilder.java 100.00% <100.00%> (ø)
... and 6 more

AlfredoG87
AlfredoG87 previously approved these changes Aug 12, 2024
Copy link
Contributor

@AlfredoG87 AlfredoG87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

- Integration with LMAX Disruptor RingBuffer
- Changed toy types to align with names and relationships like
  hedera-protobufs
- Boosted unit test coverage to 100% for most packages
- Adjusted Docker container to run as the hedera user rather than root
- Changed the consumer bidirection stream to server streaming
- Added exception handling when reading and writing to storage
- Refactored getBlock to singleBlock to align with hedera-protobufs rpc service definition
  and types
- Separated ReadBlock and WriteBlock
- Added RemoveBlock to handle removal of partially written Blocks when an exception is thrown
- Adjusted producer and consumer scripts to work with new types
- Added metrics to report: live block items counted, blocks persisted, single blocks retrieved
  and current subscribers

Signed-off-by: Matt Peterson <[email protected]>
@mattp-swirldslabs mattp-swirldslabs force-pushed the 00029-refactor-demo-disruptor branch from fdd0f5c to b40101b Compare August 12, 2024 15:43
Copy link
Contributor

@AlfredoG87 AlfredoG87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mattp-swirldslabs mattp-swirldslabs merged commit 0ca1c5e into main Aug 12, 2024
5 checks passed
@mattp-swirldslabs mattp-swirldslabs deleted the 00029-refactor-demo-disruptor branch August 12, 2024 15:48
@mattp-swirldslabs mattp-swirldslabs restored the 00029-refactor-demo-disruptor branch August 13, 2024 22:31
@mattp-swirldslabs mattp-swirldslabs deleted the 00029-refactor-demo-disruptor branch August 13, 2024 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor Live Stream Mediator to decouple producer from consumer
2 participants