New Tool: Events indexing service - Milestone 3 #95
Conversation
|
See some context here - #45 and #74 The main thing I want to highlight here is that we want to prove feasibility of the product first as a development tool used along side the Flow Emulator. Because of this we are doing some caching locally in memory for data and it is by-design that these things will not be persisted. I think the important thing to review here is the interfaces and see if they make sense and we can implement more "production" grade tech down the line (e.g. DB, Cloud, etc). Let me know if you agree with this approach. In Milestone 4 what I am really pushing for is a simple CLI you run alongside the Flow Emulator that let's you build/test dapps events very quickly. |
|
I followed the instructions here https://www.flowrider.io/install to get the project running but I'm getting: Am I doing something wrong? |
|
Hmmm seems like might be a PATH issue. Can you try running FYI: This is not the long-term UX from installation, once this is in a public NPM repo |
|
Thanks yeah I got it working. The binary ended up in The console log here seems confusing. The logic to check if we have already processed a block is after the log so it might be misleading if we're double processing all these block heights. We also need to change "Exiting" as the program is not exiting at that point. Looking at the repo generally, I think it's came together well. Thanks for the great work! |
|
Yeah that makes sense, I will play around with some dynamic console logging like using something like this - https://www.npmjs.com/package/cli-progress I am curious on your thoughts about what to do once we have read all events and are at the "latest" block height. While waiting for new blocks to seal, should we just continually loop? Or is there some average "time to seal block" time that we can sleep on so we don't continually ping that API while waiting. |
|
At that point we're just waiting for a new block to be available to process, so the message can be "Processed up to height 424242. Waiting for a new block ..." on each iteration. The Flow block time is ~1s so if you loop with that interval it would roughly match the block seal rate. In reality the number can differ (right now it's ~2.5s sometimes). |
|
OK made the terminal output cleaner, see - https://gyazo.com/d2bc83c2489f5d3e69f53cfc2e3bffc9 (the right terminal). Much more dynamic in that we will only show the latest actions. Additionally, this will work in a no TTY setup and simply just log the entire line. |
|
Let me know if you would like to see anything else specific to this Milestone, otherwise feel free to merge and ill work on the comments for Milestone 4. |
|
Nice! |
|
This will actually be supported by default if we detect a non-TTY environment, e.g. CloudWatch Logs, Docker logs, etc. See - https://gyazo.com/1bccbdb42f97b2d798e72ea58d0a5904 - for an example of this. That being said, I will make this a CLI argument as well, e.g. |
|
Oh nice thank you for the clarification. I think we're good enough for acceptance criteria here. Nice work! |
|
merge in the PR per Navid's comment |
New Tool: Events indexing service - Milestone 3
Description
This PR is for issue #15
Submission Links & Documents
Code Package: https://github.com/prpatel05/FlowRider (reach out if you need access)
> The service should deliver each event to an external consumer (i.e. a dapp backend) as a JSON payload.
>> At a minimum, the service should support event delivery via HTTP webhooks. In this scenario, the consuming application will provide an HTTP endpoint as part of the startup configuration. The consumer should respond with appropriate status consumes to indicate a successful or unsuccessful delivery — these status codes are defined by the indexer.
Event Publisher Interface - This allows different publishing mechanism beyond HTTP, e.g. SNS, SQS, etc. For now I implement two publishers -
Console Log - Really straightforward, outputs the event to the console. Useful for testing just to validate that everything is configured and working. Don't expect anyone to use this, but shows the ability to have >1 publisher and how simple it is to implement it.
HTTP - POSTs the event to a configured endpoint and expects the response to be a 200 || 204. Additionally, includes a signature so that the consumer can verify the event came from the trusted source (for now the secret is provided by the customer, this can be provided by a managed service as well).
>> The service only needs to support delivery to a single consumer.
Done here - Currently just one publisher, however, it should be simple to make this an Array and multiplex the events.
>> All events should be delivered in the same order they were emitted on-chain.
Done here - JS is single threaded and we await each publish event. Additionally, we will stop processing events if one fails and pick up from where we last tried.
>> Each event should be delivered exactly once. The indexer will need to implement logic to ensure idempotency of events.
Interface
In Memory Implementation - We moved this into the implementing logic based on feedback from Milestone 2. For now I just implemented a simple in-memory store. However, post Flip Fest, we can make this more "Production" ready and use a DB/local DB to have this data persisted.
Tests
Requirements Check
Other Details