awesome - but have a question

[walshe]

First off, great blog article and great example that you have shown ..really great. This is not an issue I am raising, more a question I wanted to ask...

I have this 'post registration' process in my app that occurs in the background after user registers.

Here i have to page an external api multiple times to get many many pages of items. Each page can return up to 250 items depending in how I request it.

I am using a simple/complex (depending how you look at it) orchestration of lambdas to process each response and re-paginate until I have all pages retrieved from the external api.

Since SQS has 10 message push limit and SNS has 1 message push limit. I am doing the following

1. GetPageOfDataFromExternalApiLambda - fetchs page of 50 items from rest api. After that it async invokes IteratePageOfDataLambda passing it the page of items in the payload and then calls itself (GetPageOfDataFromExternalApiLambda) again with incremented page number

2. IteratePageOfDataLambda - takes the page of items and async invokes EnqueueIndividualItemLambda passing one item to process along with the remaining items in the page left to process. (item to process was shift()'ed off the list)

3. EnqueueIndividualItemLambda - pushes 1 item to SNS and then async invokes IteratePageOfDataLambda again.

4. On the other side of SNS i simply use a lambda save each item to dynamodb. The lambda can scale well here to handle many items coming in from SNS. I use SNS so that I can retry any failures and use DLQ. It also gives me more options in future, e.g I can have SNS send all items to a SQS queue too.

Note- I have to use page sizes of 50 as Lambda payloads can only handle 128Kb

All the above seems a lot of work and maintenance though. I am wondering if Kenesis is the proper tool to use?

If I use Kinesis I could send all 250 items at once straight to kinesis stream. But I have the following reservations which i was hoping you could help with:

1. Is Kinesis wasted for this task ? its not as if it would be getting used heavily all day, It would be used only after registrations. But maybe I could also reuse it as a regular queue for other use cases?

2. If I have a lambda consuming the Kinesis stream and lets say I am consuming 250 at a time. Is it reasonable to invoke 250 lambdas in parallel (with promises like in your promise.ts) to handle them and would that imply a larger timeout setting for the consumer? or .. could I just set the consumer to read one record from kinesis at a time ?

3. ordering... if I sent a "end marker" message just after I sent my final batch of items to Kinesis, it is possible to consume this on the kinesis side 'in order' i.e. after all other messages have been processed so that I know when my job has actually all been done and can trigger an email etc