Complex numaflow as a step in a workflow

[ppadovani]

I have a possible use case and I'm trying to understand if numaflow will fit. In a nutshell, I'd like to:

• argo event from a source trigger an argo workflow
• argo workflow executes step(s) that either trigger or are a numaflow to process some set of data
• workflow step completes when the bounded data stream has completed processing in a numaflow
• workflow completes

Is this possible? I have just the basic understanding of each of these pieces, but must be missing something to help me figure out how to pull this together.

Thanks!

[syayi]

Thanks @ppadovani ! Can you share a little bit more about your use case and what you're trying to achieve? What are the other steps in the argo workflow?

Numaflow is for unbounded stream processing

[ppadovani]

There are really two interelated use cases.. one is a keyed unbounded stream that processes text and creates a final set of content per key from the set of text being processed. The second use case is one where a full rebuild of the content is requested, and user facing knowledge of the process/status of the rebuild is required. So the other steps are really control flow steps mostly.

I had already considered chaining pods/jobs using Keda in a similar fashion to numaflow, but really liked the built in windowing and such.

[syayi]

Ah I see, makes sense. Yes, Numaflow works for your first use case to do unbounded stream processing but having it as part of a workflow might be tricky, you'd have to track if an event has been processed by numaflow or not, so that you can use the event status for workflow

[ppadovani]

Just giving this some thought.. if I sent structured data (JSON) that contained the payload and some tracking/status info, there is no reason a custom sink couldn't emit the status/tracking event to another target(redis/sqs/etc) that the workflow is waiting on. This way, both the unbounded and bounded could use the same flow, it's just the bounded would have a bit of additional control data associated with the stream that would trigger different behavior on the sink.

The only thing I can see wrong with this would be trying to cancel/kill the bounded stream... I could stop emitting to the source, and emit a cancelled control msg, but I couldn't stop any of the inflight processing. I guess I could possibly have a control bus kafka/sqs that the different steps of the flow listen on.....

[syayi]

Yes, the first part of it makes sense. You can track the event status using additional metadata and listen for that to finish your workflow.

When you say kill? Is it killing the uber workflow? If yes, given the two systems are independent once the event is consumed by numaflow it would continue processing and there isn't a way for you to act on the kill event that is emitted. Let me know if I understood it correctly? Happy to hop on a call and discuss if you prefer (here is our slack )

[ppadovani]

I should have been more clear... consider a msg topic (think kafka) and each node in the flow is a separate consumer group. When cancelling a flow, we stop sending data, and send a cancel msg. We also put a cancel msg on the topic. Each node receives the cancel message and stops processing msgs for that 'job/id'. When the event containing the cancel shows up, each node forwards to next step(s) where eventually it will hit the sink, and the sink can emit that the job was fully cancelled.