[Workflow] How to run the workflows "distributed"?

[paule96]

Hi Dapr team,

I lately take a look into the new workflow feature of dapr.
And as far as I understand is an activity in an workflow not more then an actor.
And a workflow itself is like a component around the activities.

So for me every call of CallChildWorkflowAsync or CallActivityAsync goes to the sidecare. This looks into the placement lookup table (or updates his own copy) and send this request to the correct instance / node that hosts this Activity or workflow to run it.

It's also written in the CallChildWorkflowAsync documentation the following documentation:

     //    • You can distribute workflow logic across multiple compute nodes concurrently,
    //     which is useful if your workflow logic otherwise needs to coordinate a lot of
    //     tasks.

To make my issue easy to understand I created an example repo.

The example repo contains basicall 3 applications:

• daprWorkflowDistributedIssue.ApiService
  • is exposed in dapr as WeatherApi
  • provides the Microsoft most common WeatherAPI sample with one single easy endpoint /weatherforecast
• daprWorkflowDistributedIssue.DaprWorkflow1
  • a simple console app that starts an apphost and register some dapr things
    • a simple daprworkflow that contains just one activity called CallWeatherWorkflow
    • the activity will call the WeatherApi
• daprWorkflowDistributedIssue.DaprWorkflow2
  • a simple console app that starts an apphost and register some dapr things
    • a simple daprworkflow that will call the workflow CallWeatherWorkflow

The problem is now that the workflow CallWeatherWorkflow works just okay. But when the workflow GetWeatherAndMood tries to call CallWeatherWorkflow you will get the following response:

{
  "instanceID": "fb8c3106-a8a0-4474-a3e2-832a7e5374ce",
  "workflowName": "CallWeatherWorkflow",
  "createdAt": "2024-02-23T13:01:49.870327683Z",
  "lastUpdatedAt": "2024-02-23T13:01:49.919064012Z",
  "runtimeStatus": "FAILED",
  "properties": {
    "dapr.workflow.custom_status": "",
    "dapr.workflow.failure.error_message": "No orchestrator task named 'CallWeatherWorkflow' was found.",
    "dapr.workflow.failure.error_type": "OrchestratorTaskNotFound",
    "dapr.workflow.input": ""
  }
}

It looks for me that I can only interact with workflows / activities that are hosted in the same instance of the application. But I can't call to other workflows hosted somewhere else in my dapr application.
This creates for me also the question how to scale these workloads or even better how to scale the activities. In special in the Fan-Out pattern.

Hope someone can help me to understand, what i miss here :)

[philliphoff]

It looks for me that I can only interact with workflows / activities that are hosted in the same instance of the application. But I can't call to other workflows hosted somewhere else in my dapr application.

@paule96 This is true, only the application that hosts a workflow can start one (and use only activities that have also been registered in that application). To have other Dapr applications interact with them you must expose a new endpoint that can be called via, for example, Dapr service invocation. That endpoint can then start/stop/manage the workflow.

With respect to scaling, I believe the note you mention refers to spreading execution across multiple replicas of the service (that registers the workflow). That is, there is no affinity for running a workflow's activities to the same node that orchestrates the workflow itself.

[paule96]

Hi @philliphoff,

thanks for the explanation. So the actual activities will be run in the same application or they will be run "somewhere" inside of my dapr "network"?
Or does it just mean you have your sidecar for your application and you scale the application for example to 10 instances of your and it will spread the work to these 10 instances?

[philliphoff]

The activities will only run on the application (instances) in which they've been registered (along with the workflows themselves), but they will (should?) spread across those instances.

[paule96]

@philliphoff This makes sense. Thanks for your explanation and clarification 👍