Support for failure handling

[yarnabrina]

Do Elyra support to allow failure in specific nodes?

For example, we have a fixed (and known) number of models running on a training dataset in our kubeflow pipeline. These happen in parallel in same stage. At end of these, i.e. in the next stage (where all of these nodes are inputs), we compare validation metric, pick the best and proceed to prediction.

Now it's possible that in a scheduled pipeline, sometimes one or two models fail. It nay be due to code/data issues (hopefully not), or due to pod failure or something. For this question, reason of failure is not important. In these situation, the whole pipeline fails. And not only that, if model A fails and model B is still running, it'll wait for that to complete.

But we would want it to continue. Since we compare and pick best model on next stage, it's okay for few to fail as long as not all, and next stage can choose from the successful ones. A top level broad except may solve this, but then we'll probably loose the information that the node failed, as no one really goes through logs unless there's a failure (or at least warning) as the pipeline (or node) level.

So, the following is something that's good to have:

1. If a model fails due to some system failure (detecting may be difficult though), attempt to retry.
2. If a model fails due to code failure, allow it to fail.
3. If it's not possible to allow failure, fail fast as soon as one model fails instead of waiting for all only to fail a little late.

Does Elyra support this, or have any plan to support this?

---

This is very related to this question I asked on StackOverflow yesterday. But it seems to me that even if a solution is available to kfp.dsl natively, similar functionality in Elyra may be a bit different.

[akchinSTC]

Hi @yarnabrina
we have some somewhat related open issues involving this topic

Support for complex pipeline operations #408
Support conditions / branching on pipelines #1420

In short, this sounds like it could be more of an upstream feature request since after constructing the pipeline in our editor, we hand off hand off execution to KFP. Unfortunately AFAIK, I don't think KFP has this type of functionality yet.

[yarnabrina]

Got it. I also felt this is the case, but thought I may have been unfamiliar with few KFP features. Let me create a feature request ypstream, and explore alternatives meanwhile. Thanks for your reply :)