Allow both single-step and multi-step pipelines

[mariajgrimaldi]

Description

The current implementation of pipelines in the Open edX environment mainly features single-step pipelines. While multi-step pipelines can offer flexibility, they also introduce unnecessary complexity that may not always be needed. The objective of this feature request is to generalize the pipeline architecture to support both single-step and multi-step pipelines effectively, allowing developers to choose the appropriate level of complexity for their specific use cases.

Objectives:

• Enable the use of both single-step and multi-step pipelines within the Open edX environment.
• Provide a flexible architecture that allows developers to select the appropriate pipeline complexity based on their needs.

Expected Result:

• A flexible pipeline architecture that supports both single-step and multi-step pipelines.
• Improved developer experience by allowing the selection of pipeline complexity according to specific use cases.
• Clear documentation and guidelines to assist developers in implementing and maintaining both types of pipelines.

[mariajgrimaldi]

This issue is up for discussion, I created it based on the feedback received in the Developer Feedback survey, here's the original comment:

All the pipelines that I have come across are single hops ... in my view, the multi-hop pipeline usecase, implemented in the architecture, is turning out to be an edge case. The multi hops create unnecessary complexity and might be an overkill in the OpenEdx environment.