Implement API step executor

[joshtyf]

Description

Pipeline Execution

I created a basic pipeline executor. The flow is like this:

1. New service request created --> triggers NewServiceRequestEvent (see https://github.com/gookit/event)
2. StepExecutionManager listens for NewServiceRequestEvent and begins working on the request

The logic of executing the request steps is as follows:

1. Fetch the pipeline from the database. This is needed to know what are the steps to execute
2. Iteratively visit each step, calling the various executors registered to the StepExecutionManager. Registration is done via the WithExecutor method (see Golang options design pattern). The parameters submitted via the user when creating a new service request is passed in via context.Context. We will use the same method to pass intermediate results between steps in the future

Seeding

I've also added seeding for the pipeline.

Miscellaneous

• @Zheng-Zhi-Qiang we forgot to add the json embedding previously. Not too sure how you were creating new service requests without this embedding.
• Added default values when creating new service requests
• Added fe and be profiles for docker so that it's easier to run backend only without building the frontend.
  • docker compose --profile be -p flowforge up --build --> for running backend only
  • docker compose --profile be-seed -p flowforge up --build --> for seeding the database

Future TODOs:

• Log each step execution into the SQL database
• Figure out how to rollback/resume in the event of faults and errors
• Tidy up logger using options design pattern
• Create approval step

[Ziyang-98]

So this means different versions of the pipeline will have the same ID and different uuid?

[joshtyf]

yes

[joshtyf]

For example, the admin might create a pipeline called "Create EC2"

the first version will be uuid=1, pipelineId=1, pipelineName=createEC2, version=1

then he updates the pipeline by changing the URL.
so the 2nd version will be uuid=2, pipelineId=1, pipelineName=createEC2, version=2

Updates

I've made the following changes:

• PipelineUuid --> PipelineId
• Old PipelineId removed in favour of PrevVersionId

This way, we can still aggregate pipelines that are of the same ancestry together without the naming confusion.

[Ziyang-98]

I think string is fine since UUID can have hyphens in it

[joshtyf]

tbh, it's not really a UUID. According to the documents, the mongo ObjectID is just a '24 character hexadecimal value'.

I'm just calling it UUID cause I don't want to call it pipelineId in the service request model as it will be confused with the actual pipelineId in the pipeline model.

Maybe I'll change this

Updates

see #38 (comment)

[Ziyang-98]

LGTM. Took note of the models too, thanks. Do make sure that the event listener for NewServiceRequest can handle multiple events asynchronously. Maybe can write some test cases to ensure its working properly.