Is it ok if i create my own version based on this?

[magralo]

My current changes

Hey! I am using this library with some adjustments, basically what I am doing is the following:

1. Only allowing explicit construction of each node specifying dependencies.
2. Managing exceptions in each node

• To allow multiple runs of each node (Just like Airflow)
• To avoid failing the whole process... if node A fails, we should be able to run all the nodes that do not have that dependency.

3. Ploting the DAG with pyvis on each run to visualize the states of the DAG after the execution

• This allows us to understand the dependencies of each node
• Each node has one of 3 states, Done (green), Failed (red) or Not run(yellow)

Coming ideas

In the near future I want to build one different functionality where we can specify two different types of dependencies:

• Hard Dependency (current implementation): This should be used when we need to pass the result to the next function/node as a python Variable
• Soft Dependency: This happens when we need to run one process after another but we do not need the result as a Python variable.

The latter is useful when working with queries (which is my case)... As an example of both kinds of dependencies, let's think about a simple DAG with 2 nodes

1. Pre process the data
2. Run a simple ML model where the result of (1) is the input.

• if the result of the first node (and the input of the second one) is a pandas dataframe the dependency would be hard.
• If the result of the first node is that we overwrite a SQL table ("mlmodel1.input") that means that we do not really need the result of the first node but we need to run that node after the first one... That would be a soft dependency

How am i using this on prod?

Following the license type I made those changes, I currently run this in production where I work. What I do is that I have an API where each endpoint is a different Dag.

• I deployed the API on cloud run
• Each endpoint has an automatic scheduler using cloud scheduler

So far everything is working perfectly! Thanks a lot for this development

As a side note ill be talking about this on a local conference (Pycon Colombia 2023)

Please let me know what do you think about this fork

[magralo]

library: https://pypi.org/project/easydags/#description
repo: https://github.com/magralo/easydags