Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add evaluation matrix #141

Merged
merged 11 commits into from
Jan 8, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/use-case-evaluation/evaluation-matrix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# VEDA Auth Solution Evaluation Matrix

The below table refers to the previously [defined auth workflows](https://github.com/NASA-IMPACT/veda-auth-central/blob/main/docs/use-case-evaluation/workflows.md).

Our goal with this evaluation is to find out whether there is a solution that fulfils the workflows while relying on existing community solutions as much as possible,
only with IaC and durable configuration of the solution.

Building a replicable platform from community-maintained upstream solutions is a principle of the [VEDA Open Source Ecosystem](https://docs.openveda.cloud/open-source-ecosystem/).

The evaluated options are [KeyCloak](https://www.keycloak.org/) and [Apache Airavata Custos](https://airavata.apache.org/custos/), which uses KeyCloak under the hood.


## Explanation of the evaluation criteria

1. Can this workflow already be satisfied with this technology without having to *build* new things?
2. Did we (the NASA VEDA team) *build* this, in contrast to *deploying* a specific configuration of features built by an upstream community?
3. Level of effort to implement and document the workflow solution, in developer sprints (2 weeks)
4. Ongoing effort required to maintain this workflow in the future. This includes keeping up with upstream releases.
5. How easy is it to extend and modify this workflow in the future?


## Explanation of the values

To evaluate these workflows, each cell in the below shall be given

1. yes/no
2. yes/no
3. Estimated number of developer sprints (2 weeks)
4. Numerical score (1-5, where 5 is best)
5. Numerical score (1-5, where 5 is best)

## Matrix

| [Workflow](https://github.com/NASA-IMPACT/veda-auth-central/blob/main/docs/use-case-evaluation/workflows.md) | (E1) Already implemented | (E2) Built by VEDA | (E4) Level of Effort | (E3) Ongoing Maintenance Effort | (E6) Ease of future change |
| - | - | - | - | - | - |
| W1 KeyCloak | | | | | |
| W1 Custos | | | | | |
| W2 KeyCloak | | | | | |
| W2 Custos | | | | | |
| W3 KeyCloak | | | | | |
| W3 Custos | | | | | |
| W4 KeyCloak | | | | | |
| W4 Custos | | | | | |

## Cost estimates

The evaluation criteria include level of effort to implement. This is both for project costs but even more for project planning - how much will it take to get to a solution for a workflow.
This assumes that there is only a smaller additional effort required for implementing a concrete use case that makes use of a given workflow.

It seems like there will be no other costs than developer time:
1. It seems like there will be no subscription costs - there seem to be different ways for us to get around paying for a new CILogon subscription https://github.com/NASA-IMPACT/veda-auth-central/issues/139
2. All solutions are/will be open-source, so no license fees either.
3. Infrastructure costs should also be negligible - probably some low-powered EC2 instances for the KeyCloak and maybe Custos services + UI, etc.