Skip to content

Commit

Permalink
Revert "Merge branch 'master' into dev-vicente"
Browse files Browse the repository at this point in the history
This reverts commit 79ed3b6, reversing
changes made to dc1f7ef.
  • Loading branch information
micafer committed Nov 25, 2024
1 parent 79ed3b6 commit 7ad4c83
Show file tree
Hide file tree
Showing 76 changed files with 286 additions and 4,229 deletions.
3 changes: 1 addition & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,4 @@ about: Create a report to help us improve

### Possible Solution

Any thoughts as to potential solutions or ideas to go about finding one.
Please include links to any research.
Any thoughts as to potential solutions or ideas to go about finding one. Please include links to any research.
25 changes: 11 additions & 14 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,19 @@ orientation.
Examples of behavior that contributes to creating a positive environment
include:

- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members
- Using welcoming and inclusive language
- Being respectful of differing viewpoints and experiences
- Gracefully accepting constructive criticism
- Focusing on what is best for the community
- Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

- The use of sexualized language or imagery and unwelcome sexual attention or
advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic
address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a
professional setting
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting

## Our Responsibilities

Expand All @@ -55,7 +52,7 @@ further defined and clarified by project maintainers.
## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at <[email protected]>. All
reported by contacting the project team at [email protected]. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# OSCAR - Open Source Serverless Computing for Data-Processing Applications

[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar/v3)](https://goreportcard.com/report/github.com/grycap/oscar/v3)
[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://app.codacy.com/gh/grycap/oscar/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
[![Go Report Card](https://goreportcard.com/badge/github.com/grycap/oscar)](https://goreportcard.com/report/github.com/grycap/oscar)
[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/8145efdfb9d24af1b5b53e21c6e2df99)](https://www.codacy.com/gh/grycap/oscar/dashboard?utm_source=github.com&utm_medium=referral&utm_content=grycap/oscar&utm_campaign=Badge_Coverage)
[![tests](https://github.com/grycap/oscar/actions/workflows/tests.yaml/badge.svg?branch=master)](https://github.com/grycap/oscar/actions/workflows/tests.yaml)
[![build](https://github.com/grycap/oscar/workflows/build/badge.svg)](https://github.com/grycap/oscar/actions?query=workflow%3Abuild)
[![GitHub release (latest by date)](https://img.shields.io/github/v/release/grycap/oscar)](https://github.com/grycap/oscar/pkgs/container/oscar)
Expand Down
2 changes: 1 addition & 1 deletion deploy/ansible/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Ansible playbook to deploy K3s and the OSCAR platform

Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.
Please refer to the [docs](https://docs.oscar.grycap.net/deploy-ansible/) for instructions.
4 changes: 0 additions & 4 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,4 @@ OSCAR exposes a secure REST API available at the Kubernetes master's node IP
through an Ingress Controller. This API has been described following the
[OpenAPI Specification](https://www.openapis.org/) and it is available below.

> ℹ️
>
> The bearer token used to run a service can be either the OSCAR [service access token](invoking-sync.md#service-access-tokens) or the [user's Access Token](integration-egi.md#obtaining-an-access-token) if the OSCAR cluster is integrated with EGI Check-in.
!!swagger api.yaml!!
15 changes: 1 addition & 14 deletions docs/api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ paths:
description: List all created services
security:
- basicAuth: []
- token: []
tags:
- services
post:
Expand All @@ -51,7 +50,6 @@ paths:
description: Create a service
security:
- basicAuth: []
- token: []
requestBody:
content:
application/json:
Expand All @@ -76,7 +74,6 @@ paths:
description: Update a service
security:
- basicAuth: []
- token: []
requestBody:
content:
application/json:
Expand Down Expand Up @@ -111,7 +108,6 @@ paths:
operationId: ReadService
security:
- basicAuth: []
- token: []
description: Read a service
delete:
summary: Delete service
Expand All @@ -128,7 +124,6 @@ paths:
description: Delete a service
security:
- basicAuth: []
- token: []
tags:
- services
'/system/logs/{serviceName}':
Expand Down Expand Up @@ -162,7 +157,6 @@ paths:
operationId: ListJobs
security:
- basicAuth: []
- token: []
description: List all jobs with their status
delete:
summary: Delete jobs
Expand All @@ -179,7 +173,6 @@ paths:
description: Delete all jobs from a service.
security:
- basicAuth: []
- token: []
parameters:
- schema:
type: boolean
Expand Down Expand Up @@ -221,7 +214,6 @@ paths:
description: Get the logs from a job
security:
- basicAuth: []
- token: []
parameters:
- schema:
type: boolean
Expand All @@ -242,7 +234,6 @@ paths:
description: Delete a job
security:
- basicAuth: []
- token: []
tags:
- logs
/system/info:
Expand All @@ -265,7 +256,6 @@ paths:
description: Get system info
security:
- basicAuth: []
- token: []
/health:
get:
summary: Health
Expand Down Expand Up @@ -326,7 +316,6 @@ paths:
description: Get system configuration
security:
- basicAuth: []
- token: []
'/run/{serviceName}':
parameters:
- schema:
Expand Down Expand Up @@ -618,6 +607,4 @@ servers:
- url: 'https://localhost'
description: 'Local testing'
- url: 'https://inference.cloud.ai4eosc.eu'
description: 'AI4EOSC OSCAR cluster'
- url: 'https://inference-walton.cloud.imagine-ai.eu'
description: 'iMagine OSCAR cluster'
description: 'AI4EOSC OSCAR cluster'
Binary file removed docs/images/bucket-list.png
Binary file not shown.
Binary file removed docs/images/minio-ui.png
Binary file not shown.
Binary file removed docs/images/oidc/egi-checkin-token-portal.png
Binary file not shown.
Binary file removed docs/images/oscar-info.png
Binary file not shown.
16 changes: 1 addition & 15 deletions docs/integration-egi.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ grant access for all users from that VO.

The static web interface of OSCAR has been integrated with EGI Check-in and
published in [ui.oscar.grycap.net](https://ui.oscar.grycap.net) to facilitate
the authorization of users. To login through EGI Check-In using OIDC tokens,
the authorization of users. To login through EGI Checkín using OIDC tokens,
users only have to put the endpoint of its OSCAR cluster and click on the
"EGI CHECK-IN" button.

Expand All @@ -87,17 +87,3 @@ create a new account configuration for the
After that, clusters can be
added with the command [`oscar-cli cluster add`](oscar-cli.md#add) specifying
the oidc-agent account name with the `--oidc-account-name` flag.

### Obtaining an Access Token

Once logged in via EGI Check-In you can obtain an Access Token with one of this approaches:

* From the command-line, using `oidc-agent` with the following command:

```sh
oidc-token <account-short-name>
```
where `account-short-name` is the name of your account configuration.
* From the EGI Check-In Token Portal: [https://aai.egi.eu/token](https://aai.egi.eu/token)

![egi-checkin-token-portal.png](images/oidc/egi-checkin-token-portal.png)
8 changes: 4 additions & 4 deletions docs/invoking-async.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

For event-driven file processing, OSCAR automatically manages the creation
and [notification system](https://docs.min.io/minio/baremetal/monitoring/bucket-notifications/bucket-notifications.html#minio-bucket-notifications)
of MinIO buckets. This allow the event-driven invocation of services
using asynchronous requests for every file uploaded to the bucket, which generates a Kubernetes job for every file to be processed.
of MinIO buckets in order to allow the event-driven invocation of services
using asynchronous requests, generating a Kubernetes job for every file to be
processed.


![oscar-async.png](images/oscar-async.png)

These jobs will be queued up in the Kubernetes scheduler and will be processed whenever there are resources available. If OSCAR cluster has been deployed as an elastic Kubernetes cluster (see [Deployment with IM](https://docs.oscar.grycap.net/deploy-im-dashboard/)), then new Virtual Machines will be provisioned (up to the maximum number of nodes defined) in the underlying Cloud platform and seamlessly integrated in the Kubernetes clusters to proceed with the execution of jobs. These nodes will be terminated as the worload is reduced. Notice that the output files can be stores in MinIO or in any other storage back-end supported by the [FaaS supervisor](oscar-service.md#faas-supervisor).

If you want to process a large number of data files, consider using [OSCAR Batch](https://github.com/grycap/oscar-batch), a tool designed to perform batch-based processing in OSCAR clusters. It includes a coordinator tool where the user provides a MinIO bucket containing files for processing. This service calculates the optimal number of parallel service invocations that can be accommodated within the cluster, according to its current status, and distributes the image processing workload accordingly among the service invocations. This is mainly intended to process large amounts of files, for example, historical data.
4 changes: 2 additions & 2 deletions docs/invoking-sync.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,8 @@ base64 input.png | curl -X POST -H "Authorization: Bearer <TOKEN>" \

## Service access tokens

As detailed in the [API specification](api.md), invocation paths require either the
service access token or the Access Token of the user when the cluster is integrated with EGI Check-in, in the request header for authentication (any of them is valid). Service access
As detailed in the [API specification](api.md), invocation paths require the
service access token in the request header for authentication. Service access
tokens are auto-generated in service creation and update, and MinIO eventing
system is automatically configured to use them for event-driven file
processing. Tokens can be obtained through the API, using the
Expand Down
13 changes: 2 additions & 11 deletions docs/invoking.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,7 @@

OSCAR services can be executed:

- [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained.
- [Synchronously](invoking-sync.md), so that the invocation to the service blocks the client until the response is obtained. Useful for short-lived service invocations.
- [Asynchronously](invoking-async.md), typically in response to a file upload to MinIO or via the OSCAR API.
- As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. Jupyter Notebook)


After reading the different service execution types, take into account the following considerations to better decide the most appropriate execution type for your use case:

* **Scalability**: Asynchronous invocations provide the best throughput when dealing with multiple concurrent data processing requests, since these are processed by independent jobs which are managed by the Kubernetes scheduler. A two-level elasticity approach is used (increase in the number of pods and increase in the number of Virtual Machines, if the OSCAR cluster was configured to be elastic). This is the recommended approach when each processing request exceeds the order of tens of seconds.

* **Reduced Latency** Synchronous invocations are oriented for short-lived (< tens of seconds) bursty requests. A certain number of containers can be configured to be kept alive to avoid the performance penalty of spawning new ones while providing an upper bound limit (see [`min_scale` and `max_scale` in the FDL](fdl.md#synchronoussettings), at the expense of always consuming resources in the OSCAR cluster. If the processing file is in the order of several MBytes it may not fit in the payload of the HTTP request.

* **Easy Access** For services that provide their own user interface or their own API, exposed services provide the ability to execute them in OSCAR and benefit for an auto-scaled configuration in case they are [stateless](https://en.wikipedia.org/wiki/Service_statelessness_principle). This way, users can directly access the service using its well-known interfaces by the users.
- As an [exposed service](exposed-services.md), where the application executed already provides its own API or user interface (e.g. a Jupyter Notebook)

50 changes: 0 additions & 50 deletions docs/minio-usage.md

This file was deleted.

12 changes: 1 addition & 11 deletions docs/oscar-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ is in charge of:



### FaaS Supervisor
### Input/Output

[FaaS Supervisor](https://github.com/grycap/faas-supervisor), the component in
charge of managing the input and output of services, allows JSON or base64
Expand All @@ -37,16 +37,6 @@ The output of synchronous invocations will depend on the application itself:

This way users can adapt OSCAR's services to their own needs.

The FaaS Supervisor supports the following storage back-ends:
* [MinIO](https://min.io)
* [Amazon S3](https://aws.amazon.com/s3/)
* Webdav (and, therefore, [dCache](https://dcache.org))
* Onedata (and, therefore, [EGI DataHub](https://www.egi.eu/service/datahub/))

### Container images

Container images on asynchronous services use the tag `imagePullPolicy: Always`, which means that Kubernetes will check for the image digest on the image registry and download it if it is not present.
So, if you are using an image without a specific tag or with the latest tag, the service will automatically download and use the most recent version of the image on its executions, whenever the image is updated.

You can follow one of the
[examples](https://github.com/grycap/oscar/tree/master/examples)
Expand Down
2 changes: 1 addition & 1 deletion examples/plant-classification-theano/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,5 @@ To run this example you need:

1. Once the function is executed, the output is automatically copied to the
output bucket in minio, in this case `plant-classifier-out`. You can
download the output from here for further processing.
download the ouput from here for further processing.
![minio-out.png](img/Minio-OUT.png)
2 changes: 1 addition & 1 deletion examples/plants-classification-tensorflow/script.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

IMAGE_NAME=`basename "$INPUT_FILE_PATH" | cut -d. -f1`
IMAGE_NAME=`basename "$INPUT_FILE_PATH"`
OUTPUT_FILE="$TMP_OUTPUT_DIR/output.json"

deepaas-cli predict --files "$INPUT_FILE_PATH" 2>&1 | grep -Po '{.*}' > "$OUTPUT_FILE"
Expand Down
24 changes: 0 additions & 24 deletions examples/yolov8/README.md

This file was deleted.

Binary file removed examples/yolov8/img/cat.jpg
Binary file not shown.
4 changes: 0 additions & 4 deletions examples/yolov8/img/cat.jpg:Zone.Identifier

This file was deleted.

Loading

0 comments on commit 7ad4c83

Please sign in to comment.