Autoscaler - General solution keep EBS for 1 day

[sanderegg]

Goal: Achieve caching of data using EBS volumes

Current situation and usual scenario

• user A, B and C each start a service A, B, C respectively
• service A,B, C are pending, waiting for cluster resources
• autoscaling picks this up and provisions EC2 instances (in sim4life.io that will be 1 per service, in osparc.io it will be based on needed resources and the service might share machines)
• once the machine(s) are up and running, the service are started there

Key take aways here

• autoscaling does not know anything about user, project, node,
• autoscaling cannot attach a specific EBS volume to a machine, because it does not control where a service will finally end up

Proposed changes

Option 1: Autoscaler takes care of it

Autoscaling service:

• autoscaling can know from the docker service labels for which user/project/node the service is needed
• when connecting the node, it can label the Docker node with user/project/node ids
• autoscaling can then attach a EBS volume to the machine (note that the EBS volume shall be tagged with the necessary user_id/project_id/node_id as well)
• when the service is shutdown, the autoscaling service shall remove the user/project/node labels
• the autoscaling shall detach the EBS volume,
• the autoscaling shall terminate the EBS volume if needed or keep it until it shall be deleted

Director-v2

• when creating the docker services, there should be additional docker placement constraints such as node.labels.user_id==user_id, node.labels.project_id==project_id and/or node.labels.node_id==node_id

Pros/Cons:

• this should partially or entirely fix Risk of data loss when restarting dynamic sidecar on autoscaled instance #5862 another user cannot steal the machine anymore, if we also lock by project/node then even the same user cannot steal the machine,
• this is a good first step towards Enhancement: Lock a EC2 machine for a specific project #5669
• all EC2 related calls for autoscaling are in the autoscaling service --> simplify maintenance
• the EBS-volume is pre-mounted in the EC2 machine,
• probably next to no changes in the dynamic-sidecar,
• increase complexity of autoscaling service
• Needs modifications on both director-v2 and autoscaling and that should be maintained in sync

Option 2: Dynamic-sidecar takes care of it

Dynamic-sidecar:

• before starting the service, it shall create/re-use a EBS volume and attach it to the running EC2 instance where it is running,
• once it is up, it needs to mount the EBS volume on the machine where it is running,
• possibly format the drive, and mount the drive in the user containers,
• when the service is shutdown,
• the dynamic sidecar shall terminate or detach the drive depending on preferences,

Autoscaling:

• shall manage the lifecycle of the EBS volumes and terminate the ones that are not needed anymore

Pros/Cons:

• the handling of EBS caching is contained within the service that handles the use services (or a subsequent sidecar),
• there are almost no change in the director-v2
• there are still things to modify in autoscaling, so the complete lifecycle is divided between 2 services
• this does not help in other github issues for now

(## Changes

• Add groups_extra_properties setting to enable/disable cache feature
• Add some service label to indicate the EBS shall be kept up
• The EBS storage shall not be terminated upon EC2 shutdown procedure
• In case there is an EBS storage available for a service for UserX/ProjectY it shall be re-used
• The EBS volumes shall be auto-removed after X day(s))