From 98ecf7e739dc05fb89a0b42e0dd01e0cda2a8c4a Mon Sep 17 00:00:00 2001 From: "Carlos Manzanedo Rueda (ruecarlo@)" Date: Sun, 7 Nov 2021 18:03:27 +0000 Subject: [PATCH] changes to adhere to common hugo setup --- content/rendering-with-batch/_index.md | 16 +++++++++------- .../batch/{ => batch.files}/job_submission.py | 0 .../batch/{ => batch.files}/pottery.blend | Bin .../blendfile.blend | Bin .../diagrams.drawio | 0 .../docker-files/Dockerfile | 0 .../docker-files/render.sh | 0 .../{ => rendering-with-batch.files}/stack.yaml | 0 .../verifying_resilience.py | 0 content/rendering-with-batch/start/_index.md | 2 +- .../rendering-with-batch/start/at-an-event.md | 4 +++- .../rendering-with-batch/start/on-your-own.md | 4 ++-- .../start}/review-outputs.md | 7 +++---- 13 files changed, 18 insertions(+), 15 deletions(-) rename content/rendering-with-batch/batch/{ => batch.files}/job_submission.py (100%) rename content/rendering-with-batch/batch/{ => batch.files}/pottery.blend (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/blendfile.blend (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/diagrams.drawio (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/docker-files/Dockerfile (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/docker-files/render.sh (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/stack.yaml (100%) rename content/rendering-with-batch/{ => rendering-with-batch.files}/verifying_resilience.py (100%) rename {static/pages/rendering-with-batch => content/rendering-with-batch/start}/review-outputs.md (79%) diff --git a/content/rendering-with-batch/_index.md b/content/rendering-with-batch/_index.md index 270adf36..e9d8883a 100644 --- a/content/rendering-with-batch/_index.md +++ b/content/rendering-with-batch/_index.md @@ -6,17 +6,19 @@ weight: 80 pre: "8. " --- +{{% notice info %}} +The estimated completion time of this lab is **60 minutes**. Please note Rendering the animation presented below can incur in costs up to **$15**. +{{% /notice %}} ## Overview -In this workshop you will learn to submit jobs with [AWS Batch](https://aws.amazon.com/batch/) following Spot best practices to [render](https://en.wikipedia.org/wiki/Rendering_(computer_graphics)) a [Blender](https://www.blender.org/) file in a distributed fashion, running a Docker container that you will create and publish to Amazon Elastic Container Registry (ECR). [Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) are EC2 spare capacity offered at steep discounts compared to On-Demand instances and are a cost-effective choice for applications that can be interrupted, what makes them well-suited for the batch processing that we will run. After going through all the sections, you will have the following pipeline created: +In this workshop you will learn to submit jobs with [AWS Batch](https://aws.amazon.com/batch/) following Spot best practices to [render](https://en.wikipedia.org/wiki/Rendering_(computer_graphics)) a [Blender](https://www.blender.org/) file in a distributed way. You will be creating a docker container and publishing it in Amazon Elastic Container Registry (ECR). Then you will use that container in AWS Batch using a mix of EC2 On-Demand and Spot instances. [Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) are EC2 spare capacity offered at steep discounts compared to On-Demand instances and are a cost-effective choice for applications that can be interrupted, what makes them well-suited for the batch processing that we will run. After going through all the sections, you will have the following pipeline created: -1. A python script downloads the Blender file from S3 to extract the number of frames. -2. The script submits a Batch array job (rendering job) of dimension *n* depending on the number of frames. It also submits a single job (stitching job) with a sequential dependency on the rendering job. (You will learn more about job dependencies later). +1. A python script downloads the Blender file from S3 to extract the number of frames from the Blender project. +2. The script submits a Batch using an `array job` with as many tasks as number of frames. It also submits a single stitching job using [FFmpeg](https://ffmpeg.org/) to create a final video file. 3. Each of the jobs in the rendering job array run a Docker container that executes Blender to render a slice of frames, and after uploads them to S3. -4. The stitching job runs a Docker container that downloads all the frames from S3, executes [FFmpeg](https://ffmpeg.org/) (you will learn more about it in a future section) to create a video out of the rendered frames and finally uploads it to S3. -To have an idea of what you will be rendering, take a look to this animation. +The outcome of the workshop would be the following animation. -![Possible output](/images/rendering-with-batch/animation_example.gif) +![Pottery output](/images/rendering-with-batch/animation_example.gif) -This output is the result of rendering the project *[Pottery](https://blendswap.com/blend/28661)* by [Prokster](https://blendswap.com/profile/1012752), that is shared under the [Creative Commons 0](https://creativecommons.org/share-your-work/public-domain/cc0/) license. Head to the next page to take a look at the architecture that you will deploy. +This output is the result of rendering the project *[Pottery](https://blendswap.com/blend/28661)* by [Prokster](https://blendswap.com/profile/1012752). We thank **Prokster** for providing the project under the [Creative Commons 0](https://creativecommons.org/share-your-work/public-domain/cc0/) license. Head to the next page to take a look at the architecture that you will deploy. diff --git a/content/rendering-with-batch/batch/job_submission.py b/content/rendering-with-batch/batch/batch.files/job_submission.py similarity index 100% rename from content/rendering-with-batch/batch/job_submission.py rename to content/rendering-with-batch/batch/batch.files/job_submission.py diff --git a/content/rendering-with-batch/batch/pottery.blend b/content/rendering-with-batch/batch/batch.files/pottery.blend similarity index 100% rename from content/rendering-with-batch/batch/pottery.blend rename to content/rendering-with-batch/batch/batch.files/pottery.blend diff --git a/content/rendering-with-batch/blendfile.blend b/content/rendering-with-batch/rendering-with-batch.files/blendfile.blend similarity index 100% rename from content/rendering-with-batch/blendfile.blend rename to content/rendering-with-batch/rendering-with-batch.files/blendfile.blend diff --git a/content/rendering-with-batch/diagrams.drawio b/content/rendering-with-batch/rendering-with-batch.files/diagrams.drawio similarity index 100% rename from content/rendering-with-batch/diagrams.drawio rename to content/rendering-with-batch/rendering-with-batch.files/diagrams.drawio diff --git a/content/rendering-with-batch/docker-files/Dockerfile b/content/rendering-with-batch/rendering-with-batch.files/docker-files/Dockerfile similarity index 100% rename from content/rendering-with-batch/docker-files/Dockerfile rename to content/rendering-with-batch/rendering-with-batch.files/docker-files/Dockerfile diff --git a/content/rendering-with-batch/docker-files/render.sh b/content/rendering-with-batch/rendering-with-batch.files/docker-files/render.sh similarity index 100% rename from content/rendering-with-batch/docker-files/render.sh rename to content/rendering-with-batch/rendering-with-batch.files/docker-files/render.sh diff --git a/content/rendering-with-batch/stack.yaml b/content/rendering-with-batch/rendering-with-batch.files/stack.yaml similarity index 100% rename from content/rendering-with-batch/stack.yaml rename to content/rendering-with-batch/rendering-with-batch.files/stack.yaml diff --git a/content/rendering-with-batch/verifying_resilience.py b/content/rendering-with-batch/rendering-with-batch.files/verifying_resilience.py similarity index 100% rename from content/rendering-with-batch/verifying_resilience.py rename to content/rendering-with-batch/rendering-with-batch.files/verifying_resilience.py diff --git a/content/rendering-with-batch/start/_index.md b/content/rendering-with-batch/start/_index.md index befbdc1a..76b74a06 100644 --- a/content/rendering-with-batch/start/_index.md +++ b/content/rendering-with-batch/start/_index.md @@ -11,5 +11,5 @@ To start the workshop head to one of the following pages, depending whether you When you finish the workshop, don't forget to execute the commands described in [**Clean up**](/rendering-with-batch/cleanup.html) to prevent incurring in additional charges. {{% notice info %}} -The estimated completion time of this lab is **60 minutes**. Rendering the same animation that you have seen in the previous page, you will incur in an estimated cost of **$15**. +The estimated completion time of this lab is **60 minutes**. Rendering the animation that in this workshop you will incur in an estimated cost of up to **$15**. {{% /notice %}} diff --git a/content/rendering-with-batch/start/at-an-event.md b/content/rendering-with-batch/start/at-an-event.md index d3169bcc..08261cae 100644 --- a/content/rendering-with-batch/start/at-an-event.md +++ b/content/rendering-with-batch/start/at-an-event.md @@ -20,4 +20,6 @@ You are now logged in to the AWS console in an account that was created for you, - An instance profile for AWS Batch compute environment - The Cloud9 environment where you will run all the commands -{{% content "static/pages/rendering-with-batch/review-outputs.md" %}} + + +{{% insert-md-from-file file="rendering-with-batch/start/review-outputs.md" %}} diff --git a/content/rendering-with-batch/start/on-your-own.md b/content/rendering-with-batch/start/on-your-own.md index 9a998470..79bf0b3f 100644 --- a/content/rendering-with-batch/start/on-your-own.md +++ b/content/rendering-with-batch/start/on-your-own.md @@ -6,7 +6,7 @@ weight: 27 ## Deploying the CloudFormation stack -As a first step, **download** a [CloudFormation stack](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/rendering-with-batch/stack.yaml) that will deploy for you the following resources: +As a first step, **download** a [CloudFormation stack](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/rendering-with-batch/rendering-with-batch.files/stack.yaml) that will deploy for you the following resources: - A VPC - An S3 bucket @@ -24,4 +24,4 @@ After downloading the template, open the [CloudFormation console](https://consol The stack creation process will begin. All the resources will be ready to use when the status of the stack is `CREATE_COMPLETE`. -{{% content "static/pages/rendering-with-batch/review-outputs.md" %}} +{{% insert-md-from-file file="rendering-with-batch/start/review-outputs.md" %}} \ No newline at end of file diff --git a/static/pages/rendering-with-batch/review-outputs.md b/content/rendering-with-batch/start/review-outputs.md similarity index 79% rename from static/pages/rendering-with-batch/review-outputs.md rename to content/rendering-with-batch/start/review-outputs.md index c16b7d21..4f99a714 100644 --- a/static/pages/rendering-with-batch/review-outputs.md +++ b/content/rendering-with-batch/start/review-outputs.md @@ -1,10 +1,8 @@ ## Reviewing the Launch Template -A Launch Template has been automatically created for you when deploying the CloudFormation stack. If you had to create it yourself, you would need to follow the instructions specified below. +You can check the CloudFormation stack by downloading the following file: [CloudFormation stack](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/rendering-with-batch/rendering-with-batch.files/stack.yaml) -When creating the Batch compute environment, we need to specify some configuration parameters that will be passed on to the EC2 instances when launched, like the Security Group, the Availability Zones and bootstrapping scripts (User data). To encapsulate those properties and be able to easily reuse them, we will use a Launch Template. - -The `UserData` of the created Launch Template contains the following script: +Note the `UserData` of the created Launch Template contains the following script: ```bash MIME-Version: 1.0 @@ -24,6 +22,7 @@ echo "ECS_ENABLE_CONTAINER_METADATA=true" >> /etc/ecs/ecs.config What we are doing here is enabling [Spot Instance Draining](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-spot.html). When ECS Spot Instance draining is enabled on the instance, ECS receives the Spot Instance interruption notice and places the instance in DRAINING status. When a container instance is set to DRAINING, Amazon ECS prevents new tasks from being scheduled for placement on the container instance. To learn more about Spot instance interruption notices, visit [Spot Instance interruption notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#spot-instance-termination-notices). + ## Gathering the CloudFormation outputs You will create other AWS resources using the AWS CLI in [Cloud9](https://aws.amazon.com/cloud9/), a cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and terminal. Cloud9 comes prepackaged with essential tools for popular programming languages, including JavaScript, Python, PHP, and more.