changes to the rendering workshop

content/rendering-with-batch/_index.md

@@ -14,7 +14,7 @@ The estimated completion time of this lab is **60 minutes**. Please note Renderi
 In this workshop you will learn to submit jobs with [AWS Batch](https://aws.amazon.com/batch/) following Spot best practices to [render](https://en.wikipedia.org/wiki/Rendering_(computer_graphics)) a [Blender](https://www.blender.org/) file in a distributed way. You will be creating a docker container and publishing it in Amazon Elastic Container Registry (ECR). Then you will use that container in AWS Batch using a mix of EC2 On-Demand and Spot instances. [Spot instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html) are EC2 spare capacity offered at steep discounts compared to On-Demand instances and are a cost-effective choice for applications that can be interrupted, what makes them well-suited for the batch processing that we will run. After going through all the sections, you will have the following pipeline created:
 
 1. A python script downloads the Blender file from S3 to extract the number of frames from the Blender project. 
-2. The script submits a Batch using an `array job` with as many tasks as number of frames. It also submits a single stitching job using [FFmpeg](https://ffmpeg.org/) to create a final video file.
+2. The script submits a batch using an `array job` with as many tasks as number of frames. It also submits a single stitching job using [FFmpeg](https://ffmpeg.org/) to create a final video file.
 3. Each of the jobs in the rendering job array run a Docker container that executes Blender to render a slice of frames, and after uploads them to S3.
 
 The outcome of the workshop would be the following animation.

content/rendering-with-batch/batch/_index.md

@@ -6,25 +6,33 @@ weight: 80
 
 ## Overview
 
-AWS Batch is a set of batch management capabilities that enables you to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized compute resources) based on the volume and specific resource requirements of the batch jobs submitted. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as AWS Fargate, Amazon EC2 and Spot Instances. There is no additional charge for AWS Batch. You only pay for the AWS resources you create to store and run your batch jobs.
+AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized compute resources) based on the volume and specific resource requirements of the batch jobs submitted. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as AWS Fargate, Amazon EC2 and Spot Instances. There is no additional charge for AWS Batch. You only pay for the AWS resources you create to store and run your batch jobs.
 
-Thanks to Batch computing, you can execute a series of programs ("jobs") on one or more computers without manual intervention. Input parameters are pre-defined through scripts, command-line arguments, control files, or job control language. A given batch job may depend on the completion of preceding jobs, or on the availability of certain inputs, making the sequencing and scheduling of multiple jobs important, and incompatible with interactive processing.
+Thanks to AWS Batch computing, you can execute a series of "jobs" on one or more computers without manual intervention. Input parameters are pre-defined through scripts, command-line arguments, control files, or the AWS Batch job control language. You can make a batch job to
+depend on the completion of preceding jobs, or on the availability of certain inputs, making the sequencing and scheduling of multiple jobs easy.
 
 ## AWS Batch features
-- **Dynamic compute resource provisioning and scaling**: you only need to set up a few concepts in Batch (a Compute environment, job queue, and job definition), and you have a complete queue, scheduler, and compute architecture without managing a single piece of compute infrastructure.
-- **AWS Batch with EC2 Spot instances**: Since Batch workloads are containerized, Batch is a perfect fit for Spot Instances. If a workload is interrupted, Batch will automatically spin-up another Spot Instance you’ve specified.
-- **AWS Batch with Fargate**: AWS Batch with Fargate resources allows you to have a completely serverless architecture for your batch jobs.
-- **Integration with EC2 Launch Templates**: AWS Batch now supports EC2 Launch Templates, allowing you to build customized templates for your compute resources, and enabling Batch to scale instances with those requirements.
-- **Priority-based job scheduling**: AWS Batch enables you to set up multiple queues with different priority levels. Batch jobs are stored in the queues until compute resources are available to execute the job. The AWS Batch scheduler evaluates when, where, and how to run jobs that have been submitted to a queue based on the resource requirements of each job.
+- **Dynamic compute resource provisioning and scaling**: you only need to set up a few concepts in AWS Batch (a Compute environment, job queue, and job definition), and you have a complete queue, scheduler, and compute architecture without managing a single piece of compute infrastructure.
+- **AWS Batch with EC2 Spot instances**: AWS Batch workloads are usually a perfect fit for Spot Instances. If a workload is interrupted, AWS Batch will automatically retry the task and spin-up another Spot Instance using AWS EC2 Spot Best practices.
+- **AWS Batch with Fargate**: AWS Batch integrates with [AWS Fargate](https://aws.amazon.com/fargate/). With AWS Fargate you can run batch workloads in a completely serverless environment.
+- **Integration with EC2 Launch Templates**: AWS Batch does supports EC2 Launch Templates, allowing you to build customized compute resources, and enabling AWS Batch to scale instances with those requirements.
+- **Priority-based job scheduling**: AWS Batch enables you to set up multiple queues with different priority levels. Batch jobs are queued until compute resources are available to execute the job. The AWS Batch scheduler evaluates when, where, and how to run jobs that have been submitted to a queue based on the resource requirements of each job.
 
-These are just some examples of all the features AWS Batch has. If you want to learn in detail all its capabilities, visit [this web page](https://aws.amazon.com/batch/features/?nc=sn&loc=2).
+If you want to learn in detail all its capabilities, visit [this web page](https://aws.amazon.com/batch/features/?nc=sn&loc=2).
 
 ## Understanding Batch components
 
-### Typical job architecture
+In this section we will go through each individual AWS Batch architecture component, starting with the typical AWS Batch job architecture. In the diagram below you can see how
+
+1. The user start triggering an event by uploading a S3 object
+2. You can automate and trigger the submission of one or more jobs using [AWS Lambda](https://aws.amazon.com/lambda/) or manage [AWS Batch with Step functions](https://docs.aws.amazon.com/step-functions/latest/dg/connect-batch.html). The job gets submitted to a **Job queue** using a **Job Definition**. 
+3. AWS Batch will schedule the jobs submitted using **Compute Environments**. It will procure new resources when needed and allocate the job accordingly.
+4. The Compute Environment resources (EC2 Instances or Fargate) will run the job and in this workshop and example, it will store the output of into S3.
+
 
 ![Batch components](/images/rendering-with-batch/batch_architecture.png)
 
+
 ### Compute environment
 
 An AWS Batch Compute Environment is a collection of compute resources on which jobs are executed. AWS Batch supports two types of Compute Environments; Managed Compute Environments which are provisioned and managed by AWS and Unmanaged Compute Environments which are managed by customers.
@@ -52,12 +60,12 @@ To learn more about Job definitions, visit [AWS Batch Job definitions](https://d
 
 ## When you should use AWS Batch
 
-AWS Batch can be used in several scenarios that imply executing tasks which can be tore down to small components, jobs, thus drastically reducing execution time by running programs in parallel, and also automating the workloads. These are some example scenarios:
+AWS Batch can be used in several scenarios where you can split a workload into smaller jobs and run those in parallel thus drastically reducing execution time. These are some example scenarios:
 
-- Post-trade analytics: trading desks require batch processing of large data sets to constantly analyse day's transaction costs and execution reporting, among other areas. AWS Batch enables the automation of these workloads so that you can understand the pertinent risk going into the next day’s trading cycle and make better decisions based on data.
-- Drug screening: AWS Batch allows research scientists involved in drug discovery to more efficiently and rapidly search libraries of small molecules in order to identify those structures which are most likely to bind to a drug target, typically a protein receptor or enzyme.
-- Rendering: AWS Batch provides content producers and post-production houses with tools to automate content rendering workloads and reduces the need for human intervention due to execution dependencies or resource scheduling.
+- In Visual Effects, we cover this in this Rendering workshop: AWS Batch provides content producers and post-production houses with tools to automate content rendering workloads and reduces the need for human intervention due to execution dependencies or resource scheduling.
 - Media supply chain: AWS Batch simplifies complex media supply chain workflows by coordinating the execution of disparate and dependent jobs at different stages of processing, and supports a common framework for managing content preparation for different contributors to the media supply chain.
+- In life-sciences: Drug screening. AWS Batch allows research scientists involved in drug discovery to more efficiently and rapidly search libraries of small molecules in order to identify those structures which are most likely to bind to a drug target, typically a protein receptor or enzyme.
+- In Finance, Post-trade analytics: trading desks require batch processing of large data sets to constantly analyze day's transaction costs and execution reporting, among other areas. AWS Batch enables the automation of these workloads so that you can understand the pertinent risk going into the next day’s trading cycle and make better decisions based on data.
 
 ## Working with AWS Batch
 

content/rendering-with-batch/batch/compute_environment.md

@@ -6,13 +6,11 @@ weight: 90
 
 ![Compute environment](/images/rendering-with-batch/CE.png)
 
-We are going to create two compute environments: one using Spot instances and the other using On-Demand instances. The reason for this is that, on a real scenario, you may have to comply with several SLAs with regards to finishing the jobs under a specific time window.
-
-By using On-Demand instances, we are sure that the compute resources will remain throughout the whole execution. And by using Spot instances on top of that, we can reduce costs and execution time by acquiring compute capacity at steep discounts compared to On-Demand instances.
+We are going to create two compute environments: one using **Spot** instances and the other using **On-Demand** instances. That way if your job need to comply with an SLAs the On-Demand proportion can be used to ensure a timely completion, while the Spot will be used to accelerate the run and reduce the overall cost. 
 
 ### Spot Compute environment creation
 
-Run the following to generate the configuration file that will be used to create the Batch compute environment that uses Spot instances:
+Run the following to generate the configuration file that will be used to create the AWS Batch compute environment that uses Spot instances:
 
 ```
 export SPOT_COMPUTE_ENV_NAME="RenderingSpotComputeEnvironment"
@@ -44,13 +42,13 @@ EoF
 Let's explore the configuration parameters in the `computeResources` structure:
 
 - **type**: specifies the compute resource type to use. This compute environment will use Spot instances, hence the `SPOT` value. The other possible values are `EC2`, `FARGATE` and `FARGATE_SPOT`.
-- **allocationStrategy**: the allocation strategy to use for the compute resource if not enough instances of the best fitting instance type can be allocated. The `SPOT_CAPACITY_OPTIMIZED` allocation strategy allocates instances from the Spot Instance pools with the optimal capacity for the number of instances that are launching, making use of real-time capacity data and optimizing the selection of used Spot Instances. You can read about the benefits of using `capcity-optimized` in the blog post [Capacity-Optimized Spot Instance allocation in action at Mobileye and Skyscanner](https://aws.amazon.com/blogs/aws/capacity-optimized-spot-instance-allocation-in-action-at-mobileye-and-skyscanner/). This way, Batch will select one or more instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types that are less likely to be interrupted. To learn more about allocation strategies, see [Allocation strategies](https://docs.aws.amazon.com/batch/latest/userguide/allocation-strategies.html).
+- **allocationStrategy**: the allocation strategy to use for the compute resource if not enough instances of the best fitting instance type can be allocated. The `SPOT_CAPACITY_OPTIMIZED` allocation strategy allocates instances from the Spot Instance pools with the optimal capacity for the number of instances that are launching, making use of real-time capacity data and optimizing the selection of used Spot Instances. You can read about the benefits of using `capcity-optimized` in the blog post [Capacity-Optimized Spot Instance allocation in action at Mobileye and Skyscanner](https://aws.amazon.com/blogs/aws/capacity-optimized-spot-instance-allocation-in-action-at-mobileye-and-skyscanner/). This way, AWS Batch will select one or more instance types that are large enough to meet the requirements of the jobs in the queue, with a preference for instance types that are less likely to be interrupted. To learn more about allocation strategies, see [Allocation strategies](https://docs.aws.amazon.com/batch/latest/userguide/allocation-strategies.html).
 - **instanceTypes**: the instances types that can be launched. By specifying `optimal`, instance types from the C4, M4, and R4 instance families are selected to match the demand of your job queues.
 - **subnets**: the VPC subnets where the compute resources are launched.
 - **launchTemplate**: the launch template to use when launching compute resources. We specify the one we created earlier so that the User data script is applied to every instance that is launched.
 - **instanceRole**: the Amazon ECS instance profile applied to Amazon EC2 instances in the compute environment. AWS Batch compute environments are populated with Amazon ECS container instances, and they run the Amazon ECS container agent locally. The Amazon ECS container agent makes calls to various AWS API operations on your behalf. Therefore, container instances that run the agent require an IAM policy and role for these services to recognize that the agent belongs to you. For more information read [Amazon ECS instance role](https://docs.aws.amazon.com/batch/latest/userguide/instance_IAM_role.html).
 
-Execute this command to create the Batch compute environment and export its ARN to an environment variable. To learn more about this API, see [create-compute-environment CLI command reference](https://docs.aws.amazon.com/cli/latest/reference/batch/create-compute-environment.html).
+Run this command to create the AWS Batch compute environment and export its ARN to an environment variable. To learn more about this API, see [create-compute-environment CLI command reference](https://docs.aws.amazon.com/cli/latest/reference/batch/create-compute-environment.html).
 
 ```
 export SPOT_COMPUTE_ENV_ARN=$(aws batch create-compute-environment --cli-input-json file://spot-compute-environment-config.json | jq -r '.computeEnvironmentArn')
@@ -59,7 +57,7 @@ echo "Spot compute environment Arn: ${SPOT_COMPUTE_ENV_ARN}"
 
 ### On-Demand Compute environment creation
 
-Run the following to generate the configuration file that will be used to create the Batch compute environment that uses On-Demand instances:
+Run the following to generate the configuration file that will be used to create the AWS Batch compute environment that uses On-Demand instances:
 
 ```
 export ONDEMAND_COMPUTE_ENV_NAME="RenderingOnDemandComputeEnvironment"
@@ -88,14 +86,14 @@ cat <<EoF > ondemand-compute-environment-config.json
 EoF
 ```
 
-Let's explore the configuration parameters in the `computeResources` structure that differ from the previous Compute environment:
+Lets explore the configuration parameters in the `computeResources` structure that differ from the previous Compute environment:
 
 - **type**: specifies the compute resource type to use. This compute environment will use On-demand instances, hence the `EC2` value. The other possible values are `SPOT`, `FARGATE` and `FARGATE_SPOT`.
-- **allocationStrategy**: the allocation strategy to use for the compute resource if not enough instances of the best fitting instance type can be allocated. By specifying `BEST_FIT_PROGRESSIVE`, Batch selects an instance type that best fits the needs of the jobs with a preference for the lowest-cost instance type.
+- **allocationStrategy**: the allocation strategy to use for the compute resource if not enough instances of the best fitting instance type can be allocated. By specifying `BEST_FIT_PROGRESSIVE`, AWS Batch selects an instance type that best fits the needs of the jobs with a preference for the lowest-cost instance type.
 
 Notice how we have specified lower values for `maxvCpus` and `desiredvCpus` in comparison to the same of the Spot compute environment. We do so to make sure that only the required compute capacity to finish the job in a finite amount of time is provisioned with On-demand capacity, using more Spot resources to reduce execution time and cost.
 
-Execute this command to create the Batch compute environment and export its ARN to an environment variable.
+Execute this command to create the AWS Batch compute environment and export its ARN to an environment variable.
 
 ```
 export ONDEMAND_COMPUTE_ENV_ARN=$(aws batch create-compute-environment --cli-input-json file://ondemand-compute-environment-config.json | jq -r '.computeEnvironmentArn')