[WiP] add nextflow-draft #45

kniec · 2020-03-30T13:34:11Z

First draft of a workshop that uses nextflow to submit jobs to AWS Batch.

TODO:

Create Batch Squared part, in which a job in AWS Batch is used to execute the supervisor nextflow process submitting more jobs to a different queue.

Run local version

To run the local version, please checkout the branch and just run docker-compose up locally.

ranshn · 2020-03-31T12:47:56Z

content/nextflow-on-aws-batch/_index.md

+pre: "<b>⁃ </b>"
+---
+
+The content of this labs were orgininally created by the nice folks at [Seqera Labs](https://github.com/seqeralabs/nextflow-tutorial), all credits go to them!


typos: lab instead of labs, origininally
Can you specify what you added or modified generally? just mentioning they "created it originally" is not clear and it might seem like you are just re-hosting their workshop here

I added more color to that one.

The content of this workshop is derived from a tutorial created by the nice folks at Seqera Labs, kudos to them!
We won't create or own pipelines and tweak code, but rather jump right in with a small proof-of-concept piepline, which we will run locally in containers, submit locally to AWS Batch and run a batch job that submits to AWS Batch.

ranshn · 2020-03-31T12:48:55Z

content/nextflow-on-aws-batch/_index.md

+
+
+## Overview 
+During this tutorial you will implement a proof of concept of a RNA-seq pipeline. The goal of this workshop is not te become a Bioinformatician nor a Nextflow guru.


An RNA-seq
The goal is not "..." so what is it? to learn about underlying compute with Nextflow/Batch or something?

During this workshop you will implement a proof of concept of a RNA-seq pipeline. The goal of this workshop is not te become a Bioinformatician nor a Nextflow guru, but to get familiar with the concepts of nextflow and AWS Batch.

ranshn · 2020-03-31T12:50:01Z

content/nextflow-on-aws-batch/_index.md

+## Introduction
+
+### Conventions:  
+Throughout this labs, we provide commands for you to run in the terminal.  These commands will look like this: 


Labs = lab? workshop? tutorial? in the same page it's called many different things. stick with workshop?

let's go with workshop

content/nextflow-on-aws-batch/10_prerequisites/_index.md

content/nextflow-on-aws-batch/10_prerequisites/50_install_tools.md

content/nextflow-on-aws-batch/40_nextflow202/_index.md

content/nextflow-on-aws-batch/40_nextflow202/10_setup.md

ranshn · 2020-03-31T17:30:56Z

content/nextflow-on-aws-batch/40_nextflow202/20_simple.md

+===================================
+transcriptome: /home/ec2-user/environment/nextflow-tutorial/data/ggal/transcriptome.fa
+reads        : /home/ec2-user/environment/nextflow-tutorial/data/ggal/gut_{1,2}.fq
+outdir       : results


after this line I got "WARN: Unable to create AWS Batch helper class | credentials cannot be null"
and after same time it failed Oops .. something went wrong

ok, let's have someone else go through it with the updates - works on my cloud. :)

typo on explains (exmplains)

More content for the conclusion

Suggestions to guide and allow copy/paste of value

Update _index.md

update the role to attach to the instance

typo in code

kniec · 2020-04-23T07:20:49Z

Merged a bunch of improvements from @plample, thanks a lot!

ruecarlo

@kniec , The workshop did work well, there is mainly recommendations and cosmetic stuff required for completion, but the reason why instead of approving I'm requesting changes is the cleanup sequence. The clean-up command clean up all the images in the registry all the job definitions in AWS Batch, etc. We should not assume we can clear out all of those, users may have already applications running in their accounts and this may cause an inconvenience. We should instead target the focus cleanup to the right resources that were created during the workshop

ruecarlo · 2020-04-24T17:26:34Z

content/nextflow-on-aws-batch/_index.md

+During this workshop you will implement a proof of concept of a RNA-seq pipeline. The goal of this workshop is not te become a Bioinformatician nor a Nextflow guru, but to get familiar with the concepts of nextflow and AWS Batch.
+
+{{% notice info %}}
+The estimated cost for running this **Y hour** workshop will be less than **$X**.


This still does make a reference to Y hours and $X Dollars

put in 1.5h and $5 (will check within the next dry-run).

ruecarlo · 2020-04-24T17:41:21Z

content/nextflow-on-aws-batch/10_prerequisites/45_disable_cred.md

+weight: 45
+---
+
+## Attach the IAM role to your Workspace


This title seems misplaced as the copy&paste from the previous exercise, there might be some text lacking, explaining what does it mean the things the png is pointing; The picture refers to a set of steps that may need to be described (reference to steps 1 , 2, 3)

I'm assuming that unlike the EKS and with Kubectl in this workshop we don't need to refresh credentials , but I'd suggest adding a validation entry where people have to run

aws sts get-caller-identity

To validate that the workshop credentials are what you expected to run. This also will help to AWS SA's attending the workshop to understand if someone skipped a step

Ok, during the workshop on purpose I left enabled the AWS managed temporary credentials and it came back to the credentials of my user creating cloud-9 instead of the ones from the acquired Cloud-9 role. Definitely more instructions are needed if it takes more than 60mins to do the workshop.

Changed the headline and included get-caller-identity. good catch!

ruecarlo · 2020-04-24T17:46:52Z

content/nextflow-on-aws-batch/10_prerequisites/50_install_tools.md

+
+## Install Java and Nextflow
+
+The nextflow command-line tool uses the JVM. Thus, we will install AWS open-source variant [Amazon Corretto](https://docs.aws.amazon.com/corretto).


Perhaps adding a quick {% Notice %} with a bit of info of why coretto is so cool and why we prefer to use it rather than other JDK's may bring attention to stuff that our AWS teams are doing :)

I added the abstract from their website

ruecarlo · 2020-04-24T17:51:20Z

content/nextflow-on-aws-batch/10_prerequisites/50_install_tools.md

+
+### Nextflow
+
+Installing Nextflow using the online installer.


At this stage we are installing nextflow but we have not introduced why nextflow is so cool and what makes it a great tool to manage genomic flows. For example explaining to other why DSL's make pipeline and workflow declaration more effective, etc. Perhaps the link is further in the workshop, but I reckon at this stage a narrative on introduction that explain why nextflow would help to specific users that want to do this workshop and understand some of the concepts they will see later on. (concepts (a) Nextflow, what is it (b) DSLs and Pipelines/workflows (c)why this is required in genomics and more generally).

Narrative does not necessarily need to be huge, but pointers to back up concepts will definitely help

added a destinct 'install nextflow' page with some praises

ruecarlo · 2020-04-24T17:57:35Z

content/nextflow-on-aws-batch/20_nextflow101/10_small.md

+
+A process is defined by providing three main declarations: the process [inputs](https://www.nextflow.io/docs/latest/process.html#inputs), the process [outputs](https://www.nextflow.io/docs/latest/process.html#outputs) and finally the command [script](https://www.nextflow.io/docs/latest/process.html#script).
+
+The example workflow implements a simple RNA-seq pipeline which:


A decoration of what RNA-Seq means with references will help a lot for people to get context. https://en.wikipedia.org/wiki/RNA-Seq

Think the workshop is also public and there will be other people with little context of genomic domain that actually are doing the workshop to understand this type of things, Help them with visual aids . If anything refer at least to wikipedia for people to have fun :)

@plample can you help here?

ruecarlo · 2020-04-24T20:35:28Z

content/nextflow-on-aws-batch/50_what-we-learned/_index.md

+
+## Follow Up
+
+<div style="text-align:left;font-size: 1.2rem">


ruecarlo · 2020-04-24T20:38:46Z

content/nextflow-on-aws-batch/60_cleanup/_index.md

+#### ECR
+
+```
+for x in $(aws ecr --region=${AWS_REGION}  describe-repositories |jq -r '.repositories[] | .repositoryName' |xargs);do echo "# aws ecr --region=${AWS_REGION} delete-repository --force --repository-name=$x" ; aws ecr --region=${AWS_REGION} delete-repository --force --repository-name=$x;done


Hey, this raises a flag for me; This will clean all the repositories in my account on that region. In my case it deleted all the other projects I had in that region. We can really mess up if people run things on their own accounts. I'd suggest we filter and only list based on a regex for the project

I made the clean up a bit more carefull and put a big warning at the top

ruecarlo · 2020-04-24T20:39:12Z

content/nextflow-on-aws-batch/60_cleanup/_index.md

+##### Job Definitions
+
+```bash
+for jd in $(aws batch --region=${AWS_REGION} describe-job-definitions |jq -r '.jobDefinitions[] | .jobDefinitionArn' |xargs); do  echo "# aws batch --region=${AWS_REGION} deregister-job-definition --job-definition=${jd}" ; aws batch --region=${AWS_REGION} deregister-job-definition --job-definition=${jd}; done


Same applies here, this is ok for new accounts within AWS, we cannot assume that we can delete all job definitions

ruecarlo · 2020-04-24T20:43:09Z

content/nextflow-on-aws-batch/60_cleanup/_index.md

+Disable first...
+
+```bash
+for jq in $(aws batch --region=${AWS_REGION} describe-job-queues |jq -r '.jobQueues[] |.jobQueueName' |xargs);do echo "# aws batch --region=${AWS_REGION} update-job-queue --state=DISABLED --job-queue=${jq}" ; aws batch --region=${AWS_REGION} update-job-queue   --state=DISABLED --job-queue=${jq};done


Same applies here, we cannot destroy all the queues, we should destroy just the ones that we created!

ruecarlo · 2020-04-24T20:43:25Z

content/nextflow-on-aws-batch/60_cleanup/_index.md

+
+##### Compute Environment
+
+Disable first...


And the same here.

kniec · 2020-04-27T13:08:54Z

Thanks @ruecarlo and @ranshn for the feedback - I'll merge the PR to get it ready for the first run on Wedensday

kniec · 2020-04-27T13:16:14Z

Should have created one bit commit before merging tho...

kniec added the enhancement New feature or request label Mar 30, 2020

kniec requested a review from ruecarlo March 30, 2020 13:34

ranshn self-requested a review March 31, 2020 12:45

ranshn suggested changes Mar 31, 2020

View reviewed changes

kniec force-pushed the nextflow-workshop branch from 0f29d57 to 129b860 Compare April 8, 2020 08:25

kniec added 8 commits April 20, 2020 10:27

[WiP] add nextflow-draft

62d2fb2

update IAM AWSBatch role

0aa0169

Finished batch-squared

054a4b4

add learnings

3b16aba

Adjust wording within intro

91d6f89

more intro on what we will do within the workshop

88aeabb

included @ranshn feadback

ae03368

typo

dc79f09

kniec force-pushed the nextflow-workshop branch from 129b860 to dc79f09 Compare April 20, 2020 08:42

plample and others added 14 commits April 21, 2020 14:18

Update _index.md

f2a7c1c

typo on explains (exmplains)

update the role to attach to the instance

6d0731d

slide adjustment to job queue build

0204f0e

add sudo within packer build

bab090c

typo in code

7eb6b21

Suggestions to guide and allow copy/paste of value

7359e07

More content for the conclusion

61f61fd

Merge pull request #5 from plample/patch-5

445394f

More content for the conclusion

tweaked the what-we-learned page

8126e9c

Merge pull request #4 from plample/patch-4

2224be0

Suggestions to guide and allow copy/paste of value

adjustments to batched-squared-run

bb3af5a

Merge pull request #3 from plample/patch-1

e185659

Update _index.md

Merge pull request #2 from plample/patch-3

f9e43e0

update the role to attach to the instance

Merge pull request #1 from plample/patch-2

b788f95

typo in code

start removing AMI creation

9b26681

kniec added 2 commits April 24, 2020 11:53

update complete to run on base AMI + CleanUp

83e51dc

fixed some markdown style errors

35710af

ruecarlo requested changes Apr 24, 2020

View reviewed changes

kniec added 2 commits April 27, 2020 10:19

incorperated feedback by Carlos (PR awslabs#45)

1ed876c

fixed some markdown style errors

8488bfa

This was referenced Apr 27, 2020

Nextflow-Workshop: Elaborate on S3 #54

Open

nextflow-workshop: elaborate on why AWS Batch is used #55

Open

nextflow: more AWS Batch narration #56

Open

ruecarlo approved these changes Apr 27, 2020

View reviewed changes

kniec mentioned this pull request Apr 27, 2020

nextflow: narrative around spot #57

Open

ranshn self-requested a review April 27, 2020 09:52

ranshn approved these changes Apr 27, 2020

View reviewed changes

adding more feedback from Carlos

bea547a

kniec added 3 commits April 27, 2020 15:12

fixed some markdown style errors

b916201

fixed some markdown style errors v2

742aab2

fixed some markdown style errors v3

9c7e67d

kniec merged commit d340e20 into awslabs:master Apr 27, 2020

kniec deleted the nextflow-workshop branch April 27, 2020 13:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WiP] add nextflow-draft #45

[WiP] add nextflow-draft #45

kniec commented Mar 30, 2020 •

edited

Loading

ranshn Mar 31, 2020

kniec Apr 2, 2020

kniec Apr 2, 2020

ranshn Mar 31, 2020

kniec Apr 2, 2020

ranshn Mar 31, 2020

kniec Apr 2, 2020

kniec Apr 2, 2020

ranshn Mar 31, 2020

kniec Apr 2, 2020

kniec commented Apr 23, 2020

ruecarlo left a comment

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

ruecarlo Apr 24, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

kniec Apr 27, 2020

ruecarlo Apr 24, 2020

ruecarlo Apr 24, 2020

kniec commented Apr 27, 2020

kniec commented Apr 27, 2020



		## Overview
		During this tutorial you will implement a proof of concept of a RNA-seq pipeline. The goal of this workshop is not te become a Bioinformatician nor a Nextflow guru.


		## Install Java and Nextflow

		The nextflow command-line tool uses the JVM. Thus, we will install AWS open-source variant [Amazon Corretto](https://docs.aws.amazon.com/corretto).


		A process is defined by providing three main declarations: the process [inputs](https://www.nextflow.io/docs/latest/process.html#inputs), the process [outputs](https://www.nextflow.io/docs/latest/process.html#outputs) and finally the command [script](https://www.nextflow.io/docs/latest/process.html#script).

		The example workflow implements a simple RNA-seq pipeline which:

[WiP] add nextflow-draft #45

[WiP] add nextflow-draft #45

Conversation

kniec commented Mar 30, 2020 • edited Loading

Run local version

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kniec commented Apr 23, 2020

ruecarlo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kniec commented Apr 27, 2020

kniec commented Apr 27, 2020

kniec commented Mar 30, 2020 •

edited

Loading