Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WDL Workflow: Generate summary reports for malaria samples #456

Open
wants to merge 242 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
242 commits
Select commit Hold shift + click to select a range
04fbd9f
Intermediate commit -- adding preliminary files for malaria sequencing
bridgetknight Jul 29, 2023
d2ccd0b
Updating environments.yml in lr-malaria-reports with correct
bridgetknight Jul 29, 2023
004cf14
Updating WDL files -- separated workflow and tasks and put in respect…
bridgetknight Aug 1, 2023
b2d8bf3
Updating WDL files and Dockerfile
bridgetknight Aug 1, 2023
db1963a
Adding required inputs to WDL task file
bridgetknight Aug 4, 2023
0623e55
Intermediate commit - add argument parsing to Python script
bridgetknight Aug 4, 2023
ffaa13a
Add all current inputs and update dependencies
bridgetknight Aug 6, 2023
ef549f5
Add necessary files and change Docker imports
bridgetknight Aug 10, 2023
6b1ca7f
Minor updates to WDL files
bridgetknight Aug 11, 2023
64f668e
Dockerfile and WDL fixes + add *.zip to .gitignore
bridgetknight Aug 15, 2023
697823c
Minor updates and fixes to analysis page
bridgetknight Aug 15, 2023
5924e2a
Minor changes/fixes to WDL files
bridgetknight Aug 17, 2023
23fb735
Update WDLs and script -- task runs successfully now
bridgetknight Aug 17, 2023
f3cb984
Clean up analysis.html
bridgetknight Aug 17, 2023
584b559
Finish cleaning up template code
bridgetknight Aug 18, 2023
8de4144
Embed plots and images into HTML and consolidate templates
bridgetknight Aug 18, 2023
184aed2
Removing unnecessary files
bridgetknight Aug 22, 2023
e98c995
Updated dockstore file to include malaria reports
bridgetknight Aug 22, 2023
187a9ce
Changed WDL import statement
bridgetknight Aug 22, 2023
024e926
Updated output file reference in WDLs
bridgetknight Aug 22, 2023
640a868
Changed aligned_bases to float and fixed q-score plot
bridgetknight Aug 22, 2023
a0f34aa
Added quotes to species arg
bridgetknight Aug 23, 2023
b829229
added debug statements for species
bridgetknight Aug 23, 2023
9087585
fixed Python argument parsing for extra cases
bridgetknight Aug 23, 2023
728d78c
fixed upload date parsing
bridgetknight Aug 23, 2023
6537e96
fixed upload_date parsing
bridgetknight Aug 23, 2023
1583a53
added debug statements to WDL and changed Jinja env
bridgetknight Aug 23, 2023
1c358b2
changed jinja env
bridgetknight Aug 23, 2023
1870f8e
changed template import
bridgetknight Aug 23, 2023
0ddd130
fixed output file name in report_gen.py
bridgetknight Aug 23, 2023
5f8e10f
added debug statements to report_gen.py
bridgetknight Aug 23, 2023
7c8f5e5
fixed WDL output path
bridgetknight Aug 23, 2023
1d1ab0e
add debug statements to report_gen.py
bridgetknight Aug 23, 2023
27f08c0
edited file output
bridgetknight Aug 24, 2023
310d374
changed out file path in WDL
bridgetknight Aug 24, 2023
2de45d8
Handled cases where some info is not present
bridgetknight Aug 24, 2023
c192fa1
handled case if location is present but not coordinates
bridgetknight Aug 24, 2023
43726dd
removed unnecessary import
bridgetknight Aug 24, 2023
3a37d4a
changed some WDL inputs to optional
bridgetknight Aug 24, 2023
cae0e2b
edited optional inputs
bridgetknight Aug 24, 2023
ccd05ab
fixed import
bridgetknight Aug 24, 2023
6aa8fc5
changed optional inputs
bridgetknight Aug 24, 2023
a242e24
fixed some missing inputs
bridgetknight Aug 24, 2023
7f0823d
fixed drug_res default input value
bridgetknight Aug 24, 2023
cd4be42
fixed drug res input
bridgetknight Aug 24, 2023
44a553a
added coverage plot generation
bridgetknight Oct 16, 2023
363865e
removed unnecessary files
bridgetknight Oct 16, 2023
a84d0d2
updated Dockerfile and Makefile
bridgetknight Oct 16, 2023
996c2ff
Updated WDL files
bridgetknight Oct 20, 2023
bf44b4f
updated coverage plot code to us fastqc_path string instead
bridgetknight Oct 27, 2023
82f701e
Updated code to use fastqc_path for coverage plot
bridgetknight Oct 27, 2023
19b6185
fix header line on analysis page
bridgetknight Oct 27, 2023
31c950e
Merge branch 'bk_malaria_reports' of github.com:broadinstitute/long-r…
bridgetknight Oct 27, 2023
c4f02fc
added missing inputs in WDL workflow file
bridgetknight Oct 27, 2023
1de8c65
fix if statement for generating coverage_dir path
bridgetknight Oct 27, 2023
54cdb59
fix if statement in workflow
bridgetknight Oct 27, 2023
d0cdad5
made fastqc_path required
bridgetknight Oct 27, 2023
aecbefb
fix command order
bridgetknight Nov 8, 2023
d58be3c
fix some inputs and libraries
bridgetknight Nov 8, 2023
5baa959
add debug statements in WDL command
bridgetknight Nov 9, 2023
3f8e617
fix coverage directory parsing
bridgetknight Nov 9, 2023
18768f4
fixed coverage_dir parsing
bridgetknight Nov 9, 2023
7cbb121
fix BAM file copy regex
bridgetknight Nov 9, 2023
0c4b29c
fix coverage_regex in workflow
bridgetknight Nov 9, 2023
3ef104c
fix HRP2/HRP3 default values
bridgetknight Nov 17, 2023
ff6a078
fixed case where location might have spaces
bridgetknight Nov 17, 2023
6df2747
fix location wrapping
bridgetknight Nov 17, 2023
288cec3
fixed library references
bridgetknight Nov 17, 2023
49a9475
change some library references
bridgetknight Nov 17, 2023
f3f9c2c
fix some library references
bridgetknight Nov 17, 2023
84d541d
handle case if active_channels is missing
bridgetknight Nov 17, 2023
979e6bb
handle case if fraction_aligned_bases is not given
bridgetknight Nov 17, 2023
3df11b3
WIP - fix buttons/sidebar/header and start adding FastQC page
bridgetknight Dec 1, 2023
741c4c4
WIP - adding FastQC page
bridgetknight Dec 5, 2023
d953266
WIP -- add FastQC page generation
bridgetknight Dec 6, 2023
c1f679d
add fastqc_file
bridgetknight Dec 8, 2023
f01895b
update WDL for FastQC page
bridgetknight Dec 19, 2023
bb80e2a
update WDLs for FastQC support
bridgetknight Dec 19, 2023
168fa8e
add WDL task for getting BAM files and change flow
bridgetknight Jan 4, 2024
032c352
fix validation error with bam path
bridgetknight Jan 5, 2024
e61ad92
copy fastQC file to cloud directory
bridgetknight Jan 8, 2024
5b1aa2d
fix minor error in report_gen.py
bridgetknight Jan 8, 2024
08cb71a
update report_gen.py
bridgetknight Jan 10, 2024
5b3ff7b
update report_gen.py
bridgetknight Jan 10, 2024
cac91a9
update report_gen.py
bridgetknight Jan 10, 2024
4af1073
edit Jinja code in report.html (fastqc)
bridgetknight Jan 10, 2024
2407b71
edit fastqc html render
bridgetknight Jan 10, 2024
4046c0f
switch fqc srcdoc to encoded src
bridgetknight Jan 10, 2024
4123010
test fqc without iframe
bridgetknight Jan 10, 2024
1fe6ef0
change fqc iframe
bridgetknight Jan 12, 2024
356947a
change fqc iframe
bridgetknight Jan 12, 2024
fb4db0b
escape quotes in fastqc html
bridgetknight Jan 12, 2024
2ffb9e6
change fqc iframe
bridgetknight Jan 12, 2024
da87622
temp disable iFrame links to fix nav bug
bridgetknight Jan 23, 2024
08572ed
merging previous edits/fixing fastQC iFrame
bridgetknight Jan 23, 2024
8fa00d9
add new WIP task to malaria reporting
bridgetknight Mar 19, 2024
e59b5e1
disable links in FastQC page
bridgetknight Mar 20, 2024
430e4b4
add snapshot generation task
bridgetknight Mar 21, 2024
2ab2b28
fix inputs in snapshot workflow
bridgetknight Mar 21, 2024
8784d4e
fix snapshot output
bridgetknight Mar 21, 2024
d604e37
internally handle IGV snapshot output
bridgetknight Mar 26, 2024
7fe81d2
add template support for coverage gallery
bridgetknight Mar 28, 2024
9b879d7
Merge branch 'bk_malaria_reports' of github.com:broadinstitute/long-r…
bridgetknight Mar 28, 2024
424acc5
complete snapshot implementation
bridgetknight Mar 28, 2024
1a86350
fix params
bridgetknight Mar 28, 2024
db4a1ff
use map for reference files
bridgetknight Mar 29, 2024
f927360
fix ref map name
bridgetknight Mar 29, 2024
4cbd575
fix python command in snapshot workflow
bridgetknight Mar 29, 2024
4f25f32
update snapshots command
bridgetknight Apr 11, 2024
10d2eca
merge new snapshot command with previous file version
bridgetknight Apr 12, 2024
f97260e
remove onlysnap tag from snapshot command
bridgetknight Apr 12, 2024
87e6a99
fix refmap argument parsing
bridgetknight Apr 12, 2024
4be1e97
remove unnecessary inputs
bridgetknight Apr 12, 2024
fc46b14
fix base 64 conversion function
bridgetknight Apr 12, 2024
1f4b75f
update argument parsing for localized snapshots
bridgetknight Apr 12, 2024
65bab97
add snapshot input into script command
bridgetknight Apr 12, 2024
a061384
fix snapshot input sep in command
bridgetknight Apr 12, 2024
a69d0b0
fix issue where gallery JS was appearing as plaintext
bridgetknight Apr 18, 2024
a73bb37
add focus functionality to current snapshot
bridgetknight Apr 18, 2024
1fc2226
fix reference to first snapshot in report
bridgetknight Apr 18, 2024
63fbbae
provide title for focused snapshot
bridgetknight Apr 25, 2024
58dfdf9
fix parsing of resistance loci names
bridgetknight Apr 25, 2024
6956aea
fix parsing of resistance loci names (remove ")
bridgetknight Apr 25, 2024
4f5255e
add qc_pass and fix snapshot titles
bridgetknight Apr 25, 2024
f013236
add qc_pass to params
bridgetknight Apr 25, 2024
b363da3
fix reference to qc_status/pass
bridgetknight Apr 25, 2024
6c818ea
fix loci names display
bridgetknight May 20, 2024
4b179a1
fix loci names parsing in template
bridgetknight May 20, 2024
b34cb19
fix ref to qc_pass/status
bridgetknight May 20, 2024
b3a4fc9
fix ref to QC pass
bridgetknight May 20, 2024
fb2f52a
remove duplicate reference
bridgetknight May 20, 2024
5e5d767
clean up argument parsing
bridgetknight May 20, 2024
606b67c
fix reference to locus names
bridgetknight May 20, 2024
07d85f5
fix parsing of loci names
bridgetknight May 23, 2024
20b6053
fix loci name parsing
bridgetknight May 23, 2024
a6d4ed3
split loci names
bridgetknight May 23, 2024
0e3af3e
clean up loci names
bridgetknight May 23, 2024
1eff6e8
clean up loci names
bridgetknight May 23, 2024
b3a2c3a
clean up parsing of loci names
bridgetknight May 23, 2024
1df5453
fix parsing of loci names
bridgetknight May 23, 2024
b2fce76
fix cleanup for loci names
bridgetknight May 23, 2024
67c3a09
fix coverage gallery formatting
bridgetknight May 23, 2024
80ddbf4
round long decimals to 2 places
bridgetknight May 23, 2024
82073a7
update QC pass display
bridgetknight May 23, 2024
0e743de
replace active channels metric with barcode
bridgetknight May 24, 2024
d464c30
change sample_prep to sample_type
bridgetknight May 24, 2024
3b6db0a
update sample info contents
bridgetknight May 28, 2024
5bdd4f3
add new sample metrics
bridgetknight May 28, 2024
0f3c1ae
make read quality inputs optional
bridgetknight May 29, 2024
c8c3842
provide defaults for some new inputs
bridgetknight May 29, 2024
b4f23c3
update collection date parsing
bridgetknight May 29, 2024
21d8011
provide new input defaults
bridgetknight May 29, 2024
1978c80
add defaults to some inputs
bridgetknight May 29, 2024
87475b7
add defaults to some inputs
bridgetknight May 29, 2024
89ff7ae
remove unnecessary inputs
bridgetknight May 29, 2024
1c8a584
add new barcode input
bridgetknight May 29, 2024
e1e0f96
minor display update
bridgetknight May 30, 2024
0e78339
fix barcode display
bridgetknight May 30, 2024
a03b9e5
display fix
bridgetknight May 30, 2024
ca364e3
round some long decimals
bridgetknight May 30, 2024
da266e2
save IGV snapshot batch script as output
bridgetknight May 30, 2024
487a703
improve coverage gallery layout
bridgetknight May 30, 2024
296877f
improve coverage gallery titles and add tooltip
bridgetknight May 30, 2024
1f2fb2d
minor display fix
bridgetknight May 30, 2024
ea3828f
minor display changes
bridgetknight Jun 11, 2024
c31ad81
update coverage plot
bridgetknight Jun 20, 2024
ab72140
fix coverage plot x-axis
bridgetknight Jun 26, 2024
86299f0
fix bed file sorting
bridgetknight Jun 26, 2024
d144afa
streamline checking if certain data are present and clean up comments
bridgetknight Jul 2, 2024
dcb32ce
modularize some long code in main function and remove unused code
bridgetknight Jul 2, 2024
943c02b
add default to barcode argument
bridgetknight Jul 2, 2024
f6e63ec
fix sample name reference
bridgetknight Jul 2, 2024
a17f563
make barcode optional
bridgetknight Jul 2, 2024
4f49803
make some inputs optional
bridgetknight Jul 2, 2024
c51ac85
make sequencing date optional
bridgetknight Jul 2, 2024
893501c
fix date formatting
bridgetknight Jul 2, 2024
95f99ac
fix reference to bed file
bridgetknight Jul 2, 2024
06025f6
fix reference to drug resistance text file
bridgetknight Jul 2, 2024
72dd256
make collection date optional
bridgetknight Jul 10, 2024
3a940ca
make collection date optional
bridgetknight Jul 10, 2024
29d0909
add compatibility for ONT QC report (1)
bridgetknight Jul 10, 2024
396c938
add support for ONT QC in template
bridgetknight Jul 10, 2024
f2e3a11
fix report selection
bridgetknight Jul 10, 2024
bf27993
fix QC parsing reference
bridgetknight Jul 10, 2024
7c0c0a0
increase disk size for snapshot task
bridgetknight Jul 16, 2024
3c280c0
make coverage modular
bridgetknight Jul 16, 2024
292264e
fix qscores plot spacing
bridgetknight Jul 16, 2024
5a7e8b5
account for missing coverage data in WDL command
bridgetknight Jul 17, 2024
e5e08ba
adjust coverage_bin_size param
bridgetknight Jul 17, 2024
c61e10f
provide default for coverage bin size
bridgetknight Jul 17, 2024
537c310
check if coverage data is available during report generation
bridgetknight Jul 17, 2024
d4b5ce5
fix command regex
bridgetknight Jul 17, 2024
e49df83
adjust coverage plot parsing
bridgetknight Jul 17, 2024
f62cf37
many minor visual fixes (07/1724 feedback)
bridgetknight Jul 22, 2024
1f642df
remove some inputs from workflow
bridgetknight Jul 22, 2024
c19c1d2
remove some inputs from main workflow
bridgetknight Jul 22, 2024
45c077d
fix script inputs
bridgetknight Jul 22, 2024
a539322
fix function name
bridgetknight Jul 22, 2024
b2ea449
format long numbers
bridgetknight Jul 22, 2024
de89e8d
fix margins
bridgetknight Jul 22, 2024
97fbd54
extend sidebar
bridgetknight Jul 23, 2024
6bee893
change some floats to ints
bridgetknight Jul 23, 2024
fc0ea9e
add loci to snapshot subtitle
bridgetknight Jul 25, 2024
cc36748
adjust aligned bases and reads display
bridgetknight Jul 25, 2024
764a6ae
fix locus subtitle
bridgetknight Jul 25, 2024
28680b0
fix hidden page
bridgetknight Jul 25, 2024
87ac553
fix spacing for locus subtitle
bridgetknight Jul 25, 2024
d3d976d
improve locus subtitle
bridgetknight Jul 26, 2024
14e909f
add drug resistance bubble plot
bridgetknight Jul 26, 2024
fc66ac5
fix whitespace around drug resistance bubble plot
bridgetknight Jul 29, 2024
d1a3598
fix arrangement of DR section
bridgetknight Jul 29, 2024
28e1a95
add legend to drug resistance bubble plot
bridgetknight Jul 29, 2024
ff48884
fix DR bubbles image reference
bridgetknight Jul 29, 2024
6333642
add locus name under snapshot carousel images
bridgetknight Jul 30, 2024
5cfd93d
add zip to Jinja namespace
bridgetknight Jul 30, 2024
707cc5d
fix main image reference in snapshots
bridgetknight Jul 30, 2024
8c72aa3
fix spacing and optimize main display frame
bridgetknight Jul 30, 2024
19c9a4d
adjust framing on QC report page
bridgetknight Jul 30, 2024
694aa40
remove redundant footer code
bridgetknight Jul 30, 2024
e34d2c6
adjust height of fqc frame
bridgetknight Jul 30, 2024
05e75a0
render some numbers correctly as integers
bridgetknight Jul 31, 2024
f75ce98
fix some integer formatting
bridgetknight Jul 31, 2024
ceeb199
add tooltips with placeholder text
bridgetknight Aug 5, 2024
6f4138b
add all tooltips
bridgetknight Aug 6, 2024
302461a
edit footer
bridgetknight Aug 6, 2024
ac700bb
fix spacing in RDT regions box
bridgetknight Aug 6, 2024
deb3a3c
improve tooltip formatting and adjust footer
bridgetknight Aug 6, 2024
ce3789d
fix formatting in footer on Firefox
bridgetknight Aug 7, 2024
bc740ec
formatting fixes
bridgetknight Aug 8, 2024
55a7762
fix fastQC formatting
bridgetknight Aug 8, 2024
67c3c6e
fix links for footer on analysis page
bridgetknight Aug 8, 2024
1b59655
rearrange summary page and add HRP2/HRP3 snapshots
bridgetknight Aug 12, 2024
2ffcf63
rearrange summary page and add new location info
bridgetknight Aug 16, 2024
03fd846
fix typo
bridgetknight Aug 16, 2024
e001a39
some changes to text formatting
bridgetknight Aug 16, 2024
695d975
edit date parsing
bridgetknight Aug 16, 2024
7f7c893
fix missing import
bridgetknight Aug 16, 2024
af8dcf5
fix floating point display
bridgetknight Aug 16, 2024
c1812a2
fix floating point display (decimal place)
bridgetknight Aug 16, 2024
78ad7ff
add flag for long reads vs. short reads
bridgetknight Aug 16, 2024
ed8b0c3
add conditional header with link
bridgetknight Aug 16, 2024
e510de5
fix spacing on header link popup
bridgetknight Aug 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockstore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,3 +96,6 @@ workflows:
- name: CleanupIntermediate
subclass: wdl
primaryDescriptorPath: /wdl/pipelines/TechAgnostic/Utility/CleanupIntermediate.wdl
- name: GenerateMalariaReport
subclass: wdl
primaryDescriptorPath: /wdl/pipelines/TechAgnostic/Utility/MalariaReportsWorkflow.wdl
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ venv/
.tox
test/test_output/
__pycache__
*.zip
docker/lr-malaria-reports/report-files/Senegal_Terra_Metadata.csv
11 changes: 11 additions & 0 deletions docker/lr-malaria-reports/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
templates/map.html
test_data/
data/
/.vscode
*_report.html
*.jpeg
*.json
*.ipynb
*.txt
*.bed.gz
test.html
36 changes: 36 additions & 0 deletions docker/lr-malaria-reports/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Base image
FROM continuumio/miniconda3:4.9.2

LABEL Bridget Knight

# install gsutil
RUN apt-get --allow-releaseinfo-change update
RUN apt install -y curl git-lfs parallel
RUN curl https://sdk.cloud.google.com | bash

# Setup crcmodc for gsutil:
RUN apt-get install -y gcc python3-dev python3-setuptools && \
pip3 uninstall -y crcmod && \
pip3 install --no-cache-dir -U crcmod

#### Specific for google cloud support
RUN apt-get install -y gnupg2
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \
&& curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - \
&& apt-get update -y \
&& apt-get install google-cloud-sdk -y

# make sure pip is up to date
RUN pip3 install --upgrade pip==23.2.1

# install other packages
RUN pip3 install --no-cache-dir NumPy==1.24.4
RUN pip3 install --no-cache-dir pandas==1.3.0
RUN pip3 install --no-cache-dir argparse==1.4.0 folium==0.14.0 jinja2==3.1.2 \
MarkupSafe==2.1.3 matplotlib==3.7.3 \
plotly==5.15.0 tabulate==0.9.0

ENV PATH=/root/google-cloud-sdk/bin/:${PATH}

# copy other resources
COPY ./report-files /report-files/
18 changes: 18 additions & 0 deletions docker/lr-malaria-reports/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
IMAGE_NAME = lr-malaria-reports
VERSION = 0.0.2

TAG1 = us.gcr.io/broad-dsp-lrma/$(IMAGE_NAME):$(VERSION)
TAG2 = us.gcr.io/broad-dsp-lrma/$(IMAGE_NAME):latest

all: | build push

build:
docker build -t $(TAG1) -t $(TAG2) .

build_no_cache:
docker build --no-cache -t $(TAG1) -t $(TAG2) .

push:
docker push $(TAG1)
docker push $(TAG2)

Loading
Loading