diff --git a/.circleci/config.yml b/.circleci/config.yml
index 7454e22e..2b716ea8 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -51,7 +51,7 @@ jobs:
name: build image
command: |
source ${BASH_ENV}
- export DOCKER_CACHE_TAG=v1.1.6
+ export DOCKER_CACHE_TAG=test-v1.1.8
echo "pulling ${DOCKER_CACHE_TAG}!"
docker pull quay.io/encode-dcc/chip-seq-pipeline:${DOCKER_CACHE_TAG}
docker login -u=${QUAY_ROBOT_USER} -p=${QUAY_ROBOT_USER_TOKEN} quay.io
diff --git a/README.md b/README.md
index 3871b131..bb24c099 100644
--- a/README.md
+++ b/README.md
@@ -7,41 +7,67 @@ This ChIP-Seq pipeline is based off the ENCODE (phase-3) transcription factor an
### Features
-* **Flexibility**: Support for `docker`, `singularity` and `Conda`.
* **Portability**: Support for many cloud platforms (Google/DNAnexus) and cluster engines (SLURM/SGE/PBS).
-* **Resumability**: [Resume](utils/qc_jsons_to_tsv/README.md) a failed workflow from where it left off.
* **User-friendly HTML report**: tabulated quality metrics including alignment/peak statistics and FRiP along with many useful plots (IDR/cross-correlation measures).
- Examples: [HTML](https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/example_output/qc.html), [JSON](docs/example_output/v1.1.5/qc.json)
* **Genomes**: Pre-built database for GRCh38, hg19, mm10, mm9 and additional support for custom genomes.
-## Installation and tutorial
+## Installation
-This pipeline supports many cloud platforms and cluster engines. It also supports `docker`, `singularity` and `Conda` to resolve complicated software dependencies for the pipeline. A tutorial-based instruction for each platform will be helpful to understand how to run pipelines. There are special instructions for two major Stanford HPC servers (SCG4 and Sherlock).
+1) Install [Caper](https://github.com/ENCODE-DCC/caper#installation). Caper is a python wrapper for [Cromwell](https://github.com/broadinstitute/cromwell). Make sure that you have python3(> 3.4.1) installed on your system.
-* Cloud platforms
- * Web interface
- * [DNAnexus Platform](docs/tutorial_dx_web.md)
- * CLI (command line interface)
- * [Google Cloud Platform](docs/tutorial_google.md)
- * [DNAnexus Platform](docs/tutorial_dx_cli.md)
-* Stanford HPC servers (CLI)
- * [Stanford SCG4](docs/tutorial_scg.md)
- * [Stanford Sherlock 2.0](docs/tutorial_sherlock.md)
-* Cluster engines (CLI)
- * [SLURM](docs/tutorial_slurm.md)
- * [Sun GridEngine (SGE/PBS)](docs/tutorial_sge.md)
-* Local computers (CLI)
- * [Local system with `singularity`](docs/tutorial_local_singularity.md)
- * [Local system with `docker`](docs/tutorial_local_docker.md)
- * [Local system with `Conda`](docs/tutorial_local_conda.md)
+ ```bash
+ $ pip install caper
+ ```
-## Input JSON file
+2) Read through [Caper's README](https://github.com/ENCODE-DCC/caper) carefully.
-[Input JSON file specification](docs/input.md)
+3) Run a pipeline with Caper.
-## Output directories
+## Conda
-[Output directory specification](docs/output.md)
+We don't recommend Conda for dependency helper. Use Docker or Singularity instead. We will not take any issues about Conda. You can install Singularity locally without super-user privilege and use it for our pipeline with Caper (with `--use-singularity`).
+
+1) Install [Conda](https://docs.conda.io/en/latest/miniconda.html).
+
+2) Install Conda environment for pipeline.
+
+ ```bash
+ $ conda/install_dependencies.sh
+ ```
+
+## Tutorial
+
+Make sure that you have configured Caper correctly.
+
+```bash
+$ caper run chip.wdl -i examples/caper/ENCSR936XTK_subsampled_chr19_only.json --deepcopy --use-singularity
+```
+
+If you use Conda or Docker (on cloud platforms) then remove `--use-singularity` from the command line and activate it before running a pipeline.
+```bash
+$ conda activate encode-chip-seq-pipeline
+```
+
+## How to organize outputs
+
+Install [Croo](https://github.com/ENCODE-DCC/croo#installation). Make sure that you have python3(> 3.4.1) installed on your system.
+
+```bash
+$ pip install croo
+```
+
+Find a `metadata.json` on Caper's output directory.
+
+```bash
+$ croo [METADATA_JSON_FILE]
+```
+
+## How to build/download genome database
+
+You need to specify a genome data TSV file in your input JSON. Such TSV can be generated/downloaded with actual genome database files.
+
+Use genome database [downloader](genome/download_genome_data.sh) or [builder](docs/build_genome_database.md) for your own genome.
## Useful tools
@@ -51,10 +77,6 @@ There are some useful tools to post-process outputs of the pipeline.
[This tool](utils/qc_jsons_to_tsv/README.md) recursively finds and parses all `qc.json` (pipeline's [final output](docs/example_output/v1.1.5/qc.json)) found from a specified root directory. It generates a TSV file that has all quality metrics tabulated in rows for each experiment and replicate. This tool also estimates overall quality of a sample by [a criteria definition JSON file](utils/qc_jsons_to_tsv/criteria.default.json) which can be a good guideline for QC'ing experiments.
-### resumer
-
-[This tool](utils/resumer/README.md) parses a metadata JSON file from a previous failed workflow and generates a new input JSON file to start a pipeline from where it left off.
-
### ENCODE downloader
[This tool](https://github.com/kundajelab/ENCODE_downloader) downloads any type (FASTQ, BAM, PEAK, ...) of data from the ENCODE portal. It also generates a metadata JSON file per experiment which will be very useful to make an input JSON file for the pipeline.
\ No newline at end of file
diff --git a/chip.wdl b/chip.wdl
index 2c4589bc..74c4c70d 100644
--- a/chip.wdl
+++ b/chip.wdl
@@ -1,8 +1,12 @@
# ENCODE DCC TF/Histone ChIP-Seq pipeline
# Author: Jin Lee (leepc12@gmail.com)
+#CAPER docker quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
+#CAPER singularity docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
+#CROO out_def https://storage.googleapis.com/encode-pipeline-output-definition/chip.out_def.json
+
workflow chip {
- String pipeline_ver = 'v1.1.7'
+ String pipeline_ver = 'v1.2.0'
### sample name, description
String title = 'Untitled'
String description = 'No description'
@@ -15,6 +19,7 @@ workflow chip {
File genome_tsv # reference genome data TSV file including
# all important genome specific data file paths and parameters
Boolean paired_end
+ Boolean? ctl_paired_end
### optional but important
Boolean align_only = false # disable all post-align analysis (peak-calling, overlap, idr, ...)
@@ -47,6 +52,9 @@ workflow chip {
Int xcor_subsample_reads = 15000000 # number of reads to subsample TAGALIGN
# this will be used for xcor only
# will not affect any downstream analysis
+ Int xcor_exclusion_range_min = -500
+ Int? xcor_exclusion_range_max
+
Boolean keep_irregular_chr_in_bfilt_peak = false # when filtering with blacklist
# do not filter peaks with irregular chr name
# and just keep them in bfilt_peak file
@@ -66,12 +74,12 @@ workflow chip {
Int bwa_cpu = 4
Int bwa_mem_mb = 20000
Int bwa_time_hr = 48
- String bwa_disks = "local-disk 100 HDD"
+ String bwa_disks = "local-disk 200 HDD"
Int filter_cpu = 2
Int filter_mem_mb = 20000
Int filter_time_hr = 24
- String filter_disks = "local-disk 100 HDD"
+ String filter_disks = "local-disk 200 HDD"
Int bam2ta_cpu = 2
Int bam2ta_mem_mb = 10000
@@ -83,7 +91,7 @@ workflow chip {
Int fingerprint_cpu = 2
Int fingerprint_mem_mb = 12000
Int fingerprint_time_hr = 6
- String fingerprint_disks = "local-disk 100 HDD"
+ String fingerprint_disks = "local-disk 200 HDD"
Int xcor_cpu = 2
Int xcor_mem_mb = 16000
@@ -92,12 +100,12 @@ workflow chip {
Int macs2_mem_mb = 16000
Int macs2_time_hr = 24
- String macs2_disks = "local-disk 100 HDD"
+ String macs2_disks = "local-disk 200 HDD"
Int spp_cpu = 2
Int spp_mem_mb = 16000
Int spp_time_hr = 72
- String spp_disks = "local-disk 100 HDD"
+ String spp_disks = "local-disk 200 HDD"
#### input file definition
# pipeline can start from any type of inputs and then leave all other types undefined
@@ -166,7 +174,7 @@ workflow chip {
Array[File] ctl_pbc_qcs = []
Array[File] ctl_dup_qcs = []
Array[File] ctl_nodup_flagstat_qcs = []
- Array[File] sig_pvals = []
+ Array[File] pval_bws = []
Array[File] xcor_plots = []
Array[File] xcor_scores = []
@@ -386,7 +394,9 @@ workflow chip {
paired_end = paired_end_xcor,
subsample = xcor_subsample_reads,
mito_chr_name = mito_chr_name,
-
+ chip_seq_type = pipeline_type,
+ exclusion_range_min = xcor_exclusion_range_min,
+ exclusion_range_max = xcor_exclusion_range_max,
cpu = xcor_cpu,
mem_mb = xcor_mem_mb,
time_hr = xcor_time_hr,
@@ -442,7 +452,7 @@ workflow chip {
# merge fastqs
call merge_fastq as merge_fastq_ctl { input :
fastqs = fastq_set,
- paired_end = paired_end,
+ paired_end = if defined(ctl_paired_end) then ctl_paired_end else paired_end,
}
# align merged fastqs with bwa
call bwa as bwa_ctl { input :
@@ -462,7 +472,7 @@ workflow chip {
# filter/dedup bam
call filter as filter_ctl { input :
bam = bam,
- paired_end = paired_end,
+ paired_end = if defined(ctl_paired_end) then ctl_paired_end else paired_end,
dup_marker = dup_marker,
mapq_thresh = mapq_thresh,
no_dup_removal = no_dup_removal,
@@ -480,7 +490,7 @@ workflow chip {
# convert bam to tagalign and subsample it if necessary
call bam2ta as bam2ta_ctl { input :
bam = bam,
- paired_end = paired_end,
+ paired_end = if defined(ctl_paired_end) then ctl_paired_end else paired_end,
subsample = ctl_subsample_reads,
regex_grep_v_ta = regex_filter_reads,
mito_chr_name = mito_chr_name,
@@ -548,11 +558,21 @@ workflow chip {
chrsz = chrsz,
cap_num_peak = macs2_cap_num_peak,
pval_thresh = pval_thresh,
- make_signal = true,
fraglen = fraglen_[i],
blacklist = blacklist,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
+ mem_mb = macs2_mem_mb,
+ disks = macs2_disks,
+ time_hr = macs2_time_hr,
+ }
+ call macs2_signal_track { input :
+ tas = flatten([[tas__[i]], chosen_ctl_tas[i]]),
+ gensz = gensz,
+ chrsz = chrsz,
+ pval_thresh = pval_thresh,
+ fraglen = fraglen_[i],
+
mem_mb = macs2_mem_mb,
disks = macs2_disks,
time_hr = macs2_time_hr,
@@ -590,7 +610,6 @@ workflow chip {
pval_thresh = pval_thresh,
fraglen = fraglen_[i],
blacklist = blacklist,
- make_signal = false,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
mem_mb = macs2_mem_mb,
@@ -605,7 +624,6 @@ workflow chip {
pval_thresh = pval_thresh,
fraglen = fraglen_[i],
blacklist = blacklist,
- make_signal = false,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
mem_mb = macs2_mem_mb,
@@ -670,11 +688,21 @@ workflow chip {
chrsz = chrsz,
cap_num_peak = macs2_cap_num_peak,
pval_thresh = pval_thresh,
- make_signal = true,
fraglen = fraglen_mean.rounded_mean,
blacklist = blacklist,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
+ mem_mb = macs2_mem_mb,
+ disks = macs2_disks,
+ time_hr = macs2_time_hr,
+ }
+ call macs2_signal_track as macs2_signal_track_pooled { input :
+ tas = flatten([select_all([pool_ta.ta_pooled]), chosen_ctl_ta_pooled]),
+ gensz = gensz,
+ chrsz = chrsz,
+ pval_thresh = pval_thresh,
+ fraglen = fraglen_mean.rounded_mean,
+
mem_mb = macs2_mem_mb,
disks = macs2_disks,
time_hr = macs2_time_hr,
@@ -707,7 +735,6 @@ workflow chip {
pval_thresh = pval_thresh,
fraglen = fraglen_mean.rounded_mean,
blacklist = blacklist,
- make_signal = false,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
mem_mb = macs2_mem_mb,
@@ -725,7 +752,6 @@ workflow chip {
pval_thresh = pval_thresh,
fraglen = fraglen_mean.rounded_mean,
blacklist = blacklist,
- make_signal = false,
keep_irregular_chr_in_bfilt_peak = keep_irregular_chr_in_bfilt_peak,
mem_mb = macs2_mem_mb,
@@ -776,25 +802,25 @@ workflow chip {
Array[Pair[String,Array[File]]] peak_pairs =
if length(peaks_)<=1 then [] # 1 rep
else if length(peaks_)<=2 then # 2 reps
- [('rep1-rep2',[peaks_[0],peaks_[1]])]
+ [('rep1_rep2',[peaks_[0],peaks_[1]])]
else if length(peaks_)<=3 then # 3 reps
- [('rep1-rep2',[peaks_[0],peaks_[1]]), ('rep1-rep3',[peaks_[0],peaks_[2]]),
- ('rep2-rep3',[peaks_[1],peaks_[2]])]
+ [('rep1_rep2',[peaks_[0],peaks_[1]]), ('rep1_rep3',[peaks_[0],peaks_[2]]),
+ ('rep2_rep3',[peaks_[1],peaks_[2]])]
else if length(peaks_)<=4 then # 4 reps
- [('rep1-rep2',[peaks_[0],peaks_[1]]), ('rep1-rep3',[peaks_[0],peaks_[2]]), ('rep1-rep4',[peaks_[0],peaks_[3]]),
- ('rep2-rep3',[peaks_[1],peaks_[2]]), ('rep2-rep4',[peaks_[1],peaks_[3]]),
- ('rep3-rep4',[peaks_[2],peaks_[3]])]
+ [('rep1_rep2',[peaks_[0],peaks_[1]]), ('rep1_rep3',[peaks_[0],peaks_[2]]), ('rep1_rep4',[peaks_[0],peaks_[3]]),
+ ('rep2_rep3',[peaks_[1],peaks_[2]]), ('rep2_rep4',[peaks_[1],peaks_[3]]),
+ ('rep3_rep4',[peaks_[2],peaks_[3]])]
else if length(peaks_)<=5 then # 5 reps
- [('rep1-rep2',[peaks_[0],peaks_[1]]), ('rep1-rep3',[peaks_[0],peaks_[2]]), ('rep1-rep4',[peaks_[0],peaks_[3]]), ('rep1-rep5',[peaks_[0],peaks_[4]]),
- ('rep2-rep3',[peaks_[1],peaks_[2]]), ('rep2-rep4',[peaks_[1],peaks_[3]]), ('rep2-rep5',[peaks_[1],peaks_[4]]),
- ('rep3-rep4',[peaks_[2],peaks_[3]]), ('rep3-rep5',[peaks_[2],peaks_[4]]),
- ('rep4-rep5',[peaks_[3],peaks_[4]])]
+ [('rep1_rep2',[peaks_[0],peaks_[1]]), ('rep1_rep3',[peaks_[0],peaks_[2]]), ('rep1_rep4',[peaks_[0],peaks_[3]]), ('rep1_rep5',[peaks_[0],peaks_[4]]),
+ ('rep2_rep3',[peaks_[1],peaks_[2]]), ('rep2_rep4',[peaks_[1],peaks_[3]]), ('rep2_rep5',[peaks_[1],peaks_[4]]),
+ ('rep3_rep4',[peaks_[2],peaks_[3]]), ('rep3_rep5',[peaks_[2],peaks_[4]]),
+ ('rep4_rep5',[peaks_[3],peaks_[4]])]
else # 6 reps
- [('rep1-rep2',[peaks_[0],peaks_[1]]), ('rep1-rep3',[peaks_[0],peaks_[2]]), ('rep1-rep4',[peaks_[0],peaks_[3]]), ('rep1-rep5',[peaks_[0],peaks_[4]]), ('rep1-rep6',[peaks_[0],peaks_[5]]),
- ('rep2-rep3',[peaks_[1],peaks_[2]]), ('rep2-rep4',[peaks_[1],peaks_[3]]), ('rep2-rep5',[peaks_[1],peaks_[4]]), ('rep2-rep6',[peaks_[1],peaks_[5]]),
- ('rep3-rep4',[peaks_[2],peaks_[3]]), ('rep3-rep5',[peaks_[2],peaks_[4]]), ('rep3-rep6',[peaks_[2],peaks_[5]]),
- ('rep4-rep5',[peaks_[3],peaks_[4]]), ('rep4-rep6',[peaks_[3],peaks_[5]]),
- ('rep5-rep6',[peaks_[4],peaks_[5]])]
+ [('rep1_rep2',[peaks_[0],peaks_[1]]), ('rep1_rep3',[peaks_[0],peaks_[2]]), ('rep1_rep4',[peaks_[0],peaks_[3]]), ('rep1_rep5',[peaks_[0],peaks_[4]]), ('rep1_rep6',[peaks_[0],peaks_[5]]),
+ ('rep2_rep3',[peaks_[1],peaks_[2]]), ('rep2_rep4',[peaks_[1],peaks_[3]]), ('rep2_rep5',[peaks_[1],peaks_[4]]), ('rep2_rep6',[peaks_[1],peaks_[5]]),
+ ('rep3_rep4',[peaks_[2],peaks_[3]]), ('rep3_rep5',[peaks_[2],peaks_[4]]), ('rep3_rep6',[peaks_[2],peaks_[5]]),
+ ('rep4_rep5',[peaks_[3],peaks_[4]]), ('rep4_rep6',[peaks_[3],peaks_[5]]),
+ ('rep5_rep6',[peaks_[4],peaks_[5]])]
if ( length(peaks_)>0 ) {
scatter( pair in peak_pairs ) {
# Naive overlap on every pair of true replicates
@@ -948,7 +974,6 @@ workflow chip {
Array[File] xcor_plots_ = flatten([xcor_plots, xcor.plot_png])
Array[File] xcor_scores_ = flatten([xcor_scores, xcor.score])
- Array[File] sig_pvals_ = flatten([sig_pvals, macs2.sig_pval])
Array[File] macs2_frip_qcs_ = flatten([macs2_frip_qcs, macs2.frip_qc])
Array[File] macs2_pr1_frip_qcs_ = flatten(select_all([macs2_pr1_frip_qcs, macs2_pr1.frip_qc]))
@@ -964,6 +989,7 @@ workflow chip {
description = description,
genome = basename(genome_tsv),
paired_end = paired_end,
+ ctl_paired_end = if defined(ctl_paired_end) then ctl_paired_end else paired_end,
pipeline_type = pipeline_type,
peak_caller = peak_caller_,
macs2_cap_num_peak = macs2_cap_num_peak,
@@ -1115,8 +1141,6 @@ task filter {
String disks
command {
- ${if no_dup_removal then "touch null.dup.qc null.pbc.qc; " else ""}
- touch null
python $(which encode_filter.py) \
${bam} \
${if paired_end then "--paired-end" else ""} \
@@ -1131,8 +1155,8 @@ task filter {
File nodup_bam = glob("*.bam")[0]
File nodup_bai = glob("*.bai")[0]
File flagstat_qc = glob("*.flagstat.qc")[0]
- File dup_qc = if no_dup_removal then glob("null")[0] else glob("*.dup.qc")[0]
- File pbc_qc = if no_dup_removal then glob("null")[0] else glob("*.pbc.qc")[0]
+ File dup_qc = glob("*.dup.qc")[0]
+ File pbc_qc = glob("*.pbc.qc")[0]
}
runtime {
cpu : cpu
@@ -1224,6 +1248,10 @@ task xcor {
Int subsample # number of reads to subsample TAGALIGN
# this will be used for xcor only
# will not affect any downstream analysis
+ String? chip_seq_type
+ Int? exclusion_range_min
+ Int? exclusion_range_max
+
Int cpu
Int mem_mb
Int time_hr
@@ -1235,6 +1263,10 @@ task xcor {
${if paired_end then "--paired-end" else ""} \
${"--mito-chr-name " + mito_chr_name} \
${"--subsample " + subsample} \
+ ${"--chip-seq-type " + chip_seq_type} \
+ ${"--exclusion-range-min " + exclusion_range_min} \
+ ${"--exclusion-range-max " + exclusion_range_max} \
+ ${"--subsample " + subsample} \
${"--nth " + cpu}
}
output {
@@ -1337,7 +1369,6 @@ task macs2 {
File chrsz # 2-col chromosome sizes file
Int cap_num_peak # cap number of raw peaks called from MACS2
Float pval_thresh # p.value threshold
- Boolean make_signal
File blacklist # blacklist BED to filter raw peaks
Boolean keep_irregular_chr_in_bfilt_peak
@@ -1346,9 +1377,6 @@ task macs2 {
String disks
command {
- ${if make_signal then ""
- else "touch null.pval.signal.bigwig null.fc.signal.bigwig"}
- touch null
python $(which encode_macs2_chip.py) \
${sep=' ' tas} \
${"--gensz "+ gensz} \
@@ -1356,7 +1384,6 @@ task macs2 {
${"--fraglen " + fraglen} \
${"--cap-num-peak " + cap_num_peak} \
${"--pval-thresh "+ pval_thresh} \
- ${if make_signal then "--make-signal" else ""} \
${if keep_irregular_chr_in_bfilt_peak then "--keep-irregular-chr" else ""} \
${"--blacklist "+ blacklist}
}
@@ -1365,8 +1392,6 @@ task macs2 {
File bfilt_npeak = glob("*.bfilt.narrowPeak.gz")[0]
File bfilt_npeak_bb = glob("*.bfilt.narrowPeak.bb")[0]
Array[File] bfilt_npeak_hammock = glob("*.bfilt.narrowPeak.hammock.gz*")
- File sig_pval = if make_signal then glob("*.pval.signal.bigwig")[0] else glob("null")[0]
- File sig_fc = if make_signal then glob("*.fc.signal.bigwig")[0] else glob("null")[0]
File frip_qc = glob("*.frip.qc")[0]
}
runtime {
@@ -1377,6 +1402,38 @@ task macs2 {
}
}
+task macs2_signal_track {
+ Array[File] tas # [ta, control_ta]. control_ta is optional
+ Int fraglen # fragment length from xcor
+ String gensz # Genome size (sum of entries in 2nd column of
+ # chr. sizes file, or hs for human, ms for mouse)
+ File chrsz # 2-col chromosome sizes file
+ Float pval_thresh # p.value threshold
+
+ Int mem_mb
+ Int time_hr
+ String disks
+
+ command {
+ python $(which encode_macs2_signal_track_chip.py) \
+ ${sep=' ' tas} \
+ ${"--gensz "+ gensz} \
+ ${"--chrsz " + chrsz} \
+ ${"--fraglen " + fraglen} \
+ ${"--pval-thresh "+ pval_thresh}
+ }
+ output {
+ File pval_bw = glob("*.pval.signal.bigwig")[0]
+ File fc_bw = glob("*.fc.signal.bigwig")[0]
+ }
+ runtime {
+ cpu : 1
+ memory : "${mem_mb} MB"
+ time : time_hr
+ disks : disks
+ }
+}
+
task spp {
Array[File] tas # [ta, control_ta]. control_ta is always required
Int fraglen # fragment length from xcor
@@ -1403,7 +1460,7 @@ task spp {
output {
File rpeak = glob("*[!.][!b][!f][!i][!l][!t].regionPeak.gz")[0]
File bfilt_rpeak = glob("*.bfilt.regionPeak.gz")[0]
- File bfilt_rpeak_peak_bb = glob("*.bfilt.regionPeak.bb")[0]
+ File bfilt_rpeak_bb = glob("*.bfilt.regionPeak.bb")[0]
Array[File] bfilt_rpeak_hammock = glob("*.bfilt.regionPeak.hammock.gz*")
File frip_qc = glob("*.frip.qc")[0]
}
@@ -1557,6 +1614,7 @@ task qc_report {
#String? encode_accession_id # ENCODE accession ID of sample
# workflow params
Boolean paired_end
+ Boolean ctl_paired_end
String pipeline_type
String peak_caller
Int? macs2_cap_num_peak
@@ -1609,6 +1667,7 @@ task qc_report {
${"--genome " + genome} \
${"--multimapping " + 0} \
${if paired_end then "--paired-end" else ""} \
+ ${if ctl_paired_end then "--ctl-paired-end" else ""} \
--pipeline-type ${pipeline_type} \
--peak-caller ${peak_caller} \
${"--macs2-cap-num-peak " + macs2_cap_num_peak} \
diff --git a/conda/install_dependencies.sh b/conda/install_dependencies.sh
index 5abdcaa3..12c51d32 100755
--- a/conda/install_dependencies.sh
+++ b/conda/install_dependencies.sh
@@ -36,6 +36,9 @@ conda create -n ${CONDA_ENV_PY3} --file ${REQ_TXT_PY3} -y -c bioconda -c conda-f
echo "=== Installing packages for python2 env..."
conda create -n ${CONDA_ENV} --file ${REQ_TXT} -y -c bioconda -c conda-forge -c defaults -c r
+echo "=== Wait 30 secs..."
+sleep 30
+
echo "=== Installing additional packages for python2 env..."
source activate ${CONDA_ENV}
# all numpy packages in the conda-forge repo are linked to OpenBLAS,
@@ -93,7 +96,7 @@ source deactivate
# transfer pipeline's python scripts (src/*.py) to conda env's bin directory
# so that these pythons scripts can be accessible with UNIX which.
-cd ${SH_SCRIPT_DIR}
-bash update_conda_env.sh
+#cd ${SH_SCRIPT_DIR}
+#bash update_conda_env.sh
echo "=== All done."
diff --git a/conda/requirements_py3.txt b/conda/requirements_py3.txt
index b6a58331..ed977740 100644
--- a/conda/requirements_py3.txt
+++ b/conda/requirements_py3.txt
@@ -8,7 +8,9 @@ bedtools ==2.26.0
java-jdk ==8.0.92
# to resolve 'CXXABI_1.3.9' not found issue
libgcc==5.2.0 # this does not work with MacOS...
-matplotlib ==1.5.3 #==1.5.1
+matplotlib ==2.0.0 #==1.5.1
ncurses ==6.1
tabix==0.2.6
readline==6.2
+pyhocon
+requests
diff --git a/conda/update_conda_env.sh b/conda/update_conda_env.sh
index b20e32b0..98252e57 100755
--- a/conda/update_conda_env.sh
+++ b/conda/update_conda_env.sh
@@ -32,4 +32,4 @@ source activate ${CONDA_ENV}
chmod u+rx ${CONDA_BIN}/*.py
source deactivate
-echo "=== All done."
\ No newline at end of file
+echo "=== All done."
diff --git a/croo/chip.out_def.json b/croo/chip.out_def.json
new file mode 100644
index 00000000..af515b04
--- /dev/null
+++ b/croo/chip.out_def.json
@@ -0,0 +1,650 @@
+{
+ "chip.fingerprint": {
+ "plot": {
+ "path": "qc/${basename}",
+ "table": "QC and logs/Deeptools fingerprint plot"
+ }
+ },
+ "chip.bwa_ctl": {
+ "bam": {
+ "path": "align/ctl${i+1}/${basename}",
+ "table": "Alignment/Control ${i+1}/Raw BAM from bwa"
+ },
+ "flagstat_qc": {
+ "path": "qc/ctl${i+1}/${basename}",
+ "table": "QC and logs/Control ${i+1}/Samtools flagstat for Raw BAM"
+ }
+ },
+ "chip.filter_ctl": {
+ "nodup_bam": {
+ "path": "align/ctl${i+1}/${basename}",
+ "table": "Alignment/Control ${i+1}/Filtered BAM"
+ },
+ "flagstat_qc": {
+ "path": "qc/ctl${i+1}/${basename}",
+ "table": "QC and logs/Control ${i+1}/Samtools flagstat log for filtered BAM"
+ },
+ "dup_qc": {
+ "path": "qc/ctl${i+1}/${basename}",
+ "table": "QC and logs/Control ${i+1}/Dup QC for filtered BAM"
+ },
+ "pbc_qc": {
+ "path": "qc/ctl${i+1}/${basename}",
+ "table": "QC and logs/Control ${i+1}/PBC QC for filtered BAM"
+ },
+ "mito_dup_log": {
+ "path": "qc/ctl${i+1}/${basename}",
+ "table": "QC and logs/Control ${i+1}/Mito dup log for filtered BAM"
+ }
+ },
+ "chip.bam2ta_ctl": {
+ "ta": {
+ "path": "align/ctl${i+1}/${basename}",
+ "table": "Alignment/Control ${i+1}/TAG-ALIGN"
+ }
+ },
+ "chip.pool_ta_ctl": {
+ "ta_pooled": {
+ "path": "align/pooled-ctl/${basename}",
+ "table": "Alignment/Pooled control/TAG-ALIGN"
+ }
+ },
+ "chip.bwa": {
+ "bam": {
+ "path": "align/rep${i+1}/${basename}",
+ "table": "Alignment/Replicate ${i+1}/Raw BAM from bwa"
+ },
+ "flagstat_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Samtools flagstat for Raw BAM"
+ }
+ },
+ "chip.filter": {
+ "nodup_bam": {
+ "path": "align/rep${i+1}/${basename}",
+ "table": "Alignment/Replicate ${i+1}/Filtered BAM"
+ },
+ "flagstat_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Samtools flagstat log for filtered BAM"
+ },
+ "dup_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Dup QC for filtered BAM"
+ },
+ "pbc_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/PBC QC for filtered BAM"
+ },
+ "mito_dup_log": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Mito dup log for filtered BAM"
+ }
+ },
+ "chip.bam2ta": {
+ "ta": {
+ "path": "align/rep${i+1}/${basename}",
+ "table": "Alignment/Replicate ${i+1}/TAG-ALIGN"
+ }
+ },
+ "chip.spr": {
+ "ta_pr1": {
+ "path": "align/rep${i+1}/pseudorep1/${basename}",
+ "table": "Alignment/Replicate ${i+1}/Pseudoreplicate 1/TAG-ALIGN"
+ },
+ "ta_pr2": {
+ "path": "align/rep${i+1}/pseudorep2/${basename}",
+ "table": "Alignment/Replicate ${i+1}/Pseudoreplicate 2/TAG-ALIGN"
+ }
+ },
+ "chip.pool_ta": {
+ "ta_pooled": {
+ "path": "align/pooled-rep/${basename}",
+ "table": "Alignment/Pooled replicate/TAG-ALIGN"
+ }
+ },
+ "chip.pool_ta_pr1": {
+ "ta_pooled": {
+ "path": "align/pooled-rep/pseudorep1/${basename}",
+ "table": "Alignment/Pooled replicate/Pseudoreplicate 1/TAG-ALIGN"
+ }
+ },
+ "chip.pool_ta_pr2": {
+ "ta_pooled": {
+ "path": "align/pooled-rep/pseudorep2/${basename}",
+ "table": "Alignment/Pooled replicate/Pseudoreplicate 2/TAG-ALIGN"
+ }
+ },
+ "chip.xcor": {
+ "plot_png": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Cross-correlation Plot"
+ },
+ "score": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Cross-correlation score"
+ },
+ "fraglen": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Estimated fragment length"
+ }
+ },
+ "chip.macs2": {
+ "npeak": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.macs2_pr1": {
+ "npeak": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/pseudorep1/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudoreplicate 1/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.macs2_pr2": {
+ "npeak": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/pseudorep2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudoreplicate 2/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.macs2_pooled": {
+ "npeak": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/${basename}",
+ "table": "QC and logs/Pooled replicate/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.macs2_ppr1": {
+ "npeak": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/pseudorep1/${basename}",
+ "table": "QC and logs/Pooled replicate/Pseudoreplicate 1/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.macs2_ppr2": {
+ "npeak": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Raw MACS2 narrowpeak"
+ },
+ "bfilt_npeak": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak"
+ },
+ "bfilt_npeak_bb": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak (BigBed)"
+ },
+ "bfilt_npeak_hammock": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered MACS2 narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/pseudorep2/${basename}",
+ "table": "QC and logs/Pooled replicate/Pseudoreplicate 2/Fraction of MACS2 peaks in reads"
+ }
+ },
+ "chip.spp": {
+ "rpeak": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/rep${i+1}/${basename}",
+ "table": "Peak/Replicate ${i+1}/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.spp_pr1": {
+ "rpeak": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/rep${i+1}/pseudorep1/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/pseudorep1/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudoreplicate 1/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.spp_pr2": {
+ "rpeak": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/rep${i+1}/pseudorep2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}/pseudorep2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudoreplicate 2/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.spp_pooled": {
+ "rpeak": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/pooled-rep/${basename}",
+ "table": "Peak/Pooled replicate/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/${basename}",
+ "table": "QC and logs/Pooled replicate/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.spp_ppr1": {
+ "rpeak": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/pooled-rep/pseudorep1/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 1/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/pseudorep1/${basename}",
+ "table": "QC and logs/Pooled replicate/Pseudoreplicate 1/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.spp_ppr2": {
+ "rpeak": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Raw SPP narrowpeak"
+ },
+ "bfilt_rpeak": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak"
+ },
+ "bfilt_rpeak_bb": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak (BigBed)"
+ },
+ "bfilt_rpeak_hammock": {
+ "path": "peak/pooled-rep/pseudorep2/${basename}",
+ "table": "Peak/Pooled replicate/Pseudoreplicate 2/Blacklist-filtered SPP narrowpeak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-rep/pseudorep2/${basename}",
+ "table": "QC and logs/Pooled replicate/Pseudoreplicate 2/Fraction of SPP peaks in reads"
+ }
+ },
+ "chip.macs2_signal_track": {
+ "pval_bw": {
+ "path": "signal/rep${i+1}/${basename}",
+ "table": "Signal/Replicate ${i+1}/MACS2 signal track (p-val)"
+ },
+ "fc_bw": {
+ "path": "signal/rep${i+1}/${basename}",
+ "table": "Signal/Replicate ${i+1}/MACS2 signal track (fold-enrichment)"
+ }
+ },
+ "chip.macs2_signal_track_pooled": {
+ "pval_bw": {
+ "path": "signal/pooled-rep/${basename}",
+ "table": "Signal/Pooled replicate/MACS2 signal track (p-val)"
+ },
+ "fc_bw": {
+ "path": "signal/pooled-rep/${basename}",
+ "table": "Signal/Pooled replicate/MACS2 signal track (fold-enrichment)"
+ }
+ },
+ "chip.count_signal_track": {
+ "pos_bw": {
+ "path": "signal/rep${i+1}/${basename}",
+ "table": "Signal/Replicate ${i+1}/Count signal track (positive)"
+ },
+ "neg_bw": {
+ "path": "signal/rep${i+1}/${basename}",
+ "table": "Signal/Replicate ${i+1}/Count signal track (negative)"
+ }
+ },
+ "chip.count_signal_track_pooled": {
+ "pos_bw": {
+ "path": "signal/pooled-rep/${basename}",
+ "table": "Signal/Pooled replicate/Count signal track (positive)"
+ },
+ "neg_bw": {
+ "path": "signal/pooled-rep/${basename}",
+ "table": "Signal/Pooled replicate/Count signal track (negative)"
+ }
+ },
+ "chip.idr": {
+ "bfilt_idr_peak": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered IDR peak"
+ },
+ "bfilt_idr_peak_bb": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered IDR peak (BigBed)"
+ },
+ "bfilt_idr_peak_hammock": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered IDR peak (hammock)"
+ },
+ "idr_unthresholded_peak": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Unthresholded raw IDR peak"
+ },
+ "idr_plot": {
+ "path": "qc/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "QC and logs/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/IDR plot"
+ },
+ "idr_log": {
+ "path": "qc/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "QC and logs/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/IDR log"
+ },
+ "frip_qc": {
+ "path": "qc/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "QC and logs/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Fraction of IDR peaks in reads"
+ }
+ },
+ "chip.idr_ppr": {
+ "bfilt_idr_peak": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered IDR peak"
+ },
+ "bfilt_idr_peak_bb": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered IDR peak (BigBed)"
+ },
+ "bfilt_idr_peak_hammock": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered IDR peak (hammock)"
+ },
+ "idr_unthresholded_peak": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Unthresholded raw IDR peak"
+ },
+ "idr_plot": {
+ "path": "qc/pooled-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Pooled pseudoreplicate 1 vs. 2/IDR plot"
+ },
+ "idr_log": {
+ "path": "qc/pooled-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Pooled pseudoreplicate 1 vs. 2/IDR log"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Pooled pseudoreplicate 1 vs. 2/Fraction of IDR peaks in reads"
+ }
+ },
+ "chip.idr_pr": {
+ "bfilt_idr_peak": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered IDR peak"
+ },
+ "bfilt_idr_peak_bb": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered IDR peak (BigBed)"
+ },
+ "bfilt_idr_peak_hammock": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered IDR peak (hammock)"
+ },
+ "idr_unthresholded_peak": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Unthresholded raw IDR peak"
+ },
+ "idr_plot": {
+ "path": "qc/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/IDR plot"
+ },
+ "idr_log": {
+ "path": "qc/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/IDR log"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Fraction of IDR peaks in reads"
+ }
+ },
+ "chip.overlap": {
+ "bfilt_overlap_peak": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered overlap peak"
+ },
+ "bfilt_overlap_peak_bb": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered overlap peak (BigBed)"
+ },
+ "bfilt_overlap_peak_hammock": {
+ "path": "peak/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "Peak/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Blacklist-filtered overlap peak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/${basename.split('.')[0].replace('_','_vs_').replace('-','_vs_')}/${basename}",
+ "table": "QC and logs/${basename.split('.')[0].replace('_',' vs. ').replace('-',' vs. ').capitalize()}/Fraction of overlap peaks in reads"
+ }
+ },
+ "chip.overlap_ppr": {
+ "bfilt_overlap_peak": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered overlap peak"
+ },
+ "bfilt_overlap_peak_bb": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered overlap peak (BigBed)"
+ },
+ "bfilt_overlap_peak_hammock": {
+ "path": "peak/pooled-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Pooled pseudoreplicate 1 vs. 2/Blacklist-filtered overlap peak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/pooled-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Pooled pseudoreplicate 1 vs. 2/Fraction of overlap peaks in reads"
+ }
+ },
+ "chip.overlap_pr": {
+ "bfilt_overlap_peak": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered overlap peak"
+ },
+ "bfilt_overlap_peak_bb": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered overlap peak (BigBed)"
+ },
+ "bfilt_overlap_peak_hammock": {
+ "path": "peak/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "Peak/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Blacklist-filtered overlap peak (hammock)"
+ },
+ "frip_qc": {
+ "path": "qc/rep${i+1}-pseudorep1_vs_2/${basename}",
+ "table": "QC and logs/Replicate ${i+1}/Pseudorep 1 vs. pseudorep 2/Fraction of overlap peaks in reads"
+ }
+ },
+ "chip.reproducibility_idr": {
+ "optimal_peak": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Optimal peak"
+ },
+ "optimal_peak_bb": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Optimal peak (BigBed)"
+ },
+ "optimal_peak_hammock": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Optimal peak (hammock)"
+ },
+ "conservative_peak": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Conservative peak"
+ },
+ "conservative_peak_bb": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Conservative peak (BigBed)"
+ },
+ "conservative_peak_hammock": {
+ "path": "peak/idr_reproducibility/${basename}",
+ "table": "Peak/IDR reproducibility/Conservative peak (hammock)"
+ },
+ "reproducibility_qc": {
+ "path": "qc/${basename}",
+ "table": "QC and logs/Reproducibility QC for overlap peaks"
+ }
+ },
+ "chip.reproducibility_overlap": {
+ "optimal_peak": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Optimal peak"
+ },
+ "optimal_peak_bb": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Optimal peak (BigBed)"
+ },
+ "optimal_peak_hammock": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Optimal peak (hammock)"
+ },
+ "conservative_peak": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Conservative peak"
+ },
+ "conservative_peak_bb": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Conservative peak (BigBed)"
+ },
+ "conservative_peak_hammock": {
+ "path": "peak/overlap_reproducibility/${basename}",
+ "table": "Peak/Overlap reproducibility/Conservative peak (hammock)"
+ },
+ "reproducibility_qc": {
+ "path": "qc/${basename}",
+ "table": "QC and logs/Reproducibility QC for IDR peaks"
+ }
+ },
+ "chip.qc_report": {
+ "report": {
+ "path": "qc/${basename}",
+ "table": "QC and logs/Final HTML report"
+ },
+ "qc_json": {
+ "path": "qc/${basename}",
+ "table": "QC and logs/Final QC JSON file"
+ }
+ }
+}
diff --git a/docker_image/Dockerfile b/docker_image/Dockerfile
index f2045b2d..cdc46206 100644
--- a/docker_image/Dockerfile
+++ b/docker_image/Dockerfile
@@ -74,8 +74,8 @@ RUN git clone --branch 2.0.4.2 --single-branch https://github.com/kundajelab/idr
cd idr && python3 setup.py install && cd ../ && rm -rf idr*
# Install UCSC tools (Kent utils) latest
-RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bedClip && chmod +x bedClip
-RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bedGraphToBigWig && chmod +x bedGraphToBigWig
+RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64.v369/bedClip && chmod +x bedClip
+RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64.v369/bedGraphToBigWig && chmod +x bedGraphToBigWig
# Install samtools 1.2
RUN git clone --branch 1.2 --single-branch https://github.com/samtools/samtools.git && \
@@ -126,7 +126,7 @@ RUN pip install --no-cache-dir --no-dependencies deeptools==2.5.4
RUN pip install --no-cache-dir pyfaidx==0.4.7.1
# Install UCSC tools (Kent utils) latest for peak2bigbed conversion
-RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bedToBigBed && chmod +x bedToBigBed
+RUN wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64.v369/bedToBigBed && chmod +x bedToBigBed
# Install bgzip/tabix for Wash U browser track (hammock type)
#RUN apt-get update && apt-get install -y tabix && rm -rf /var/lib/apt/lists/*
diff --git a/docs/dev.md b/docs/dev.md
index 4993fd6b..b8bb380c 100644
--- a/docs/dev.md
+++ b/docs/dev.md
@@ -2,8 +2,8 @@
## Command line for version change
```bash
-PREV_VER=v1.1.7
-NEW_VER=v1.1.7
+PREV_VER=v1.2.0
+NEW_VER=v1.2.0
for f in $(grep -rl ${PREV_VER} --include=*.{wdl,md,sh})
do
sed -i "s/${PREV_VER}/${NEW_VER}/g" ${f}
@@ -24,7 +24,7 @@ Run the following command line locally to build out DX workflows for this pipeli
```bash
# version
-VER=v1.1.7
+VER=v1.2.0
# general
java -jar ~/dxWDL-0.77.jar compile chip.wdl -project "ENCODE Uniform Processing Pipelines" -extras workflow_opts/docker.json -f -folder /ChIP-seq2/workflows/$VER/general -defaults examples/dx/template_general.json
diff --git a/docs/input.md b/docs/input.md
index 58924db7..cb936b14 100644
--- a/docs/input.md
+++ b/docs/input.md
@@ -42,6 +42,10 @@ Let us take a close look at the following template JSON. Comments are not allowe
// Read endedness
"chip.paired_end" : true,
+
+ // Read endedness for controls
+ // If not defined, "chip.paired_end" will be used for controls too
+ "chip.ctl_paired_end" : true,
// If you start from FASTQs then define these, otherwise remove from this file.
// You can define up to 6 replicates.
diff --git a/docs/tutorial_dx_web.md b/docs/tutorial_dx_web.md
index 1f20705a..4b0e1b01 100644
--- a/docs/tutorial_dx_web.md
+++ b/docs/tutorial_dx_web.md
@@ -15,8 +15,8 @@ This document describes instruction for the item 2).
3. Move to one of the following workflow directories according to the platform you have chosen for your project (AWS or Azure). These DX workflows are pre-built with all parameters defined.
-* [AWS test workflow](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/test_ENCSR936XTK_subsampled_chr19_only)
-* [Azure test workflow](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/test_ENCSR936XTK_subsampled_chr19_only)
+* [AWS test workflow](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/test_ENCSR936XTK_subsampled_chr19_only)
+* [Azure test workflow](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/test_ENCSR936XTK_subsampled_chr19_only)
4. Copy it to your project by right-clicking on the DX workflow `chip` and choose "Copy".
@@ -40,16 +40,16 @@ This document describes instruction for the item 2).
1. DNAnexus allows only one copy of a workflow per project. The example workflow in the previous section is pre-built for the subsampled test sample [ENCSR936XTK](https://www.encodeproject.org/experiments/ENCSR936XTK/) with all parameters defined already.
2. Copy one of the following workflows according to the platform you have chosen for your project (AWS or Azure).
-* [AWS general](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/general) without pre-defined reference genome.
-* [AWS hg38](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/hg38) with pre-defined hg38 reference genome.
-* [AWS hg19](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/hg19) with pre-defined hg19 reference genome.
-* [AWS mm10](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/mm10) with pre-defined mm10 reference genome.
-* [AWS mm9](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.1.7/mm9) with pre-defined mm9 reference genome.
-* [Azure general](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/general) without pre-defined reference genome.
-* [Azure hg38](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/hg38) with pre-defined hg38 reference genome.
-* [Azure hg19](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/hg19) with pre-defined hg19 reference genome.
-* [Azure mm10](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/mm10) with pre-defined mm10 reference genome.
-* [Azure mm9](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.1.7/mm9) with pre-defined mm9 reference genome.
+* [AWS general](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/general) without pre-defined reference genome.
+* [AWS hg38](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/hg38) with pre-defined hg38 reference genome.
+* [AWS hg19](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/hg19) with pre-defined hg19 reference genome.
+* [AWS mm10](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/mm10) with pre-defined mm10 reference genome.
+* [AWS mm9](https://platform.DNAnexus.com/projects/BKpvFg00VBPV975PgJ6Q03v6/data/ChIP-seq2/workflows/v1.2.0/mm9) with pre-defined mm9 reference genome.
+* [Azure general](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/general) without pre-defined reference genome.
+* [Azure hg38](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/hg38) with pre-defined hg38 reference genome.
+* [Azure hg19](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/hg19) with pre-defined hg19 reference genome.
+* [Azure mm10](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/mm10) with pre-defined mm10 reference genome.
+* [Azure mm9](https://platform.DNAnexus.com/projects/F6K911Q9xyfgJ36JFzv03Z5J/data/ChIP-seq2/workflows/v1.2.0/mm9) with pre-defined mm9 reference genome.
3. Click on the DX workflow `chip`.
diff --git a/docs/tutorial_google.md b/docs/tutorial_google.md
index e596aac5..137a635e 100644
--- a/docs/tutorial_google.md
+++ b/docs/tutorial_google.md
@@ -32,8 +32,8 @@ All test samples and genome data are shared on our public Google Cloud buckets.
8. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
9. Git clone this pipeline and move into it.
@@ -49,7 +49,7 @@ All test samples and genome data are shared on our public Google Cloud buckets.
$ BUCKET=gs://[YOUR_BUCKET_NAME]/ENCSR936XTK_subsampled
$ INPUT=examples/google/ENCSR936XTK_subsampled_chr19_only.json
- $ java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=google -Dbackend.providers.google.config.project=${PROJECT} -Dbackend.providers.google.config.root=${BUCKET} cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/docker.json
+ $ java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=google -Dbackend.providers.google.config.project=${PROJECT} -Dbackend.providers.google.config.root=${BUCKET} cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/docker.json
```
11. It will take about 6 hours. You will be able to find all outputs on your Google Cloud bucket. Final QC report/JSON will be written on `gs://[YOUR_BUCKET_NAME]/ENCSR936XTK_subsampled/chip/[SOME_HASH_STRING]/call-qc_report/execution/glob*/qc.html` or `qc.json`. See [output directory structure](output.md) for details.
diff --git a/docs/tutorial_local_conda.md b/docs/tutorial_local_conda.md
index 35280454..e79c5cf5 100644
--- a/docs/tutorial_local_conda.md
+++ b/docs/tutorial_local_conda.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into it.
@@ -43,7 +43,7 @@
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/local/ENCSR936XTK_subsampled_chr19_only.json
$ PIPELINE_METADATA=metadata.json
- $ java -jar -Dconfig.file=backends/backend.conf cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
+ $ java -jar -Dconfig.file=backends/backend.conf cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
```
8. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
diff --git a/docs/tutorial_local_docker.md b/docs/tutorial_local_docker.md
index a46b412a..f46f768c 100644
--- a/docs/tutorial_local_docker.md
+++ b/docs/tutorial_local_docker.md
@@ -2,8 +2,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into it.
@@ -28,7 +28,7 @@
```bash
$ INPUT=examples/local/ENCSR936XTK_subsampled_chr19_only.json
$ PIPELINE_METADATA=metadata.json
- $ java -jar -Dconfig.file=backends/backend.conf cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/docker.json -m ${PIPELINE_METADATA}
+ $ java -jar -Dconfig.file=backends/backend.conf cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/docker.json -m ${PIPELINE_METADATA}
```
6. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
diff --git a/docs/tutorial_local_singularity.md b/docs/tutorial_local_singularity.md
index 272514d4..d7d350f0 100644
--- a/docs/tutorial_local_singularity.md
+++ b/docs/tutorial_local_singularity.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into it.
@@ -33,14 +33,14 @@
6. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
```
7. Run a pipeline for the test sample.
```bash
$ INPUT=examples/local/ENCSR936XTK_subsampled_chr19_only.json
$ PIPELINE_METADATA=metadata.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=singularity cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=singularity cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
```
8. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -53,7 +53,7 @@
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR2,..."
}
}
diff --git a/docs/tutorial_scg.md b/docs/tutorial_scg.md
index d8cf9207..f82cf10d 100644
--- a/docs/tutorial_scg.md
+++ b/docs/tutorial_scg.md
@@ -12,8 +12,8 @@ All test samples and genome data are shared on Stanford SCG4 cluster based on SL
2. Download [cromwell](https://github.com/broadinstitute/cromwell) on your `$HOME` directory.
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
3. Git clone this pipeline and move into its directory.
@@ -63,7 +63,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/reference/ENCODE,/scratch,/srv/gsfs0,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR1,..."
}
}
diff --git a/docs/tutorial_scg_backend.md b/docs/tutorial_scg_backend.md
index eb6b565b..ac2fb119 100644
--- a/docs/tutorial_scg_backend.md
+++ b/docs/tutorial_scg_backend.md
@@ -10,8 +10,8 @@ All test samples and genome data are shared on Stanford SCG cluster based on SLU
2. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
3. Git clone this pipeline and move into it.
@@ -46,7 +46,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
$ module load java miniconda/3
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/scg/ENCSR936XTK_subsampled_scg.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json
```
7. It will take about 2 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -58,7 +58,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
5. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
$ sdev # SCG cluster does not allow building a container on login node
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
$ exit
```
@@ -66,7 +66,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/scg/ENCSR936XTK_subsampled_scg.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json
```
7. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -77,7 +77,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/scratch/users,/srv/gsfs0,/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR1,..."
}
}
@@ -95,12 +95,12 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
For Conda users,
```bash
$ source activate encode-chip-seq-pipeline
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-38.jar server
```
For singularity users,
```bash
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-38.jar server
```
diff --git a/docs/tutorial_sge.md b/docs/tutorial_sge.md
index 31ab8ca7..f4bf458f 100644
--- a/docs/tutorial_sge.md
+++ b/docs/tutorial_sge.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell) on your `$HOME` directory.
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into its directory.
@@ -61,7 +61,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
7. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
```
8. Run a pipeline for the test sample. If your parallel environment (PE) found from step 5) has a different name from `shm` then edit the following shell script to change the PE name.
@@ -83,7 +83,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR2,..."
}
}
diff --git a/docs/tutorial_sge_backend.md b/docs/tutorial_sge_backend.md
index 82281420..eb35c562 100644
--- a/docs/tutorial_sge_backend.md
+++ b/docs/tutorial_sge_backend.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into it.
@@ -52,7 +52,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/local/ENCSR936XTK_subsampled.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=sge cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sge.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=sge cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sge.json
```
9. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -68,13 +68,13 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
7. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1
```
8. Run a pipeline for the test sample.
```bash
$ INPUT=examples/local/ENCSR936XTK_subsampled.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=sge_singularity cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sge.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=sge_singularity cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sge.json
```
9. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -102,11 +102,11 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
For Conda users,
```bash
$ source activate encode-chip-seq-pipeline
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=sge cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=sge cromwell-38.jar server
```
For singularity users,
```bash
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=sge_singularity cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=sge_singularity cromwell-38.jar server
```
2. You can modify `backend.providers.sge.concurrent-job-limit` or `backend.providers.sge_singularity.concurrent-job-limit` in `backends/backend.conf` to increase maximum concurrent jobs. This limit is **not per sample**. It's for all sub-tasks of all submitted samples.
diff --git a/docs/tutorial_sherlock.md b/docs/tutorial_sherlock.md
index 85d9567f..48f3bf07 100644
--- a/docs/tutorial_sherlock.md
+++ b/docs/tutorial_sherlock.md
@@ -12,8 +12,8 @@ All test samples and genome data are shared on Stanford Sherlock cluster based o
2. Download [cromwell](https://github.com/broadinstitute/cromwell) on your `$HOME` directory.
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
3. Git clone this pipeline and move into its directory.
@@ -68,7 +68,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/scratch,/lscratch,/oak/stanford,/home/groups/cherry/encode,/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR1,..."
}
}
diff --git a/docs/tutorial_sherlock_backend.md b/docs/tutorial_sherlock_backend.md
index a2fc5231..e8e09a21 100644
--- a/docs/tutorial_sherlock_backend.md
+++ b/docs/tutorial_sherlock_backend.md
@@ -10,8 +10,8 @@ All test samples and genome data are shared on Stanford Sherlock cluster. You do
2. Download [cromwell](https://github.com/broadinstitute/cromwell).
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
3. Git clone this pipeline and move into it.
@@ -46,7 +46,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/sherlock/ENCSR936XTK_subsampled_sherlock.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm -Dbackend.providers.slurm.config.concurrent-job-limit=1 cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm -Dbackend.providers.slurm.config.concurrent-job-limit=1 cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json
```
8. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -63,7 +63,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
6. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`. Stanford Sherlock does not allow building a container on login nodes. Wait until you get a command prompt after `sdev`.
```bash
$ sdev # sherlock cluster does not allow building a container on login node
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
$ exit # exit from an interactive node
```
@@ -71,7 +71,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/sherlock/ENCSR936XTK_subsampled_sherlock.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity -Dbackend.providers.slurm_singularity.config.concurrent-job-limit=1 cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity -Dbackend.providers.slurm_singularity.config.concurrent-job-limit=1 cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json
```
8. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -82,7 +82,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/scratch,/oak/stanford,/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR1,..."
}
}
@@ -100,11 +100,11 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
For Conda users,
```bash
$ source activate encode-chip-seq-pipeline
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-38.jar server
```
For singularity users,
```bash
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-38.jar server
```
2. You can modify `backend.providers.slurm.concurrent-job-limit` or `backend.providers.slurm_singularity.concurrent-job-limit` in `backends/backend.conf` to increase maximum concurrent jobs. This limit is **not per sample**. It's for all sub-tasks of all submitted samples.
diff --git a/docs/tutorial_slurm.md b/docs/tutorial_slurm.md
index 60b8a9d2..bf9ec12b 100644
--- a/docs/tutorial_slurm.md
+++ b/docs/tutorial_slurm.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell) on your `$HOME` directory.
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into its directory.
@@ -56,7 +56,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
7. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
```
8. Run a pipeline for the test sample. If your cluster requires to specify any of them then add one to the command line.
@@ -78,7 +78,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR2,..."
}
}
diff --git a/docs/tutorial_slurm_backend.md b/docs/tutorial_slurm_backend.md
index 4a26cdfd..fdd6557c 100644
--- a/docs/tutorial_slurm_backend.md
+++ b/docs/tutorial_slurm_backend.md
@@ -3,8 +3,8 @@
1. Download [cromwell](https://github.com/broadinstitute/cromwell) on your `$HOME` directory.
```bash
$ cd
- $ wget https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
- $ chmod +rx cromwell-34.jar
+ $ wget https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
+ $ chmod +rx cromwell-38.jar
```
2. Git clone this pipeline and move into it.
@@ -52,7 +52,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/local/ENCSR936XTK_subsampled.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/slurm.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/slurm.json
```
9. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -68,14 +68,14 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
7. Pull a singularity container for the pipeline. This will pull pipeline's docker container first and build a singularity one on `~/.singularity`.
```bash
- $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.1.7.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ $ mkdir -p ~/.singularity && cd ~/.singularity && SINGULARITY_CACHEDIR=~/.singularity SINGULARITY_PULLFOLDER=~/.singularity singularity pull --name chip-seq-pipeline-v1.2.0.simg -F docker://quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
```
8. Run a pipeline for the test sample.
```bash
$ source activate encode-chip-seq-pipeline # IMPORTANT!
$ INPUT=examples/local/ENCSR936XTK_subsampled.json
- $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/slurm.json
+ $ java -jar -Xmx1G -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/slurm.json
```
9. It will take about 6 hours. You will be able to find all outputs on `cromwell-executions/chip/[RANDOM_HASH_STRING]/`. See [output directory structure](output.md) for details.
@@ -86,7 +86,7 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
```javascript
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/your/,YOUR_OWN_DATA_DIR1,YOUR_OWN_DATA_DIR2,..."
}
}
@@ -104,11 +104,11 @@ Our pipeline supports both [Conda](https://conda.io/docs/) and [Singularity](htt
For Conda users,
```bash
$ source activate encode-chip-seq-pipeline
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm cromwell-38.jar server
```
For singularity users,
```bash
- $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-34.jar server
+ $ _JAVA_OPTIONS="-Xmx5G" java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=slurm_singularity cromwell-38.jar server
```
2. You can modify `backend.providers.slurm.concurrent-job-limit` or `backend.providers.slurm_singularity.concurrent-job-limit` in `backends/backend.conf` to increase maximum concurrent jobs. This limit is **not per sample**. It's for all sub-tasks of all submitted samples.
diff --git a/examples/caper/ENCSR936XTK_subsampled_chr19_only.json b/examples/caper/ENCSR936XTK_subsampled_chr19_only.json
new file mode 100644
index 00000000..3a61fa62
--- /dev/null
+++ b/examples/caper/ENCSR936XTK_subsampled_chr19_only.json
@@ -0,0 +1,26 @@
+{
+ "chip.pipeline_type" : "tf",
+ "chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM_caper.tsv",
+ "chip.fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"
+ ],
+
+ "chip.paired_end" : true,
+
+ "chip.always_use_pooled_ctl" : true,
+ "chip.title" : "ENCSR936XTK (subsampled 1/67, chr19 and chrM Only)",
+ "chip.description" : "ZNF143 ChIP-seq on human GM12878"
+}
diff --git a/examples/caper/ENCSR936XTK_subsampled_chr19_only.local.json b/examples/caper/ENCSR936XTK_subsampled_chr19_only.local.json
new file mode 100644
index 00000000..8672937e
--- /dev/null
+++ b/examples/caper/ENCSR936XTK_subsampled_chr19_only.local.json
@@ -0,0 +1,32 @@
+{
+ "chip.pipeline_type": "tf",
+ "chip.genome_tsv": "/srv/scratch/leepc12/cromweller_tmp_dir/4055223617984292143/hg38_chr19_chrM_caper.local.tsv",
+ "chip.fastqs_rep1_R1": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/8929619045197156503/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/5415527068319646597/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/-7034041647294296992/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/-382765533061561480/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/7558395090902605926/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/8591456472921953331/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/-547718864866627392/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2": [
+ "/srv/scratch/leepc12/cromweller_tmp_dir/-5954423518746074743/ctl2-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.paired_end": true,
+ "chip.always_use_pooled_ctl": true,
+ "chip.title": "ENCSR936XTK (subsampled 1/67, chr19 and chrM Only)",
+ "chip.description": "ZNF143 ChIP-seq on human GM12878"
+}
\ No newline at end of file
diff --git a/examples/dx/ENCSR000DYI_dx.json b/examples/dx/ENCSR000DYI_dx.json
index d640427a..05220eaa 100644
--- a/examples/dx/ENCSR000DYI_dx.json
+++ b/examples/dx/ENCSR000DYI_dx.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-genome-data/hg38_dx.tsv",
- "chip.fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx/ENCSR000DYI_subsampled_chr19_only_dx.json b/examples/dx/ENCSR000DYI_subsampled_chr19_only_dx.json
index fe39b590..11829311 100644
--- a/examples/dx/ENCSR000DYI_subsampled_chr19_only_dx.json
+++ b/examples/dx/ENCSR000DYI_subsampled_chr19_only_dx.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-genome-data/hg38_chr19_chrM_dx.tsv",
- "chip.fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx/ENCSR000DYI_subsampled_chr19_only_old_fraglen_dx.json b/examples/dx/ENCSR000DYI_subsampled_chr19_only_old_fraglen_dx.json
index 4d3bce6e..629ff785 100644
--- a/examples/dx/ENCSR000DYI_subsampled_chr19_only_old_fraglen_dx.json
+++ b/examples/dx/ENCSR000DYI_subsampled_chr19_only_old_fraglen_dx.json
@@ -1,14 +1,15 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-genome-data/hg38_chr19_chrM_dx.tsv",
- "chip.fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"
],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
+ ],
+
"chip.fraglen" : [95,105],
"chip.paired_end" : false,
diff --git a/examples/dx/ENCSR000DYI_subsampled_dx.json b/examples/dx/ENCSR000DYI_subsampled_dx.json
index f4af1231..61743b3a 100644
--- a/examples/dx/ENCSR000DYI_subsampled_dx.json
+++ b/examples/dx/ENCSR000DYI_subsampled_dx.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-genome-data/hg38_dx.tsv",
- "chip.fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx/ENCSR000DYI_subsampled_rep1_dx.json b/examples/dx/ENCSR000DYI_subsampled_rep1_dx.json
index 0da8fb52..753f1193 100644
--- a/examples/dx/ENCSR000DYI_subsampled_rep1_dx.json
+++ b/examples/dx/ENCSR000DYI_subsampled_rep1_dx.json
@@ -1,11 +1,9 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-genome-data/hg38_dx.tsv",
- "chip.fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]]
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-BKpvFg00VBPV975PgJ6Q03v6:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx_azure/ENCSR000DYI_dx_azure.json b/examples/dx_azure/ENCSR000DYI_dx_azure.json
index 1c567974..9113fe48 100644
--- a/examples/dx_azure/ENCSR000DYI_dx_azure.json
+++ b/examples/dx_azure/ENCSR000DYI_dx_azure.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-genome-data/hg38_dx_azure.tsv",
- "chip.fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx_azure/ENCSR000DYI_subsampled_chr19_only_dx_azure.json b/examples/dx_azure/ENCSR000DYI_subsampled_chr19_only_dx_azure.json
index d8bdabbb..1767227e 100644
--- a/examples/dx_azure/ENCSR000DYI_subsampled_chr19_only_dx_azure.json
+++ b/examples/dx_azure/ENCSR000DYI_subsampled_chr19_only_dx_azure.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-genome-data/hg38_chr19_chrM_dx_azure.tsv",
- "chip.fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/dx_azure/ENCSR000DYI_subsampled_dx_azure.json b/examples/dx_azure/ENCSR000DYI_subsampled_dx_azure.json
index 195d0ab2..365ee102 100644
--- a/examples/dx_azure/ENCSR000DYI_subsampled_dx_azure.json
+++ b/examples/dx_azure/ENCSR000DYI_subsampled_dx_azure.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-genome-data/hg38_dx_azure.tsv",
- "chip.fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["dx://project-F6K911Q9xyfgJ36JFzv03Z5J:pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/klab/ENCSR000DYI_klab.json b/examples/klab/ENCSR000DYI_klab.json
index b06e4937..98390c12 100644
--- a/examples/klab/ENCSR000DYI_klab.json
+++ b/examples/klab/ENCSR000DYI_klab.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38/hg38_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/klab/ENCSR000DYI_subsampled_chr19_only_keep_irregular_chr_klab.json b/examples/klab/ENCSR000DYI_subsampled_chr19_only_keep_irregular_chr_klab.json
index d2bcb5a3..5a872992 100644
--- a/examples/klab/ENCSR000DYI_subsampled_chr19_only_keep_irregular_chr_klab.json
+++ b/examples/klab/ENCSR000DYI_subsampled_chr19_only_keep_irregular_chr_klab.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38_chr19_chrM/hg38_chr19_chrM_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.20.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.20.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/klab/ENCSR000DYI_subsampled_chr19_only_klab.json b/examples/klab/ENCSR000DYI_subsampled_chr19_only_klab.json
index c3044fbb..5ab35351 100644
--- a/examples/klab/ENCSR000DYI_subsampled_chr19_only_klab.json
+++ b/examples/klab/ENCSR000DYI_subsampled_chr19_only_klab.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38_chr19_chrM/hg38_chr19_chrM_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.20.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.20.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/klab/ENCSR000DYI_subsampled_klab.json b/examples/klab/ENCSR000DYI_subsampled_klab.json
index 75eff2a7..3bf4a848 100644
--- a/examples/klab/ENCSR000DYI_subsampled_klab.json
+++ b/examples/klab/ENCSR000DYI_subsampled_klab.json
@@ -1,13 +1,13 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38/hg38_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.15.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.subsampled.15.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/examples/klab/ENCSR936XTK_klab.json b/examples/klab/ENCSR936XTK_klab.json
index d5e74b23..e53447f6 100644
--- a/examples/klab/ENCSR936XTK_klab.json
+++ b/examples/klab/ENCSR936XTK_klab.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38/hg38_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/klab/ENCSR936XTK_subsampled_chr19_only_klab.json b/examples/klab/ENCSR936XTK_subsampled_chr19_only_klab.json
index c5a363e1..6dfd5718 100644
--- a/examples/klab/ENCSR936XTK_subsampled_chr19_only_klab.json
+++ b/examples/klab/ENCSR936XTK_subsampled_chr19_only_klab.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38_chr19_chrM/hg38_chr19_chrM_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/klab/ENCSR936XTK_subsampled_klab.json b/examples/klab/ENCSR936XTK_subsampled_klab.json
index 4bb003bb..3e7d9c0d 100644
--- a/examples/klab/ENCSR936XTK_subsampled_klab.json
+++ b/examples/klab/ENCSR936XTK_subsampled_klab.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/mnt/data/pipeline_genome_data/hg38/hg38_klab.tsv",
- "chip.fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/mnt/data/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_conda.sh b/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_conda.sh
index f59acf2a..83384c82 100644
--- a/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_conda.sh
+++ b/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_conda.sh
@@ -66,4 +66,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_singularity.sh b/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_singularity.sh
index 6078933a..4df78341 100644
--- a/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_singularity.sh
+++ b/examples/local/ENCSR936XTK_subsampled_chr19_only_sge_singularity.sh
@@ -63,4 +63,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_conda.sh b/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_conda.sh
index 9993ea58..231d6108 100644
--- a/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_conda.sh
+++ b/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_conda.sh
@@ -62,4 +62,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_singularity.sh b/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_singularity.sh
index 221a2b74..4b4a578a 100644
--- a/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_singularity.sh
+++ b/examples/local/ENCSR936XTK_subsampled_chr19_only_slurm_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_sge_conda.sh b/examples/local/ENCSR936XTK_subsampled_sge_conda.sh
index 90f146a4..e625f36e 100644
--- a/examples/local/ENCSR936XTK_subsampled_sge_conda.sh
+++ b/examples/local/ENCSR936XTK_subsampled_sge_conda.sh
@@ -66,4 +66,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_sge_singularity.sh b/examples/local/ENCSR936XTK_subsampled_sge_singularity.sh
index e57163aa..50617ab6 100644
--- a/examples/local/ENCSR936XTK_subsampled_sge_singularity.sh
+++ b/examples/local/ENCSR936XTK_subsampled_sge_singularity.sh
@@ -63,4 +63,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_slurm_conda.sh b/examples/local/ENCSR936XTK_subsampled_slurm_conda.sh
index f44ca45a..e7bdf937 100644
--- a/examples/local/ENCSR936XTK_subsampled_slurm_conda.sh
+++ b/examples/local/ENCSR936XTK_subsampled_slurm_conda.sh
@@ -62,4 +62,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/local/ENCSR936XTK_subsampled_slurm_singularity.sh b/examples/local/ENCSR936XTK_subsampled_slurm_singularity.sh
index 542544f1..f0a163f1 100644
--- a/examples/local/ENCSR936XTK_subsampled_slurm_singularity.sh
+++ b/examples/local/ENCSR936XTK_subsampled_slurm_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/singularity.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg.json b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg.json
index 62229e7f..5c23ce80 100644
--- a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg.json
+++ b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/reference/ENCODE/pipeline_genome_data/hg38_chr19_chrM_scg.tsv",
- "chip.fastqs" : [
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_conda.sh b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_conda.sh
index d0656c1f..2eaae870 100644
--- a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_conda.sh
+++ b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_conda.sh
@@ -63,4 +63,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_singularity.sh b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_singularity.sh
index bb0775b5..28e5acd3 100644
--- a/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_singularity.sh
+++ b/examples/scg/ENCSR936XTK_subsampled_chr19_only_scg_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/scg/ENCSR936XTK_subsampled_scg.json b/examples/scg/ENCSR936XTK_subsampled_scg.json
index c08b2d5e..37fbeedf 100644
--- a/examples/scg/ENCSR936XTK_subsampled_scg.json
+++ b/examples/scg/ENCSR936XTK_subsampled_scg.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/reference/ENCODE/pipeline_genome_data/hg38_scg.tsv",
- "chip.fastqs" : [
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/reference/ENCODE/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/scg/ENCSR936XTK_subsampled_scg_conda.sh b/examples/scg/ENCSR936XTK_subsampled_scg_conda.sh
index 385dc219..2e5c335f 100644
--- a/examples/scg/ENCSR936XTK_subsampled_scg_conda.sh
+++ b/examples/scg/ENCSR936XTK_subsampled_scg_conda.sh
@@ -63,4 +63,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/scg/ENCSR936XTK_subsampled_scg_singularity.sh b/examples/scg/ENCSR936XTK_subsampled_scg_singularity.sh
index 09b1dce3..11c05254 100644
--- a/examples/scg/ENCSR936XTK_subsampled_scg_singularity.sh
+++ b/examples/scg/ENCSR936XTK_subsampled_scg_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/scg.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock.json b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock.json
index a91bfb45..52455d42 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock.json
+++ b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/home/groups/cherry/encode/pipeline_genome_data/hg38_chr19_chrM_sherlock.tsv",
- "chip.fastqs" : [
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_conda.sh b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_conda.sh
index 916cd3c9..a7bf3c7d 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_conda.sh
+++ b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_conda.sh
@@ -62,4 +62,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_singularity.sh b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_singularity.sh
index decb9da1..215a86d8 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_singularity.sh
+++ b/examples/sherlock/ENCSR936XTK_subsampled_chr19_only_sherlock_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_sherlock.json b/examples/sherlock/ENCSR936XTK_subsampled_sherlock.json
index d96b2831..7168e6bd 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_sherlock.json
+++ b/examples/sherlock/ENCSR936XTK_subsampled_sherlock.json
@@ -1,17 +1,21 @@
{
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "/home/groups/cherry/encode/pipeline_genome_data/hg38_sherlock.tsv",
- "chip.fastqs" : [
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"]],
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"]],
- [["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz",
- "/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["/home/groups/cherry/encode/pipeline_test_samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_sherlock_conda.sh b/examples/sherlock/ENCSR936XTK_subsampled_sherlock_conda.sh
index 6b68c1eb..a899d5cf 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_sherlock_conda.sh
+++ b/examples/sherlock/ENCSR936XTK_subsampled_sherlock_conda.sh
@@ -62,4 +62,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf \
-Dbackend.providers.Local.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/sherlock/ENCSR936XTK_subsampled_sherlock_singularity.sh b/examples/sherlock/ENCSR936XTK_subsampled_sherlock_singularity.sh
index c087c9eb..ebebcf82 100644
--- a/examples/sherlock/ENCSR936XTK_subsampled_sherlock_singularity.sh
+++ b/examples/sherlock/ENCSR936XTK_subsampled_sherlock_singularity.sh
@@ -59,4 +59,4 @@ NUM_CONCURRENT_TASK=2
# you can monitor your jobs with "squeue -u $USER"
java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=singularity \
-Dbackend.providers.singularity.config.concurrent-job-limit=${NUM_CONCURRENT_TASK} \
-$HOME/cromwell-34.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json -m ${PIPELINE_METADATA}
\ No newline at end of file
+$HOME/cromwell-38.jar run chip.wdl -i ${INPUT} -o workflow_opts/sherlock.json -m ${PIPELINE_METADATA}
\ No newline at end of file
diff --git a/examples/template_pe.json b/examples/template_pe.json
index 40db4e1c..7ada2c63 100644
--- a/examples/template_pe.json
+++ b/examples/template_pe.json
@@ -1,6 +1,7 @@
{
"chip.genome_tsv" : "/path_to_genome_data/hg38/hg38.tsv",
"chip.paired_end" : true,
+ "chip.ctl_paired_end" : true,
"chip.fastqs_rep1_R1" : [ "rep1_R1_L1.fastq.gz", "rep1_R1_L2.fastq.gz", "rep1_R1_L3.fastq.gz" ],
"chip.fastqs_rep1_R2" : [ "rep1_R2_L1.fastq.gz", "rep1_R2_L2.fastq.gz", "rep1_R2_L3.fastq.gz" ],
@@ -78,12 +79,12 @@
"chip.bwa_cpu" : 4,
"chip.bwa_mem_mb" : 20000,
"chip.bwa_time_hr" : 48,
- "chip.bwa_disks" : "local-disk 100 HDD",
+ "chip.bwa_disks" : "local-disk 200 HDD",
"chip.filter_cpu" : 2,
"chip.filter_mem_mb" : 20000,
"chip.filter_time_hr" : 24,
- "chip.filter_disks" : "local-disk 100 HDD",
+ "chip.filter_disks" : "local-disk 200 HDD",
"chip.bam2ta_cpu" : 2,
"chip.bam2ta_mem_mb" : 10000,
@@ -95,7 +96,7 @@
"chip.fingerprint_cpu" : 2,
"chip.fingerprint_mem_mb" : 12000,
"chip.fingerprint_time_hr" : 6,
- "chip.fingerprint_disks" : "local-disk 100 HDD",
+ "chip.fingerprint_disks" : "local-disk 200 HDD",
"chip.xcor_cpu" : 2,
"chip.xcor_mem_mb" : 16000,
@@ -104,10 +105,10 @@
"chip.macs2_mem_mb" : 16000,
"chip.macs2_time_hr" : 24,
- "chip.macs2_disks" : "local-disk 100 HDD",
+ "chip.macs2_disks" : "local-disk 200 HDD",
"chip.spp_cpu" : 2,
"chip.spp_mem_mb" : 16000,
"chip.spp_time_hr" : 72,
- "chip.spp_disks" : "local-disk 100 HDD"
+ "chip.spp_disks" : "local-disk 200 HDD"
}
\ No newline at end of file
diff --git a/examples/template_se.json b/examples/template_se.json
index e3d17194..a791600a 100644
--- a/examples/template_se.json
+++ b/examples/template_se.json
@@ -1,6 +1,7 @@
{
"chip.genome_tsv" : "/path_to_genome_data/hg38/hg38.tsv",
"chip.paired_end" : false,
+ "chip.ctl_paired_end" : true,
"chip.fastqs_rep1_R1" : [ "rep1_R1_L1.fastq.gz", "rep1_R1_L2.fastq.gz", "rep1_R1_L3.fastq.gz" ],
"chip.fastqs_rep2_R1" : [ "rep2_R1_L1.fastq.gz", "rep2_R1_L2.fastq.gz" ],
@@ -73,12 +74,12 @@
"chip.bwa_cpu" : 4,
"chip.bwa_mem_mb" : 20000,
"chip.bwa_time_hr" : 48,
- "chip.bwa_disks" : "local-disk 100 HDD",
+ "chip.bwa_disks" : "local-disk 200 HDD",
"chip.filter_cpu" : 2,
"chip.filter_mem_mb" : 20000,
"chip.filter_time_hr" : 24,
- "chip.filter_disks" : "local-disk 100 HDD",
+ "chip.filter_disks" : "local-disk 200 HDD",
"chip.bam2ta_cpu" : 2,
"chip.bam2ta_mem_mb" : 10000,
@@ -90,7 +91,7 @@
"chip.fingerprint_cpu" : 2,
"chip.fingerprint_mem_mb" : 12000,
"chip.fingerprint_time_hr" : 6,
- "chip.fingerprint_disks" : "local-disk 100 HDD",
+ "chip.fingerprint_disks" : "local-disk 200 HDD",
"chip.xcor_cpu" : 2,
"chip.xcor_mem_mb" : 16000,
@@ -99,10 +100,10 @@
"chip.macs2_mem_mb" : 16000,
"chip.macs2_time_hr" : 24,
- "chip.macs2_disks" : "local-disk 100 HDD",
+ "chip.macs2_disks" : "local-disk 200 HDD",
"chip.spp_cpu" : 2,
"chip.spp_mem_mb" : 16000,
"chip.spp_time_hr" : 72,
- "chip.spp_disks" : "local-disk 100 HDD"
+ "chip.spp_disks" : "local-disk 200 HDD"
}
\ No newline at end of file
diff --git a/genome/caper/hg19_caper.tsv b/genome/caper/hg19_caper.tsv
new file mode 100644
index 00000000..ec688ed0
--- /dev/null
+++ b/genome/caper/hg19_caper.tsv
@@ -0,0 +1,13 @@
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/hg19/male.hg19.fa
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg19/bowtie2_index/male.hg19.fa.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg19/bwa_index/male.hg19.fa.tar
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/hg19/hg19.chrom.sizes
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/hg19/wgEncodeDacMapabilityConsensusExcludable.bed.gz
+gensz hs
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/hg19_gencode_tss_unique.bed.gz
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.bed.gz
+prom https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/reg2map_honeybadger2_dnase_prom_p2.bed.gz
+enh https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/reg2map_honeybadger2_dnase_enh_p2.bed.gz
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/dnase_avgs_reg2map_p10_merged_named.pvals.gz
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/null
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/hg19/ataqc/eid_to_mnemonic.txt
\ No newline at end of file
diff --git a/genome/caper/hg38_caper.tsv b/genome/caper/hg38_caper.tsv
new file mode 100644
index 00000000..54321dbd
--- /dev/null
+++ b/genome/caper/hg38_caper.tsv
@@ -0,0 +1,13 @@
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/hg38/hg38.chrom.sizes
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/hg38/hg38.blacklist.bed.gz
+gensz hs
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_gencode_tss_unique.bed.gz
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.hg19_to_hg38.bed.gz
+prom https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_prom_p2.hg19_to_hg38.bed.gz
+enh https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_enh_p2.hg19_to_hg38.bed.gz
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt
\ No newline at end of file
diff --git a/genome/caper/hg38_chr19_chrM_caper.tsv b/genome/caper/hg38_chr19_chrM_caper.tsv
new file mode 100644
index 00000000..62082d93
--- /dev/null
+++ b/genome/caper/hg38_chr19_chrM_caper.tsv
@@ -0,0 +1,13 @@
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM/hg38.blacklist.bed.gz
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM/hg38_chr19_chrM.chrom.sizes
+gensz hs
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.chr19_chrM.fasta.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.chr19_chrM.fasta.tar
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/hg38_chr19_chrM/GRCh38_no_alt_analysis_set_GCA_000001405.15.chr19_chrM.fasta
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+prom https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+enh https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/null
diff --git a/genome/caper/mm10_caper.tsv b/genome/caper/mm10_caper.tsv
new file mode 100644
index 00000000..d52a27d1
--- /dev/null
+++ b/genome/caper/mm10_caper.tsv
@@ -0,0 +1,13 @@
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/mm10/mm10_no_alt_analysis_set_ENCODE.fasta.gz
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm10/bowtie2_index/mm10_no_alt_analysis_set_ENCODE.fasta.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm10/bwa_index/mm10_no_alt_analysis_set_ENCODE.fasta.tar
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/mm10/mm10.chrom.sizes
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/mm10/mm10.blacklist.bed.gz
+gensz mm
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_gencode_tss_unique.bed.gz
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_univ_dhs_ucsc.bed.gz
+prom https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/tss_mm10_master.bed.gz
+enh https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_enh_dhs_ucsc.bed.gz
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_dnase_avg_fseq_signal_formatted.txt.gz
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_celltype_compare_subsample.bed.gz
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/mm10_dnase_avg_fseq_signal_metadata.txt
\ No newline at end of file
diff --git a/genome/caper/mm10_chr19_chrM_caper.tsv b/genome/caper/mm10_chr19_chrM_caper.tsv
new file mode 100644
index 00000000..13cc9afa
--- /dev/null
+++ b/genome/caper/mm10_chr19_chrM_caper.tsv
@@ -0,0 +1,13 @@
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/mm10_chr19_chrM/mm10.blacklist.bed.gz
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/mm10_chr19_chrM/mm10_chr19_chrM.chrom.sizes
+gensz mm
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm10_chr19_chrM/bowtie2_index/mm10_no_alt_analysis_set_ENCODE.chr19_chrM.fasta.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm10_chr19_chrM/bwa_index/mm10_no_alt_analysis_set_ENCODE.chr19_chrM.fasta.tar
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/mm10_chr19_chrM/mm10_no_alt_analysis_set_ENCODE.chr19_chrM.fasta
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+prom https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+enh https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/mm10/ataqc/null
diff --git a/genome/caper/mm9_caper.tsv b/genome/caper/mm9_caper.tsv
new file mode 100644
index 00000000..f7a0fcfd
--- /dev/null
+++ b/genome/caper/mm9_caper.tsv
@@ -0,0 +1,13 @@
+ref_fa https://storage.googleapis.com/encode-pipeline-genome-data/mm9/mm9.fa
+bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm9/bowtie2_index/mm9.fa.tar
+bwa_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/mm9/bwa_index/mm9.fa.tar
+chrsz https://storage.googleapis.com/encode-pipeline-genome-data/mm9/mm9.chrom.sizes
+blacklist https://storage.googleapis.com/encode-pipeline-genome-data/mm9/mm9-blacklist.bed.gz
+gensz mm
+tss_enrich https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/mm9_gencode_tss_unique.bed.gz
+dnase https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/mm9_univ_dhs_ucsc.from_mm10.bed.gz
+prom https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/tss_mm9_master.from_mm10.bed.gz
+enh https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/mm9_enh_dhs_ucsc.from_mm10.bed.gz
+reg2map https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/dnase_avgs_merged_named.fseq.vals.gz
+reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/null
+roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/mm9/ataqc/accession_to_name.txt
\ No newline at end of file
diff --git a/genome/google/hg38_mint_google.tsv b/genome/google/hg38_mint_google.tsv
new file mode 100644
index 00000000..23d8e125
--- /dev/null
+++ b/genome/google/hg38_mint_google.tsv
@@ -0,0 +1,13 @@
+ref_fa gs://encode-pipeline-genome-data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
+bowtie2_idx_tar gs://encode-pipeline-genome-data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
+bwa_idx_tar gs://encode-pipeline-genome-data/hg38/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
+chrsz gs://encode-pipeline-genome-data/hg38/hg38.chrom.sizes
+blacklist gs://encode-pipeline-genome-data/hg38/GRCh38.combined_mint.blacklist.bed.gz
+gensz hs
+tss_enrich gs://encode-pipeline-genome-data/hg38/ataqc/hg38_gencode_tss_unique.bed.gz
+dnase gs://encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.hg19_to_hg38.bed.gz
+prom gs://encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_prom_p2.hg19_to_hg38.bed.gz
+enh gs://encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_enh_p2.hg19_to_hg38.bed.gz
+reg2map gs://encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz
+reg2map_bed gs://encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz
+roadmap_meta gs://encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt
\ No newline at end of file
diff --git a/src/encode_common_html.py b/src/encode_common_html.py
index 54684aee..4e4eb917 100755
--- a/src/encode_common_html.py
+++ b/src/encode_common_html.py
@@ -7,6 +7,7 @@
import re
from encode_common_log_parser import get_long_keyname
from encode_common import *
+from collections import OrderedDict
def html_heading(lvl, label):
html = '{label}\n'
@@ -37,50 +38,45 @@ def html_parse_body_from_file(f):
else:
return ''
-def html_vert_table_multi_rep(json_objs, paired_end=False, row_header=[]): # json_objs=list of OrderedDict
- if len(json_objs)==0: return ''
+def html_vert_table_multi_rep(json_objs, row_header=[]): # json_objs=list of OrderedDict
html = '
\n'
- # make row header list
- if row_header:
- row_header = [' '] + row_header
- else:
- row_header = [' '] + ['rep'+str(i+1)
- for i, json_obj in enumerate(json_objs)]
- # make column header list
- col_header = json_objs[0].keys()
+
+ valid_row_header = []
+ valid_json_objs = []
+ col_header_dict = OrderedDict()
+ for i, json_obj in enumerate(json_objs):
+ if not json_obj:
+ continue
+ if row_header:
+ valid_row_header.append(row_header[i])
+ else:
+ valid_row_header.append('rep'+str(i+1))
+ valid_json_objs.append(json_obj)
+ # find column header
+ col_header_dict.update(json_obj)
+
+ col_header = col_header_dict.keys()
+ if not col_header:
+ return ''
# make row header
- header = '| '+' | '.join(row_header)+' |
|---|
\n'
+ header = '| '+' | '.join([' ']+valid_row_header)+' |
|---|
\n'
# contents
content = ''
for row in col_header:
content += '| {} | {} |
|---|
\n'.format(
- get_long_keyname(row, paired_end),
+ get_long_keyname(row),
''.join(
- [str_float_4_dec_pts(json_obj[row]) for json_obj in json_objs]))
- return html.format(header=header, content=content)
-
-def html_horz_table(json_obj, paired_end=False):
- html = '
\n'
-
- # make row header
- header = ' | | '+\
- ' | '.join(
- [get_long_keyname(key, paired_end) for key in json_obj])+' |
|---|
\n'
- # contents
- content = '| '+\
- ' | '.join([str_float_4_dec_pts(json_obj[col]) for col in json_obj])+' |
\n'
+ [str_float_4_dec_pts(json_obj[row]) if row in json_obj else 'N/A'
+ for json_obj in valid_json_objs]))
return html.format(header=header, content=content)
-def html_help_filter(multimapping, paired_end):
+def html_help_filter(multimapping):
html = """
Filtered out (samtools view -F 1804):
- read unmapped (0x4)
- """
- if paired_end:
- html += "- mate unmapped (0x8)
"
- html += """
+ - mate unmapped (0x8, for paired-end)
"
- not primary alignment (0x100)
- read fails platform/vendor quality checks (0x200)
- read is PCR or optical duplicate (0x400)
diff --git a/src/encode_common_log_parser.py b/src/encode_common_log_parser.py
index c86a1d90..b7d10383 100755
--- a/src/encode_common_log_parser.py
+++ b/src/encode_common_log_parser.py
@@ -7,6 +7,7 @@
def parse_flagstat_qc(txt):
result = OrderedDict()
+ if not txt: return result
total = ''
total_qc_failed = ''
duplicates = ''
@@ -161,6 +162,7 @@ def parse_flagstat_qc(txt):
def parse_dup_qc(txt):
result = OrderedDict()
+ if not txt: return result
paired_reads = ''
unpaired_reads = ''
unmapped_reads = ''
@@ -250,6 +252,7 @@ def float_or_none(var):
def parse_pbc_qc(txt):
result = OrderedDict()
+ if not txt: return result
with open(txt, 'r') as f:
for line in f:
arr = line.strip().split('\t')
@@ -265,6 +268,7 @@ def parse_pbc_qc(txt):
def parse_xcor_score(txt):
result = OrderedDict()
+ if not txt: return result
with open(txt, 'r') as f:
arr = f.readlines()[0].strip().split('\t')
result['num_reads'] = int(arr[1])
@@ -280,6 +284,7 @@ def parse_xcor_score(txt):
def parse_jsd_qc(txt):
result = OrderedDict()
+ if not txt: return result
with open(txt, 'r') as f:
arr = f.readlines()[0].strip().split('\t')
result['pct_gen_enrich'] = float(arr[0])
@@ -296,6 +301,7 @@ def parse_jsd_qc(txt):
return result
def parse_reproducibility_qc(txt):
+ if not txt: return OrderedDict()
with open(txt, 'r') as f:
lines = f.readlines()
header = lines[0].strip()
@@ -311,6 +317,7 @@ def parse_reproducibility_qc(txt):
def parse_frip_qc(txt):
result = OrderedDict()
+ if not txt: return result
with open(txt, 'r') as f:
frip = f.readlines()[0].strip()
result['FRiP'] = float(frip)
@@ -318,6 +325,7 @@ def parse_frip_qc(txt):
def parse_multi_col_txt(txt):
result = OrderedDict()
+ if not txt: return result
with open(txt, 'r') as f:
lines = f.readlines()
for line in lines:
@@ -342,6 +350,7 @@ def num(s): # try to parse a number string (int->float->string)
return s.strip()
pre_parsed = parse_multi_col_txt(txt)
result = OrderedDict()
+ if not txt: return result
for key in pre_parsed:
val = pre_parsed[key]
if type(val)==list:
@@ -358,7 +367,7 @@ def num(s): # try to parse a number string (int->float->string)
result[key] = num(val)
return result
-def get_long_keyname(key, paired_end=False):
+def get_long_keyname(key): #, paired_end=False):
short_to_long = {
'total' : 'Total',
'total_qc_failed' : 'Total(QC-failed)',
@@ -421,30 +430,38 @@ def get_long_keyname(key, paired_end=False):
'syn_jsd' : 'Synthetic JS Distance',
'syn_x_intcpt' : 'Synthetic X-intercept',
'x_intcpt' : 'X-intercept',
- 'diff_enrich' : 'diff. enrichment'}
-
- short_to_long_se = {
- 'total_read_pairs' : 'Total Reads',
- 'distinct_read_pairs' : 'Distinct Reads',
- 'one_read_pair' : 'One Read',
- 'two_read_pair' : 'Two Reads',
- 'NRF' : 'NRF = Distinct/Total',
- 'PBC1' : 'PBC1 = OneRead/Distinct',
- 'PBC2' : 'PBC2 = OneRead/TwoReads',}
-
- short_to_long_pe = {
- 'total_read_pairs' : 'Total Read Pairs',
- 'distinct_read_pairs' : 'Distinct Read Pairs',
- 'one_read_pair' : 'One Read Pair',
- 'two_read_pair' : 'Two Read Pairs',
+ 'diff_enrich' : 'diff. enrichment',
+ 'total_read_pairs' : 'Total Reads (Pairs)',
+ 'distinct_read_pairs' : 'Distinct Reads (Pairs)',
+ 'one_read_pair' : 'One Read (Pair)',
+ 'two_read_pair' : 'Two Reads (Pairs)',
'NRF' : 'NRF = Distinct/Total',
'PBC1' : 'PBC1 = OnePair/Distinct',
- 'PBC2' : 'PBC2 = OnePair/TwoPair',}
+ 'PBC2' : 'PBC2 = OnePair/TwoPair'
+ }
+
+ # short_to_long_se = {
+ # 'total_read_pairs' : 'Total Reads',
+ # 'distinct_read_pairs' : 'Distinct Reads',
+ # 'one_read_pair' : 'One Read',
+ # 'two_read_pair' : 'Two Reads',
+ # 'NRF' : 'NRF = Distinct/Total',
+ # 'PBC1' : 'PBC1 = OneRead/Distinct',
+ # 'PBC2' : 'PBC2 = OneRead/TwoReads',}
+
+ # short_to_long_pe = {
+ # 'total_read_pairs' : 'Total Read Pairs',
+ # 'distinct_read_pairs' : 'Distinct Read Pairs',
+ # 'one_read_pair' : 'One Read Pair',
+ # 'two_read_pair' : 'Two Read Pairs',
+ # 'NRF' : 'NRF = Distinct/Total',
+ # 'PBC1' : 'PBC1 = OnePair/Distinct',
+ # 'PBC2' : 'PBC2 = OnePair/TwoPair',}
if key in short_to_long:
return short_to_long[key]
- if paired_end and key in short_to_long_pe:
- return short_to_long_pe[key]
- if not paired_end and key in short_to_long_se:
- return short_to_long_se[key]
+ # if paired_end and key in short_to_long_pe:
+ # return short_to_long_pe[key]
+ # if not paired_end and key in short_to_long_se:
+ # return short_to_long_se[key]
return key.replace('_',' ').capitalize()
diff --git a/src/encode_filter.py b/src/encode_filter.py
index 993f540e..440d8eac 100755
--- a/src/encode_filter.py
+++ b/src/encode_filter.py
@@ -317,21 +317,21 @@ def main():
args.bam, args.multimapping, args.mapq_thresh,
args.nth, args.out_dir)
+ log.info('Marking dupes with {}...'.format(args.dup_marker))
+ if args.dup_marker=='picard':
+ dupmark_bam, dup_qc = mark_dup_picard(
+ filt_bam, args.out_dir)
+ elif args.dup_marker=='sambamba':
+ dupmark_bam, dup_qc = mark_dup_sambamba(
+ filt_bam, args.nth, args.out_dir)
+ else:
+ raise argparse.ArgumentTypeError(
+ 'Unsupported --dup-marker {}'.format(args.dup_marker))
+
if args.no_dup_removal:
- nodup_bam = filt_bam
+ nodup_bam = filt_bam
else:
- log.info('Marking dupes with {}...'.format(args.dup_marker))
- if args.dup_marker=='picard':
- dupmark_bam, dup_qc = mark_dup_picard(
- filt_bam, args.out_dir)
- elif args.dup_marker=='sambamba':
- dupmark_bam, dup_qc = mark_dup_sambamba(
- filt_bam, args.nth, args.out_dir)
- else:
- raise argparse.ArgumentTypeError(
- 'Unsupported --dup-marker {}'.format(args.dup_marker))
temp_files.append(filt_bam)
-
log.info('Removing dupes...')
if args.paired_end:
nodup_bam = rm_dup_pe(
@@ -340,8 +340,8 @@ def main():
nodup_bam = rm_dup_se(
dupmark_bam, args.nth, args.out_dir)
samtools_index(dupmark_bam)
- temp_files.append(dupmark_bam)
temp_files.append(dupmark_bam+'.bai')
+ temp_files.append(dupmark_bam)
# initialize multithreading
log.info('Initializing multi-threading...')
@@ -349,12 +349,6 @@ def main():
log.info('Number of threads={}.'.format(num_process))
pool = multiprocessing.Pool(num_process)
- # log.info('samtools index...')
- # ret_val_1 = pool.apply_async(samtools_index,
- # (nodup_bam, args.out_dir))
- # log.info('samtools flagstat...')
- # ret_val_2 = pool.apply_async(samtools_flagstat,
- # (nodup_bam, args.out_dir))
log.info('sambamba index...')
ret_val_1 = pool.apply_async(sambamba_index,
(nodup_bam, args.nth, args.out_dir))
@@ -363,25 +357,23 @@ def main():
(nodup_bam, args.nth, args.out_dir))
log.info('Generating PBC QC log...')
- if not args.no_dup_removal:
- if args.paired_end:
- ret_val_3 = pool.apply_async(pbc_qc_pe,
- (dupmark_bam, args.mito_chr_name,
- max(1,args.nth-2),
- args.out_dir))
- else:
- ret_val_3 = pool.apply_async(pbc_qc_se,
- (dupmark_bam, args.mito_chr_name, args.out_dir))
-
+ if args.paired_end:
+ ret_val_3 = pool.apply_async(pbc_qc_pe,
+ (dupmark_bam, args.mito_chr_name,
+ max(1,args.nth-2),
+ args.out_dir))
+ else:
+ ret_val_3 = pool.apply_async(pbc_qc_se,
+ (dupmark_bam, args.mito_chr_name, args.out_dir))
+
# gather
nodup_bai = ret_val_1.get(BIG_INT)
nodup_flagstat_qc = ret_val_2.get(BIG_INT)
- if not args.no_dup_removal:
- pbc_qc = ret_val_3.get(BIG_INT)
+ pbc_qc = ret_val_3.get(BIG_INT)
- log.info('Making mito dup log...')
- mito_dup_log = make_mito_dup_log(dupmark_bam, args.out_dir)
+ log.info('Making mito dup log...')
+ mito_dup_log = make_mito_dup_log(dupmark_bam, args.out_dir)
log.info('Closing multi-threading...')
pool.close()
diff --git a/src/encode_macs2_chip.py b/src/encode_macs2_chip.py
index c33248f0..c2c0253e 100755
--- a/src/encode_macs2_chip.py
+++ b/src/encode_macs2_chip.py
@@ -29,8 +29,6 @@ def parse_arguments():
help='P-Value threshold.')
parser.add_argument('--cap-num-peak', default=500000, type=int,
help='Capping number of peaks by taking top N peaks.')
- parser.add_argument('--make-signal', action="store_true",
- help='Generate signal tracks for P-Value and fold enrichment.')
parser.add_argument('--blacklist', type=str, required=True,
help='Blacklist BED file.')
parser.add_argument('--keep-irregular-chr', action="store_true",
@@ -49,7 +47,7 @@ def parse_arguments():
return args
def macs2(ta, ctl_ta, chrsz, gensz, pval_thresh, shift, fraglen, cap_num_peak,
- make_signal, out_dir):
+ out_dir):
basename_ta = os.path.basename(strip_ext_ta(ta))
if ctl_ta:
basename_ctl_ta = os.path.basename(strip_ext_ta(ctl_ta))
@@ -100,91 +98,12 @@ def macs2(ta, ctl_ta, chrsz, gensz, pval_thresh, shift, fraglen, cap_num_peak,
npeak)
run_shell_cmd(cmd2)
rm_f(npeak_tmp)
-
- if make_signal:
- cmd3 = 'macs2 bdgcmp -t "{}"_treat_pileup.bdg '
- cmd3 += '-c "{}"_control_lambda.bdg '
- cmd3 += '--o-prefix "{}" -m FE '
- cmd3 = cmd3.format(
- prefix,
- prefix,
- prefix)
- run_shell_cmd(cmd3)
-
- cmd4 = 'bedtools slop -i "{}"_FE.bdg -g {} -b 0 | '
- cmd4 += 'awk \'{{if ($3 != -1) print $0}}\' |'
- cmd4 += 'bedClip stdin {} {}'
- cmd4 = cmd4.format(
- prefix,
- chrsz,
- chrsz,
- fc_bedgraph)
- run_shell_cmd(cmd4)
-
- # sort and remove any overlapping regions in bedgraph by comparing two lines in a row
- cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | ' \
- 'awk \'BEGIN{{OFS="\\t"}}{{if (NR==1 || NR>1 && (prev_chr!=$1 || prev_chr==$1 && prev_chr_e<=$2)) ' \
- '{{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'.format(
- fc_bedgraph,
- fc_bedgraph_srt)
- run_shell_cmd(cmd5)
-
- cmd6 = 'bedGraphToBigWig {} {} {}'
- cmd6 = cmd6.format(
- fc_bedgraph_srt,
- chrsz,
- fc_bigwig)
- run_shell_cmd(cmd6)
-
- # sval counts the number of tags per million in the (compressed) BED file
- sval = float(get_num_lines(ta))/1000000.0
-
- cmd7 = 'macs2 bdgcmp -t "{}"_treat_pileup.bdg '
- cmd7 += '-c "{}"_control_lambda.bdg '
- cmd7 += '--o-prefix {} -m ppois -S {}'
- cmd7 = cmd7.format(
- prefix,
- prefix,
- prefix,
- sval)
- run_shell_cmd(cmd7)
-
- cmd8 = 'bedtools slop -i "{}"_ppois.bdg -g {} -b 0 | '
- cmd8 += 'awk \'{{if ($3 != -1) print $0}}\' |'
- cmd8 += 'bedClip stdin {} {}'
- cmd8 = cmd8.format(
- prefix,
- chrsz,
- chrsz,
- pval_bedgraph)
- run_shell_cmd(cmd8)
-
- # sort and remove any overlapping regions in bedgraph by comparing two lines in a row
- cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | ' \
- 'awk \'BEGIN{{OFS="\\t"}}{{if (NR==1 || NR>1 && (prev_chr!=$1 || prev_chr==$1 && prev_chr_e<=$2)) ' \
- '{{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'.format(
- pval_bedgraph,
- pval_bedgraph_srt)
- run_shell_cmd(cmd9)
-
- cmd10 = 'bedGraphToBigWig {} {} {}'
- cmd10 = cmd10.format(
- pval_bedgraph_srt,
- chrsz,
- pval_bigwig)
- run_shell_cmd(cmd10)
- else:
- # make empty signal bigwigs (WDL wants it in output{})
- fc_bigwig = '/dev/null'
- pval_bigwig = '/dev/null'
-
+
# remove temporary files
- temp_files.extend([fc_bedgraph,fc_bedgraph_srt,
- pval_bedgraph,pval_bedgraph_srt])
temp_files.append("{}_*".format(prefix))
rm_f(temp_files)
- return npeak, fc_bigwig, pval_bigwig
+ return npeak
def main():
# read params
@@ -194,10 +113,9 @@ def main():
mkdir_p(args.out_dir)
log.info('Calling peaks and generating signal tracks with MACS2...')
- npeak, fc_bigwig, pval_bigwig = macs2(
+ npeak = macs2(
args.tas[0], args.tas[1], args.chrsz, args.gensz, args.pval_thresh,
- args.shift, args.fraglen, args.cap_num_peak, args.make_signal,
- args.out_dir)
+ args.shift, args.fraglen, args.cap_num_peak, args.out_dir)
log.info('Blacklist-filtering peaks...')
bfilt_npeak = blacklist_filter(
diff --git a/src/encode_macs2_signal_track_chip.py b/src/encode_macs2_signal_track_chip.py
new file mode 100755
index 00000000..6acfa2bf
--- /dev/null
+++ b/src/encode_macs2_signal_track_chip.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python
+
+# ENCODE DCC MACS2 signal track wrapper
+# Author: Jin Lee (leepc12@gmail.com)
+
+import sys
+import os
+import argparse
+from encode_common import *
+
+def parse_arguments():
+ parser = argparse.ArgumentParser(prog='ENCODE DCC MACS2 signal track',
+ description='')
+ parser.add_argument('tas', type=str, nargs='+',
+ help='Path for TAGALIGN file (first) and control TAGALIGN file (second; optional).')
+ parser.add_argument('--fraglen', type=int, required=True,
+ help='Fragment length.')
+ parser.add_argument('--shift', type=int, default=0,
+ help='macs2 callpeak --shift.')
+ parser.add_argument('--chrsz', type=str,
+ help='2-col chromosome sizes file.')
+ parser.add_argument('--gensz', type=str,
+ help='Genome size (sum of entries in 2nd column of \
+ chr. sizes file, or hs for human, ms for mouse).')
+ parser.add_argument('--pval-thresh', default=0.01, type=float,
+ help='P-Value threshold.')
+ parser.add_argument('--out-dir', default='', type=str,
+ help='Output directory.')
+ parser.add_argument('--log-level', default='INFO',
+ choices=['NOTSET','DEBUG','INFO',
+ 'WARNING','CRITICAL','ERROR','CRITICAL'],
+ help='Log level')
+ args = parser.parse_args()
+ if len(args.tas)==1:
+ args.tas.append('')
+ log.setLevel(args.log_level)
+ log.info(sys.argv)
+ return args
+
+def macs2_signal_track(ta, ctl_ta, chrsz, gensz, pval_thresh, shift, fraglen, out_dir):
+ basename_ta = os.path.basename(strip_ext_ta(ta))
+ if ctl_ta:
+ basename_ctl_ta = os.path.basename(strip_ext_ta(ctl_ta))
+ basename_prefix = '{}_x_{}'.format(basename_ta, basename_ctl_ta)
+ if len(basename_prefix) > 200: # UNIX cannot have len(filename) > 255
+ basename_prefix = '{}_x_control'.format(basename_ta)
+ else:
+ basename_prefix = basename_ta
+ prefix = os.path.join(out_dir, basename_prefix)
+ fc_bigwig = '{}.fc.signal.bigwig'.format(prefix)
+ pval_bigwig = '{}.pval.signal.bigwig'.format(prefix)
+ # temporary files
+ fc_bedgraph = '{}.fc.signal.bedgraph'.format(prefix)
+ fc_bedgraph_srt = '{}.fc.signal.srt.bedgraph'.format(prefix)
+ pval_bedgraph = '{}.pval.signal.bedgraph'.format(prefix)
+ pval_bedgraph_srt = '{}.pval.signal.srt.bedgraph'.format(prefix)
+
+ temp_files = []
+
+ cmd0 = ' macs2 callpeak '
+ cmd0 += '-t {} {} -f BED -n {} -g {} -p {} '
+ cmd0 += '--nomodel --shift {} --extsize {} --keep-dup all -B --SPMR'
+ cmd0 = cmd0.format(
+ ta,
+ '-c {}'.format(ctl_ta) if ctl_ta else '',
+ prefix,
+ gensz,
+ pval_thresh,
+ 0,
+ fraglen)
+ run_shell_cmd(cmd0)
+
+ cmd3 = 'macs2 bdgcmp -t "{}"_treat_pileup.bdg '
+ cmd3 += '-c "{}"_control_lambda.bdg '
+ cmd3 += '--o-prefix "{}" -m FE '
+ cmd3 = cmd3.format(
+ prefix,
+ prefix,
+ prefix)
+ run_shell_cmd(cmd3)
+
+ cmd4 = 'bedtools slop -i "{}"_FE.bdg -g {} -b 0 | '
+ cmd4 += 'awk \'{{if ($3 != -1) print $0}}\' |'
+ cmd4 += 'bedClip stdin {} {}'
+ cmd4 = cmd4.format(
+ prefix,
+ chrsz,
+ chrsz,
+ fc_bedgraph)
+ run_shell_cmd(cmd4)
+
+ # sort and remove any overlapping regions in bedgraph by comparing two lines in a row
+ cmd5 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | ' \
+ 'awk \'BEGIN{{OFS="\\t"}}{{if (NR==1 || NR>1 && (prev_chr!=$1 || prev_chr==$1 && prev_chr_e<=$2)) ' \
+ '{{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'.format(
+ fc_bedgraph,
+ fc_bedgraph_srt)
+ run_shell_cmd(cmd5)
+ rm_f(fc_bedgraph)
+
+ cmd6 = 'bedGraphToBigWig {} {} {}'
+ cmd6 = cmd6.format(
+ fc_bedgraph_srt,
+ chrsz,
+ fc_bigwig)
+ run_shell_cmd(cmd6)
+ rm_f(fc_bedgraph_srt)
+
+ # sval counts the number of tags per million in the (compressed) BED file
+ sval = float(get_num_lines(ta))/1000000.0
+
+ cmd7 = 'macs2 bdgcmp -t "{}"_treat_pileup.bdg '
+ cmd7 += '-c "{}"_control_lambda.bdg '
+ cmd7 += '--o-prefix {} -m ppois -S {}'
+ cmd7 = cmd7.format(
+ prefix,
+ prefix,
+ prefix,
+ sval)
+ run_shell_cmd(cmd7)
+
+ cmd8 = 'bedtools slop -i "{}"_ppois.bdg -g {} -b 0 | '
+ cmd8 += 'awk \'{{if ($3 != -1) print $0}}\' |'
+ cmd8 += 'bedClip stdin {} {}'
+ cmd8 = cmd8.format(
+ prefix,
+ chrsz,
+ chrsz,
+ pval_bedgraph)
+ run_shell_cmd(cmd8)
+
+ # sort and remove any overlapping regions in bedgraph by comparing two lines in a row
+ cmd9 = 'LC_COLLATE=C sort -k1,1 -k2,2n {} | ' \
+ 'awk \'BEGIN{{OFS="\\t"}}{{if (NR==1 || NR>1 && (prev_chr!=$1 || prev_chr==$1 && prev_chr_e<=$2)) ' \
+ '{{print $0}}; prev_chr=$1; prev_chr_e=$3;}}\' > {}'.format(
+ pval_bedgraph,
+ pval_bedgraph_srt)
+ run_shell_cmd(cmd9)
+ rm_f(pval_bedgraph)
+
+ cmd10 = 'bedGraphToBigWig {} {} {}'
+ cmd10 = cmd10.format(
+ pval_bedgraph_srt,
+ chrsz,
+ pval_bigwig)
+ run_shell_cmd(cmd10)
+ rm_f(pval_bedgraph_srt)
+
+ # remove temporary files
+ temp_files.append("{}_*".format(prefix))
+ rm_f(temp_files)
+
+ return fc_bigwig, pval_bigwig
+
+def main():
+ # read params
+ args = parse_arguments()
+
+ log.info('Initializing and making output directory...')
+ mkdir_p(args.out_dir)
+
+ log.info('Calling peaks and generating signal tracks with MACS2...')
+ fc_bigwig, pval_bigwig = macs2_signal_track(
+ args.tas[0], args.tas[1], args.chrsz, args.gensz, args.pval_thresh,
+ args.shift, args.fraglen, args.out_dir)
+
+ log.info('List all files in output directory...')
+ ls_l(args.out_dir)
+
+ log.info('All done.')
+
+if __name__=='__main__':
+ main()
\ No newline at end of file
diff --git a/src/encode_qc_report.py b/src/encode_qc_report.py
index 6607070d..bb00e99b 100755
--- a/src/encode_qc_report.py
+++ b/src/encode_qc_report.py
@@ -53,6 +53,22 @@
'''
+def split_entries_and_extend(l, delim='_:_'):
+ result = []
+ for a in l:
+ if type(a)==str:
+ result.extend(a.split(delim))
+ else:
+ result.append(a)
+ test_not_all_empty = [r for r in result if r]
+ # if all empty then return []
+ return result if test_not_all_empty else []
+
+def str2bool(s):
+ if s not in ['false', 'true', 'False', 'True']:
+ raise ValueError('Not a valid boolean string')
+ return s=='True' or s=='true'
+
def parse_arguments():
parser = argparse.ArgumentParser(
prog='ENCODE DCC generate HTML report and QC JSON.',
@@ -67,8 +83,14 @@ def parse_arguments():
help='Pipeline version.')
parser.add_argument('--multimapping', default=0, type=int,
help='Multimapping reads.')
- parser.add_argument('--paired-end', action="store_true",
- help='Paired-end sample.')
+ parser.add_argument('--paired-end', action='store_true',
+ help='Endedness of all replicates.')
+ parser.add_argument('--ctl-paired-end', action='store_true',
+ help='Endedness of all controls.')
+ parser.add_argument('--paired-ends', type=str2bool, nargs='*',
+ help='List of true/false for paired endedness of sample.')
+ parser.add_argument('--ctl-paired-ends', type=str2bool, nargs='*',
+ help='List of true/false for paired endedness of control.')
parser.add_argument('--pipeline-type', type=str, required=True,
help='Pipeline type.')
parser.add_argument('--peak-caller', type=str, required=True,
@@ -170,6 +192,23 @@ def parse_arguments():
help='Log level')
args = parser.parse_args()
+ # _:_
+ for a in vars(args):
+ if type(eval('args.{}'.format(a)))==list:
+ exec('args.{} = split_entries_and_extend(args.{})'.format(a,a))
+
+ if args.paired_ends is None:
+ if args.paired_end:
+ args.paired_ends = [True]*20
+ else:
+ args.paired_ends = [False]*20
+
+ if args.ctl_paired_ends is None:
+ if args.ctl_paired_end:
+ args.ctl_paired_ends = [True]*20
+ else:
+ args.ctl_paired_ends = args.paired_ends
+
log.setLevel(args.log_level)
log.info(sys.argv)
return args
@@ -185,11 +224,29 @@ def get_full_name(pipeline_type):
return 'Histone ChIP-Seq'
return pipeline_type
-def get_rep_labels(arr):
- return (['rep'+str(i+1) for i, a in enumerate(arr)])
-
-def get_ctl_labels(arr):
- return (['ctl'+str(i+1) for i, a in enumerate(arr)])
+def get_rep_labels(arr, paired_ends=[]):
+ result = []
+ for i, a in enumerate(arr):
+ s = 'rep'+str(i+1)
+ if paired_ends:
+ if paired_ends[i]:
+ s+= ' (PE)'
+ else:
+ s+= ' (SE)'
+ result.append(s)
+ return result
+
+def get_ctl_labels(arr, paired_ends=[]):
+ result = []
+ for i, a in enumerate(arr):
+ s = 'ctl'+str(i+1)
+ if paired_ends:
+ if paired_ends[i]:
+ s+= ' (PE)'
+ else:
+ s+= ' (SE)'
+ result.append(s)
+ return result
def main():
# read params
@@ -206,13 +263,15 @@ def main():
json_all['general']['genome'] = args.genome
json_all['general']['description'] = args.desc
json_all['general']['title'] = args.title
- json_all['general']['paired_end'] = args.paired_end
+ json_all['general']['paired_end'] = args.paired_ends
+ if args.ctl_paired_ends:
+ json_all['general']['ctl_paired_end'] = args.ctl_paired_ends
html += html_heading(1, args.title)
html += html_paragraph(args.desc)
html += html_paragraph('Pipeline version: {}'.format(args.pipeline_ver))
html += html_paragraph('Report generated at {}'.format(now()))
- html += html_paragraph('Paired-end: {}'.format(args.paired_end))
+ html += html_paragraph('Paired-end: {}'.format(args.paired_ends))
html += html_paragraph('Pipeline type: {}'.format(get_full_name(args.pipeline_type)))
html += html_paragraph('Genome: {}'.format(args.genome))
html += html_paragraph('Peak caller: {}'.format(args.peak_caller.upper()))
@@ -234,19 +293,19 @@ def main():
json_objs = [parse_flagstat_qc(qc) for qc in args.flagstat_qcs]
json_all['flagstat_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_rep_labels(json_objs))
+ row_header.extend(get_rep_labels(json_objs, args.paired_ends))
if args.ctl_flagstat_qcs:
json_objs = [parse_flagstat_qc(qc) for qc in args.ctl_flagstat_qcs]
json_all['ctl_flagstat_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_ctl_labels(json_objs))
+ row_header.extend(get_ctl_labels(json_objs, args.ctl_paired_ends))
- html_align += html_vert_table_multi_rep(json_objs_all, args.paired_end, row_header)
+ html_align += html_vert_table_multi_rep(json_objs_all, row_header)
if args.dup_qcs and get_num_lines(args.dup_qcs[0]) \
or args.ctl_dup_qcs and get_num_lines(args.ctl_dup_qcs[0]):
html_align += html_heading(2, 'Marking duplicates (filtered BAM)')
- html_align += html_help_filter(args.multimapping, args.paired_end)
+ html_align += html_help_filter(args.multimapping)
# check if file is empty (when filter.no_dup_removal is on)
# if empty then skip
json_objs_all = []
@@ -255,13 +314,13 @@ def main():
json_objs = [parse_dup_qc(qc) for qc in args.dup_qcs]
json_all['dup_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_rep_labels(json_objs))
+ row_header.extend(get_rep_labels(json_objs, args.paired_ends))
if args.ctl_dup_qcs and get_num_lines(args.ctl_dup_qcs[0]):
json_objs = [parse_dup_qc(qc) for qc in args.ctl_dup_qcs]
json_all['ctl_dup_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_ctl_labels(json_objs))
- html_align += html_vert_table_multi_rep(json_objs_all, args.paired_end, row_header)
+ row_header.extend(get_ctl_labels(json_objs, args.ctl_paired_ends))
+ html_align += html_vert_table_multi_rep(json_objs_all, row_header)
if args.pbc_qcs and get_num_lines(args.pbc_qcs[0]) \
or args.ctl_pbc_qcs and get_num_lines(args.ctl_pbc_qcs[0]):
@@ -274,13 +333,13 @@ def main():
json_objs = [parse_pbc_qc(qc) for qc in args.pbc_qcs]
json_all['pbc_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_rep_labels(json_objs))
+ row_header.extend(get_rep_labels(json_objs, args.paired_ends))
if args.ctl_pbc_qcs and get_num_lines(args.ctl_pbc_qcs[0]):
json_objs = [parse_pbc_qc(qc) for qc in args.ctl_pbc_qcs]
json_all['ctl_pbc_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_ctl_labels(json_objs))
- html_align += html_vert_table_multi_rep(json_objs_all, args.paired_end, row_header)
+ row_header.extend(get_ctl_labels(json_objs, args.ctl_paired_ends))
+ html_align += html_vert_table_multi_rep(json_objs_all, row_header)
html_align += html_help_pbc()
if args.nodup_flagstat_qcs or args.ctl_nodup_flagstat_qcs:
@@ -292,13 +351,13 @@ def main():
json_objs = [parse_flagstat_qc(qc) for qc in args.nodup_flagstat_qcs]
json_all['nodup_flagstat_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_rep_labels(json_objs))
+ row_header.extend(get_rep_labels(json_objs, args.paired_ends))
if args.ctl_nodup_flagstat_qcs:
json_objs = [parse_flagstat_qc(qc) for qc in args.ctl_nodup_flagstat_qcs]
json_all['ctl_nodup_flagstat_qc'] = json_objs
json_objs_all.extend(json_objs)
- row_header.extend(get_ctl_labels(json_objs))
- html_align += html_vert_table_multi_rep(json_objs_all, args.paired_end, row_header)
+ row_header.extend(get_ctl_labels(json_objs, args.ctl_paired_ends))
+ html_align += html_vert_table_multi_rep(json_objs_all, row_header)
# IDR plots
if args.idr_plots or args.idr_plots_pr or args.idr_plot_ppr:
@@ -308,11 +367,13 @@ def main():
num_rep = infer_n_from_nC2(len(args.idr_plots))
# embed PNGs into HTML
for i, png in enumerate(args.idr_plots):
- pair_label = infer_pair_label_from_idx(num_rep, i)
- html_peakcall += html_embedded_png(png, pair_label)
+ if png:
+ pair_label = infer_pair_label_from_idx(num_rep, i)
+ html_peakcall += html_embedded_png(png, pair_label)
if args.idr_plots_pr:
for i, png in enumerate(args.idr_plots_pr):
- html_peakcall += html_embedded_png(png, 'rep{}-pr'.format(i+1))
+ if png:
+ html_peakcall += html_embedded_png(png, 'rep{}-pr'.format(i+1))
if args.idr_plot_ppr:
png = args.idr_plot_ppr[0]
html_peakcall += html_embedded_png(png, 'ppr')
@@ -340,7 +401,6 @@ def main():
row_header_reproducibility_qc.append('IDR')
html_peakcall += html_vert_table_multi_rep(
json_objs_reproducibility_qc,
- args.paired_end,
row_header_reproducibility_qc)
if args.overlap_reproducibility_qc:
html_peakcall += html_help_overlap()
@@ -350,13 +410,18 @@ def main():
if args.xcor_plots and args.xcor_scores:
html_enrich += html_heading(2, 'Strand cross-correlation measures')
json_objs = [parse_xcor_score(qc) for qc in args.xcor_scores]
+ num_reads = 0 # json_objs[0]['num_reads']
+ for _, j in enumerate(json_objs):
+ if j:
+ num_reads = j['num_reads']
html_enrich += html_paragraph('Performed on subsampled reads ({})'.format(
- human_readable_number(json_objs[0]['num_reads'])))
+ human_readable_number(num_reads)))
- html_enrich += html_vert_table_multi_rep(json_objs, args.paired_end)
+ html_enrich += html_vert_table_multi_rep(json_objs)
html_enrich += html_help_xcor()
for i, xcor_plot in enumerate(args.xcor_plots):
- html_enrich += html_embedded_png(xcor_plot, 'rep{}'.format(i+1), 60)
+ if xcor_plot:
+ html_enrich += html_embedded_png(xcor_plot, 'rep{}'.format(i+1), 60)
json_all['xcor_score'] = json_objs
# frip (Enrichment QC) for MACS2 raw peaks
@@ -397,7 +462,7 @@ def main():
row_header_frip.append('ppr2')
json_all['frip_macs2_qc'] = OrderedDict(
zip(row_header_frip,json_objs_frip))
- # html_enrich += html_vert_table_multi_rep(json_objs_frip,args.paired_end,row_header_frip)
+ # html_enrich += html_vert_table_multi_rep(json_objs_frip,row_header_frip)
# html_enrich += html_help_FRiP(args.peak_caller)
# frip (Enrichment QC) for SPP raw peaks
@@ -438,7 +503,7 @@ def main():
row_header_frip.append('ppr2')
json_all['frip_spp_qc'] = OrderedDict(
zip(row_header_frip,json_objs_frip))
- # html_enrich += html_vert_table_multi_rep(json_objs_frip,args.paired_end,row_header_frip)
+ # html_enrich += html_vert_table_multi_rep(json_objs_frip,row_header_frip)
# html_enrich += html_help_FRiP(args.peak_caller)
# frip (Enrichment QC) for overlap
@@ -465,7 +530,6 @@ def main():
json_all['overlap_frip_qc'] = OrderedDict(zip(row_header_frip_overlap,json_objs_frip_overlap))
html_enrich += html_vert_table_multi_rep(
json_objs_frip_overlap,
- args.paired_end,
row_header_frip_overlap)
html_enrich += html_help_overlap_FRiP()
@@ -493,14 +557,13 @@ def main():
json_all['idr_frip_qc'] = OrderedDict(zip(row_header_frip_idr,json_objs_frip_idr))
html_enrich += html_vert_table_multi_rep(
json_objs_frip_idr,
- args.paired_end,
row_header_frip_idr)
html_enrich += html_help_idr_FRiP()
if args.jsd_plot:
html_etc += html_heading(2, 'Fingerprint and Jensen-Shannon distance')
json_objs = [parse_jsd_qc(qc) for qc in args.jsd_qcs]
- html_etc += html_vert_table_multi_rep(json_objs, args.paired_end)
+ html_etc += html_vert_table_multi_rep(json_objs)
html_etc += html_embedded_png(args.jsd_plot[0], '', 40)
json_all['jsd_qc'] = json_objs
@@ -511,9 +574,10 @@ def main():
html_ataqc += html_heading(2, 'Summary table')
html_ataqc += html_vert_table_multi_rep(json_objs)
for i, ataqc_html in enumerate(args.ataqc_htmls):
- html_ataqc += html_heading(2, 'Replicate {}'.format(i+1))
- html_ataqc += html_parse_body_from_file(ataqc_html).replace('ATAqC
','')
- html_ataqc += '\n
'
+ if ataqc_html:
+ html_ataqc += html_heading(2, 'Replicate {}'.format(i+1))
+ html_ataqc += html_parse_body_from_file(ataqc_html).replace('ATAqC
','')
+ html_ataqc += '\n
'
if html_align:
html_align = html_heading(1, 'Alignment')+'
'+html_align
diff --git a/src/encode_xcor.py b/src/encode_xcor.py
index 7f98adc8..6e0a8201 100755
--- a/src/encode_xcor.py
+++ b/src/encode_xcor.py
@@ -21,6 +21,14 @@ def parse_arguments():
parser.add_argument('--speak', type=int, default=-1,
help='User-defined cross-corr. peak strandshift \
(-speak= in run_spp.R). Disabled if -1.')
+ parser.add_argument('--exclusion-range-min', type=int,
+ help='User-defined exclusion range minimum used for '
+ '-x=${xcor_exclusion_range_min}:${xcor_exclusion_range_max}')
+ parser.add_argument('--exclusion-range-max', type=int,
+ help='User-defined exclusion range maximum used for '
+ '-x=${xcor_exclusion_range_min}:${xcor_exclusion_range_max}')
+ parser.add_argument('--chip-seq-type', choices=['tf', 'histone'],
+ help='Type of ChIP-seq pipeline (histone of tf)')
parser.add_argument('--paired-end', action="store_true",
help='Paired-end TAGALIGN.')
parser.add_argument('--nth', type=int, default=1,
@@ -37,15 +45,48 @@ def parse_arguments():
log.info(sys.argv)
return args
-def xcor(ta, speak, mito_chr_name, nth, out_dir):
+
+def get_exclusion_range_max(ta, chip_seq_type):
+ # estimate read length from TA
+ cmd0 = "zcat -f {} > {}.tmp".format(ta, ta)
+ run_shell_cmd(cmd0)
+
+ cmd = "head -n 100 {}.tmp | awk 'function abs(v) "
+ cmd += "{{return v < 0 ? -v : v}} BEGIN{{sum=0}} "
+ cmd += "{{sum+=abs($3-$2)}} END{{print int(sum/NR)}}'"
+ cmd = cmd.format(ta)
+ read_len = int(run_shell_cmd(cmd))
+ rm_f(ta+'.tmp')
+
+ if chip_seq_type == 'tf':
+ return max(read_len + 10, 50)
+ elif chip_seq_type == 'histone':
+ return max(read_len + 10, 100)
+ else:
+ raise NotImplementedError('chip_seq_type not supported')
+
+
+def xcor(ta, speak, mito_chr_name,
+ nth, out_dir, chip_seq_type=None,
+ exclusion_range_min=None, exclusion_range_max=None):
prefix = os.path.join(out_dir,
os.path.basename(strip_ext_ta(ta)))
xcor_plot_pdf = '{}.cc.plot.pdf'.format(prefix)
xcor_score = '{}.cc.qc'.format(prefix)
fraglen_txt = '{}.cc.fraglen.txt'.format(prefix)
+ if chip_seq_type is not None and exclusion_range_min is not None:
+ if exclusion_range_max is None:
+ exclusion_range_max = get_exclusion_range_max(ta, chip_seq_type)
+
+ exclusion_range_param = ' -x={}:{}'.format(
+ exclusion_range_min, exclusion_range_max)
+ else:
+ exclusion_range_param = ''
+
cmd1 = 'Rscript --max-ppsize=500000 $(which run_spp.R) -rf -c={} -p={} '
cmd1 += '-filtchr="{}" -savp={} -out={} {}'
+ cmd1 += exclusion_range_param
cmd1 = cmd1.format(
ta,
nth,
@@ -88,7 +129,8 @@ def main():
log.info('Cross-correlation analysis...')
xcor_plot_pdf, xcor_plot_png, xcor_score, fraglen_txt = xcor(
- ta_subsampled, args.speak, args.mito_chr_name, args.nth, args.out_dir)
+ ta_subsampled, args.speak, args.mito_chr_name, args.nth, args.out_dir,
+ args.chip_seq_type, args.exclusion_range_min, args.exclusion_range_max)
log.info('Removing temporary files...')
rm_f(temp_files)
diff --git a/test/test_task/test.sh b/test/test_task/test.sh
index 02a28e4d..9de014d9 100755
--- a/test/test_task/test.sh
+++ b/test/test_task/test.sh
@@ -12,7 +12,7 @@ INPUT=$2
if [ $# -gt 2 ]; then
DOCKER_IMAGE=$3
else
- DOCKER_IMAGE=quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ DOCKER_IMAGE=quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
fi
if [ $# -gt 3 ]; then
NUM_TASK=$4
@@ -20,12 +20,12 @@ else
NUM_TASK=1
fi
-if [ -f "cromwell-34.jar" ]; then
+if [ -f "cromwell-38.jar" ]; then
echo "Skip downloading cromwell."
else
- wget -N -c https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
+ wget -N -c https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
fi
-CROMWELL_JAR=cromwell-34.jar
+CROMWELL_JAR=cromwell-38.jar
BACKEND_CONF=../../backends/backend.conf
BACKEND=Local
EXTRA_PARAM="-Dbackend.providers.Local.config.concurrent-job-limit=2"
diff --git a/test/test_task/test_macs2.json b/test/test_task/test_macs2.json
index f188ab2d..6b03e643 100644
--- a/test/test_task/test_macs2.json
+++ b/test/test_task/test_macs2.json
@@ -11,8 +11,6 @@
"test_macs2.ref_se_macs2_npeak" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.pval0.01.500K.narrowPeak.gz",
"test_macs2.ref_se_macs2_bfilt_npeak" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.pval0.01.500K.bfilt.narrowPeak.gz",
"test_macs2.ref_se_macs2_frip_qc" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.pval0.01.500K.bfilt.frip.qc",
- "test_macs2.ref_se_macs2_sig_pval" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.pval.signal.bigwig",
- "test_macs2.ref_se_macs2_sig_fc" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.fc.signal.bigwig",
"test_macs2.cap_num_peak" : 500000,
"test_macs2.pval_thresh" : 0.01
diff --git a/test/test_task/test_macs2.wdl b/test/test_task/test_macs2.wdl
index a4982bfe..c2f3000b 100644
--- a/test/test_task/test_macs2.wdl
+++ b/test/test_task/test_macs2.wdl
@@ -14,7 +14,6 @@ workflow test_macs2 {
String ref_se_macs2_npeak # raw narrow-peak
String ref_se_macs2_bfilt_npeak # blacklist filtered narrow-peak
String ref_se_macs2_frip_qc
- String ref_se_macs2_sig_pval # p-val signal
String se_blacklist
String se_chrsz
@@ -31,7 +30,6 @@ workflow test_macs2 {
fraglen = fraglen,
cap_num_peak = cap_num_peak,
pval_thresh = pval_thresh,
- make_signal = true,
blacklist = se_blacklist,
keep_irregular_chr_in_bfilt_peak = false,
@@ -45,19 +43,16 @@ workflow test_macs2 {
'se_macs2_npeak',
'se_macs2_bfilt_npeak',
'se_macs2_frip_qc',
- 'se_macs2_sig_pval',
],
files = [
se_macs2.npeak,
se_macs2.bfilt_npeak,
se_macs2.frip_qc,
- se_macs2.sig_pval,
],
ref_files = [
ref_se_macs2_npeak,
ref_se_macs2_bfilt_npeak,
ref_se_macs2_frip_qc,
- ref_se_macs2_sig_pval,
],
}
}
diff --git a/test/test_task/test_macs2_signal_track.json b/test/test_task/test_macs2_signal_track.json
new file mode 100644
index 00000000..eef1e929
--- /dev/null
+++ b/test/test_task/test_macs2_signal_track.json
@@ -0,0 +1,13 @@
+{
+ "test_macs2_signal_track.se_chrsz" : "chip-seq-pipeline-test-data/genome_data/hg38_chr19_chrM/hg38_chr19_chrM.chrom.sizes",
+ "test_macs2_signal_track.se_gensz" : "hs",
+
+ "test_macs2_signal_track.se_ta" : "chip-seq-pipeline-test-data/input/se/tas/rep1/rep1.subsampled.25.merged.nodup.tagAlign.gz",
+ "test_macs2_signal_track.se_ctl_ta" : "chip-seq-pipeline-test-data/input/se/tas/pooled_ctl/ctl1.subsampled.25.merged.nodup.pooled.tagAlign.gz",
+
+ "test_macs2_signal_track.fraglen" : 95,
+
+ "test_macs2_signal_track.ref_se_macs2_pval_bw" : "chip-seq-pipeline-test-data/ref_output/test_macs2/rep1.subsampled.25.merged.nodup_x_ctl1.subsampled.25.merged.nodup.pooled.pval.signal.bigwig",
+
+ "test_macs2_signal_track.pval_thresh" : 0.01
+}
diff --git a/test/test_task/test_macs2_signal_track.wdl b/test/test_task/test_macs2_signal_track.wdl
new file mode 100644
index 00000000..900a6b9c
--- /dev/null
+++ b/test/test_task/test_macs2_signal_track.wdl
@@ -0,0 +1,44 @@
+# ENCODE DCC ChIP-Seq pipeline tester for task macs2
+# Author: Jin Lee (leepc12@gmail.com)
+import "../../chip.wdl" as chip
+
+workflow test_macs2_signal_track {
+ Float pval_thresh
+
+ Int fraglen
+ # test macs2 for SE set only
+ String se_ta
+ String se_ctl_ta
+
+ String ref_se_macs2_pval_bw # p-val signal
+ String se_chrsz
+ String se_gensz
+
+ Int macs2_mem_mb = 16000
+ Int macs2_time_hr = 24
+ String macs2_disks = "local-disk 100 HDD"
+
+ call chip.macs2_signal_track as se_macs2_signal_track { input :
+ tas = [se_ta, se_ctl_ta],
+ gensz = se_gensz,
+ chrsz = se_chrsz,
+ fraglen = fraglen,
+ pval_thresh = pval_thresh,
+
+ mem_mb = macs2_mem_mb,
+ time_hr = macs2_time_hr,
+ disks = macs2_disks,
+ }
+
+ call chip.compare_md5sum { input :
+ labels = [
+ 'se_macs2_pval_bw',
+ ],
+ files = [
+ se_macs2_signal_track.pval_bw,
+ ],
+ ref_files = [
+ ref_se_macs2_pval_bw,
+ ],
+ }
+}
diff --git a/test/test_workflow/ENCSR000DYI.json b/test/test_workflow/ENCSR000DYI.json
index f63fa8a2..c64baacc 100644
--- a/test/test_workflow/ENCSR000DYI.json
+++ b/test/test_workflow/ENCSR000DYI.json
@@ -2,13 +2,13 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR000DYI/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep1.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/rep2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq/ctl2.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/test/test_workflow/ENCSR000DYI_subsampled.json b/test/test_workflow/ENCSR000DYI_subsampled.json
index 652b5e68..4f548317 100644
--- a/test/test_workflow/ENCSR000DYI_subsampled.json
+++ b/test/test_workflow/ENCSR000DYI_subsampled.json
@@ -2,13 +2,13 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR000DYI_subsampled/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.15.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/test/test_workflow/ENCSR000DYI_subsampled_chr19_only.json b/test/test_workflow/ENCSR000DYI_subsampled_chr19_only.json
index 0a10b32c..47126777 100644
--- a/test/test_workflow/ENCSR000DYI_subsampled_chr19_only.json
+++ b/test/test_workflow/ENCSR000DYI_subsampled_chr19_only.json
@@ -2,13 +2,13 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR000DYI_subsampled_chr19_only/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_chr19_chrM_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep1.subsampled.25.fastq.gz"
],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"]]
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/rep2.subsampled.20.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl1.subsampled.25.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR000DYI/fastq_subsampled/ctl2.subsampled.25.fastq.gz"
],
"chip.paired_end" : false,
diff --git a/test/test_workflow/ENCSR936XTK.json b/test/test_workflow/ENCSR936XTK.json
index c162d552..50b54c2f 100644
--- a/test/test_workflow/ENCSR936XTK.json
+++ b/test/test_workflow/ENCSR936XTK.json
@@ -2,17 +2,21 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR936XTK/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R1.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep1-R2.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R1.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/rep2-R2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl1-R2.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R1.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq/ctl2-R2.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/test/test_workflow/ENCSR936XTK_subsampled.json b/test/test_workflow/ENCSR936XTK_subsampled.json
index 034b470a..9b772945 100644
--- a/test/test_workflow/ENCSR936XTK_subsampled.json
+++ b/test/test_workflow/ENCSR936XTK_subsampled.json
@@ -2,17 +2,21 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR936XTK_subsampled/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.67.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.67.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.67.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/test/test_workflow/ENCSR936XTK_subsampled_chr19_only.json b/test/test_workflow/ENCSR936XTK_subsampled_chr19_only.json
index f1e463bf..bb30a8fe 100644
--- a/test/test_workflow/ENCSR936XTK_subsampled_chr19_only.json
+++ b/test/test_workflow/ENCSR936XTK_subsampled_chr19_only.json
@@ -2,17 +2,21 @@
"chip.qc_report.qc_json_ref" : "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ref_output/v1.1.5/ENCSR936XTK_subsampled_chr19_only/qc.json",
"chip.pipeline_type" : "tf",
"chip.genome_tsv" : "gs://encode-pipeline-genome-data/hg38_chr19_chrM_google.tsv",
- "chip.fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.67.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.67.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.67.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.67.fastq.gz"]]
- ],
- "chip.ctl_fastqs" : [
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"]],
- [["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz",
- "gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"]]
+ "chip.fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.67.fastq.gz"
+ ],
+ "chip.fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.67.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep1_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R1" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz"
+ ],
+ "chip.ctl_fastqs_rep2_R2" : ["gs://encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz"
],
"chip.paired_end" : true,
diff --git a/test/test_workflow/test_chip.sh b/test/test_workflow/test_chip.sh
index 2291095e..35304890 100755
--- a/test/test_workflow/test_chip.sh
+++ b/test/test_workflow/test_chip.sh
@@ -8,17 +8,17 @@ fi
if [ $# -gt 2 ]; then
DOCKER_IMAGE=$3
else
- DOCKER_IMAGE=quay.io/encode-dcc/chip-seq-pipeline:v1.1.7
+ DOCKER_IMAGE=quay.io/encode-dcc/chip-seq-pipeline:v1.2.0
fi
INPUT=$1
GCLOUD_SERVICE_ACCOUNT_SECRET_JSON_FILE=$2
PREFIX=$(basename $INPUT .json)
-CROMWELL_JAR="cromwell-34.jar"
+CROMWELL_JAR="cromwell-38.jar"
if [ -f ${CROMWELL_JAR} ]; then
echo "Skip downloading cromwell."
else
- wget -N -c https://github.com/broadinstitute/cromwell/releases/download/34/cromwell-34.jar
+ wget -N -c https://github.com/broadinstitute/cromwell/releases/download/38/cromwell-38.jar
fi
# Write workflow option JSON file
diff --git a/utils/qc_jsons_to_tsv/qc_jsons_to_tsv.py b/utils/qc_jsons_to_tsv/qc_jsons_to_tsv.py
index bfddb915..9d015348 100755
--- a/utils/qc_jsons_to_tsv/qc_jsons_to_tsv.py
+++ b/utils/qc_jsons_to_tsv/qc_jsons_to_tsv.py
@@ -9,6 +9,7 @@
import argparse
import json
import csv
+import hashlib
from collections import OrderedDict, defaultdict
def parse_arguments():
@@ -41,15 +42,33 @@ def find_workflow_id_from_path(path):
m = re.findall(pattern, abspath)
return m[0] if m else None
+def calc_md5(fname):
+ """https://stackoverflow.com/a/3431838
+ """
+ hash_md5 = hashlib.md5()
+ with open(fname, "rb") as f:
+ for chunk in iter(lambda: f.read(4096), b""):
+ hash_md5.update(chunk)
+ return hash_md5.hexdigest()
+
def recursively_read_qc_jsons(search_dir, qc_json_file_basename,
map_qc_json_path_to_title=None, map_workflow_id_to_title=None):
# find all qc.json recursively
json_files = [y for x in os.walk(search_dir) \
for y in glob.glob(os.path.join(x[0], qc_json_file_basename))]
+ uniq_json_files = []
+ json_files_md5sum = set()
+ for json_file in json_files:
+ md5 = calc_md5(json_file)
+ if md5 in json_files_md5sum:
+ continue
+ json_files_md5sum.add(md5)
+ uniq_json_files.append(json_file)
+
# read all qc.json files
qc_jsons = []
- for json_file in json_files:
+ for json_file in uniq_json_files:
with open(json_file,'r') as fp:
qc = json.load(fp, object_pairs_hook=OrderedDict)
diff --git a/utils/resumer/README.md b/utils/resumer/README.md
index 31ed960f..c32cd02d 100644
--- a/utils/resumer/README.md
+++ b/utils/resumer/README.md
@@ -1,5 +1,7 @@
# resumer
+** RESUMER HAS BEEN DEPRECATED. PIPELINE NO LONGER SUPPORTS RESUMING. USE [CAPER](https://github.com/ENCODE-DCC/caper) WITH MYSQL DATABASE TO USE CROMWELL'S CALL-CACHING. WE WILL KEEP IT FOR A COUPLE OF NEXT RELEASES BUT WILL BE REMOVED SOON AFTER THAT **
+
## Introduction
This python script parses a metadata JSON file from a previous failed workflow and generates a new input JSON file to start a pipeline from where it left off.
@@ -8,7 +10,7 @@ This python script parses a metadata JSON file from a previous failed workflow a
Before running this script, you should have a metadata JSON file for a previous failed workflow. You can get it by adding a parameter `-m metadata.json` to the cromwell Java command line. If you stop a workflow (CTRL+C or kill) metadata then JSON file will not be generated.
```bash
-$ java -jar ... cromwell-34.jar run chip.wdl -i original_input.json ... -m metadata.json
+$ java -jar ... cromwell-38.jar run chip.wdl -i original_input.json ... -m metadata.json
```
Unfortunately your workflow failed for some reasons but you can fix the problem and want to resume it from where it left off.
@@ -18,7 +20,7 @@ $ python resumer.py metadata.json
You will get a new input JSON file `resume.FAILED_WORKFLOW_ID.json` and run cromwell with it instead of the original one `original_input.json`.
```bash
-$ java -jar ... cromwell-34.jar run chip.wdl -i resume.FAILED_WORKFLOW_ID.json ...
+$ java -jar ... cromwell-38.jar run chip.wdl -i resume.FAILED_WORKFLOW_ID.json ...
```
## Usage
@@ -132,7 +134,7 @@ An original input JSON file to start from fastqs.
Run a pipeline with this original input JSON.
```bash
-$ java -jar cromwell-34.jar run toy_chip.wdl -i org_input.json -m metadata.json
+$ java -jar cromwell-38.jar run toy_chip.wdl -i org_input.json -m metadata.json
```
Pipeline fails due to some errors in `call_peak` task. Run `resumer.py` to make a new input JSON file to resume.
@@ -151,7 +153,7 @@ Then `result.WORKFLOW_ID.json` will be generated.
You feed it to the cromwell java command line after fixing the problem. Then pipeline will start from ``scatter` block for `call_peak` tasks.
```bash
-$ java -jar cromwell-34.jar run toy_chip.wdl -i resume.WORKFLOW_ID.json
+$ java -jar cromwell-38.jar run toy_chip.wdl -i resume.WORKFLOW_ID.json
```
## Output definition JSON file (for developers)
diff --git a/utils/resumer/chip.json b/utils/resumer/chip.json
index d5855d42..3fb67c7d 100644
--- a/utils/resumer/chip.json
+++ b/utils/resumer/chip.json
@@ -34,7 +34,10 @@
"chip.macs2" : {
"npeak" : "chip.peaks",
"frip_qc" : "chip.macs2_frip_qcs",
- "sig_pval" : "chip.sig_pvals"
+ "sig_pval" : "chip.pval_bws"
+ },
+ "chip.macs2_signal_track" : {
+ "pval_bw" : "chip.pval_bws"
},
"chip.macs2_pr1" : {
"npeak" : "chip.peaks_pr1",
diff --git a/workflow_opts/docker.json b/workflow_opts/docker.json
index 44a68f05..8ff1066c 100644
--- a/workflow_opts/docker.json
+++ b/workflow_opts/docker.json
@@ -1,6 +1,6 @@
{
"default_runtime_attributes" : {
- "docker" : "quay.io/encode-dcc/chip-seq-pipeline:v1.1.7",
+ "docker" : "quay.io/encode-dcc/chip-seq-pipeline:v1.2.0",
"zones": "us-west1-a us-west1-b us-west1-c us-central1-c us-central1-b",
"failOnStderr" : false,
"continueOnReturnCode" : 0,
diff --git a/workflow_opts/scg.json b/workflow_opts/scg.json
index 6d1b0a0f..eb4da35d 100644
--- a/workflow_opts/scg.json
+++ b/workflow_opts/scg.json
@@ -1,7 +1,7 @@
{
"default_runtime_attributes" : {
"slurm_account" : "YOUR_SLURM_ACCOUNT",
- "singularity_container" : "/reference/ENCODE/pipeline_singularity_images/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "/reference/ENCODE/pipeline_singularity_images/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/reference/ENCODE,/scratch,/srv/gsfs0"
}
}
diff --git a/workflow_opts/sge.json b/workflow_opts/sge.json
index 2d711d09..593c22ff 100644
--- a/workflow_opts/sge.json
+++ b/workflow_opts/sge.json
@@ -1,6 +1,6 @@
{
"default_runtime_attributes" : {
"sge_pe" : "shm",
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg"
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg"
}
}
diff --git a/workflow_opts/sherlock.json b/workflow_opts/sherlock.json
index 8caad810..7747c03b 100644
--- a/workflow_opts/sherlock.json
+++ b/workflow_opts/sherlock.json
@@ -1,7 +1,7 @@
{
"default_runtime_attributes" : {
"slurm_partition" : "normal",
- "singularity_container" : "/home/groups/cherry/encode/pipeline_singularity_images/chip-seq-pipeline-v1.1.7.simg",
+ "singularity_container" : "/home/groups/cherry/encode/pipeline_singularity_images/chip-seq-pipeline-v1.2.0.simg",
"singularity_bindpath" : "/scratch,/lscratch,/oak/stanford,/home/groups/cherry/encode"
}
}
diff --git a/workflow_opts/singularity.json b/workflow_opts/singularity.json
index 050b7e10..677911a2 100644
--- a/workflow_opts/singularity.json
+++ b/workflow_opts/singularity.json
@@ -1,5 +1,5 @@
{
"default_runtime_attributes" : {
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg"
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg"
}
}
diff --git a/workflow_opts/slurm.json b/workflow_opts/slurm.json
index cb8f7c14..fc29e223 100644
--- a/workflow_opts/slurm.json
+++ b/workflow_opts/slurm.json
@@ -2,6 +2,6 @@
"default_runtime_attributes" : {
"slurm_partition" : "YOUR_SLURM_PARTITION",
"slurm_account" : "YOUR_SLURM_ACCOUNT",
- "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.1.7.simg"
+ "singularity_container" : "~/.singularity/chip-seq-pipeline-v1.2.0.simg"
}
}