Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dp24 module update 2 #334

Closed
wants to merge 11 commits into from
Closed

Dp24 module update 2 #334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
cb5d998
Updates for modules, README and config
DLBPointon Nov 15, 2024
79bd32f
Updating SAMTOOLS
DLBPointon Nov 20, 2024
217bd32
linting
DLBPointon Nov 20, 2024
30a576c
Updating all easy to update modules
DLBPointon Nov 20, 2024
e93fd9a
Updating all easy to update modules
DLBPointon Nov 20, 2024
dc1e076
Updating all easy to update modules
DLBPointon Nov 20, 2024
3c660bb
Updating all easy to update modules
DLBPointon Nov 20, 2024
32152a5
Fixes to modules
DLBPointon Nov 20, 2024
1beeeb5
linting
DLBPointon Nov 20, 2024
9fc4135
Testing the removal of BUSCO through steps
DLBPointon Nov 21, 2024
23d1eec
Merge branch 'main' into dp24_module_update_2
DLBPointon Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,4 @@ jobs:
- name: Singularity - Run FULL pipeline with test data
# Remember that you can parallelise this by using strategy.matrix
run: |
nextflow run ./sanger-treeval/${{ steps.branch-names.outputs.current_branch }}/main.nf -profile test_github,singularity --outdir ./Sing-Full
nextflow run ./sanger-treeval/${{ steps.branch-names.outputs.current_branch }}/main.nf -profile test_github,singularity --outdir ./Sing-Full --steps busco
76 changes: 76 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,75 @@
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.2.0] - Ancient Destiny - [2024-11-15]

Our 3rd release for sanger-tol/treeval.

### Enhancements & Fixes

- Togglable subworkflows
- Adds a JBrowse Only workflow (this will lead to an update to the FULL workflow which can now call JBROWSE_ONLY and RAPID).
- Updates to containers (local modules) to remove Anaconda dependencies following policy changes.
- Updates to modules to remove Anaconda dependencies following policy changes
- CONDA warnings for modules which cannot use CONDA.
- Removable of a liberal use of spaces.
- reformat_intersect was previously not outputing version data.
- Adding arch specification to Pretext GitHub actions runner. Hopefully this will stop the spurious errors we see on there.
- Addition of steps into schema.

### Parameters

| Old Parameter | New Parameter |
| ------------- | ------------- |
| - | --steps |

### Software dependencies

Note, since the pipeline is using Nextflow DSL2, each process will be run with its own Biocontainer. This means that on occasion it is entirely possible for the pipeline to be using different versions of the same tool. However, the overall software dependency changes compared to the last release have been listed below for reference.

| Module | Old Version | New Versions |
| -------------------------------------- | ---------------- | ------------ |
| bamtobed_sort ( bedtools + samtools ) | 2.31.0 + 1.17 | |
| bedtools | 2.31.1 | |
| busco | 5.5.0 | |
| bwa-mem2 | 2.2.1 | |
| cat | 2.3.4 | |
| chunk_fasta ( pyfasta ) | 0.5.2-1 | |
| cooler | 0.9.2 | |
| cram_filter_align_bwamem2_fixmate_sort | - | |
| ^ ( samtools + bwamem2 ) ^ | 1.17 + 2.2.1 | |
| coreutils | 9.1 | |
| fastk | 1.0.1 | |
| gcc | 10.4.0 | |
| find_telomere_windows ( java-jdk ) | 8.0.112 | |
| generate_cram_csv ( samtools ) | 1.17 | |
| gnu-sort | 8.25 | |
| juicer_tools_pre ( java-jdk ) | 8.0.112 | |
| perl | 5.26.2 | |
| merquryfk | 1.0.1 | |
| minimap2 + samtools | 2.24 + 1.14 | |
| miniprot | 0.11--he4a0461_2 | |
| mummer | 3.23 | |
| paftools ( minimap2 + samtools ) | 2.24 + 1.14 | |
| pretextmap + samtools | 0.0.2 + 1.17 | |
| python | 3.9 | - |
| - pandas | 1.5.2 | - |
| samtools | 1.18 | 1.21 |
| selfcomp_splitfasta ( perl-bioperl ) | 1.7.8-1 | |
| seqtk | 1.4 | |
| tabix | 1.11 | |
| ucsc | 377 | |
| windowmasker (blast) | 2.14.0 | |

## [1.1.1] - Ancient Aurora (H1) - [2024-04-26]

### Enhancements & Fixes

- Generate CRAM CSV fix to allow for multi-readgroup cram files
- Removing KMER_READCOV
- tmp directory was being used
- Output file adjustment (names and location)

## [1.1.0] - Ancient Aurora - [2024-04-26]

The second release for sanger-tol, created with the [nf-core](https://nf-co.re/) template.
Expand Down Expand Up @@ -40,6 +109,13 @@ This builds on the initial release by adding subworkflows which generate kmer ba
- Fix a bug in build_alignment_blocks.py to avoid indexing errors happening in large genomes.
- Change output BEDGRAPH from EXTRACT_TELO module.

#### Hot Fix 1

- Generate CRAM CSV fix to allow for multi-readgroup cram files
- Removing KMER_READCOV
- tmp directory was being used
- Output file adjustment (names and location)

### Parameters

| Old Parameter | New Parameter |
Expand Down
4 changes: 1 addition & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

## Introduction

**sanger-tol/treeval [1.1.0 - Ancient Aurora]** is a bioinformatics best-practice analysis pipeline for the generation of data supplemental to the curation of reference quality genomes. This pipeline has been written to generate flat files compatible with [JBrowse2](https://jbrowse.org/jb2/) as well as HiC maps for use in Juicebox, PretextView and HiGlass.
**sanger-tol/treeval [1.2.0 - Ancient Destiny-]** is a bioinformatics best-practice analysis pipeline for the generation of data supplemental to the curation of reference quality genomes. This pipeline has been written to generate flat files compatible with [JBrowse2](https://jbrowse.org/jb2/) as well as HiC maps for use in Juicebox, PretextView and HiGlass.

The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!

Expand Down Expand Up @@ -80,8 +80,6 @@ If you would like to contribute to this pipeline, please see the [contributing g

## Citations

<!--TODO: Citation-->

If you use sanger-tol/treeval for your analysis, please cite it using the following doi: [10.5281/zenodo.10047653](https://doi.org/10.5281/zenodo.10047653).

### Tools
Expand Down
4 changes: 2 additions & 2 deletions conf/base.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,14 +225,14 @@ process {
memory = { check_max( 100.GB * task.attempt, 'memory' ) }
}

withName: BUSCO {
withName: BUSCO_BUSCO {
cpus = { check_max( 16 * task.attempt, 'cpus' ) }
memory = { check_max( 50.GB * task.attempt, 'memory' ) }
time = { check_max( 20.h * task.attempt, 'time' ) }
}

// Large Genomes > 4Gb
//withName: BUSCO {
//withName: BUSCO_BUSCO {
// cpus = { check_max( 30 * task.attempt, 'cpus' ) }
// memory = { check_max( 100.GB * task.attempt, 'memory' ) }
// time = { check_max( 300.h * task.attempt, 'time' ) }
Expand Down
2 changes: 1 addition & 1 deletion conf/modules.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ process {
ext.prefix = { "${meta.id}_ancestral" }
}

withName: "BUSCO" {
withName: "BUSCO_BUSCO" {
ext.args = "--offline"
}

Expand Down
60 changes: 30 additions & 30 deletions modules.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"nf-core": {
"bedtools/bamtobed": {
"branch": "master",
"git_sha": "1d1cb7bfef6cf67fbc7faafa6992ad8bdc3045b3",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"bedtools/genomecov": {
Expand All @@ -17,12 +17,12 @@
},
"bedtools/intersect": {
"branch": "master",
"git_sha": "575e1bc54b083fb15e7dd8b5fcc40bea60e8ce83",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"bedtools/makewindows": {
"branch": "master",
"git_sha": "3b248b84694d1939ac4bb33df84bf6233a34d668",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"bedtools/map": {
Expand All @@ -32,27 +32,27 @@
},
"bedtools/merge": {
"branch": "master",
"git_sha": "575e1bc54b083fb15e7dd8b5fcc40bea60e8ce83",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"bedtools/sort": {
"branch": "master",
"git_sha": "575e1bc54b083fb15e7dd8b5fcc40bea60e8ce83",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"busco": {
"busco/busco": {
"branch": "master",
"git_sha": "e3126f437c336c826f242842fe51769cfce0ec2d",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"bwamem2/index": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"cat/cat": {
"branch": "master",
"git_sha": "81f27e75847087865299cc46605deb3b09b4e0a2",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"cooler/cload": {
Expand All @@ -67,12 +67,12 @@
},
"custom/dumpsoftwareversions": {
"branch": "master",
"git_sha": "8ec825f465b9c17f9d83000022995b4f7de6fe93",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"custom/getchromsizes": {
"branch": "master",
"git_sha": "1b0ffa4e5aed5b7e3cd4311af31bd3b2c8345051",
"git_sha": "49f4e50534fe4b64101e62ea41d5dc43b1324358",
"installed_by": ["modules"],
"patch": "modules/nf-core/custom/getchromsizes/custom-getchromsizes.diff"
},
Expand All @@ -83,7 +83,7 @@
},
"gnu/sort": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"merquryfk/merquryfk": {
Expand All @@ -99,17 +99,17 @@
},
"minimap2/index": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"miniprot/align": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"miniprot/index": {
"branch": "master",
"git_sha": "8d737766e8f3c1417212b4b56acb959f3c356d26",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"mummer": {
Expand All @@ -120,79 +120,79 @@
},
"paftools/sam2paf": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"pretextmap": {
"branch": "master",
"git_sha": "decfb802f2e573efb7b44ff06b11ecf16853054d",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"],
"patch": "modules/nf-core/pretextmap/pretextmap.diff"
},
"pretextsnapshot": {
"branch": "master",
"git_sha": "911696ea0b62df80e900ef244d7867d177971f73",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"],
"patch": "modules/nf-core/pretextsnapshot/pretextsnapshot.diff"
},
"samtools/faidx": {
"branch": "master",
"git_sha": "ce0b1aed7d504883061e748f492a31bf44c5777c",
"git_sha": "b13f07be4c508d6ff6312d354d09f2493243e208",
"installed_by": ["modules"]
},
"samtools/index": {
"branch": "master",
"git_sha": "a64788f5ad388f1d2ac5bd5f1f3f8fc81476148c",
"git_sha": "b13f07be4c508d6ff6312d354d09f2493243e208",
"installed_by": ["modules"]
},
"samtools/markdup": {
"branch": "master",
"git_sha": "ce0b1aed7d504883061e748f492a31bf44c5777c",
"git_sha": "b13f07be4c508d6ff6312d354d09f2493243e208",
"installed_by": ["modules"]
},
"samtools/merge": {
"branch": "master",
"git_sha": "ce0b1aed7d504883061e748f492a31bf44c5777c",
"git_sha": "b13f07be4c508d6ff6312d354d09f2493243e208",
"installed_by": ["modules"]
},
"samtools/sort": {
"branch": "master",
"git_sha": "ce0b1aed7d504883061e748f492a31bf44c5777c",
"git_sha": "b7800db9b069ed505db3f9d91b8c72faea9be17b",
"installed_by": ["modules"]
},
"samtools/view": {
"branch": "master",
"git_sha": "ce0b1aed7d504883061e748f492a31bf44c5777c",
"git_sha": "669eb24fd82a9d3cb18ad0e73673ecb26827f683",
"installed_by": ["modules"]
},
"seqtk/cutn": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"tabix/bgziptabix": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "f448e846bdadd80fc8be31fbbc78d9f5b5131a45",
"installed_by": ["modules"]
},
"ucsc/bedgraphtobigwig": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "49f4e50534fe4b64101e62ea41d5dc43b1324358",
"installed_by": ["modules"]
},
"ucsc/bedtobigbed": {
"branch": "master",
"git_sha": "9e51255c4f8ec69fb6ccf68593392835f14fecb8",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"windowmasker/mkcounts": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
},
"windowmasker/ustat": {
"branch": "master",
"git_sha": "3f5420aa22e00bd030a2556dfdffc9e164ec0ec5",
"git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
"installed_by": ["modules"]
}
}
Expand Down
2 changes: 0 additions & 2 deletions modules/nf-core/bedtools/bamtobed/environment.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

43 changes: 23 additions & 20 deletions modules/nf-core/bedtools/bamtobed/meta.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading