Fixes for production, March 2024 #95

muffato · 2024-03-26T07:56:08Z

TOLIT-2021

This pull-request includes all the changes I'm planning for a v0.4 release. It is mostly about:

fixing errors faced in production
handling large genomes
tidying up the generation of the output directories

Summary of the changes:

Upgraded all nf-core modules
Use the newer genomehubs/blobtoolkit Docker image
To address Updates in place and broken Nextflow job cache #90, I have updated the modules that contribute to the blobdir to not update their input blobdir in place. However, complete cache reusability is not achievable because the blobdir needs the list of software in a Yaml file, which comes from a Nextflow collectFile call which is not reusable (each call creates a new temporary file)
Files in the blobdir are now compressed (181 MB → 225 KB on the cricket blobdir)
Only the relevant output Busco files are published, and the sequences are tar.gz-ed
The busco_diamond_blastp.nf subworkflow is completely restructured to allow the above
To handle large genomes, I brought some configuration bits from the read-mapping and genome-note pipeline. To make things unambiguous, I have removed the process_* label from all modules that have their own withName entry. That's what most of the .diff files are for. After complete optimisation (TOLIT-1931), since there won't be any withLabel process_* in conf/base.config, we'll be able to undo most of those .diff
This required bringing a few steps to get the genome size and the read counts at the start of the pipeline
The Busco settings are slightly different from the genome-note pipeline as we're finding that interrupted runs (MEMLIMIT/RUNLIMIT) may leave a lot of temporary files, which on the Ubuntu 18.04 farms, take disk space + RAM and prevents other jobs from running on the machines. Once confirmed they work well, they should be backported to the genome-note pipeline.
Quick optimisation: I've patched the seqtk/subseq module to not compress the Fasta file since we had GUNZIP to uncompress it right after
More trace fields by default (the same as in the other pipelines)
More complete and accurate list of recognised file extensions for the reads

Test runs

Initially failures:

/lustre/scratch123/tol/share/weskit/data/prod/5267/5267b490-b605-4afc-a27d-8d0faf02a932
/lustre/scratch123/tol/share/weskit/data/prod/e3bb/e3bb1a93-c08f-4c25-b8b0-fdd8a945a6f8
/lustre/scratch123/tol/share/weskit/data/prod/5504/55048677-c46b-456a-a624-3c6e17331617
/lustre/scratch123/tol/share/weskit/data/prod/5363/5363f999-4a99-4301-b53b-a196cc6b72e2
/lustre/scratch123/tol/share/weskit/data/prod/dda7/dda74568-7e9a-482c-bd44-b220c885051e

And my runs on the same input data:

/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/TOLIT-2008/5267
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/TOLIT-2008/dda7
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/TOLIT-2008/5504
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/TOLIT-2008/5363
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/TOLIT-2008/dda7

I've also tried three assemblies that had failed previously and left some files in /tmp.

/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/GCA_946902985.2  # still running
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/GCA_963580285.1
/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/GCA_963921795.1

and the full test

/lustre/scratch123/tol/teams/tolit/users/mm49/nextflow/btk/results_full

PR checklist

…ong accession being used

…elines

github-actions · 2024-03-26T07:57:27Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 8af7fa8

+| ✅ 134 tests passed       |+
#| ❔  23 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 0.4.0

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
template_strings - template_strings
merge_markers - merge_markers

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowBlobtoolkit.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-blobtoolkit_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: sanger_test.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.11
Run at 2024-04-17 09:22:50

This should not affect the normal/long assignment much

…an pair all outputs from the same genome

bin/check_samplesheet.py

muffato added 26 commits February 9, 2024 12:54

Upgraded the BTK container version to enable better reporting of a wr…

196427a

…ong accession being used

Updated the minimap2/align module

b65e399

Same format for the trace files as the readmapping and genomenote pip…

0f509b4

…elines

bugfix: put the summary.json in the blobdir so that it is published too

0e3a4a0

Upgraded the BTK container version to handle None in CSV files

fc7c576

Make minimap2 support large genomes

f04b749

Pull the genome size as a meta field to allow usage in every process

40fbde5

Don't modify the blobdir in place

46d231d

Compress the blobdir

bb428b2

Updated the changelog

9ee35c0

Updated the documentation

2c1bd1f

Version bump

d7fa228

Updated the resource requirements of fasta_windows

b53b497

Updated the resource requirements of BUSCO

f99b542

Updated the resource requirements of WINDOWSTATS_INPUT

72a639b

Updated the resource requirements of BLOBTOOLKIT_WINDOWSTATS

54862de

Added missing FastQ extensions

424d7e0

Also support FastA files

f8c3cd0

Updated the input JSON

8f7e029

Updated the CHANGELOG

75bbc21

Only BAM and CRAM are accepted for aligned reads

7dd24b3

There is no patch for cat/cat

a71b759

Updated all nf-core modules

857d094

No need to compress the file if we need to decompress it right after

ff43207

Count the number of reads in each input file

b967b51

Optimised settings for minimap2, taken from the read-mapping pipeline

bf82511

muffato self-assigned this Mar 26, 2024

Updated these versions too

fd86c4e

muffato added 8 commits March 26, 2024 23:01

Slightly increased the runtime

c8253f5

This should not affect the normal/long assignment much

Missing space

eecf569

bugfix: meta needs to have the lineage information so that the join c…

cf1dc98

…an pair all outputs from the same genome

bugfix: these files can be missing if no gene is found

328e905

Tidy up the busco directory before publication

4c723c7

Actually need more resources for BUSCO

e273433

Better explanation

f4e236f

Updated the BUSCO resources

7417c5b

muffato requested a review from BethYates April 10, 2024 12:54

muffato marked this pull request as ready for review April 10, 2024 12:54

This was linked to issues Apr 12, 2024

Updates in place and broken Nextflow job cache #90

Closed

summary.json missing from the output blobdir #94

Closed

BethYates reviewed Apr 17, 2024

View reviewed changes

bin/check_samplesheet.py Show resolved Hide resolved

Updated the README

8af7fa8

BethYates approved these changes Apr 17, 2024

View reviewed changes

muffato merged commit f519c4e into main Apr 17, 2024
6 checks passed

muffato deleted the prod_fix branch April 17, 2024 11:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for production, March 2024 #95

Fixes for production, March 2024 #95

muffato commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Fixes for production, March 2024 #95

Fixes for production, March 2024 #95

Conversation

muffato commented Mar 26, 2024 • edited Loading

Test runs

PR checklist

github-actions bot commented Mar 26, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

muffato commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️