Remove input read indices, make reference index optional and combine same sample together #59

gq1 · 2023-10-07T16:09:56Z

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

…ex file.

…s well

…file optional

github-actions · 2023-10-07T16:11:49Z

`nf-core lint` overall result: Passed ✅

Posted for pipeline commit 3e8705c

+| ✅ 126 tests passed       |+
#| ❔  26 tests were ignored |#

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-variantcalling_logo_light.png
files_exist - File is ignored: docs/images/nf-core-variantcalling_logo_light.png
files_exist - File is ignored: docs/images/nf-core-variantcalling_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: assets/multiqc_config.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-variantcalling_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-variantcalling_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-variantcalling_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/variantcalling/variantcalling/.github/workflows/awstest.yml
multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowVariantcalling.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-variantcalling_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '1.1.0-dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (82 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: sanger_test.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2023-11-02 11:44:04

gq1 · 2023-10-07T18:50:47Z

@nf-core-bot fix linting

muffato

Push the updated samtools merge to nf-core
Update bin/check_samplesheet.py not to append the _T* suffixes so that you don't need the replaceAll

conf/test.config

priyanka-surana

Initial review. I think we should clean up the pipeline logic a bit before adding the next set of features. Thinking about what we need and what is just adding complexity. Yet to run tests.

modules/nf-core/samtools/merge/main.nf

modules/nf-core/samtools/merge/meta.yml

subworkflows/local/input_filter_split.nf

nextflow.config

subworkflows/local/input_merge.nf

workflows/variantcalling.nf

modules/nf-core/samtools/merge/samtools-merge.diff

Co-authored-by: Matthieu Muffato <[email protected]>

with conda environment file and maintainers list

Combine gzi and fai parameters as one.

gq1 · 2023-11-02T12:35:53Z

Push the updated samtools merge to nf-core
The patch being pushed to nf-core and updated. samtools/merge - emit crai file when cram file created nf-core/modules#4258

Update bin/check_samplesheet.py not to append the _T* suffixes so that you don't need the replaceAll
The script being updated only validate the samplesheet not transform it. d6fa00f

gq1 · 2023-11-02T14:14:56Z

patch for samtools/merge module being pushed to nf-core
combine fai and gzi input fields for samtool/merge module but didn't update the pattern, following all other cases in samtools modules
combine fai and gzi parameters for the whole workflow
remove sort_input parameters and sorting input bam/cram before merging
not changing the sample names in sample_checking script

gq1 added 19 commits October 5, 2023 14:15

Remove index file from the samplesheet and update checking script

3f89203

Remove indexfile from Input_check workflow

cef527b

add sample name to the meta data

67826e3

nf-core modules install samtools/merge

2d30d82

Add input merge sub workflow

953658f

comments

0d0d06e

patch samtools_merge module to allow using fasta.gz file with gzi ind…

224c7f2

…ex file.

use original sample name for id if just 1, otherwise add _combined.

36f4c67

Path samtools_merge module again add indexing, emit crai index file a…

a808cf2

…s well

emit crai files as well

c646b8b

nf-core modules install samtools/sort

c4a762f

Add an option to sort input if not sorted.

9fa3696

combine merged bam/cram together, add with their index files as well.

5cbea5b

add filtered to distinguish the samtools input and output name

3ac73e9

use the merged read for the rest of pipeline

bc6e5a0

covert all input files into channels, and make reference fasta index …

a7131a8

…file optional

use the first for the reference fasta channel

0265cbf

move write-index flag to the config file

d4d38cc

make sure work file when no interval file given

ea046b9

formating and documents

7506e81

[automated] Fix linting with Prettier

4dacfa8

gq1 marked this pull request as ready for review October 7, 2023 18:55

gq1 requested review from priyanka-surana and muffato October 9, 2023 08:50

muffato reviewed Oct 13, 2023

View reviewed changes

conf/test.config Outdated Show resolved Hide resolved

priyanka-surana suggested changes Oct 13, 2023

View reviewed changes

gq1 and others added 2 commits October 13, 2023 09:49

Update conf/test.config

ebaa7e4

Co-authored-by: Matthieu Muffato <[email protected]>

nf-core modules update samtools/merge

1b2c45e

with conda environment file and maintainers list

gq1 added 4 commits November 1, 2023 19:22

Update samtools/merge module and remove its patch.

a76298f

Combine gzi and fai parameters as one.

remove sort_input params. Always sort the input before merging.

3cf953d

only validate the sample sheet not transform the sample names

d6fa00f

update fai file for the full test and formating

3e8705c

muffato approved these changes Nov 3, 2023

View reviewed changes

gq1 merged commit aa99735 into dev Nov 3, 2023
6 checks passed

This was referenced Nov 3, 2023

Combine multiple libraries from the same sample #44

Closed

Create indices #43

Closed

gq1 deleted the input_index branch November 6, 2023 10:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove input read indices, make reference index optional and combine same sample together #59

Remove input read indices, make reference index optional and combine same sample together #59

gq1 commented Oct 7, 2023

github-actions bot commented Oct 7, 2023 •

edited

Loading

❔ Tests ignored:

✅ Tests passed:

Run details

gq1 commented Oct 7, 2023

muffato left a comment

priyanka-surana left a comment

gq1 commented Nov 2, 2023

gq1 commented Nov 2, 2023

Remove input read indices, make reference index optional and combine same sample together #59

Remove input read indices, make reference index optional and combine same sample together #59

Conversation

gq1 commented Oct 7, 2023

PR checklist

github-actions bot commented Oct 7, 2023 • edited Loading

nf-core lint overall result: Passed ✅

❔ Tests ignored:

✅ Tests passed:

Run details

gq1 commented Oct 7, 2023

muffato left a comment

Choose a reason for hiding this comment

priyanka-surana left a comment

Choose a reason for hiding this comment

gq1 commented Nov 2, 2023

gq1 commented Nov 2, 2023

github-actions bot commented Oct 7, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅