The Big Merge #150

DLBPointon · 2023-09-25T13:10:17Z

The Final Merge of dev into Main before tagging!

…g pipeline_execution file

…y functions

Mm49 docs reformat

Adding pretext ingestion docs

Fix genome size

github-actions · 2023-09-25T13:12:58Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 2413fe5

+| ✅ 126 tests passed       |+
#| ❔  17 tests were ignored |#
!| ❗   8 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes.config
files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
nextflow_config - Config manifest.version should end in dev: '1.0.0'
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in methods_description_template.yml: ## Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
system_exit - System.exit in WorkflowTreeval.groovy: System.exit(1) [line 17]

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-treeval_logo_light.png
files_exist - File is ignored: conf/test_full.config
files_exist - File is ignored: docs/images/nf-core-treeval_logo_light.png
files_exist - File is ignored: docs/images/nf-core-treeval_logo_dark.png
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-treeval_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-treeval_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-treeval_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/treeval/treeval/.github/workflows/awstest.yml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowTreeval.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-treeval_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (227 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2023-09-27 11:09:56

muffato · 2023-09-25T15:40:26Z

This merge is a "fast-forward" at the git level because the dev branch is cleanly attached to to the latest main. In other words, the code that would be on main after the merge is exactly the same as what's currently on dev and has been tested/reviewed in the recent pull-requests.

First check 🤝

muffato · 2023-09-25T19:23:36Z

I ran the full test myself since the run through Tower could not be initiated.

Just checking, @DLBPointon, is it expected that these processes don't have any job ?

SANGERTOL_TREEVAL:TREEVAL:BUSCO_ANNOTATION:ANCESTRAL_GENE:*
SANGERTOL_TREEVAL:TREEVAL:LONGREAD_COVERAGE:MINIMAP2_ALIGN_SPLIT
SANGERTOL_TREEVAL:TREEVAL:SYNTENY:MINIMAP2_ALIGN

DLBPointon · 2023-09-25T21:09:40Z

I ran the full test myself since the run through Tower could not be initiated.

Just checking, @DLBPointon, is it expected that these processes don't have any job ?

SANGERTOL_TREEVAL:TREEVAL:BUSCO_ANNOTATION:ANCESTRAL_GENE:*

SANGERTOL_TREEVAL:TREEVAL:LONGREAD_COVERAGE:MINIMAP2_ALIGN_SPLIT

SANGERTOL_TREEVAL:TREEVAL:SYNTENY:MINIMAP2_ALIGN

Hi @muffato,
In these cases (fungi on github and nematode on farm) yes, those 3 processes will not have any jobs.
We only have synteny jobs for birds, align_split is only for genomes > 4gb and Ancestral is only for leps.
Thanks

…on the farm

… test data too

muffato · 2023-09-27T05:57:01Z

Once #152 is merged, I'm happy to approve this PR

into mm49_github_test

Refactoring of the test and test_github profiles

DLBPointon added 30 commits July 25, 2023 15:34

Prettier Formatting

bd2ef78

Finished the reporting script

d6497c1

Finished the reporting script

da4a37c

Added a param so allow for propogation of trace_timestamp for catchin…

8f89150

…g pipeline_execution file

Added a param so allow for propogation of trace_timestamp for catchin…

165a0ca

…g pipeline_execution file

Updated project for TreeValProject summary reporting

a472ece

Updated project for TreeValProject summary reporting

c0b72fd

Updated project for TreeValProject summary reporting

a7a2e42

Updated project for TreeValProject summary reporting

19e0188

Minor bug fix

dda688f

Minor bug fix

df80a5b

Additional small changes

850a7f4

Additional small changes

3980f4a

Linting

166513e

Linting

5312727

Additional small changes and linting

eb30318

Additional small changes and linting

d125902

Fixed bug when coming across multiple cram or pacbio files for summar…

b84be0c

…y functions

Fixed bug when coming across multiple cram or pacbio files for summar…

6410c6c

…y functions

Adding reporting to rapid workflow

15c0699

Adding reporting to rapid workflow

f6a954a

Resetting permissions

55a1511

Resetting permissions

deb0af8

Adding spaces to fix github bug

fd76899

Adding spaces to fix github bug

a614604

Adding spaces to fix github bug

a370943

Adding spaces to fix github bug

1d03353

Adding spaces to fix github bug

20e7f16

Adding spaces to fix github bug

af14d00

Adding spaces to fix github bug

d62dfe7

muffato and others added 7 commits September 25, 2023 09:45

Changed genomic_alignment_data

7b58017

Update usage.md

29e6598

Merge pull request #149 from sanger-tol/mm49_docs_reformat

3d5ea17

Mm49 docs reformat

Merge branch 'More-docs' into fix_genome_size

5b4bb92

Merge branch 'More-docs' into fix_genome_size

580ee1c

Merge pull request #145 from sanger-tol/More-docs

6df2644

Adding pretext ingestion docs

Merge pull request #146 from sanger-tol/fix_genome_size

28309b7

Fix genome size

DLBPointon added documentation Improvements or additions to documentation enhancement New feature or request Release 1 Issues required for first release labels Sep 25, 2023

DLBPointon requested review from muffato and priyanka-surana September 25, 2023 13:10

DLBPointon self-assigned this Sep 25, 2023

muffato added this to the Release 1 milestone Sep 25, 2023

muffato added 6 commits September 26, 2023 09:16

Fixed the data path

271ee00

Reintroduced the github_test profile, and made the test profile work …

27d2e75

…on the farm

typo

ceddbba

Added instructions for people to run the test

f434f33

Added some missing citations, as realised when reviewing curationpretext

7b98305

Updated the test instructions, as the GitHub path is hardcoded in the…

702cf96

… test data too

DLBPointon and others added 3 commits September 27, 2023 10:37

Updating for more control of CI jobs

0092471

Merge branch 'mm49_github_test' of https://github.com/sanger-tol/treeval

370361f

into mm49_github_test

Merge pull request #152 from sanger-tol/mm49_github_test

2413fe5

Refactoring of the test and test_github profiles

muffato approved these changes Sep 27, 2023

View reviewed changes

DLBPointon merged commit a064360 into main Sep 27, 2023
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The Big Merge #150

The Big Merge #150

DLBPointon commented Sep 25, 2023

github-actions bot commented Sep 25, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

muffato commented Sep 25, 2023

muffato commented Sep 25, 2023

DLBPointon commented Sep 25, 2023

muffato commented Sep 27, 2023

The Big Merge #150

The Big Merge #150

Conversation

DLBPointon commented Sep 25, 2023

github-actions bot commented Sep 25, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

muffato commented Sep 25, 2023

muffato commented Sep 25, 2023

DLBPointon commented Sep 25, 2023

muffato commented Sep 27, 2023

github-actions bot commented Sep 25, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️