Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Singularity #77

Merged
merged 23 commits into from
May 7, 2024
Merged

Singularity #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
67e576f
simplify conda finding
pdimens May 2, 2024
aac9b14
rename R env
pdimens May 2, 2024
6c46fb0
add containerizeation
pdimens May 6, 2024
0f37184
specify apptainer
pdimens May 6, 2024
b3176b3
fix test
pdimens May 6, 2024
9e547e8
fixes
pdimens May 6, 2024
d1110ce
rm apptainer code
pdimens May 6, 2024
69b21da
try to add clear space run
pdimens May 6, 2024
87e8810
add missing conda arg
pdimens May 6, 2024
74e085e
try to cache image
pdimens May 6, 2024
5a08506
change trigger
pdimens May 6, 2024
6d8ed78
try this approach
pdimens May 6, 2024
94fd381
include sanity check
pdimens May 6, 2024
a9d83f6
fix
pdimens May 6, 2024
ccb0f2e
pull artifact into all jobs
pdimens May 6, 2024
c6118c5
use this simpler(?) phasing annotation
pdimens May 7, 2024
c2fec15
simplify the split code with bcftools
pdimens May 7, 2024
dcac0ab
update summary to match workflow
pdimens May 7, 2024
af014da
add mutation rate
pdimens May 7, 2024
8082e03
fix
pdimens May 7, 2024
2da18f1
split out the main.py stuff into respective modules
pdimens May 7, 2024
467a819
rm lambda
pdimens May 7, 2024
d0bf048
rm extra quote
pdimens May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added .cache/conda/notices/notices.cache
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions .condarc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
channels:
- conda-forge
channel_priority: strict
149 changes: 130 additions & 19 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,19 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache
- name: pull image
shell: micromamba-shell {0}
run: harpy qc --skipreports -a test/fastq/sample1.*.fq.gz
- name: create singularity artifact
uses: actions/upload-artifact@v4
with:
name: deps-image
path: .snakemake/singularity/*.simg
retention-days: 1

test_dmux_gen1:
dmux_gen1:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.demux == 'true' && needs.pkgbuild.result == 'success' }}
name: test demux gen1
Expand All @@ -97,11 +108,18 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: harpy demultiplex
run: harpy demultiplex gen1 --schema test/demux/samples.schema test/demux/Undetermined_S0_L004_R* test/demux/Undetermined_S0_L004_I*
shell: micromamba-shell {0}
run: harpy demultiplex gen1 --schema test/demux/samples.schema test/demux/Undetermined_S0_L004_R* test/demux/Undetermined_S0_L004_I*

test_preflight:
preflight:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.preflight == 'true' && needs.pkgbuild.result == 'success' }}
name: test preflight
Expand All @@ -128,6 +146,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: test preflight fastq
shell: micromamba-shell {0}
run: harpy preflight fastq test/fastq
Expand All @@ -136,7 +161,7 @@ jobs:
shell: micromamba-shell {0}
run: harpy preflight bam test/bam

test_qc:
qc:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.qc == 'true' && needs.pkgbuild.result == 'success' }}
name: test qc
Expand All @@ -163,6 +188,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: harpy qc
shell: micromamba-shell {0}
run: harpy qc -x "--trim_poly_g" --snakemake "--show-failed-logs" test/fastq
Expand All @@ -171,7 +203,7 @@ jobs:
shell: micromamba-shell {0}
run: harpy qc -a --snakemake "--show-failed-logs" test/fastq

test_bwa:
bwa:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.bwa == 'true' && needs.pkgbuild.result == 'success' }}
name: test align BWA
Expand All @@ -198,11 +230,18 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: test bwa
shell: micromamba-shell {0}
run: harpy align bwa -g test/genome/genome.fasta.gz --snakemake "--show-failed-logs" -x "-A 2" test/fastq

test_ema:
ema:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.ema == 'true' && needs.pkgbuild.result == 'success' }}
name: test align EMA
Expand All @@ -229,11 +268,18 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: test ema
shell: micromamba-shell {0}
run: harpy align ema --ema-bins 150 -g test/genome/genome.fasta.gz --snakemake "--show-failed-logs" -x "-d" test/fastq

test_minimap:
minimap:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.minimap == 'true' && needs.pkgbuild.result == 'success' }}
name: test align minimap
Expand All @@ -260,11 +306,18 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: test minimap
shell: micromamba-shell {0}
run: harpy align minimap -g test/genome/genome.fasta.gz --snakemake "--show-failed-logs" -x "--seed 13" test/fastq

test_mpileup:
mpileup:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.mpileup == 'true' && needs.pkgbuild.result == 'success' }}
name: test mpileup
Expand All @@ -291,14 +344,21 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: snp mpileup
shell: micromamba-shell {0}
run: harpy snp mpileup -r test/positions.bed -g test/genome/genome.fasta.gz -x "--ignore-RG" test/bam
- name: snp mpileup-pop
shell: micromamba-shell {0}
run: harpy snp mpileup -r test/positions.bed -o SNP/poptest -g test/genome/genome.fasta.gz -p test/samples.groups test/bam
run: harpy snp mpileup -r test/positions.bed -o SNP/poptest -g test/genome/genome.fasta.gz -p test/samples.groups --snakemake "--show-failed-logs" test/bam

test_freebayes:
freebayes:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.freebayes == 'true' && needs.pkgbuild.result == 'success' }}
name: test freebayes
Expand All @@ -325,14 +385,21 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: snp freebayes
shell: micromamba-shell {0}
run: harpy snp freebayes -r test/positions.bed -g test/genome/genome.fasta.gz -x "-g 200" test/bam
- name: snp freebayes-pop
shell: micromamba-shell {0}
run: harpy snp freebayes -r test/positions.bed -o SNP/poptest -g test/genome/genome.fasta.gz -p test/samples.groups test/bam
run: harpy snp freebayes -r test/positions.bed -o SNP/poptest -g test/genome/genome.fasta.gz -p test/samples.groups --snakemake "--show-failed-logs" test/bam

test_impute:
impute:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.impute == 'true' && needs.pkgbuild.result == 'success' }}
name: test impute
Expand All @@ -359,6 +426,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: impute
shell: micromamba-shell {0}
run: harpy impute --vcf test/vcf/test.bcf -p test/stitch.params --snakemake "--show-failed-logs" test/bam
Expand All @@ -367,7 +441,7 @@ jobs:
if: always()
run: harpy impute --vcf-samples -o vcfImpute --vcf test/vcf/test.bcf -p test/stitch.params --snakemake "--show-failed-logs" test/bam

test_phase:
phase:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.phase == 'true' && needs.pkgbuild.result == 'success' }}
name: test phase
Expand All @@ -394,6 +468,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: phase
shell: micromamba-shell {0}
run: harpy phase --vcf test/vcf/test.bcf --snakemake "--show-failed-logs" -x "--max_iter 10001" test/bam
Expand All @@ -406,7 +487,7 @@ jobs:
if: always()
run: harpy phase --vcf-samples -o phasevcf --vcf test/vcf/test.bcf --snakemake "--show-failed-logs" test/bam

test_leviathan:
leviathan:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.leviathan == 'true' && needs.pkgbuild.result == 'success' }}
name: test sv leviathan
Expand All @@ -433,6 +514,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: leviathan
shell: micromamba-shell {0}
run: harpy sv leviathan -n 100 -b 1 -g test/genome/genome.fasta.gz --snakemake "--show-failed-logs" -x "-M 2002" test/bam
Expand All @@ -442,7 +530,7 @@ jobs:
shell: micromamba-shell {0}
run: harpy sv leviathan -n 100 -b 1 -g test/genome/genome.fasta.gz -o SV/leviathanpop -p test/samples.groups --snakemake "--show-failed-logs" test/bam

test_naibr:
naibr:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.naibr == 'true' && needs.pkgbuild.result == 'success' }}
name: test sv naibr
Expand All @@ -469,6 +557,13 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: naibr
shell: micromamba-shell {0}
run: harpy sv naibr -g test/genome/genome.fasta.gz -o SV/naibr --snakemake "--show-failed-logs" -x "-min_sv 5000" test/bam_phased && rm -r Genome
Expand All @@ -487,7 +582,7 @@ jobs:
run: harpy sv naibr -g test/genome/genome.fasta.gz -o SV/phasepop -v test/vcf/test.phased.bcf -p test/samples.groups --snakemake "--show-failed-logs" test/bam && rm -r Genome


test_simulate_variants:
simulate_variants:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.simvars == 'true' && needs.pkgbuild.result == 'success' }}
name: test simulate variants
Expand All @@ -509,11 +604,18 @@ jobs:
post-cleanup: 'all'
log-level: error
- name: Install harpy
shell: micromamba-shell {0}
run: |
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
shell: micromamba-shell {0}
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: simulate random snps/indels
shell: micromamba-shell {0}
run: |
Expand All @@ -538,7 +640,7 @@ jobs:
harpy simulate translocation --count 10 -z 0.5 test/genome/genome.fasta.gz
harpy simulate translocation --prefix Simulate/transvcf --vcf Simulate/translocation/sim.translocation.hap1.vcf --snakemake "--show-failed-logs" test/genome/genome.fasta.gz

test_simulate_linkedreads:
simulate_linkedreads:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.simreads == 'true' && needs.pkgbuild.result == 'success' }}
name: test simulate linkedreads
Expand All @@ -565,11 +667,18 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: Download singularity artifact
uses: actions/download-artifact@v4
with:
name: deps-image
path: .snakemake/singularity
- name: simulate linked reads
shell: micromamba-shell {0}
run: harpy simulate linkedreads --snakemake "--show-failed-logs" -t 4 -n 2 -l 100 -p 50 test/genome/genome.fasta.gz test/genome/genome2.fasta.gz

test_extras:
extras:
needs: [changes, pkgbuild]
if: ${{ needs.changes.outputs.extras == 'true' && needs.pkgbuild.result == 'success' }}
name: test harpy extras
Expand All @@ -596,6 +705,8 @@ jobs:
python3 -m pip install --upgrade build && python3 -m build
pip install dist/*.whl
resources/buildforCI.sh
- name: Clear space
run: rm -rf /opt/hostedtoolcache && mkdir -p .snakemake/singularity
- name: harpy stitchparams
shell: micromamba-shell {0}
run: harpy stitchparams -o params.file
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "harpy"
version = "0.10.0"
version = "1.0.0"
authors = [
{name = "Pavel Dimens"}
]
Expand Down
Loading
Loading