make_chains.py in pipeline: modules.error_classes.PipelineFileNotFoundError: Error! No non-empty files found at make_lastz_chains/test_out/temp_concat_lastz_output. The failed operation label is: cat_step

[SomePersonSomeWhereInTheWorld]

On RHEL8, nextflow 23.10.0, anaconda/3-2023.09 on a shared HPC cluster, running the below example from Running the pipeline, results in the below. Did I miss a step?

./make_chains.py target query test_data/test_reference.fa test_data/test_query.fa --pd test_out -f --chaining_memory 16
# Make Lastz Chains #
Version 2.0.8
Commit: 187e313afc10382fe44c96e47f27c4466d63e114
Branch: main

* found run_lastz.py at /path/to/me/make_lastz_chains/standalone_scripts/run_lastz.py
* found run_lastz_intermediate_layer.py at /path/to/me/make_lastz_chains/standalone_scripts/run_lastz_intermediate_layer.py
* found chain_gap_filler.py at /path/to/me/make_lastz_chains/standalone_scripts/chain_gap_filler.py
* found faToTwoBit at /path/to/me/make_lastz_chains/HL_kent_binaries/faToTwoBit
* found twoBitToFa at /path/to/me/make_lastz_chains/HL_kent_binaries/twoBitToFa
* found pslSortAcc at /path/to/me/make_lastz_chains/HL_kent_binaries/pslSortAcc
* found axtChain at /path/to/me/make_lastz_chains/HL_kent_binaries/axtChain
* found axtToPsl at /path/to/me/make_lastz_chains/HL_kent_binaries/axtToPsl
* found chainAntiRepeat at /path/to/me/make_lastz_chains/HL_kent_binaries/chainAntiRepeat
* found chainMergeSort at /path/to/me/make_lastz_chains/HL_kent_binaries/chainMergeSort
* found chainCleaner at /path/to/me/make_lastz_chains/HL_kent_binaries/chainCleaner
* found chainSort at /path/to/me/make_lastz_chains/HL_kent_binaries/chainSort
* found chainScore at /path/to/me/make_lastz_chains/HL_kent_binaries/chainScore
* found chainNet at /path/to/me/make_lastz_chains/HL_kent_binaries/chainNet
* found chainFilter at /path/to/me/make_lastz_chains/HL_kent_binaries/chainFilter
* found lastz at /path/to/me/make_lastz_chains/HL_kent_binaries/lastz
* found nextflow at /burg/opt/nextflow/23.10.0/nextflow
All necessary executables found.
Making chains for test_data/test_reference.fa and test_data/test_query.fa files, saving results to /path/to/me/make_lastz_chains/test_out
Pipeline started at 2023-12-14 10:35:02.145466
* Setting up genome sequences for target
genomeID: target
input sequence file: test_data/test_reference.fa
is 2bit: False
planned genome dir location: /path/to/me/make_lastz_chains/test_out/target.2bit
Initial fasta file test_data/test_reference.fa saved to /path/to/me/make_lastz_chains/test_out/target.2bit
For target (target) sequence file: /path/to/me/make_lastz_chains/test_out/target.2bit; chrom sizes saved to: /path/to/me/make_lastz_chains/test_out/target.chrom.sizes
* Setting up genome sequences for query
genomeID: query
input sequence file: test_data/test_query.fa
is 2bit: False
planned genome dir location: /path/to/me/make_lastz_chains/test_out/query.2bit
Initial fasta file test_data/test_query.fa saved to /path/to/me/make_lastz_chains/test_out/query.2bit
For query (query) sequence file: /path/to/me/make_lastz_chains/test_out/query.2bit; chrom sizes saved to: /path/to/me/make_lastz_chains/test_out/query.chrom.sizes

### Partition Step ###

# Partitioning for target
Saving partitions and creating 1 buckets for lastz output
In particular, 0 partitions for bigger chromosomes
And 1 buckets for smaller scaffolds
Saving target partitions to: /path/to/me/make_lastz_chains/test_out/target_partitions.txt
# Partitioning for query
Saving partitions and creating 1 buckets for lastz output
In particular, 0 partitions for bigger chromosomes
And 1 buckets for smaller scaffolds
Saving query partitions to: /path/to/me/make_lastz_chains/test_out/query_partitions.txt
Num. target partitions: 0
Num. query partitions: 0
Num. lastz jobs: 0

### Lastz Alignment Step ###

LASTZ: making jobs
LASTZ: saved 1 jobs to /path/to/me/make_lastz_chains/test_out/temp_lastz_run/lastz_joblist.txt
Parallel manager: pushing job /burg/opt/nextflow/23.10.0/nextflow /path/to/me/make_lastz_chains/parallelization/execute_joblist.nf --joblist /path/to/me/make_lastz_chains/test_out/temp_lastz_run/lastz_joblist.txt -c /path/to/me/make_lastz_chains/test_out/temp_lastz_run/lastz_config.nf
N E X T F L O W  ~  version 23.10.0
Launching `/path/to/me/make_lastz_chains/parallelization/execute_joblist.nf` [boring_faggin] DSL2 - revision: 0483b29723
executor >  local (1)
[c0/1be141] process > execute_jobs (1) [100%] 1 of 1 ✔


### Nextflow process lastz finished successfully
Found 1 output files from the LASTZ step
Please note that lastz_step.py does not produce output in case LASTZ could not find any alignment

### Concatenating Lastz Results (Cat) Step ###

Concatenating LASTZ output from 1 buckets
* skip bucket bucket_ref_bulk_1: nothing to concat
An error occurred while executing cat: Error! No non-empty files found at /path/to/me/make_lastz_chains/test_out/temp_concat_lastz_output. The failed operation label is: cat_step
Traceback (most recent call last):
  File "/path/to/me/make_lastz_chains/modules/step_manager.py", line 70, in execute_steps
    step_result = step_to_function[step](params, project_paths, step_executables)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/me/make_lastz_chains/modules/pipeline_steps.py", line 58, in cat_step
    do_cat(params, project_paths, executables)
  File "/path/to/me/make_lastz_chains/steps_implementations/cat_step.py", line 51, in do_cat
    has_non_empty_file(project_paths.cat_out_dirname, "cat_step")
  File "/path/to/me/make_lastz_chains/modules/common.py", line 51, in has_non_empty_file
    raise PipelineFileNotFoundError(err_msg)
modules.error_classes.PipelineFileNotFoundError: Error! No non-empty files found at /path/to/me/make_lastz_chains/test_out/temp_concat_lastz_output. The failed operation label is: cat_step

ls -l test_out/temp_concat_lastz_output
total 0

[Iris-QAQ]

I've encountered similar bugs in the past. To resolve the issue, I specified the path to Python 3.11 in the shebang line at the top of each *.py script. For example, I changed the first line to #!/path/to/python3.11, ensuring that this line was added regardless of whether a path was previously present. This adjustment seems to have resolved the problems for me.
I hope this information helps anyone else experiencing the same issues.

[MichaelHiller]

Thx a lot for reporting this.
@kirilenkobm Would that be an easy fix that is generally stable and applicable? These python version issues are annoying :-)