Failed to find loop device[BUG]

[Zifeng-L]

Describe the bug
I tried the nextflow with singularity and there were something wrong with nextflow.container.SingularityCache.

To Reproduce
Steps to reproduce the behavior:

1. Configure with these options:

manifest {
   name = 'vib-singlecell-nf/vsn-pipelines'
   description = 'A repository of pipelines for single-cell data in Nextflow DSL2'
   homePage = 'https://github.com/vib-singlecell-nf/vsn-pipelines'
   version = '0.26.1'
   mainScript = 'main.nf'
   defaultBranch = 'master'
   nextflowVersion = '!>=20.10.0'
}

params {
   global {
      project_name = '10x_PBMC'
      outdir = 'out'
   }
   misc {
      test {
         enabled = false
      }
   }
   utils {
      container = 'vibsinglecellnf/utils:0.4.0'
      publish {
         compressionLevel = 6
         annotateWithBatchVariableName = false
         mode = 'link'
      }
   }
   sc {
      file_converter {
         off = 'h5ad'
         tagCellWithSampleId = true
         remove10xGEMWell = false
         useFilteredMatrix = true
         makeVarIndexUnique = false
      }
      scanpy {
         container = 'vibsinglecellnf/scanpy:0.5.2'
         report {
            annotations_to_plot = []
         }
         feature_selection {
            report_ipynb = '/src/scanpy/bin/reports/sc_select_variable_genes_report.ipynb'
            method = 'mean_disp_plot'
            minMean = 0.0125
            maxMean = 3
            minDisp = 0.5
            off = 'h5ad'
         }
         feature_scaling {
            method = 'zscore_scale'
            maxSD = 10
            off = 'h5ad'
         }
         neighborhood_graph {
            nPcs = 50
            off = 'h5ad'
         }
         dim_reduction {
            report_ipynb = '/src/scanpy/bin/reports/sc_dim_reduction_report.ipynb'
            pca {
               method = 'pca'
               nComps = 50
               off = 'h5ad'
            }
            umap {
               method = 'umap'
               off = 'h5ad'
            }
            tsne {
               method = 'tsne'
               off = 'h5ad'
            }
         }
         clustering {
            preflight_checks = true
            report_ipynb = '/src/scanpy/bin/reports/sc_clustering_report.ipynb'
            method = 'louvain'
            resolution = 0.8
            off = 'h5ad'
         }
         marker_genes {
            method = 'wilcoxon'
            ngenes = 0
            groupby = 'louvain'
            off = 'h5ad'
         }
         filter {
            report_ipynb = '/src/scanpy/bin/reports/sc_filter_qc_report.ipynb'
            cellFilterStrategy = 'fixedthresholds'
            cellFilterMinNGenes = 200
            cellFilterMaxNGenes = 4000
            cellFilterMaxPercentMito = 0.15
            geneFilterMinNCells = 3
            off = 'h5ad'
            outdir = 'out'
         }
         data_transformation {
            method = 'log1p'
            off = 'h5ad'
         }
         normalization {
            method = 'cpx'
            countsPerCellAfter = 10000
            off = 'h5ad'
         }
      }
      scope {
         genome = ''
         tree {
            level_1 = ''
            level_2 = ''
            level_3 = ''
         }
      }
   }
   data {
      tenx {
         cellranger_mex = 'data/10x/1k_pbmc/1k_pbmc_*/outs/'
      }
   }
}

process {
   executor = 'local'
   cpus = 2
   memory = '60 GB'
   clusterOptions = '-A cluster_account'
   withLabel:compute_resources__default {
      time = '1h'
   }
   withLabel:compute_resources__minimal {
      cpus = 1
      memory = '1 GB'
   }
   withLabel:compute_resources__mem {
      cpus = 4
      memory = '160 GB'
   }
   withLabel:compute_resources__cpu {
      cpus = 20
      memory = '80 GB'
   }
   withLabel:compute_resources__report {
      maxForks = 2
      cpus = 1
      memory = '160 GB'
   }
   withLabel:compute_resources__24hqueue {
      time = '24h'
   }
}

timeline {
   enabled = true
   file = 'out/nextflow_reports/execution_timeline.html'
}

report {
   enabled = true
   file = 'out/nextflow_reports/execution_report.html'
}

trace {
   enabled = true
   file = 'out/nextflow_reports/execution_trace.txt'
}

dag {
   enabled = true
   file = 'out/nextflow_reports/pipeline_dag.svg'
}

min {
   enabled = false
}

vsc {
   enabled = false
}

singularity {
   enabled = true
   autoMounts = true
   runOptions = '--cleanenv -H $PWD -B ${HOME}'
}

...

2. Run using this entry point:

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=2-00:00:00
#SBATCH --comment project_name

#################################################################
source activate scnf
export NXF_SINGULARITY_CACHEDIR=/home/u21111240011/.singularity
cd /home/u21111240011/test/single_sample_test_1
nextflow -C single_sample.config \
    run vib-singlecell-nf/vsn-pipelines \
        -entry single_sample

3. See error:

[1k_pbmc_v3_chemistry, /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v3_chemistry/outs, 10x_cellranger_mex_outs, h5ad, NULL]
[1k_pbmc_v2_chemistry, /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v2_chemistry/outs, 10x_cellranger_mex_outs, h5ad, NULL]
Error executing process > 'single_sample:SINGLE_SAMPLE:SC__FILE_CONVERTER (2)'

Caused by:
  Process `single_sample:SINGLE_SAMPLE:SC__FILE_CONVERTER (2)` terminated with an error exit status (255)

Command executed:

  /home/u21111240011/.nextflow/assets/vib-singlecell-nf/vsn-pipelines/src/utils/bin/sc_file_converter.py             --sample-id "1k_pbmc_v2_chemistry"                                                    --tag-cell-with-sample-id true                                       --input-format 10x_cellranger_mex             --output-format h5ad             /home/u21111240011/test/single_sample_test_1/data/10x/1k_pbmc/1k_pbmc_v2_chemistry/outs/filtered_feature_bc_matrix             "1k_pbmc_v2_chemistry.SC__FILE_CONVERTER.h5ad"

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

Work dir:
  /home/u21111240011/test/single_sample_test_1/work/60/045b8d1ad8acd214dd3af8f2bcb582

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`


WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

Expected behavior
It seemed something wrong with singularity. When I tested it in local, it worked well. When I submit it into slurm system, it can not work.
Screenshots
If applicable, add screenshots to help explain your problem.

Please complete the following information:

• OS: Linux workstation 4.18.0-80.7.1.el8_0.x86_64
• Nextflow Version: 21.04.0
• vsn-pipelines Version: 962111a