Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quantification step fails with skimage.morphology.convex_hull ModuleNotFoundError error #565

Open
emir-radkevich opened this issue Nov 23, 2024 · 22 comments

Comments

@emir-radkevich
Copy link

emir-radkevich commented Nov 23, 2024

Hi.

I am using mcmicro on our lab's image from Orion machine.
I am using your pipeline on HPC starting at segmentation. So far, unmicst and s3seg are working.

I am getting this error during quantification step though:

Command executed:

  shopt -s nullglob
  mcquant --image HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif        --masks cell*.tif *_cp_masks*.tif --output . --channel_names markers.csv

Command exit status:
  1

Command output:
  {'masks': ['cell.ome.tif'], 'image': 'HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif', 'channel_names': 'markers.csv', 'output': '.', 'intensity_props': {'intensity_mean'}, 'mask_props': None}
  Extracting single-cell data for HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif...
  Finished 0
  Finished 1
  Finished 2
  Finished 3
  Finished 4
  Finished 5
  Finished 6
  Finished 7
  Finished 8
  Finished 9
  Finished 10
  Finished 11
  Finished 12
  Finished 13
  Finished 14
  Finished 15
  Finished 16
  Finished 17
  Finished 18
  Finished 19

Command error:
  WARNING: Could not find any nv files on this host!
  INFO:    Converting SIF file to temporary sandbox...
  Traceback (most recent call last):
    File "/usr/local/bin/mcquant", line 8, in <module>
    File "/usr/local/lib/python3.11/site-packages/mcquant/cli.py", line 11, in main
    File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 272, in MultiExtractSingleCells
    File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 245, in ExtractSingleCells
    File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 194, in MaskZstack
    File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 62, in MaskIDs
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 1097, in regionprops_table
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 882, in _props_to_dict
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 666, in solidity
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 242, in wrapper
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 452, in area_convex
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 242, in wrapper
    File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 457, in image_convex
  ModuleNotFoundError: No module named 'skimage.morphology.convex_hull'
  INFO:    Cleaning up image...

I tried to look into the error inside the contatiner:

>singularity shell work/singularity/labsyspharm-quantification-1.6.0.img

Singularity> cat /usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py | grep -n convex_hull
457:        from ..morphology.convex_hull import convex_hull_image
459:        return convex_hull_image(self.image)
499:        identity_convex_hull = np.pad(
504:                find_contours(identity_convex_hull, 0.5, fully_connected='high')
507:            coordinates, _, _, _ = marching_cubes(identity_convex_hull, level=0.5)

Singularity> ls /usr/local/lib/python3.11/site-packages/skimage/morphology/ | grep convex
_convex_hull.cpython-311-x86_64-linux-gnu.so
convex_hull.py

Looks like this module exists.
Do not understand the error.

Could you please help me?

@jmuhlich
Copy link
Contributor

What kind of computer and operating system are you running mcmicro on?

@jmuhlich
Copy link
Contributor

jmuhlich commented Nov 23, 2024

I re-read your report and I see now you said "HPC" so I figure it's x86_64 machines running Linux, but it would still be good to verify that.

@emir-radkevich
Copy link
Author

I re-read your report and I see now you said "HPC" so I figure it's x86_64 machines running Linux, but it would still be good to verify that.

Yes, you are right.

@ArtemSokolov
Copy link
Collaborator

Hi @emir-radkevich,

That's a super weird error message. We suspect it may have to do with how the container got converted to Singularity.

Do you get this error with every image you try to process or just that one? Also, do you get the error with exemplar-001?

@emir-radkevich
Copy link
Author

Hi @emir-radkevich,

That's a super weird error message. We suspect it may have to do with how the container got converted to Singularity.

Do you get this error with every image you try to process or just that one? Also, do you get the error with exemplar-001?

No, exemplar-001 ran smoothly.

@ArtemSokolov
Copy link
Collaborator

Would you be able to share a small image that's giving you issues, so we can try to reproduce the issue on our end?

@emir-radkevich
Copy link
Author

Do you mean slide image or singularity image [work/singularity/labsyspharm-quantification-1.6.0.img]?
Slide is very big, 267 GB.

@ArtemSokolov
Copy link
Collaborator

ArtemSokolov commented Nov 23, 2024

The slide image, but ideally a small (<1GB) crop from it that still gives you the issue.

If you are not getting an issue with exemplar-001, then there is something different about your data that is triggering this error. If we can reproduce the issue on our end, we can help figure out what the problem is.

@jmuhlich
Copy link
Contributor

The nextflow output should have mentioned a path within your work directory that ends with a long alphanumeric string. Could you zip up all the files in that directory whose names begin with .command (they are all very small) and upload that zip file here? Those files may help us diagnose the issue better.

@emir-radkevich
Copy link
Author

The nextflow output should have mentioned a path within your work directory that ends with a long alphanumeric string. Could you zip up all the files in that directory whose names begin with .command (they are all very small) and upload that zip file here? Those files may help us diagnose the issue better.

commands.zip

@emir-radkevich
Copy link
Author

emir-radkevich commented Nov 23, 2024

The slide image, but ideally a small (<1GB) crop from it that still gives you the issue.

If you are not getting an issue with exemplar-001, then there is something different about your data that is triggering this error. If we can reproduce the issue on our end, we can help figure out what the problem is.

I cropped the slide to a very small piece (<200 MB). Pipeline starting from segmentation or quantification works.
I re-ran the whole slide starting from quantification and received the same issue described in the subject.

I noticed something weird -- the whole slide is pink when loading in QuPath-0.5.1, the cropped image is also pinkish. See screenshots attached.

Screenshot 2024-11-23 at 1 53 12 PM Screenshot 2024-11-23 at 1 54 13 PM

@jmuhlich
Copy link
Contributor

Could you try starting an interactive job on the HPC system with 48GB of memory and running the following command in it to run the quantification step in Singularity directly?
bash /sc/arion/projects/HIMC/software/mcmicro_orion/work/47/398663ebda873a13b8165763e5b3b5/.command.run
It will take around 30 minutes to run based on the logs you shared. Please report back whether you get the same ModuleNotFoundError message or not.

@emir-radkevich
Copy link
Author

Could you try starting an interactive job on the HPC system with 48GB of memory and running the following command in it to run the quantification step in Singularity directly? bash /sc/arion/projects/HIMC/software/mcmicro_orion/work/47/398663ebda873a13b8165763e5b3b5/.command.run It will take around 30 minutes to run based on the logs you shared. Please report back whether you get the same ModuleNotFoundError message or not.

(base) [radkee01@lg03a01 mcmicro_orion]$ bash /sc/arion/projects/HIMC/software/mcmicro_orion/work/47/398663ebda873a13b8165763e5b3b5/.command.run
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (517) bind mounts
{'masks': ['cell.ome.tif'], 'image': 'HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif', 'channel_names': 'markers.csv', 'output': '.', 'intensity_props': {'intensity_mean'}, 'mask_props': None}
Extracting single-cell data for HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif...
Finished 0
Finished 1
Finished 2
Finished 3
Finished 4
Finished 5
Finished 6
Finished 7
Finished 8
Finished 9
Finished 10
Finished 11
Finished 12
Finished 13
Finished 14
Finished 15
Finished 16
Finished 17
Finished 18
Finished 19
Traceback (most recent call last):
  File "/usr/local/bin/mcquant", line 8, in <module>
  File "/usr/local/lib/python3.11/site-packages/mcquant/cli.py", line 11, in main
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 272, in MultiExtractSingleCells
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 258, in ExtractSingleCells
  File "/usr/local/lib/python3.11/site-packages/pandas/util/_decorators.py", line 333, in wrapper
  File "/usr/local/lib/python3.11/site-packages/pandas/core/generic.py", line 3967, in to_csv
  File "/usr/local/lib/python3.11/site-packages/pandas/io/formats/format.py", line 987, in to_csv
ModuleNotFoundError: No module named 'pandas.io.formats.csvs'
INFO:    Cleaning up image...

@jmuhlich
Copy link
Contributor

Can you try running that same command one more time? I am wondering if it will show the same exact error, or something different. I can't figure out how you would see different ModuleNotFoundError messages on different runs when the container didn't change.

@emir-radkevich
Copy link
Author

Can you try running that same command one more time? I am wondering if it will show the same exact error, or something different. I can't figure out how you would see different ModuleNotFoundError messages on different runs when the container didn't change.

(base) [radkee01@lg03a01 mcmicro_orion]$ bash /sc/arion/projects/HIMC/software/mcmicro_orion/work/47/398663ebda873a13b8165763e5b3b5/.command.run
INFO:    Converting SIF file to temporary sandbox...
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (517) bind mounts
{'masks': ['cell.ome.tif'], 'image': 'HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif', 'channel_names': 'markers.csv', 'output': '.', 'intensity_props': {'intensity_mean'}, 'mask_props': None}
Extracting single-cell data for HIMC_52231_Ovarian_200x_MSSM_EM_Q143_BV_Test_000226--unmicst.ome.tif...
Finished 0
Finished 1
Finished 2
Finished 3
Finished 4
Finished 5
Finished 6
Finished 7
Finished 8
Finished 9
Finished 10
Finished 11
Finished 12
Finished 13
Finished 14
Finished 15
Finished 16
Finished 17
Finished 18
Finished 19
Traceback (most recent call last):
  File "/usr/local/bin/mcquant", line 8, in <module>
  File "/usr/local/lib/python3.11/site-packages/mcquant/cli.py", line 11, in main
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 272, in MultiExtractSingleCells
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 245, in ExtractSingleCells
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 194, in MaskZstack
  File "/usr/local/lib/python3.11/site-packages/mcquant/SingleCellDataExtraction.py", line 62, in MaskIDs
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 1097, in regionprops_table
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 882, in _props_to_dict
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 666, in solidity
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 242, in wrapper
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 452, in area_convex
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 242, in wrapper
  File "/usr/local/lib/python3.11/site-packages/skimage/measure/_regionprops.py", line 457, in image_convex
ModuleNotFoundError: No module named 'skimage.morphology.convex_hull'
INFO:    Cleaning up image...

@jmuhlich
Copy link
Contributor

Please re-run mcmicro on the whole slide with --start-at quantification and also add the following arguments to the nextflow run command line: -e.PYTHONVERBOSE=2 -e.PYTHONUNBUFFERED=1 . Then, upload the .command.log file from the work subdirectory for the quantification process. You can get the exact subdirectory path with this command:

nextflow log last -f workdir -F 'name =~ /quant.*/'

The log file will include information on how python is locating files for import which should help us learn more about what's going on.

@emir-radkevich
Copy link
Author

Please re-run mcmicro on the whole slide with --start-at quantification and also add the following arguments to the nextflow run command line: -e.PYTHONVERBOSE=2 -e.PYTHONUNBUFFERED=1 . Then, upload the .command.log file from the work subdirectory for the quantification process. You can get the exact subdirectory path with this command:

nextflow log last -f workdir -F 'name =~ /quant.*/'

The log file will include information on how python is locating files for import which should help us learn more about what's going on.

logfile.log

@jmuhlich
Copy link
Contributor

The log message "INFO: Converting SIF file to temporary sandbox..." indicates that singularity isn't running the .img file directly but rather expanding it into a temporary directory (within /tmp by default) first. This is apparently the mode of operation for certain singularity installations. If there's not enough space in the temporary location, I'm not sure what would happen.

I reached out to a colleague at Mount Sinai who has run mcmicro successfully on Minerva, and he said he can only run a couple of modules, then has to remove work/singularity/ as well as ~/.singularity/cache to free up space, then run the remaining modules. He was also getting strange unexplainable errors that seemed to stem from files missing inside the container.

@jmuhlich
Copy link
Contributor

The logfile.log that you uploaded indicates there are files missing from the .../skimage/morphology/ directory. Could you run this command to enter the container with explicit sandbox mode:
singularity shell --unsquash work/singularity/labsyspharm-quantification-1.6.0.img
Then run this command and post the output here:
ls -l /usr/local/lib/python3.11/site-packages/skimage/morphology{,/__pycache__}

@emir-radkevich
Copy link
Author

The logfile.log that you uploaded indicates there are files missing from the .../skimage/morphology/ directory. Could you run this command to enter the container with explicit sandbox mode: singularity shell --unsquash work/singularity/labsyspharm-quantification-1.6.0.img Then run this command and post the output here: ls -l /usr/local/lib/python3.11/site-packages/skimage/morphology{,/__pycache__}

(base) [radkee01@li04e03 mcmicro_orion]$ singularity shell work/singularity/labsyspharm-quantification-1.6.0.img
INFO:    Converting SIF file to temporary sandbox...
Singularity> ls -l /usr/local/lib/python3.11/site-packages/skimage/morphology{,/__pycache__}
/usr/local/lib/python3.11/site-packages/skimage/morphology:
total 3523
-rw-r--r-- 1 radkee01 nogroup   2288 Aug  9 11:36 __init__.py
drwxr-xr-x 2 radkee01 nogroup   4096 Aug  9 11:36 __pycache__
-rwxr-xr-x 1 radkee01 nogroup 236456 Aug  9 11:36 _convex_hull.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 radkee01 nogroup 338824 Aug  9 11:36 _extrema_cy.cpython-311-x86_64-linux-gnu.so
-rw-r--r-- 1 radkee01 nogroup  10733 Aug  9 11:36 _flood_fill.py
-rwxr-xr-x 1 radkee01 nogroup 414448 Aug  9 11:36 _flood_fill_cy.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 radkee01 nogroup 290512 Aug  9 11:36 _grayreconstruct.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 radkee01 nogroup 928360 Aug  9 11:36 _max_tree.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 radkee01 nogroup 337536 Aug  9 11:36 _misc_cy.cpython-311-x86_64-linux-gnu.so
-rw-r--r-- 1 radkee01 nogroup  23548 Aug  9 11:36 _skeletonize.py
-rwxr-xr-x 1 radkee01 nogroup 254584 Aug  9 11:36 _skeletonize_3d_cy.cpython-311-x86_64-linux-gnu.so
-rwxr-xr-x 1 radkee01 nogroup 259648 Aug  9 11:36 _skeletonize_cy.cpython-311-x86_64-linux-gnu.so
-rw-r--r-- 1 radkee01 nogroup  12019 Aug  9 11:36 _util.py
-rw-r--r-- 1 radkee01 nogroup    431 Aug  9 11:36 ball_decompositions.npy
-rw-r--r-- 1 radkee01 nogroup  12112 Aug  9 11:36 binary.py
-rw-r--r-- 1 radkee01 nogroup   8419 Aug  9 11:36 convex_hull.py
-rw-r--r-- 1 radkee01 nogroup    881 Aug  9 11:36 disk_decompositions.npy
-rw-r--r-- 1 radkee01 nogroup  20797 Aug  9 11:36 extrema.py
-rw-r--r-- 1 radkee01 nogroup  40077 Aug  9 11:36 footprints.py
-rw-r--r-- 1 radkee01 nogroup  25742 Aug  9 11:36 gray.py
-rw-r--r-- 1 radkee01 nogroup   9350 Aug  9 11:36 grayreconstruct.py
-rw-r--r-- 1 radkee01 nogroup   7879 Aug  9 11:36 isotropic.py
-rw-r--r-- 1 radkee01 nogroup  26983 Aug  9 11:36 max_tree.py
-rw-r--r-- 1 radkee01 nogroup  16678 Aug  9 11:36 misc.py
drwxr-xr-x 3 radkee01 nogroup   4096 Aug  9 11:36 tests

/usr/local/lib/python3.11/site-packages/skimage/morphology/__pycache__:
total 417
-rw-r--r-- 1 radkee01 nogroup  2667 Aug  9 11:36 __init__.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 11982 Aug  9 11:36 _flood_fill.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 29424 Aug  9 11:36 _skeletonize.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 14424 Aug  9 11:36 _util.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 12638 Aug  9 11:36 binary.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup  9712 Aug  9 11:36 convex_hull.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 21129 Aug  9 11:36 extrema.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 47527 Aug  9 11:36 footprints.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 27842 Aug  9 11:36 gray.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 11911 Aug  9 11:36 grayreconstruct.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup  8625 Aug  9 11:36 isotropic.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 28970 Aug  9 11:36 max_tree.cpython-311.pyc
-rw-r--r-- 1 radkee01 nogroup 18063 Aug  9 11:36 misc.cpython-311.pyc

@emir-radkevich
Copy link
Author

The log message "INFO: Converting SIF file to temporary sandbox..." indicates that singularity isn't running the .img file directly but rather expanding it into a temporary directory (within /tmp by default) first. This is apparently the mode of operation for certain singularity installations. If there's not enough space in the temporary location, I'm not sure what would happen.

I reached out to a colleague at Mount Sinai who has run mcmicro successfully on Minerva, and he said he can only run a couple of modules, then has to remove work/singularity/ as well as ~/.singularity/cache to free up space, then run the remaining modules. He was also getting strange unexplainable errors that seemed to stem from files missing inside the container.

I removed images from work/singularity/ and cache from custom path, where I store singularity cache singularity_layers_emir/cache/, not ~/.singularity/cache, because I have only 20 GB in my home directory.

After that, I got the same error as in this thread's subject. .command.log is attached:
command3.log

@jmuhlich
Copy link
Contributor

Could you share your submit script so I could compare with the one from my MSKCC colleague? Here is the body of his:

module purge
module load proxies
module load java/11.0.2
module load singularity/3.6.4

$HOME/bin/nextflow run labsyspharm/mcmicro -profile singularity -w /sc/arion/scratch/.../work --in /sc/arion/scratch/... 

(I don't think any of the #BSUB option lines would matter so I omitted those)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants