-
Notifications
You must be signed in to change notification settings - Fork 604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. #3342
Comments
Same issue here. My HD data structure is similar to 10x Mouse Small Intestine default structure, which contains ['feature_slice.h5', 'metrics_summary.csv', 'probe_set.csv', 'possorted_genome_bam.bam', 'spatial', 'binned_outputs', 'molecule_info.h5', 'possorted_genome_bam.bam.bai', 'web_summary.html', 'cloupe_008um.cloupe'] if there could be any tutorial how to read it and then convert to zarr will be great :> thank you again for this amazing package development! |
Hi @ankshe91 , i tried to directly download 10x Mouse Small Intestine data from their website, and used it as input (remember to unzip some of the .tar.gz files) nsong@gemini-data1:/home/Visium_HD_Mouse_Small_Intestine |
Hi Nina!
Thank you for the update!
But for some reason my output data does not have the sample id on the feature_slice file.
Like this one - Visium_HD_Mouse_Small_Intestine_feature_slice.h5
And it seems that is creating a problem with the reading. It keeps asking me a value for dataset_id.
Iv tried "None", "", and the posisble sample id. But then it says - no such directory found.
I don't know if theres a problem with my output but this is how space ranger delivered it.
Regards
Ankitha
…________________________________
From: Nina-Song ***@***.***>
Sent: 07 November 2024 17:36
To: scverse/scanpy ***@***.***>
Cc: Shetty, Ankitha ***@***.***>; Mention ***@***.***>
Subject: Re: [scverse/scanpy] Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. (Issue #3342)
Hi @ankshe91 , i tried to directly download 10x Mouse Small Intestine data from their website, and used it as input (remember to unzip some of the .tar. gz files) now the visium_hd function works. i guess our previous naming issue causing error.
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi @ankshe91<https://urldefense.com/v3/__https://github.com/ankshe91__;!!LQC6Cpwp!pfpax61KtvNJY1BGgnz7hzQeTh9SbHQwHBbB8NvQCwiDkw_6qWx2vlHPk9FIfcZf2YHqBhbqXLD5RrlZwJUkvsO7QlpHJw$> , i tried to directly download 10x Mouse Small Intestine data from their website, and used it as input (remember to unzip some of the .tar.gz files)
now the visium_hd function works. i guess our previous naming issue causing error.
***@***.***:/home/Visium_HD_Mouse_Small_Intestine $ ls binned_outputs spatial Visium_HD_Mouse_Small_Intestine_cloupe_008um.cloupe Visium_HD_Mouse_Small_Intestine_feature_slice.h5 Visium_HD_Mouse_Small_Intestine_metrics_summary.csv Visium_HD_Mouse_Small_Intestine_molecule_info.h5 Visium_HD_Mouse_Small_Intestine_spatial.tar.gz Visium_HD_Mouse_Small_Intestine_web_summary.html
sdata = spatialdata_io.visium_hd(path_read) sdata SpatialData object ├── Images │ ├── 'Visium_HD_Mouse_Small_Intestine_hires_image': DataArray[cyx] (3, 5575, 6000) │ └── 'Visium_HD_Mouse_Small_Intestine_lowres_image': DataArray[cyx] (3, 558, 600) ├── Shapes │ ├── 'Visium_HD_Mouse_Small_Intestine_square_002um': GeoDataFrame shape: (5479660, 1) (2D shapes) │ ├── 'Visium_HD_Mouse_Small_Intestine_square_008um': GeoDataFrame shape: (351817, 1) (2D shapes) │ └── 'Visium_HD_Mouse_Small_Intestine_square_016um': GeoDataFrame shape: (91033, 1) (2D shapes) └── Tables ├── 'square_002um': AnnData (5479660, 19059) ├── 'square_008um': AnnData (351817, 19059) └── 'square_016um': AnnData (91033, 19059) with coordinate systems: ▸ 'downscaled_hires', with elements: Visium_HD_Mouse_Small_Intestine_hires_image (Images), Visium_HD_Mouse_Small_Intestine_square_002um (Shapes), Visium_HD_Mouse_Small_Intestine_square_008um (Shapes), Visium_HD_Mouse_Small_Intestine_square_016um (Shapes) ▸ 'downscaled_lowres', with elements: Visium_HD_Mouse_Small_Intestine_lowres_image (Images), Visium_HD_Mouse_Small_Intestine_square_002um (Shapes), Visium_HD_Mouse_Small_Intestine_square_008um (Shapes), Visium_HD_Mouse_Small_Intestine_square_016um (Shapes) ▸ 'global', with elements: Visium_HD_Mouse_Small_Intestine_square_002um (Shapes), Visium_HD_Mouse_Small_Intestine_square_008um (Shapes), Visium_HD_Mouse_Small_Intestine_square_016um (Shapes)
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/scverse/scanpy/issues/3342*issuecomment-2463579384__;Iw!!LQC6Cpwp!pfpax61KtvNJY1BGgnz7hzQeTh9SbHQwHBbB8NvQCwiDkw_6qWx2vlHPk9FIfcZf2YHqBhbqXLD5RrlZwJUkvsPYAGfc-w$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BMWIN5F2LG3PSCF3KIW5CJ3Z7QISBAVCNFSM6AAAAABRHZKH2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTGU3TSMZYGQ__;!!LQC6Cpwp!pfpax61KtvNJY1BGgnz7hzQeTh9SbHQwHBbB8NvQCwiDkw_6qWx2vlHPk9FIfcZf2YHqBhbqXLD5RrlZwJUkvsOhbS57yA$>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Hi @ankshe91 , does Visium_HD_Mouse_Small_Intestine demo works in your script? |
I haven't tried the sample dataset yet. But I see that your own dataset also doesn't have the dataset_ids. |
Maybe start with https://www.10xgenomics.com/datasets/visium-hd-cytassist-gene-expression-libraries-of-mouse-intestine (batch download) could be a good idea, i mimic their folder structure and now worked on my own data as well. (the screenshot attached previously was from this demo data not my own data but both of them work now) |
Thank you! |
Hi,
Did they delete the code from github? I can no longer find the code
Regards,
Ankitha Shetty
Post doc - Ansel Lab
UCSF
…________________________________
From: ankshe ***@***.***>
Sent: 07 November 2024 23:58
To: scverse/scanpy ***@***.***>
Cc: Shetty, Ankitha ***@***.***>; Mention ***@***.***>
Subject: Re: [scverse/scanpy] Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. (Issue #3342)
Thank you! I'll try doing that! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <scverse/scanpy/issues/3342/2463990505@ github. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you!
I'll try doing that!
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/scverse/scanpy/issues/3342*issuecomment-2463990505__;Iw!!LQC6Cpwp!tN3aV21pYO1Iw7kf4zHziJpaq95AN3kTMTdZelgX0AABm3tpogecQ2S3criMwM4Q_WrUB92FJbf1DORS8Yj5k1-VjJNqVg$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BMWIN5HRD5IFOLIQEH2MF4DZ7RVIRAVCNFSM6AAAAABRHZKH2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTHE4TANJQGU__;!!LQC6Cpwp!tN3aV21pYO1Iw7kf4zHziJpaq95AN3kTMTdZelgX0AABm3tpogecQ2S3criMwM4Q_WrUB92FJbf1DORS8Yj5k1837xitAQ$>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
@ankshe91 is this the function that you refer to? https://spatialdata.scverse.org/projects/io/en/latest/generated/spatialdata_io.visium_hd.html |
No, there was another code page. Never mind, they took it down.
Thanks for your insights.
I followed you and read in the demo file successfully with no issues.
But my data still does not read in, I made the labelling and format exactly as the demo file.
But now I get a new error saying
OSError: Unable to synchronously open file (file signature not found)
…---------------------------------------------------------------------------
OSError Traceback (most recent call last)
Cell In[19], line 1
----> 1 sdata = spatialdata_io.visium_hd(path_read2)
File ~\miniconda3\envs\myenv\Lib\site-packages\spatialdata_io\readers\visium_hd.py:109, in visium_hd(path, dataset_id, filtered_counts_file, bin_size, bins_as_squares, fullres_image_file, load_all_images, imread_kwargs, image_models_kwargs, anndata_kwargs)
98 def load_image(path: Path, suffix: str, scale_factors: list[int] | None = None) -> None:
99 _load_image(
100 path=path,
101 images=images,
(...)
106 scale_factors=scale_factors,
107 )
--> 109 metadata, hd_layout = _parse_metadata(path, filename_prefix)
110 transform_matrices = _get_transform_matrices(metadata, hd_layout)
111 file_format = hd_layout[VisiumHDKeys.FILE_FORMAT]
File ~\miniconda3\envs\myenv\Lib\site-packages\spatialdata_io\readers\visium_hd.py:407, in _parse_metadata(path, filename_prefix)
406 def _parse_metadata(path: Path, filename_prefix: str) -> tuple[dict[str, Any], dict[str, Any]]:
--> 407 with h5py.File(path / f"{filename_prefix}{VisiumHDKeys.FEATURE_SLICE_FILE.value}", "r") as f5:
408 metadata = json.loads(dict(f5.attrs)[VisiumHDKeys.METADATA_JSON])
409 hd_layout = json.loads(metadata[VisiumHDKeys.HD_LAYOUT_JSON])
File ~\miniconda3\envs\myenv\Lib\site-packages\h5py\_hl\files.py:561, in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, **kwds)
552 fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
553 locking, page_buf_size, min_meta_keep, min_raw_keep,
554 alignment_threshold=alignment_threshold,
555 alignment_interval=alignment_interval,
556 meta_block_size=meta_block_size,
557 **kwds)
558 fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
559 fs_persist=fs_persist, fs_threshold=fs_threshold,
560 fs_page_size=fs_page_size)
--> 561 fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
563 if isinstance(libver, tuple):
564 self._libver = libver
File ~\miniconda3\envs\myenv\Lib\site-packages\h5py\_hl\files.py:235, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
233 if swmr and swmr_support:
234 flags |= h5f.ACC_SWMR_READ
--> 235 fid = h5f.open(name, flags, fapl=fapl)
236 elif mode == 'r+':
237 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
File h5py\\_objects.pyx:54, in h5py._objects.with_phil.wrapper()
File h5py\\_objects.pyx:55, in h5py._objects.with_phil.wrapper()
File h5py\\h5f.pyx:102, in h5py.h5f.open()
OSError: Unable to synchronously open file (file signature not found)
I definitely think there is something wrong with the space ranger output that I have.
what do you think?
Regards,
Ankitha Shetty
Post doc - Ansel Lab
UCSF
________________________________
From: Nina-Song ***@***.***>
Sent: 12 November 2024 15:26
To: scverse/scanpy ***@***.***>
Cc: Shetty, Ankitha ***@***.***>; Mention ***@***.***>
Subject: Re: [scverse/scanpy] Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. (Issue #3342)
@ ankshe91 is this the function that you refer to? https: //spatialdata. scverse. org/projects/io/en/latest/generated/spatialdata_io. visium_hd. html — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
@ankshe91<https://urldefense.com/v3/__https://github.com/ankshe91__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh1hWqeX3A$> is this the function that you refer to? https://spatialdata.scverse.org/projects/io/en/latest/generated/spatialdata_io.visium_hd.html<https://urldefense.com/v3/__https://spatialdata.scverse.org/projects/io/en/latest/generated/spatialdata_io.visium_hd.html__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh0o7_dOSg$>
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/scverse/scanpy/issues/3342*issuecomment-2471832850__;Iw!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh2HiJo3GQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BMWIN5HR5F7Y7E3MJVZRPPT2AKFA7AVCNFSM6AAAAABRHZKH2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRHAZTEOBVGA__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh289KOthg$>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I used exactly the same labeling and file structure now for my own data.
Still no success.
[cid:080f2d7c-e4e8-4c26-8637-bf36ca43de8f]
Regards,
Ankitha Shetty
Post doc - Ansel Lab
UCSF
…________________________________
From: Shetty, Ankitha ***@***.***>
Sent: 12 November 2024 15:36
To: scverse/scanpy ***@***.***>; scverse/scanpy ***@***.***>
Cc: Mention ***@***.***>
Subject: Re: [scverse/scanpy] Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. (Issue #3342)
No, there was another code page. Never mind, they took it down.
Thanks for your insights.
I followed you and read in the demo file successfully with no issues.
But my data still does not read in, I made the labelling and format exactly as the demo file.
But now I get a new error saying
OSError: Unable to synchronously open file (file signature not found)
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
Cell In[19], line 1
----> 1 sdata = spatialdata_io.visium_hd(path_read2)
File ~\miniconda3\envs\myenv\Lib\site-packages\spatialdata_io\readers\visium_hd.py:109, in visium_hd(path, dataset_id, filtered_counts_file, bin_size, bins_as_squares, fullres_image_file, load_all_images, imread_kwargs, image_models_kwargs, anndata_kwargs)
98 def load_image(path: Path, suffix: str, scale_factors: list[int] | None = None) -> None:
99 _load_image(
100 path=path,
101 images=images,
(...)
106 scale_factors=scale_factors,
107 )
--> 109 metadata, hd_layout = _parse_metadata(path, filename_prefix)
110 transform_matrices = _get_transform_matrices(metadata, hd_layout)
111 file_format = hd_layout[VisiumHDKeys.FILE_FORMAT]
File ~\miniconda3\envs\myenv\Lib\site-packages\spatialdata_io\readers\visium_hd.py:407, in _parse_metadata(path, filename_prefix)
406 def _parse_metadata(path: Path, filename_prefix: str) -> tuple[dict[str, Any], dict[str, Any]]:
--> 407 with h5py.File(path / f"{filename_prefix}{VisiumHDKeys.FEATURE_SLICE_FILE.value}", "r") as f5:
408 metadata = json.loads(dict(f5.attrs)[VisiumHDKeys.METADATA_JSON])
409 hd_layout = json.loads(metadata[VisiumHDKeys.HD_LAYOUT_JSON])
File ~\miniconda3\envs\myenv\Lib\site-packages\h5py\_hl\files.py:561, in File.__init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, **kwds)
552 fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
553 locking, page_buf_size, min_meta_keep, min_raw_keep,
554 alignment_threshold=alignment_threshold,
555 alignment_interval=alignment_interval,
556 meta_block_size=meta_block_size,
557 **kwds)
558 fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
559 fs_persist=fs_persist, fs_threshold=fs_threshold,
560 fs_page_size=fs_page_size)
--> 561 fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
563 if isinstance(libver, tuple):
564 self._libver = libver
File ~\miniconda3\envs\myenv\Lib\site-packages\h5py\_hl\files.py:235, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
233 if swmr and swmr_support:
234 flags |= h5f.ACC_SWMR_READ
--> 235 fid = h5f.open(name, flags, fapl=fapl)
236 elif mode == 'r+':
237 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
File h5py\\_objects.pyx:54, in h5py._objects.with_phil.wrapper()
File h5py\\_objects.pyx:55, in h5py._objects.with_phil.wrapper()
File h5py\\h5f.pyx:102, in h5py.h5f.open()
OSError: Unable to synchronously open file (file signature not found)
I definitely think there is something wrong with the space ranger output that I have.
what do you think?
Regards,
Ankitha Shetty
Post doc - Ansel Lab
UCSF
________________________________
From: Nina-Song ***@***.***>
Sent: 12 November 2024 15:26
To: scverse/scanpy ***@***.***>
Cc: Shetty, Ankitha ***@***.***>; Mention ***@***.***>
Subject: Re: [scverse/scanpy] Cannot read Visium HD data using spatialdata-io (Recurrent error). Data is non-zarr format. (Issue #3342)
@ ankshe91 is this the function that you refer to? https: //spatialdata. scverse. org/projects/io/en/latest/generated/spatialdata_io. visium_hd. html — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
@ankshe91<https://urldefense.com/v3/__https://github.com/ankshe91__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh1hWqeX3A$> is this the function that you refer to? https://spatialdata.scverse.org/projects/io/en/latest/generated/spatialdata_io.visium_hd.html<https://urldefense.com/v3/__https://spatialdata.scverse.org/projects/io/en/latest/generated/spatialdata_io.visium_hd.html__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh0o7_dOSg$>
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/scverse/scanpy/issues/3342*issuecomment-2471832850__;Iw!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh2HiJo3GQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BMWIN5HR5F7Y7E3MJVZRPPT2AKFA7AVCNFSM6AAAAABRHZKH2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRHAZTEOBVGA__;!!LQC6Cpwp!uaRwzklHIm-dbojpqlGGnFyMHkxs1GyXeIk9ELZS_Mme0TOkMBhWcbpLpSOQkMybH4XzYAknHJo4rOOFcCxRJh289KOthg$>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Please make sure these conditions are met
What happened?
I have non zarr format Visium HD data.
I tried reading it with sdata = visium_hd(path_read)
it keeps asking me for a dataset_id which is not there in the feature_slice file name or my folder.
Nonetheless, I kept setting it to None or "" or other possible dataset id values.
I cannot find any tech support on the error either.
(I also tried specifying the file path to the different binned folders)
Minimal code sample
path_read = '/Users/DarthRNA/Downloads/1299_1_XS_VHD_v2_outs'
sdata = visium_hd(path_read)
Error output
ValueError Traceback (most recent call last)
Cell In[54], line 1
----> 1 sdata = visium_hd(path_read)
File /Volumes/Ankitha/Conda/miniconda3/envs/myenv/lib/python3.12/site-packages/spatialdata_io/readers/visium_hd.py:95, in visium_hd(path, dataset_id, filtered_counts_file, bin_size, bins_as_squares, fullres_image_file, load_all_images, imread_kwargs, image_models_kwargs, anndata_kwargs)
92 images: dict[str, Any] = {}
94 if dataset_id is None:
---> 95 dataset_id = infer_dataset_id(path)
96 filename_prefix = f"{dataset_id}"
98 def load_image(path: Path, suffix: str, scale_factors: list[int] | None = None) -> None:
File /Volumes/Ankitha/Conda/miniconda3/envs/myenv/lib/python3.12/site-packages/spatialdata_io/readers/visium_hd.py:361, in _infer_dataset_id(path)
359 files = [f for f in os.listdir(path) if os.path.isfile(os.path.join(path, f)) and f.endswith(suffix)]
360 if len(files) == 0 or len(files) > 1:
--> 361 raise ValueError(
362 f"Cannot infer
dataset_id
from the feature slice file in {path}, please passdataset_id
as an argument."363 )
364 return files[0].replace(suffix, "")
ValueError: Cannot infer
dataset_id
from the feature slice file in /Users/DarthRNA/Downloads/1299_1_XS_VHD_v2_outs, please passdataset_id
as an argument.Versions
The text was updated successfully, but these errors were encountered: