"Unable to open file (file signature not found)" - bm_raster

[ehinman]

Bug Report

Description

I am reviewing the package and giving it a try on a hydrologic unit region in Wisconsin. It appears as though the issue is with the "COLLATING RESULTS" step, and the bm_raster command fails.

Reproducibility

• The bug is reproducible.
• The bug is intermittent.
• The bug occurs only under specific conditions.

Steps to Reproduce

1. List the steps to reproduce the bug.

import os
import datetime

import colorcet as cc
import contextily as cx
import geopandas
import matplotlib.pyplot as plt
import pandas as pd
from bokeh.plotting import figure, output_notebook, show
from bokeh.models import HoverTool, Title
from pynhd import WaterData

from blackmarble.extract import bm_extract
from blackmarble.raster import bm_raster

%load_ext autoreload
%autoreload 2

plt.rcParams["figure.figsize"] = (18, 10)

# An environment variable can obfuscate to secure a secret
import os
 
bearer = os.getenv("BLACKMARBLE_TOKEN")

gdf = WaterData('huc08').byid('huc8',['07090001'])

VNP46A2_20210205 = bm_raster(
    gdf, product_id="VNP46A2", date_range="2024-02-05", bearer=bearer
)
VNP46A2_20210205

3. Be as detailed as possible to help with replication.

Screenshots / Error Messages (if applicable)

---

OSError Traceback (most recent call last)
Cell In[14], line 2
1 # Daily data: raster for February 5, 2021
----> 2 VNP46A2_20210205 = bm_raster(
3 gdf, product_id="VNP46A2", date_range="2024-02-05", bearer=bearer
4 )
5 VNP46A2_20210205

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/pydantic/validate_call_decorator.py:60, in validate_call..validate..wrapper_function(*args, **kwargs)
58 @functools.wraps(function)
59 def wrapper_function(*args, **kwargs):
---> 60 return validate_call_wrapper(*args, **kwargs)

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/pydantic/_internal/_validate_call.py:96, in ValidateCallWrapper.call(self, *args, **kwargs)
95 def call(self, *args: Any, **kwargs: Any) -> Any:
---> 96 res = self.pydantic_validator.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
97 if self.return_pydantic_validator:
98 return self.return_pydantic_validator(res)

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/blackmarble/raster.py:357, in bm_raster(gdf, product_id, date_range, bearer, variable, drop_values_by_quality_flag, check_all_tiles_exist, output_directory, output_skip_if_exists)
351 filenames = _pivot_paths_by_date(pathnames).get(date)
353 try:
354 # Open each GeoTIFF file as a DataArray and store in a list
355 da = [
356 rioxarray.open_rasterio(
--> 357 h5_to_geotiff(
358 f,
359 variable=variable,
360 drop_values_by_quality_flag=drop_values_by_quality_flag,
361 output_directory=d,
362 ),
363 )
364 for f in filenames
365 ]
366 ds = merge_arrays(da)
367 clipped_dataset = ds.rio.clip(
368 gdf.geometry.apply(mapping), gdf.crs, drop=True
369 )

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/blackmarble/raster.py:177, in h5_to_geotiff(f, variable, drop_values_by_quality_flag, output_directory)
174 if variable is None:
175 variable = VARIABLE_DEFAULT.get(product_id)
--> 177 with h5py.File(f, "r") as h5_data:
178 attrs = h5_data.attrs
179 data_field_key = "HDFEOS/GRIDS/VNP_Grid_DNB/Data Fields"

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/h5py/_hl/files.py:561, in File.init(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, **kwds)
552 fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
553 locking, page_buf_size, min_meta_keep, min_raw_keep,
554 alignment_threshold=alignment_threshold,
555 alignment_interval=alignment_interval,
556 meta_block_size=meta_block_size,
557 **kwds)
558 fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
559 fs_persist=fs_persist, fs_threshold=fs_threshold,
560 fs_page_size=fs_page_size)
--> 561 fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
563 if isinstance(libver, tuple):
564 self._libver = libver

File /opt/homebrew/Caskroom/miniforge/base/envs/blackmarblepy-env/lib/python3.12/site-packages/h5py/_hl/files.py:235, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
233 if swmr and swmr_support:
234 flags |= h5f.ACC_SWMR_READ
--> 235 fid = h5f.open(name, flags, fapl=fapl)
236 elif mode == 'r+':
237 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5f.pyx:102, in h5py.h5f.open()

OSError: Unable to open file (file signature not found)

Environment

• Operating System: [e.g., Windows 10, macOS, Linux] macOS
• Browser (if applicable): [e.g., Google Chrome, Firefox, Safari] Edge
• Application Version/Commit: [e.g., v1.2.3, Git commit hash] 2024.8.2.dev3+gff9c83f
• Additional Environment Details: [e.g., hardware specifications, network conditions] I am using a homebrew installed version of miniforge with python 3.12.0.

Additional Context

My mamba environment contains the following:

Package               Version
--------------------- ----------------------
affine                2.4.0
aiodns                3.0.0
aiohappyeyeballs      2.4.3
aiohttp               3.10.10
aiohttp-client-cache  0.12.4
aiosignal             1.3.1
aiosqlite             0.20.0
annotated-types       0.7.0
anyio                 4.6.2.post1
appnope               0.1.4
asttokens             2.4.1
async-retriever       0.18.0
attrs                 24.2.0
backoff               2.2.1
blackmarblepy         2024.8.2.dev3+gff9c83f
bokeh                 3.6.0
bounded-pool-executor 0.0.3
branca                0.7.2
Brotli                1.1.0
cattrs                24.1.2
certifi               2024.8.30
cffi                  1.17.1
cftime                1.6.4
charset-normalizer    3.4.0
click                 8.1.7
click-plugins         1.1.1
cligj                 0.7.2
colorcet              3.1.0
comm                  0.2.2
contextily            1.6.2
contourpy             1.3.0
cycler                0.12.1
cytoolz               1.0.0
debugpy               1.8.7
decorator             5.1.1
defusedxml            0.7.1
exceptiongroup        1.2.2
executing             2.1.0
fiona                 1.10.1
folium                0.18.0
fonttools             4.54.1
frozenlist            1.5.0
GDAL                  3.9.3
geographiclib         2.0
geopandas             1.0.1
geopy                 2.4.1
h11                   0.14.0
h2                    4.1.0
h5py                  3.12.1
hpack                 4.0.0
httpcore              1.0.6
httpx                 0.27.2
hyperframe            6.0.1
idna                  3.10
importlib_metadata    8.5.0
ipykernel             6.29.5
ipython               8.29.0
ipywidgets            8.1.5
itsdangerous          2.2.0
jedi                  0.19.1
Jinja2                3.1.4
joblib                1.4.2
jupyter_client        8.6.3
jupyter_core          5.7.2
jupyterlab_widgets    3.0.13
kiwisolver            1.4.7
lxml                  5.3.0
mapclassify           2.8.1
MarkupSafe            3.0.2
matplotlib            3.9.2
matplotlib-inline     0.1.7
mercantile            1.2.1
multidict             6.1.0
munkres               1.1.4
nest_asyncio          1.6.0
netCDF4               1.7.1
networkx              3.4.2
numpy                 2.1.3
OWSLib                0.32.0
packaging             24.1
pandas                2.2.3
parso                 0.8.4
pexpect               4.9.0
pickleshare           0.7.5
pillow                11.0.0
pip                   24.3.1
platformdirs          4.3.6
pqdm                  0.2.0
prompt_toolkit        3.0.48
propcache             0.2.0
psutil                6.1.0
ptyprocess            0.7.0
pure_eval             0.2.3
pyarrow               18.0.0
pycares               4.0.0
pycparser             2.22
pydantic              2.9.2
pydantic_core         2.23.4
pygeoogc              0.18.0
pygeoutils            0.18.0
Pygments              2.18.0
pynhd                 0.18.0
pyogrio               0.10.0
pyparsing             3.2.0
pyproj                3.7.0
PySocks               1.7.1
python-dateutil       2.9.0
pytz                  2024.1
PyYAML                6.0.2
pyzmq                 26.2.0
rasterio              1.4.2
rasterstats           0.20.0
requests              2.32.3
requests-cache        1.2.1
rioxarray             0.17.0
scikit-learn          1.5.2
scipy                 1.14.1
setuptools            75.3.0
shapely               2.0.6
simplejson            3.19.3
six                   1.16.0
sniffio               1.3.1
snuggs                1.4.7
stack-data            0.6.2
threadpoolctl         3.5.0
toolz                 1.0.0
tornado               6.4.1
tqdm                  4.66.6
traitlets             5.14.3
typing_extensions     4.12.2
tzdata                2024.2
ujson                 5.10.0
unicodedata2          15.1.0
url-normalize         1.4.3
urllib3               2.2.3
wcwidth               0.2.13
wheel                 0.44.0
widgetsnbextension    4.0.13
xarray                2024.10.0
xyzservices           2024.9.0
yarl                  1.16.0
zipp                  3.20.2
zstandard             0.23.0

Possible Fix

If you have suggestions for a fix, please describe them here.

[ehinman]

Whoops, I realized that this is a duplicate of #96. Closing.