Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve predictability of DataQuery, DataID, and dependency tree #3018

Draft
wants to merge 22 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5db7d99
Cleanup dependency tree tests
djhoese Dec 3, 2024
66b714a
Update dep tree tests to be more realistic
djhoese Dec 3, 2024
cab16b9
Convert combine metadata tests to pytest
djhoese Dec 3, 2024
1e88269
Fix inconsistency with DataID "resolution" transitive property
djhoese Dec 7, 2024
dd46a4e
Convert dataquery tests to pytest
djhoese Dec 9, 2024
251524b
Fix DataQuery equality to require query keys to match
djhoese Dec 9, 2024
ef07c55
Fix inconsistency with DataID "resolution" transitive property
djhoese Dec 9, 2024
438de2c
Split data query tests to be more explicit
djhoese Dec 9, 2024
ce01aa6
Cleanup satpy internals documentation regarding DataQuery equality
djhoese Dec 9, 2024
ed2867a
Change DataQuery equality to require all query keys to be equal
djhoese Dec 13, 2024
d223f10
Add test querying for a wavelength on no-wavelength DataIDs
djhoese Dec 16, 2024
aba5dc5
Merge combine times test cases
djhoese Dec 16, 2024
24a1068
Refactor ID key types to separate module
djhoese Dec 16, 2024
2085b50
Extract ID update with query function
djhoese Dec 16, 2024
6afc26d
Remove unused _match_query_value method
djhoese Dec 16, 2024
633802f
Add shared_key option to DataQuery equality checks
djhoese Dec 17, 2024
0f77b4e
Refactor get_keys_from_config for simpler conditionals
djhoese Dec 17, 2024
67cb550
Attempt to refactor DataQuery equality
djhoese Dec 17, 2024
52469d4
Refactor DataQuery equality checks
djhoese Dec 17, 2024
68880a4
Another try at making CodeScene happy with get_keys_from_config
djhoese Dec 17, 2024
de36fb8
Remove duplicated test code in test_satpy_cf_nc.py
djhoese Dec 17, 2024
c45ed8d
Remove accidental holoviews import in dependency tree
djhoese Dec 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions doc/source/dev_guide/satpy_internals.rst
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,9 @@ DataID and DataQuery interactions
Different DataIDs and DataQuerys can have different metadata items defined. As such
we define equality between different instances of these classes, and across the classes
as equality between the sorted key/value pairs shared between the instances.
If a DataQuery has one or more values set to `'*'`, the corresponding key/value pair will be omitted from the comparison.
Instances sharing no keys will no be equal.
If a DataQuery has one or more values set to ``'*'``, the corresponding key/value pair will be omitted from the comparison.
If any key in a DataQuery (not equal to ``"*"``) is missing from a DataID they
are considered not equal.


Breaking changes from DatasetIDs
Expand All @@ -151,8 +152,9 @@ Breaking changes from DatasetIDs
Creating DataID for tests
=========================

Sometimes, it is useful to create `DataID` instances for testing purposes. For these cases, the `satpy.tests.utils` module
now has a `make_dsid` function that can be used just for this::
Sometimes, it is useful to create ``DataID`` instances for testing purposes.
For these cases, the ``satpy.tests.utils`` module now has a ``make_dataid``
function that can be used just for this::

from satpy.tests.utils import make_dataid
did = make_dataid(name='camembert', modifiers=('runny',))
2 changes: 1 addition & 1 deletion satpy/composites/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
import satpy
from satpy.aux_download import DataDownloadMixin
from satpy.dataset import DataID, combine_metadata
from satpy.dataset.dataid import minimal_default_keys_config
from satpy.dataset.id_keys import minimal_default_keys_config
from satpy.utils import unify_chunks
from satpy.writers import get_enhanced_image

Expand Down
2 changes: 1 addition & 1 deletion satpy/composites/config_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
import satpy
from satpy import DataID, DataQuery
from satpy._config import config_search_paths, get_entry_points_config_dirs, glob_config
from satpy.dataset.dataid import minimal_default_keys_config
from satpy.dataset.id_keys import minimal_default_keys_config
from satpy.utils import recursive_dict_update

logger = logging.getLogger(__name__)
Expand Down
3 changes: 2 additions & 1 deletion satpy/dataset/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,6 @@

from .anc_vars import dataset_walker, replace_anc # noqa
from .data_dict import DatasetDict, get_key # noqa
from .dataid import DataID, DataQuery, ModifierTuple, WavelengthRange, create_filtered_query # noqa
from .dataid import DataID, DataQuery, create_filtered_query # noqa
from .id_keys import ModifierTuple, WavelengthRange # noqa
from .metadata import combine_metadata # noqa
3 changes: 2 additions & 1 deletion satpy/dataset/anc_vars.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""Utilities for dealing with ancillary variables."""

from .dataid import DataID, default_id_keys_config
from .dataid import DataID
from .id_keys import default_id_keys_config


def dataset_walker(datasets):
Expand Down
23 changes: 13 additions & 10 deletions satpy/dataset/data_dict.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@

import numpy as np

from .dataid import DataID, create_filtered_query, minimal_default_keys_config
from .dataid import DataID, create_filtered_query
from .id_keys import minimal_default_keys_config


class TooManyResults(KeyError):
Expand Down Expand Up @@ -143,16 +144,18 @@ def get_key(self, match_key, num_results=1, best=True, **dfilter): # noqa: D417
"""Get multiple fully-specified keys that match the provided query.

Args:
key (DataID): DataID of query parameters to use for
searching. Any parameter that is `None`
is considered a wild card and any match is
accepted. Can also be a string representing the
dataset name or a number representing the dataset
wavelength.
match_key (DataID): DataID of query parameters to use for
searching. Any parameter that is `None`
is considered a wild card and any match is
accepted. Can also be a string representing the
dataset name or a number representing the dataset
wavelength.
num_results (int): Number of results to return. If `0` return all,
if `1` return only that element, otherwise
return a list of matching keys.
**dfilter (dict): See `get_key` function for more information.
if `1` return only that element, otherwise
return a list of matching keys.
best (bool): Sort results to get "best" result first
(default: True). See `get_best_dataset_key` for details.
**dfilter (dict): See :func:`get_key` function for more information.

"""
return get_key(match_key, self.keys(), num_results=num_results,
Expand Down
Loading
Loading