Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch default Quantity: AttrSeries → SparseDataArray #23

Open
khaeru opened this issue Feb 7, 2021 · 3 comments
Open

Switch default Quantity: AttrSeries → SparseDataArray #23

khaeru opened this issue Feb 7, 2021 · 3 comments
Labels
discuss Under/solely for discussion enh New feature or request

Comments

@khaeru
Copy link
Owner

khaeru commented Feb 7, 2021

Inherited from iiasa/ixmp#191:

xarray 0.13 includes support for converting pd.DataFrame to a pydata/sparse data structure.
This should mostly obviate the need for the custom AttrSeries class.
A PR should be opened to make the change, test performance, and make any necessary adjustments.

Resources:

As of genno 1.0, all code is tested with both AttrSeries and SparseDataArray to minimize surprises on switching.

#27 should probably be done first.

@khaeru khaeru added the enh New feature or request label Feb 7, 2021
@khaeru
Copy link
Owner Author

khaeru commented Mar 7, 2021

#27 should probably be done first.

While doing #37, it turns out that sparse.COO currently lacks some methods that xarray needs to implement its basic DataArray API. These include:

  • numpy.flip, for xarray.bfill.
  • numpy.nancumprod, for DataArray.cumprod.
  • numpy.pad, for DataArray.shift.

These cases are now noted in the test suite, e.g.:

def test_bfill(self, tri):
"""Test Quantity.bfill()."""
if Quantity._get_class() is SparseDataArray:
pytest.xfail(reason="sparse.COO.flip() not implemented")

genno is not the right place to implement them, so this issue needs to wait on those being implemented upstream.

@khaeru
Copy link
Owner Author

khaeru commented Mar 7, 2021

@khaeru
Copy link
Owner Author

khaeru commented Nov 3, 2021

https://xarray.pydata.org/en/stable/user-guide/duckarrays.html gives a summary of missing operations.

@khaeru khaeru added the discuss Under/solely for discussion label Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Under/solely for discussion enh New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant