Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust for pandas 2.1 #99

Merged
merged 6 commits into from
Aug 30, 2023
Merged

Adjust for pandas 2.1 #99

merged 6 commits into from
Aug 30, 2023

Conversation

khaeru
Copy link
Owner

@khaeru khaeru commented Aug 30, 2023

At iiasa/message-ix-models#117, some tests are failing with:

Traceback
>       assert (
            expected == qty_out.sel(node_loc=node_loc, technology="coal_ppl", year=2022)
        ).all()

message_ix_models/tests/util/test_node.py:106: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../genno/genno/core/attrseries.py:455: in sel
    data = data.droplevel(list(to_drop & set(data.index.names)))
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/generic.py:898: in droplevel
    return self.set_axis(new_labels, axis=axis, copy=None)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/series.py:4962: in set_axis
    return super().set_axis(labels, axis=axis, copy=copy)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/generic.py:745: in set_axis
    return self._set_axis_nocheck(labels, axis, inplace=False, copy=copy)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/generic.py:756: in _set_axis_nocheck
    obj = self.copy(deep=copy and not using_copy_on_write())
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/generic.py:6685: in copy
    return self._constructor_from_mgr(data, axes=data.axes).__finalize__(
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/series.py:589: in _constructor_from_mgr
    return self._constructor(ser, copy=False)
../../genno/genno/core/attrseries.py:130: in __init__
    data, name = Quantity._single_column_df(data, name, copy=kwargs.get("copy"))
../../genno/genno/core/quantity.py:102: in _single_column_df
    data.set_axis(pd.MultiIndex.from_product([data.index]), copy=copy),
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/arrays/categorical.py:449: in __init__
    codes, categories = factorize(values, sort=True)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/algorithms.py:763: in factorize
    return values.factorize(sort=sort, use_na_sentinel=use_na_sentinel)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/base.py:1195: in factorize
    codes, uniques = algorithms.factorize(
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/algorithms.py:795: in factorize
    codes, uniques = factorize_array(
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/algorithms.py:592: in factorize_array
    hash_klass, values = _get_hashtable_algo(values)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/algorithms.py:273: in _get_hashtable_algo
    values = _ensure_data(values)
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/algorithms.py:150: in _ensure_data
    elif is_bool_dtype(values.dtype):
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/dtypes/common.py:1235: in is_bool_dtype
    if isinstance(arr_or_dtype, ABCIndex):
../../../.venv/3.11/lib/python3.11/site-packages/pandas/core/dtypes/generic.py:44: in _instancecheck
    return _check(inst) and not isinstance(inst, type)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

inst = dtype('int64')

    def _check(inst) -> bool:
>       return getattr(inst, attr, "_typ") in comp
E       RecursionError: maximum recursion depth exceeded while calling a Python object

This occurs with pandas 2.1 (released today) but not pandas < 2.1. Additionally, it is was not triggered by the genno test suite—but only because no workflow run had occurred with the 2.1.0 release.

PR checklist

  • Add a test that reproduces the exception.
  • Checks all ✅
  • Update documentation N/A, internals only.
  • Update doc/whatsnew.rst

@khaeru khaeru added the bug Something isn't working label Aug 30, 2023
@khaeru khaeru self-assigned this Aug 30, 2023
@khaeru
Copy link
Owner Author

khaeru commented Aug 30, 2023

The test runs also give the following new warnings:

genno/tests/test_computations.py::test_aggregate[attrseries-True]
  /opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/xarray/core/dataarray.py:885: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    return key in self.data

genno/tests/test_util.py::test_parse_units0[['kg', 'kg']-'kg']
genno/tests/test_util.py::test_parse_units0[['foo/bar']-'foo/bar']
genno/tests/test_util.py::test_parse_units0[[]-'dimensionless']
genno/tests/test_util.py::test_parse_units1
  /home/runner/work/genno/genno/genno/util.py:98: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version.
    unit = pd.unique(data)

genno/tests/compat/test_pyam.py: 12 warnings
  /opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/pyam/core.py:2691: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version.
    index = pd.unique(index)

genno/tests/core/test_quantity.py::TestQuantity::test_bfill[attrseries]
genno/tests/core/test_quantity.py::TestQuantity::test_bfill[attrseries]
  /home/runner/work/genno/genno/genno/core/attrseries.py:189: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
    .fillna(method="bfill", axis=1, limit=limit)

genno/tests/core/test_quantity.py::TestQuantity::test_ffill[attrseries]
genno/tests/core/test_quantity.py::TestQuantity::test_ffill[attrseries]
  /home/runner/work/genno/genno/genno/core/attrseries.py:274: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
    .fillna(method="ffill", axis=1, limit=limit)

@codecov-commenter
Copy link

codecov-commenter commented Aug 30, 2023

Codecov Report

Merging #99 (fdef63e) into main (bdead68) will not change coverage.
The diff coverage is 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #99   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           45        45           
  Lines         3819      3823    +4     
=========================================
+ Hits          3819      3823    +4     
Files Changed Coverage Δ
genno/core/quantity.py 100.00% <ø> (ø)
genno/tests/core/test_quantity.py 100.00% <ø> (ø)
genno/tests/test_computations.py 100.00% <ø> (ø)
genno/core/attrseries.py 100.00% <100.00%> (ø)
genno/tests/compat/test_pyam.py 100.00% <100.00%> (ø)
genno/util.py 100.00% <100.00%> (ø)

khaeru added 2 commits August 30, 2023 20:47
Don't force Series return value to be MultiIndex'ed; both AttrSeries and
SparseDataArray handle this independently.
@khaeru khaeru merged commit 0777db8 into main Aug 30, 2023
@khaeru khaeru deleted the fix/pandas-2.1 branch August 31, 2023 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants