-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CachingBackend, address performance of JDBCBackend.item_get_elements #213
Conversation
@OFR-IIASA notes that this change brings the time to load the He also reports that the caching appears to work. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks fine; I ran the tests and additional tests of my own to check that the speed of retrieving data as well as caching works as desired - all OK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Would be great to compare memory allocation after switching to categorical dtypes. Thanks @khaeru!
This PR:
[:]
slice syntax when accessing arrays via JPype. This was added in 2017 (554c9be) but removed in Add Backend, JDBCBackend, Model, GAMSModel classes #182. It is significantly faster than accessing the object without the slice; see Fast conversion from Java byte array to Numpy uint8 array jpype-project/jpype#71. Although that issue is closed, this workaround still seems necessary. It is not discussed in the JPype documentation. This PR also adds a comment clearly stating this is necessary. Incorporates WIP: Convert java arrays to Series on fetching model elements #214.In the longer run, we will need to add performance tests (e.g. using pytest-benchmark) to ensure that the ixmp test suite directly tests that ixmp has satisfactory performance for data on the scale of MESSAGE-GLOBIOM (~10 dimensions, ~10⁶ rows). That will be a separate issue.
How to review
test_report_size
to 8 and run only that test withpytest -k report_size
.PR checklist