✨ Add ordering options to askar scan and fetch_all methods

[ff137]

✨ Adds order_by and descending options to paginated scan or fetch_all methods

❗ This PR is waiting for next official Askar release

🚧 At present, uses TestPyPI package to test ordering functionality introduced in Askar: openwallet-foundation/askar#291

To-do: test coverage, and point to next aries-askar release that includes this change.

Notes: currently it only appears to ever make sense to order by the "id" column, since record field values are encrypted.
So, it does feel a bit weird allowing users to select an "order_by" column when there's only one valid choice. But, I still believe it's better than nothing. We at least add the functionality for control over ordering, and it can be disabled (for whatever reason) by requesting None. Ordering by id is effectively the same as ordering by time created (AFAICT), so the newest records can be received at the top by selecting descending=True. Perhaps this can be made default behavior.

At present (before this Askar PR is merged), there is no guaranteed ordering when fetching records, for the scan (paginated queries) or fetch all methods. This means there's no solid guarantee that all records will be retrieved while scanning (with increasing offsets), and ordering of records may be inconsistent across many fetch_all calls. So, resolving that is the motive for this contribution.

Will close #3001

[ff137]

Marking ready for review just to see integration test results

[jamshale]

@ff137 Is this working for you to test changes to the aries-askar library? I've been trying to do the same thing, with a fork of aries-askar, committing the libaries_askar.* files and installing it with poetry. However, I'm always getting Library not found in path error even though it seems to be there.

I think i might have something wrong with some meta data or my python env.

[ff137]

@jamshale Hmm, I got the same error in the integration run: https://github.com/hyperledger/aries-cloudagent-python/actions/runs/10451102652/job/28936801431
even after committing all the libraries_askar.* files (in the wrappers/python/aries_askar dir)

At first I couldn't run the PR tests locally, but resolved when I added just libraries_askar.so file to the wrappers/python/aries_askar dir. Just needed to reinstall aries_askar. But it seems somethings up in the integration test context, such that having all the library resources committed doesn't resolve it. Not sure what's up there - I'm new to testing askar like this.

But, we could get our end-to-end tests passing here: didx-xyz/aries-cloudapi-python#989, using this ACA-Py branch, so it seems it builds fine in different contexts

[ff137]

For interest, the following commit shows the end-to-end tests that asserts unique records are returned across pages, when scanning with limit/offset: didx-xyz/aries-cloudapi-python@00c3f0f

Those tests had to be commented out because lack of ordering made results inconsistent, and it seems the askar changes + this branch resolves that 😃

Will wait for askar release including those changes, and perhaps adding tests here, before marking this PR ready

[ff137]

@jamshale I managed to get integration tests passing - using an aries_askar TestPyPI package 👀 90cd53c

Last couple commits here show how I uploaded to TestPyPI: ff137/aries-askar#1

[jamshale]

@jamshale I managed to get integration tests passing - using an aries_askar TestPyPI package 👀 90cd53c

Last couple commits here show how I uploaded to TestPyPI: ff137/aries-askar#1

Great, thanks. I'm going to give this a try.

[sonarcloud]

Quality Gate failed

Failed conditions
77.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

[sonarcloud]

Quality Gate failed

Failed conditions
76.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

[swcurran]

Any status update on this PR so we can raise it in the ACA-Pug meeting tomorrow -- 2024.11.16 @ 8:00 Pacific / 17:00 Central Europe?

[ff137]

Essentially just needs an official askar release (currently using my test-pypi package / fork of askar). And some minor rebasing.
We're also using this in our fork for a few months now

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
3.0% Duplication on New Code

See analysis details on SonarQube Cloud