Scenario testing - Restarts and upgrades

[jamshale]

In the acapy tools repo there is some really useful testing using docker containers that allows testing of upgrade scripts and restarts. https://github.com/hyperledger/aries-acapy-tools/tree/main/askar_tools/tests/e2e.

It should be possible to support this in the scenarios directory from acapy itself. It would be awesome to be able to test a supported upgrade path from the last LTS version and test a few different restarts with changed configurations such as making public dids and using seeds etc.

[ianco]

@jamshale are there instructions for running the e2e tests?

[jamshale]

In [acapy-tools](https://github.com/openwallet-foundation/acapy-tools) I don't think there is any instructions :/ . It's pretty simple though. I just poetry install and then navigate to askar_tools and run poetry run pytest ..

You can use other pytest commands to run an individual test.

[ianco]

ok thanks, I'll add this to the README

[ianco]

Probably more useful to be able to test this with a back-end postgres database rather than having to worry about import/export for sqlite wallets (and then the tests could run in a separate docker container, and just point to the external database).

[jamshale]

I agree. For this scenario testing against a postgres db would be easier and better. The sqlite testing was much more difficult with restarts in acapy-tools. Also would be good to have postgres getting tested more here in general.

[ianco]

@jamshale I'm getting errors running the tests.

I ran poetry install --with=dev (to include the dev dependencies, basically acapy-controller), which failed with the following error:

HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /indicio-tech/acapy-minimal-example.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

I updated the project to use acapy-controller = "~=0.2.0" (the published version of the library) and then the install worked.

After running poetry run pytest . however everything failed:

FAILED tests/e2e/test_export.py::TestPgExport::test_export_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...
FAILED tests/e2e/test_export.py::TestPgExport::test_export_sqlite - docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/create?name=alice: Conflict ("Conflict. The container name "/alice" is already in use by container "81137e408a975071c15cfb032...
FAILED tests/e2e/test_mt_convert_to_mw.py::TestMultitenantConvertToMultiwallet::test_conversion_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...
FAILED tests/e2e/test_mt_convert_to_mw.py::TestMultitenantConvertToMultiwallet::test_conversion_sqlite - docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/create?name=admin: Conflict ("Conflict. The container name "/admin" is already in use by container "14b1b2cf31f1c7e6c9f075f70...
FAILED tests/e2e/test_tenant_import.py::TestTenantImport::test_tenant_import_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...

Any thoughts? Looks like a certificate error and then the container ... already is use is because a previous test failed.

[ianco]

(the above is in the askar_tools directory, for the tests under acapy_wallet_upgrade some work and some fail)

(most of the tests fail)

[jamshale]

It doesn't clean up very well when there's a fail. Sometimes you need to manually delete the containers or the network.

I'm not sure about the certificate error though. I'm not getting any and the tests are passing for me. You might be able to switch the genesis file. I'm just trying that now.

[ianco]

What's running at selfserve.indiciotech.io:443?

[jamshale]

The acapy-tools tests use the indicio indy test ledger. I guess that's where it's trying to register a did. I changed the scenario tests in acapy to use http://test.bcovrin.vonx.io/. There's something automatic that the acapy-controller indy_anoncred_onboard function does that requires https that prevents just switching easily :/

Not sure why your getting certificate problems though.

[ianco]

Looks like all that is hard-coded into the unit tests. I'll update to use env vars (if available) and also it looks like some tests have a postgres dependency.

Eventually would be nice to run the tests in docker (I always have to mess around when running locally to get all the proper versions, like python 3.10 which isn't a default on my local ...)

[jamshale]

Ya. These test in acapy-tools aren't the best, but the container process that allows restarts is what I wanted to replicate for acapy.

I got around the hardcoded stuff in the library for the ledger here https://github.com/openwallet-foundation/acapy/blob/main/scenarios/examples/presenting_revoked_credential/example.py#L50.

[ianco]

I notice that the test container logic starts up a bunch of services within the python code. Typically in other projects we do this within a docker-compose file, and then pass parameters in (overrides) via environment variables.

I'd like to convert this project to use a similar structure - docker compose and then build a docker image to run the unit tests.

Before I dive into the code, does this make sense? It should make it easier to use these components within other repo's (like aca-py).

@jamshale @swcurran @dbluhm

[jamshale]

Yes. That would be good. In the scenario tests in acapy we use docker compose files. It's really just about how to persist the storage containers for the length of the test and restart the agents with a different image and configs. The acapy-tools e2e tests do this so I thought we could use a similar design.

[ianco]

It's really just about how to persist the storage containers for the length of the test and restart the agents with a different image and configs.

I'll work on updating the acapy-tools tests to use docker and think about how to do this for the aca-py scenario tests ...

[dbluhm]

The reason the tests start the container in python is because that seemed like the best way to control starting and stopping containers. I also generally prefer to keep it simple and just use docker compose wherever possible but, at the time, I was not finding good solutions that permitted both simple docker compose and controlling startup parameters and starting and stopping the containers. So docker compose got nixed.