aiida-firecrest [IN-DEVELOPMENT]

AiiDA Transport/Scheduler plugins for interfacing with FirecREST, via pyfirecrest.

It is currently tested against FirecREST v2.6.0.

Installation

Install via GitHub or PyPI:

pip install git+https://github.com/aiidateam/aiida-firecrest.git
# pip not yet available
pip install aiida-firecrest

You should then be able to see the firecrest transport and scheduler plugins loaded in AiiDA:

$ verdi plugin list aiida.transports firecrest
A plugin to connect to a FirecREST server.
It must be used together with the 'firecrest' scheduler plugin.
Authentication parameters:
  url: URL to the FirecREST server
  token_uri: URI for retrieving FirecREST authentication tokens
  client_id: FirecREST client ID
  client_secret: FirecREST client secret
  client_machine: FirecREST machine secret
  small_file_size_mb: Below this size, file bytes will be sent in a single API call.

$ verdi plugin list aiida.schedulers firecrest
A plugin to connect to a FirecREST server.
It must be used together with the 'firecrest' transport plugin.

You can then create a Computer in AiiDA:

$ verdi computer setup
Report: enter ? for help.
Report: enter ! to ignore the default and set no value.
Computer label: firecrest-client                 # your choice
Hostname: unused                                 # your choice, irrelevant
Description []: My FirecREST client plugin       # your choice
Transport plugin: firecrest
Scheduler plugin: firecrest
Shebang line (first line of each script, starting with #!) [#!/bin/bash]:
Work directory on the computer [/scratch/{username}/aiida/]:
Mpirun command [mpirun -np {tot_num_mpiprocs}]:
Default number of CPUs per machine:  2           # depending on your compute resource
Default amount of memory per machine (kB).: 100  # depending on your compute resource
Escape CLI arguments in double quotes [y/N]:
Success: Computer<3> firecrest-client created
Report: Note: before the computer can be used, it has to be configured with the command:
Report:   verdi -p MYPROFILE computer configure firecrest firecrest-client

$ verdi -p MYPROFILE computer configure firecrest firecrest-client
Report: enter ? for help.
Report: enter ! to ignore the default and set no value.
Server URL: https://firecrest.cscs.ch          # this for CSCS
Token URI: https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token
Client ID: username-client
Client Secret: xyz
Compute resource (Machine): daint
Temp directory on server: /scratch/something/ # "A temp directory on user's space on the server for creating temporary files (compression, extraction, etc.)"
FirecREST api version [Enter 0 to get this info from server] [0]: 0
Maximum file size for direct transfer (MB) [Enter 0 to get this info from server] [0]: 0
Report: Configuring computer firecrest-client for user [email protected].
Success: firecrest-client successfully configured for [email protected]

You can always check your config with

$ verdi computer show firecrest-client

See also the pyfirecrest CLI, for directly interacting with a FirecREST server.

After this, everything should function normally through AiiDA with no problems. See tests/test_calculation.py for a working example of how to use the plugin, via the AiiDA API.

If you encounter any problems/bug, please don't hesitate to open an issue on this repository.

Current Issues

Calculations are now running successfully, however, there are still issues regarding efficiency, Could be improved:

1. Monitoring / management of API request rates could to be improved. Currently this is left up to pyfirecrest.
2. Each transfer request includes 2 seconds of sleep time, imposed by pyfirecrest. One can takes use of their async client, but with current design of aiida-core, the gain will be minimum. (see the closing comment of issue#94 on pyfirecrest and PR#6079 on aiida-core )

For developers

git clone
cd aiida-firecrest
pip install -e .[dev]

Code Style

To format the code and lint it, run pre-commit:

pre-commit run --all-files

Testing

It is recommended to run the tests via tox.

tox

By default, the tests are run using a monkey patched pyfirecrest. This allows for quick testing and debugging of the plugin, without needing to connect to a real server, but is obviously not guaranteed to be fully representative of the real behaviour.

To have a guaranteed proof, you may also provide connections details to a real FirecREST server:

tox -- --firecrest-config=".firecrest-demo-config.json"

If a config file is provided, tox sets up a client environment with the information in the config file and uses pyfirecrest to communicate with the server.

┌─────────────────┐───►┌─────────────┐───►┌──────────────────┐
│ aiida_firecrest │    │ pyfirecrest │    │ FirecREST server │
└─────────────────┘◄───└─────────────┘◄───└──────────────────┘

if a config file is not provided, it monkeypatches pyfirecrest so we never actually communicate with a server.

┌─────────────────┐───►┌─────────────────────────────┐
│ aiida_firecrest │    │ pyfirecrest (monkeypatched) │
└─────────────────┘◄───└─────────────────────────────┘

The format of the .firecrest-demo-config.json file, for example is like:

 {
    "url": "https://firecrest-tds.cscs.ch",
    "token_uri": "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token",
    "client_id": "username-client",
    "client_secret": "path-to-secret-file",
    "compute_resource": "daint",
    "temp_directory": "/scratch/snx3000/username/",
    "small_file_size_mb": 5.0,
    "workdir": "/scratch/snx3000/username/",
    "api_version": "1.16.0"
}

In this mode, if you want to inspect the generated files, after a failure, you can use:

tox -- --firecrest-config=".firecrest-demo-config.json" --firecrest-no-clean

These tests were successful against FirecREST v1.16.0, except those who require to list directories in a symlink directory, which fail due to a bug in FirecREST. An issue is open on FirecREST repo about this.

Instead of a real server (which requires an account and credential), tests can also run against a docker image provided by FirecREST. See firecrest_demo.py for how to start up a demo server, and also server-tests.yml for how the tests are run against the demo server on GitHub Actions.

Notes on using the demo server on MacOS

A few issues have been noted when using the demo server on MacOS (non-Mx):

docker-compose up can fail, with an error that port 7000 is already in use. Running lsof -i :7000 you may see it is used by afs3-fileserver, which can be fixed by turning off the Airplay receiver (see cookiecutter/cookiecutter-django#3499)

Large file uploads can fail, because the server provides a URL with 192.168.220.19 that actually needs to be localhost. To fix this, ensure that you set FIRECREST_LOCAL_TESTING = true in your environment (set by default if running tox).

Large file downloads has the same problem, but even with this fix, it will still fail with a 403 HTTP error, due to a signature mismatch. No automatic workaround has been found for this yet, although it is of note that you can find these files directly where you your firecrest Github repo is cloned, /path/to/firecrest/deploy/demo/minio/ plus the path of the URL.

🐛 Fishing Bugs 🐛

First, start with running tests locally with no config file given, that would monkeypatch pyfirecrest. These set of test do not guarantee that the whole firecrest protocol is working, but it's very useful to quickly check if aiida-firecrest is behaving as it's expected to do. To run just simply use pytest or tox.

If these tests pass and the bug persists, consider providing a config file to run the tests on a docker image or directly on a real server. Be aware of versioning, pyfirecrest doesn't check which version of api it's interacting with. (see eth-cscs/pyfirecrest#116)

If the bug persists and test still passes, then most certainly it's a problem of aiida-firecrest. If not, probably the issue is from FirecREST, you might open an issue to pyfirecrest or FirecREST.

Acknowledgment:

This project is supported by SwissTwins fundings.