Skip to content

AiiDA release roadmap

Jump to bottom
Sebastiaan Huber edited this page Nov 18, 2020 · 69 revisions

Where to look

Information on upcoming AiiDA releases and features being actively worked on:

  • GitHub milestones: We use milestones to collect issues we plan to resolve in specific upcoming releases of aiida-core.

  • GitHub projects: We use GitHub projects to group issues by topic where appropriate. Projects may be tackled in focussed AiiDA coding days, or they may extend over longer periods of time.

Suggested features & future plans:

Please note: All of these resources are working drafts and subject to change.

Roadmap

Tentative roadmap of upcoming releases with major features/changes that will be included:

  • v1.6.0 (December 31, 2020)
    • Major change is migration from tornado to asyncio
    • Calculation job file stashing
    • Transfer calcjob
  • v2.0.0 (April 2021)
    • New file repository implementation
    • Archive file format
    • Repository maintenance interface
    • Resetting of the database migrations
    • Moving out all materials science specific code
    • Remove all deprecated code
    • ....

Development roadmap

This is a short overview of directions under discussion for future AiiDA development. Most of these items require significant development efforts, and will be achieved more quickly with dedicated efforts from external contributors.

In short, help is always welcome!

Solution for HPC centres with two-factor authentication

AiiDA communicates with HPC centres via SSH keys. The SSH agent already allows AiiDA to deal with password-protected SSH keys, but so far there is no support for HPC centres that require two-factor authentication (2FA). With the cyberattack on European HPC centres in May 2020, this is becoming a pressing issue.

There are multiple routes to explore -- simpler installation on login nodes of HPC centers as well as solutions for scoping SSH key access (for certain time periods).

For more details, see https://github.com/aiidateam/aiida-core/issues/3929

Efficient file repository for large numbers of files

AiiDA uses the computer's file system to store files that are too large to be stored in the database. While a simple solution, this can create issues when the number of files grows large (reaching the limit of inodes of the file system). It also currently does not provide a way to compress files in order to save space.

See the enhancement proposal https://github.com/aiidateam/AEP/pull/11

Contacts: @giovannipizzi

Support object store API for static file repositories

When serving static AiiDA provenance graphs (such as on materialscloud.org), it can be convenient to have a web API for accessing files from AiiDA file repositories, so that multiple servers can access the same repository. The natural solution would seem to be to support specifying an object store as the location of the AiiDA file repository.

One open question is whether explicit support in AiiDA is needed, given that there are adapters like the swift virtual file system that allow mounting an object store container on disk.

Automatic integration tests for plugins registered on AiiDA registry

The AiiDA registry could run specific tests of AiiDA plugins to check whether they work with certain python / AiiDA versions, providing a basic level of quality control.

Early implementation: https://github.com/aiidateam/aiida-plugin-ci

Presentation on the topic

An important open question is whether it would be better to start with a "soft touch" approach, asking plugin developers to simply pytest.mark some of their tests to be run by the AiiDA registry, or whether a custom framework is necessary to inspect tests more deeply (e.g. to ensure that a full integration test of a simulation code plugin works with a specific version of the code).

Support for python-based simulation codes

AiiDA has two classes of processes: the locally running process functions, and the AiiDA will treat simulation codes written in python just as any other executable with input files and output files. While you can use python packages inside work functions and calculation functions that run locally on your computer, for codes running on remote machines AiiDA supports only file-based interaction.

For python codes, it seems wasteful to serialize/deserialize python objects from/to files, and a direct reuse of python objects would seem to be useful.

Related projects: pyOS, mincePy

Support for containerized simulation codes

See the Google Summer of Code project

Contacts: @ltalirz, @giovannipizzi

Performance optmizations at the ORM level

See the Google Summer of Code project

Contacts: @ltalirz, @giovannipizzi

Simplify generation of CLI/GUI for ORM classes

Creating a command-line interface (CLI) or a graphical user interface (GUI) to provide the information needed to create instances of the AiiDA ORM, e.g. a Computer, a Code or even an Int, currently involves boilerplate code. It would be useful, if the Computer/Code/... classes could be annotated in a way that makes the creation of a CLI/GUI automatic.

One open question in this respect is how to deal with validation - most of the validation currently occurs at the click level. In order to avoid duplication of validation checks, these would need to move into the ORM classes themselves.

Task farming: Disguise multiple jobs as one

AiiDA originated from the ab initio electronic structure community, where individual calculation jobs usually occupy a full compute node. As users from neighboring disciplines start using AiiDA (e.g. for force-field calculations), this is no longer the case, and one node may need to be shared between multiple AiiDA jobs.

Many HPC centers have queues that allow for node-sharing of multiple serial jobs, but there a centers that only allow one job per node. For such cases, it would be useful if AiiDA could pack multiple jobs into one.

See section "10. Task farming" in the report of the 2020 AiiDA hackathon for more details.

Contacts: @giovannipizzi, @pzarabadip

Cross-computer scheduling

While AiiDA makes it easy to submit jobs to different computers, it is the responsibility of the user to do so. It would be useful, if AiiDA would include some basic cross-computer scheduling features, such as setting a maximum number of jobs running on one computer at any given time.

"git push/pull" for sharing AiiDA graphs with collaborators

AiiDA provides export files as a means of exchanging AiiDA graphs. It would be useful if AiiDA supported pushing/pulling changes in AiiDA graphs from/to collaborators, transferring only the "delta" of the differences.

Presentation on the topic

Early implementation: https://github.com/szoupanos/aiida_core/tree/sharing_v1

Hierarchical visualization of AiiDA graphs

AiiDA provenance graphs can grow very complex. A hierarchical view of AiiDA graphs (e.g. by giving users the ability to "fold/unfold" workflow nodes) will be critical to make them useful for grasping the high-level structure of complex workflows.

This applies both to static visualization (verdi node graph generate) and for interactive visualization (e.g. the provenance browser.

Contact: @ltalirz

Command line interface for exporting complete AiiDA profile (e.g. backup)

The verdi export command allows to export your AiiDA graph based on provenance rules, starting from a set of nodes specified by the user. A common use case is to export the entire AiiDA graph, e.g. for backup purposes. While we document how to create backups, it involves different tools, and there is currently no shorthand in verdi export to export the entire profile.

See https://github.com/aiidateam/aiida-core/issues/974 for more details

Intuitive command-line interface for browsing the file repository

See mailing list and implementation candidate

Contacts: @giovannipizzi, @bonanzhu

Support for hiding nodes in verdi export

Sometimes, data that is necessary for provenance is licensed and cannot be shared publicly (e.g. atomic structures from commercial databases, pseudopotential files, ...) While this is contrary to the open science credo, it is a relevant use case and AiiDA should offer a convenient way to support it.

Online repository of computer/code configurations

AiiDA can read computer & code configurations from yaml files. It would be useful to create an online registry of computer and code setups for major high-performance computing clusters.

An open question is how to best deal with customizations needed on an individual user level (e.g. which queues to use, etc.). Do we need an additional templating language in the yaml files (like jinja2) or can we get away without it?

First draft: https://github.com/ltalirz/aiida-code-registry

Contact: @ltalirz

Domain-agnostic AiiDA core

Most of AiiDA's API is already domain-agnostic but some materials science specific classes remain in aiida-core (e.g. StructureData, KpointsData, ...). In order to make aiida-core more friendly to other disciplines, we would like to move these classes out to separate packages (e.g. aiida-pseudopotenials, aiida-atomic, ...).

One important open question is how to handle database schema migrations for data types defined by plugins.

REST API plugins

Plugins can add to AiiDA in many ways, but one major way that is still missing is to be able to add plugins to the AiiDA REST API (e.g. for new data types).

This would require adding a new entry point group aiida.rest and using this group to announce both existing and new REST endpoints.

One important open question in this regard is whether a switch from REST to GraphQL would make this easier (see aiida-graphql prototype and presentation ).

Contacts: @ltalirz, @dev-zero

Workflow management through the REST API

While AiiDA has a web interface to query the AiiDA graph (the AiiDA REST API), it does currently not have a web interface for workflow management. In order to integrate AiiDA into existing workflow management platforms, a web interface for workflow management (mimicking e.g. what can currently be achieved via the verdi command line interface) is necessary.

Contacts: @ltalirz, @dgiovannipizzi

Clone this wiki locally