-
Notifications
You must be signed in to change notification settings - Fork 192
AiiDA release roadmap
Information on upcoming AiiDA releases and features being actively worked on:
-
GitHub milestones: We use milestones to collect issues we plan to resolve in specific upcoming releases of
aiida-core
. -
GitHub projects: We use GitHub projects to group issues by topic where appropriate. Projects may be tackled in focussed AiiDA coding days, or they may extend over longer periods of time.
Suggested features & future plans:
-
AiiDA Enhancement Proposals (AEPs): AEPs are the preferred vehicle for proposing substantial new features and design choices. Note in particular the open pull requests.
-
Google Summer of Code projects: Projects listed here are considered to be suitable for newcomers. Help is welcome!
-
GitHub issues: Open issues with
type/feature request
ortype/enhancement
-
Development roadmap (below)
Please note: All of these resources are working drafts and subject to change.
Tentative roadmap of upcoming releases with major features/changes that will be included:
- v2.0.0: milestone
- Released on April 27
This is a short overview of directions under discussion for future AiiDA development. Most of these items require significant development efforts, and will be achieved more quickly with dedicated efforts from external contributors.
In short, help is always welcome!
AiiDA communicates with HPC centres via SSH keys. The SSH agent already allows AiiDA to deal with password-protected SSH keys, but so far there is no support for HPC centres that require two-factor authentication (2FA). With the cyberattack on European HPC centres in May 2020, this is becoming a pressing issue.
There are multiple routes to explore -- simpler installation on login nodes of HPC centers as well as solutions for scoping SSH key access (for certain time periods).
For more details, see https://github.com/aiidateam/aiida-core/issues/3929
AiiDA uses the computer's file system to store files that are too large to be stored in the database. While a simple solution, this can create issues when the number of files grows large (reaching the limit of inodes of the file system). It also currently does not provide a way to compress files in order to save space.
See the enhancement proposal https://github.com/aiidateam/AEP/pull/11 This was released with v2.0
Contacts: @giovannipizzi
When serving static AiiDA provenance graphs (such as on materialscloud.org), it can be convenient to have a web API for accessing files from AiiDA file repositories, so that multiple servers can access the same repository. The natural solution would seem to be to support specifying an object store as the location of the AiiDA file repository.
One open question is whether explicit support in AiiDA is needed, given that there are adapters like the swift virtual file system that allow mounting an object store container on disk.
Note: The basic API for this is included in 2.0 and the interface to the disk-objectstore. Further tests with network-based object stores are performed by @espenfl
Contacts: @giovannipizzi, @espenfl
The AiiDA registry could run specific tests of AiiDA plugins to check whether they work with certain python / AiiDA versions, providing a basic level of quality control.
Early implementation: https://github.com/aiidateam/aiida-plugin-ci
An important open question is whether it would be better to start with a "soft touch" approach, asking plugin developers to simply pytest.mark
some of their tests to be run by the AiiDA registry, or whether a custom framework is necessary to inspect tests more deeply (e.g. to ensure that a full integration test of a simulation code plugin works with a specific version of the code).
Contact: @giovannipizzi, @ltalirz
AiiDA has two classes of processes: the locally running process functions, and the AiiDA will treat simulation codes written in python just as any other executable with input files and output files. While you can use python packages inside work functions and calculation functions that run locally on your computer, for codes running on remote machines AiiDA supports only file-based interaction.
For python codes, it seems wasteful to serialize/deserialize python objects from/to files, and a direct reuse of python objects would seem to be useful.
Related projects: pyOS, mincePy, aiida-dynamic-workflows
See the Google Summer of Code project
Note: Basic support for containerized codes is scheduled to be released with AiiDA 2.1. This is being dealt with in #5507
Contacts: @giovannipizzi, Contact: @unkcpz
See the Google Summer of Code project
Contacts: @chrisjsewell, @giovannipizzi
Creating a command-line interface (CLI) or a graphical user interface (GUI) to provide the information needed to create instances of the AiiDA ORM, e.g. a Computer
, a Code
or even an Int
, currently involves boilerplate code.
It would be useful, if the Computer/Code/... classes could be annotated in a way that makes the creation of a CLI/GUI automatic.
One open question in this respect is how to deal with validation - most of the validation currently occurs at the click
level.
In order to avoid duplication of validation checks, these would need to move into the ORM classes themselves.
This is added for creating codes in #5510. The ORM class provides a simple spec of its properties and the CLI is generated automatically from this. This is done using the aiida.cmdline.groups.dynamic.DynamicEntryPointCommandGroup
.
AiiDA originated from the ab initio electronic structure community, where individual calculation jobs usually occupy a full compute node. As users from neighboring disciplines start using AiiDA (e.g. for force-field calculations), this is no longer the case, and one node may need to be shared between multiple AiiDA jobs.
Many HPC centers have queues that allow for node-sharing of multiple serial jobs, but there a centers that only allow one job per node. For such cases, it would be useful if AiiDA could pack multiple jobs into one.
See section "10. Task farming" in the report of the 2020 AiiDA hackathon for more details.
We also need to make sure that the plugin is well-documented, and clearly advertised on the main AiiDA documentation etc so it is easy to find.
Contacts: @mbercx, @giovannipizzi, @pzarabadip
While AiiDA makes it easy to submit jobs to different computers, it is the responsibility of the user to do so. It would be useful, if AiiDA would include some basic cross-computer scheduling features, such as setting a maximum number of jobs running on one computer at any given time.
AiiDA provides export files as a means of exchanging AiiDA graphs.
It would be useful if AiiDA supported pushing
/pulling
changes in AiiDA graphs from/to collaborators, transferring only the "delta" of the differences.
Early implementation: https://github.com/szoupanos/aiida_core/tree/sharing_v1
AiiDA provenance graphs can grow very complex. A hierarchical view of AiiDA graphs (e.g. by giving users the ability to "fold/unfold" workflow nodes) will be critical to make them useful for grasping the high-level structure of complex workflows.
This applies both to static visualization (verdi node graph generate
) and for interactive visualization (e.g. the provenance browser.
Contact: @ltalirz
The verdi export
command allows to export your AiiDA graph based on provenance rules, starting from a set of nodes specified by the user.
A common use case is to export the entire AiiDA graph, e.g. for backup purposes.
While we document how to create backups, it involves different tools, and there is currently no shorthand in verdi export
to export the entire profile.
See https://github.com/aiidateam/aiida-core/issues/974 for more details
See mailing list and implementation candidate
Contacts: @giovannipizzi, @bonanzhu
Sometimes, data that is necessary for provenance is licensed and cannot be shared publicly (e.g. atomic structures from commercial databases, pseudopotential files, ...) While this is contrary to the open science credo, it is a relevant use case and AiiDA should offer a convenient way to support it.
AiiDA can read computer & code configurations from yaml
files.
It would be useful to create an online registry of computer and code setups for major high-performance computing clusters.
An open question is how to best deal with customizations needed on an individual user level (e.g. which queues to use, etc.).
Do we need an additional templating language in the yaml
files (like jinja2) or can we get away without it?
First draft: https://github.com/ltalirz/aiida-code-registry
Contact: @ltalirz
Most of AiiDA's API is already domain-agnostic but some materials science specific classes remain in aiida-core
(e.g. StructureData
, KpointsData
, ...).
In order to make aiida-core
more friendly to other disciplines, we would like to move these classes out to separate packages (e.g. aiida-pseudopotenials
, aiida-atomic
, ...).
One important open question is how to handle database schema migrations for data types defined by plugins.
Plugins can add to AiiDA in many ways, but one major way that is still missing is to be able to add plugins to the AiiDA REST API (e.g. for new data types).
This would require adding a new entry point group aiida.rest
and using this group to announce both existing and new REST endpoints.
One important open question in this regard is whether a switch from REST to GraphQL would make this easier (see aiida-graphql prototype and presentation ).
See also: https://github.com/aiidateam/aiida-restapi
While AiiDA has a web interface to query the AiiDA graph (the AiiDA REST API), it does currently not have a web interface for workflow management.
In order to integrate AiiDA into existing workflow management platforms, a web interface for workflow management (mimicking e.g. what can currently be achieved via the verdi
command line interface) is necessary.
Prototype implementation: https://github.com/aiidateam/aiida-restapi
Contacts: @ltalirz, @chrisjsewell