AiiDA 2.0 plugin migration guide

This page will contain important information to help migrate plugins from AiiDA v1.x to v2.0. First, it will give a quick overview of minimal things to do to update your environment and plugin to quickly get you started. The rest of the page will provide more in depth details about relevant changes that may affect your plugin.

NOTE: https://github.com/aiidateam/aiida-upgrade/ is currently in development, to provide an automated migration for some API changes.

Quick Start

To start migrating your plugin to AiiDA v2.0, follow these steps:

Update pip: pip install --upgrade pip. See this section for more details
Install aiida-core: pip install aiida-core[tests]==2.0 --pre. Read this section if you use editable installs.
Create a new branch for your plugin: git checkout -b fix/compatibility-aiida-2.0
Run the automated migration plugin, see https://github.com/aiidateam/aiida-upgrade/
Run your test suite
Check the rest of this document to see the changes introduced in AiiDA 2.0, check which apply to your plugin, and update it accordingly

These minimal steps should give you a working environment with aiida-core==2.0 installed. If your tests are failing, please refer to the detailed discussion of changes in AiiDA v2.0 below to find the potential source of the errors. If you cannot find the answer, feel free to contact us on Slack, open a new discussion or open a new issue.

Detailed discussion

Installation

Update pip

To be able to install aiida-core==2.0 in editable mode with pip, you need pip>=21.0.0. To update pip, simply run:

pip install --upgrade pip

The reason for this requirement is that as of v2.0, AiiDA adopts PEP 621 (see PR #5312). This means that the setup.py, which has been deprecated for a long time, and the old bespoke setup.json, are fully replaced by the pyproject.toml file. This is the new standard in Python packaging and full support was added in pip v21.

Delete old editable installs

If you are using an editable install and are reinstalling in the same repository that you used for an installation of AiiDA v1.x, you need to remove the old .egg-info file:

rm -r ./*.egg-info

Failing to do so, may cause problems with entry points as the old ones will still be registered. When there are duplicates, one will be chosen at random and so sometimes you can end up with the wrong entry point.

Deprecation warnings

AiiDA v2.0 comes with quite a few changes, but the bulk of them are backwards compatible. When a changed or removed resource is used, a deprecation warning is emitted which includes the future version in which the deprecated resource will be removed. These are hidden by default, but they can be activated using the AIIDA_WARN_v3 environmental variable:

export AIIDA_WARN_v3=1

We recommend you add this environment variable to your continuous integration environment so you see the deprecations and can address them.

`verdi`

Tab-completion

The library click, which is what verdi is built with, was upgraded. It now comes with tab-completion built-in, which means we could drop the additional dependency click-completion. The completion works the same, except that the string that should be put in the activation script to enable it is now shell-dependent. See the documentation to find out what string you should use for your shell. See this PR for more details.

`verdi code setup`

There is a small change in verdi code setup where the order of prompts has changed. If you have scripts that use the interactive mode for this command, they might start to fail, since the wrong values are passed for the wrong arguments. However, it is in general not advisable to use the interactive (prompting) mode for automated scripts. Please use the --non-interactive flag to ensure the command doesn't prompt and simply use the various parameter flags to specify the values, .e.g.:

verdi code setup --non-interactive -L label -D "description" .....

Entry points

The entry point system allows external packages to extend the functionality of aiida-core. This concept was formally introduced in v1.0 and since there have been unwritten guidelines and naming conventions for entry points. Particularly, entry points defined by a plugin package are encouraged to be prefixed with the name of the plugin package. For example, the entry points of aiida-quantumespresso all start with the prefix quantumespresso.. This ensures that entry points are properly namespaced and there is minimal risk that the entry points of different plugin packages overlap and therefore cannot be uniquely resolved, rendering them unusable.

To this day, however, aiida-core itself has not been respecting this guideline and provides many entry points that are not namespaced with core.. This not only causes many namespaces to essentially be blocked for use for any potential plugin packages, it also makes it unclear where certain entry points come from. Therefore, the decision was made to change the entry point in aiida-core in v2.0 and properly prefix them with core.. The change was implemented in PR #5073.

This change has been made largely backward compatible, by updating the various plugin factories (imported from the aiida.plugins module) with a special condition that detects the old entry point names. If detected, it emits a deprecation warning and then proceeds to actually load the new entry point. For example, the following code:

from aiida.plugins import DataFactory
Int = DataFactory('int')

will emit the following warning in v2.0:

In [1]: Int = DataFactory('int')
aiida/plugins/factories.py:40: AiidaDeprecationWarning: The entry point `int` is deprecated. Please replace it with `core.int`.

To get rid of the deprecation warning, simply update the entry point by prefixing it with core.:

from aiida.plugins import DataFactory
Int = DataFactory('core.int')

Note that entry point names are also used on the command line. For example, when creating a new computer, let's say the localhost configured with the DirectScheduler, this used to be done with

verdi computer setup -L localhost -T local -S direct

which should now become

verdi computer setup -L localhost -T core.local -S core.direct

The old entry points will continue to work for v2.0, but will also cause the deprecation warning to be printed since the CLI goes through the plugin factories to load the entry points behind the scenes.

Given that entry point names are also stored in the database in certain places (for example the node_type attribute of Data nodes, and the scheduler_type of Computer instances), the data of existing databases will be automatically migrated.

Note that the new entry points do not only apply when they are used as command arguments, but also if the entry point is itself a command, the full entry point name needs to be used. A good example are the subcommands of the verdi data command, which are themselves entry points. For example, what used to be:

verdi data bands list

has now become:

verdi data core.bands list

Remember that you can always use tab-completion to automatically discover the subcommands that are available.

Node namespace restructuring

The Node class, and thus its subclasses, have many methods and attributes in its public namespace. This has been noted as being a problem for those using auto-completion, since it makes it difficult to select suitable methods and attributes. The restructuring is fully backwards-compatible, and existing methods/attributes will continue to work, until aiida-core v3.0.

The methods/attributes of the Node class have been partitioned into "sub-namespaces" for specific purposes:

Node.base.attributes: Interface to the attributes of a node instance.
Node.base.caching: Interface to control caching of a node instance.
Node.base.comments: Interface for comments of a node instance.
Node.base.extras: Interface to the extras of a node instance.
Node.base.links: Interface for links of a node instance.
Node.base.repository: Interface to the file repository of a node instance.

The table below gives a complete overview of the changes in the Node namespace:

Current name	New name
`Collection`	Deprecated, use `NodeCollection` directly
`add_comment`	`Node.base.comments.add`
`add_incoming`	`Node.base.links.add_incoming`
`attributes`	`Node.base.attributes.all`
`attributes_items`	`Node.base.attributes.items`
`attributes_keys`	`Node.base.attributes.keys`
`check_mutability`	`Node._check_mutability_attributes`
`clear_attributes`	`Node.base.attributes.clear`
`clear_extras`	`Node.base.extras.clear`
`clear_hash`	`Node.base.caching.clear_hash`
`copy_tree`	`Node.base.repository.copy_tree`
`delete_attribute`	`Node.base.attributes.delete`
`delete_attribute_many`	`Node.base.attributes.delete_many`
`delete_extra`	`Node.base.extras.delete`
`delete_extra_many`	`Node.base.extras.delete_many`
`delete_object`	`Node.base.repository.delete_object`
`erase`	`Node.base.repository.erase`
`extras`	`Node.base.extras.all`
`extras_items`	`Node.base.extras.items`
`extras_keys`	`Node.base.extras.keys`
`get`	Deprecated, use `Node.objects.get`
`get_all_same_nodes`	`Node.base.caching.get_all_same_nodes`
`get_attribute`	`Node.base.attributes.get`
`get_attribute_many`	`Node.base.attributes.get_many`
`get_cache_source`	`Node.base.caching.get_cache_source`
`get_comment`	`Node.base.comments.get`
`get_comments`	`Node.base.comments.all`
`get_extra`	`Node.base.extras.get`
`get_extra_many`	`Node.base.extras.get_many`
`get_hash`	`Node.base.caching.get_hash`
`get_incoming`	`Node.base.links.get_incoming`
`get_object`	`Node.base.repository.get_object`
`get_object_content`	`Node.base.repository.get_object_content`
`get_outgoing`	`Node.base.links.get_outgoing`
`get_stored_link_triples`	`Node.base.links.get_stored_link_triples`
`glob`	`Node.base.repository.glob`
`has_cached_links`	`Node.base.caching.has_cached_links`
`id`	Deprecated, use `pk`
`is_created_from_cache`	`Node.base.caching.is_created_from_cache`
`is_valid_cache`	`Node.base.caching.is_valid_cache`
`list_object_names`	`Node.base.repository.list_object_names`
`list_objects`	`Node.base.repository.list_objects`
`objects`	`collection`
`open`	`Node.base.repository.open`
`put_object_from_file`	`Node.base.repository.put_object_from_file`
`put_object_from_filelike`	`Node.base.repository.put_object_from_filelike`
`put_object_from_tree`	`Node.base.repository.put_object_from_tree`
`rehash`	`Node.base.caching.rehash`
`remove_comment`	`Node.base.comments.remove`
`repository_metadata`	`Node.base.repository.metadata`
`repository_serialize`	`Node.base.repository.serialize`
`reset_attributes`	`Node.base.attributes.reset`
`reset_extras`	`Node.base.extras.reset`
`set_attribute`	`Node.base.attributes.set`
`set_attribute_many`	`Node.base.attributes.set_many`
`set_extra`	`Node.base.extras.set`
`set_extra_many`	`Node.base.extras.set_many`
`update_comment`	`Node.base.comments.update`
`validate_incoming`	`Node.base.links.validate_incoming`
`validate_outgoing`	`Node.base.links.validate_outgoing`
`validate_storability`	`Node._validate_storability`
`verify_are_parents_stored`	`Node._verify_are_parents_stored`
`walk`	`Node.base.repository.walk`

Repository

The file repository arguably underwent the greatest change of all components of AiiDA in v2.0 and as such various backwards incompatible changes had to be introduced.

FileType: moved from aiida.orm.utils.repository to aiida.repository.common
File: moved from aiida.orm.utils.repository to aiida.repository.common
File: changed from namedtuple to class
File: can no longer be iterated over
File: type attribute was renamed to file_type
Node.put_object_from_tree: path argument was renamed to filepath
Node.put_object_from_file: path argument was renamed to filepath
Node.put_object_from_tree: key argument was renamed to path
Node.put_object_from_file: key argument was renamed to path
Node.put_object_from_filelike: key argument was renamed to path
Node.get_object: key argument was renamed to path
Node.get_object_content: key argument was renamed to path
Node.open: key argument was renamed to path
Node.list_objects: key argument was renamed to path
Node.list_object_names: key argument was renamed to path
SinglefileData.open: key argument was renamed to path
Node.open: can no longer be called without context manager
Node.open: only mode r and rb are supported, use put_object_from_ methods instead
Node.get_object_content: only mode r and rb are supported
Node.put_object_from_tree: argument contents_only was removed
Node.put_object_from_tree: argument force was removed
Node.put_object_from_file: argument force was removed
Node.put_object_from_filelike: argument force was removed
Node.delete_object: argument force was removed

Using `open` in a context manager

In AiiDA v1.0 it was possible to call Node.open without a context manager, for example:

handle = node.open('filename.txt')
content = handle.read()
handle.close()

In AiiDA v2.0, this will raise and instead it should be used in a context manager

with node.open('filename.txt') as handle:
    content = handle.read()

This is good practice in any case, because in this case the file handle will be properly closed even if the read call excepts for some reason. In normal Python, although ill-advised, it is possible to call open on a file on the file system without a context manager, but in AiiDA v2.0 this raises. The reason is that by requiring a context manager, the file repository can be implemented in a more efficient manner, making the reading of files faster.

Writing cross-compatible code

Despite the changes listed above, it should be possible to write code that is compatible with both AiiDA 1.x and 2.x. The most important things to consider are:

Always use .open() with a context manager (as detailed above).

Use key or path as positional arguments, not keyword arguments. For example, write

with node.open('filename.txt') as in_f:
    <...>

instead of

with node.open(key='filename.txt') as in_f:
    <...>

Use try / except clauses to handle imports that have moved. For example:

try:
    from aiida.orm.utils.repository import FileType
except ImportError:
    from aiida.repository.common import FileType

To access the type / file_type attribute of a File, you can again use try / except clauses:

some_file = File(<...>)
try:
    file_type = some_file.file_type
except AttributeError:
    file_type = some_file.type

Or alternatively, getattr chaining:

some_file = File(<...>)
file_type = getattr(some_file, 'file_type', getattr(some_file, 'type'))

Points 3 & 4 are needed only for cross-compatibility between AiiDA versions <=1.3, and >=2.0. The 1.4 release is compatible with both the old and new syntax, but will show DeprecationWarning if the old syntax is used.

When using these workarounds (3 & 4), we recommend placing a comment into your code. For example:

# Workaround for compatibility with AiiDA version < 1.4

This will let you know to remove the workaround once your code no longer needs to be compatible with older AiiDA versions. Make sure the comment is always exactly the same, to simplify searching for it.

`QueryBuilder`

For the Computer class, the attribute name was already deprecated in AiiDA v1.0 and was replaced by label. However, the attribute name remained in the database table. This meant that in the QueryBuilder one had to continue using name. In AiiDA v2.0, the database table is now updated to match the ORM. If before you did the following:

QueryBuilder().append(Computer, filters={'name': 'localhost'}, project=['name']).all()

now you have to use

QueryBuilder().append(Computer, filters={'label': 'localhost'}, project=['label']).all()

REST API

The attribute name for the entity Computer was renamed to label.

Transport plugins

In PR #3787 a change to the API of transport plugins has been introduced, to support also transferring bytes (rather than only Unicode strings) in the stdout/stderr of "remote" commands (via the transport).

The required changes in your plugin (if you wrote a transport plugin) are:

rename the exec_command_wait function in your plugin implementation with exec_command_wait_bytes
ensure that you have a stdin in the parameters (the signature should be exec_command_wait_bytes(self, command, stdin=None, **kwargs)) and that you (also) accepts bytes in input in the stdin parameter. Ideally, if you get bytes, you shouldn't do any encoding/decoding, to ensure your plugin works also if the stdin contains binary data.
return bytes for stdout and stderr (most probably internally you are already getting bytes - just do not decode them to strings)

See e.g. the changes to the local transport plugin to see an example what needs to be changes.

Note that one can still call exec_command_wait that is now defined in the parent Transport class (that now has an encoding optional parameter with default=utf8, as it used to be), and takes care of the decoding. More details can be found in the PR and in the corresponding commit message, including how to support both v1.6 and v2.0 of AiiDA (by still defining also the exec_command_wait in your plugin, during the transition period).

Equality comparison of `Dict` nodes

Since AiiDA v1.6.0, nodes of all types compare equal when they have the same UUID (See PR #4753). However, most of the Pythonic base data types (Bool, Int, Float, Str and List) already went one step further and also compared equal to other nodes based on the node content. The only base type that was the exception here was Dict. After some discussion (see #5187 for a summary), it was decided to make the way compare equal to be consistent among the base types and hence make Dict nodes compare equal when they have the same content (see PR #5251).

In case your code relies on Dict nodes only comparing equal when it is strictly the same node, you can use the uuid property of the nodes. For example, when you define two different Dict nodes based on the same dictionary:

In [1]: d1 = Dict({'a': 1})

In [2]: d2 = Dict({'a': 1})

They will now be equal according to the == operator:

In [3]: d1 == d2
Out[3]: True

However, you can still see if they are the same node using the uuid property:

In [4]: d1.uuid == d2.uuid
Out[4]: False

Default cell in `StructureData` nodes

The default cell for StructureData nodes used to be 3 unit vectors, which is a valid cell. This made it easy for users to forget to set the actual cell of their structure. In https://github.com/aiidateam/aiida-core/pull/5341, the default has now been changed to [[0,0,0],[0,0,0],[0,0,0]] , which is no longer a valid cell for a periodic structure (only for pbc=False). Users therefore need to set a valid cell when creating a StructureData.

# Used to work, fails now
s = orm.StructureData().store()

# Correct
s = orm.StructureData(cell=[v1,v2,v3]).store()

# Quick fix for tests
s = orm.StructureData(pbc=False).store()

Note: The validity check is performed only upon storing. Any intermediate manipulation of the periodicity/cell will continue to work.

Schedulers

Scheduler plugins implementing the Scheduler class, had to implement the _get_submit_script_header method, which was also responsible for writing the environment variable declarations if the job_environment variable was set on the job template. This functionality has now been factored out to the method _get_submit_script_environment_variables (see PR 5283). Instead of formatting the environment variables themselves, it is advised that plugin simply call this function from _get_submit_script_header and include the generated string in the returned string.

Miscellaneous

The Transport.get_valid_transports() method has been removed, use get_entry_point_names('aiida.transports') instead, with aiida.plugins.entry_point.get_entry_point_names.
The Scheduler.get_valid_transports() method has been removed, use get_entry_point_names('aiida.schedulers') instead, with aiida.plugins.entry_point.get_entry_point_names.

Unit tests

This affects only plugins still using the PluginTestCase class.

Background

Since 2017 (v0.11.0), AiiDA offered a PluginTestCase class that made it easy for plugin developers set up a fully functioning test environment. The test class was originally designed to work with the unittest package, but testing in aiida-core (as well as most plugins) moved to pytest.

The PluginTestCase class could still be run through pytest (and the aiida-plugin-cutter included an example of this), but as testing through unittest is being deprecated, the PluginTestCase only adds extra code to maintain and will be removed.

Migrating to `pytest`

The canonical way of writing tests in pytest is through simple test functions and pytest fixtures. See the pytest documentation for details.

However, pytest also offers support for test classes with unittest-style setup methods. For a minimalist approach to removing the dependency on the PluginTestCase, see this migration diff from the aiida-plugin-cutter.

The fixtures clear_database, clear_database_after_test, clear_database_before_test are now deprecated, in favour of the aiida_profile_clean fixture, which ensures (before the test) the default profile is reset with clean storage, and that all previous resources are closed. If you only require the profile to be reset before a class of tests, then you can use aiida_profile_clean_class.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AiiDA 2.0 plugin migration guide

Quick Start

Detailed discussion

Installation

Update pip

Delete old editable installs

Deprecation warnings

`verdi`

Tab-completion

`verdi code setup`

Entry points

Node namespace restructuring

Repository

Using `open` in a context manager

Writing cross-compatible code

`QueryBuilder`

REST API

Transport plugins

Equality comparison of `Dict` nodes

Default cell in `StructureData` nodes

Schedulers

Miscellaneous

Unit tests

Background

Migrating to `pytest`

Clone this wiki locally

AiiDA 2.0 plugin migration guide

Quick Start

Detailed discussion

Installation

Update pip

Delete old editable installs

Deprecation warnings

verdi

Tab-completion

verdi code setup

Entry points

Node namespace restructuring

Repository

Using open in a context manager

Writing cross-compatible code

QueryBuilder

REST API

Transport plugins

Equality comparison of Dict nodes

Default cell in StructureData nodes

Schedulers

Miscellaneous

Unit tests

Background

Migrating to pytest

Clone this wiki locally

`verdi`

`verdi code setup`

Using `open` in a context manager

`QueryBuilder`

Equality comparison of `Dict` nodes

Default cell in `StructureData` nodes

Migrating to `pytest`