Skip to content

AiiDA Dependency Management

Jump to bottom
Carl Simon Adorf edited this page Mar 9, 2020 · 44 revisions

AiiDA Dependency Management

About

This page contains guidelines, concrete procedures, and any other information relevant to the management of dependencies for the aiida-core package and plugins.

General information

The dependencies for the aiida-core package must be managed according to AEP 002.

Dependency Manager (DM):

The current dependency manager should be a member of the dependency-manager team.

Specification of AiiDA core dependencies

The dependencies for the aiida-core package are specified in the setup.json file are divided into install_requires and extras_require categories with various sub-categories.

Furthermore, dependencies are also specified in

  • environment.yml: to create a conda environment
  • docs/requirements_for_rtd.txt: for building documentation
  • pyproject.toml: to specify requirements for building the package
  • requirements.txt: An automatically updated file, that contains a complete Python environment, that is tested to work.

The specification of a dependency within the setup.json is to be considered authoritative.

In addition to above mentioned files, the DM must also be concerned with the modification of the following files:

  • setup.py: Indirectly specifies install dependencies sourced from setup.json
  • util/dependency_management.py: Utility script used for dependency management related tasks.
  • .github/workflows/dm.yml: Specifies the DM-related CI workflow

Utility script

The utils/dependency_management.py script within the aiida-core repository provides commands to validate the consistency of aforementioned files with the setup.json file as well as commands to re-generate them after updating a dependency.

Adding new dependencies

To add a new dependency for the aiida-core package as part of a related change, make sure that the dependency adheres to the following requirements:

The newly introduced dependency

  • is needed to close a non-trivial feature gap that could not be resolved easily otherwise,
  • supports all Python versions supported by aiida-core (as specified in setup.json),
  • is available on PyPI and conda-forge [1],
  • appears to be within a stable development stage, e.g., has reached a version 1.0 for projects that follow semantic versioning.

[1] In case that a dependency is not yet available through aforementioned channels, but is considered critical enough to be included anyways, the following steps can be taken:

  1. For lightweight dependencies, consider to vendor the package if permissible by license.
  2. Create request to current maintainer to make the dependency available.
  3. Maintain the PyPI project and conda-forge recipes ourselves.

Updating dependencies

To update dependencies, modify the affected entries in setup.json and then update all dependent files with

./util/dependency_management.py generate-all

For packages that are named differently between PyPI and conda-forge, you might need to add an entry to the SETUPTOOLS_CONDA_MAPPINGS variable within the same script. Pushing a modification to these files will automatically trigger the dependency-management workflow for GitHub actions, which executes extended tests regarding the validity and feasibility of the update (see also the next section).

Continuous Integration

The consistency of the various files with setup.json is checked with a pre-commit hook that is executing the validate-* commands of utils/dependency_management.py. This hook is also executed for all commits as part of the aiida-core CI workflow.

Furthermore, a dedicated workflow (dm) to check whether the current dependencies can be installed with pip and conda, and tests pass for all supported Python versions, is defined in the .github/workflows/dm.yml file.

The DM-dedicated workflow is executed nightly, as well as for all commits on the develop branch and all branches that are prefixed with dm/ in case that any of the dependency-related files have been modified.

As part of the release workflow (executed on branches with prefix release/, a PR for a revision of the requirements.txt file is automatically created, in case that it differs from the current version.

Problematic dependencies

This is a list of dependencies that might be problematic for some reason.

Impact on the build and install chain of aiida-core:

  1. Critical disruption/blocker
  2. Major disruption with user impact
  3. Minor disruption without user impact
  4. Concern, but so far no impact.
  • [1] tornado: old major version <5 required, causing issues for example in the context of AiiDA lab
  • [1] pymatgen: Their use of specifying numpy as a requirement in setup_requires in the call to setuptools.setup() is causing issues, because it will trigger an installation (and eventual build!) of numpy within the build environment for pymatgen, which will fail on Python 3.5 with a version of setuptools<38.2.0 and is suspected to cause other potential problems in the future.
  • [2] sqlalchemy-utils: not available through conda-forge
  • [4] shortuiid: A broken release (0.5.1) was removed from PyPI after upload.
Clone this wiki locally