Address virtualenv compatibility issues (pip check)

[chrisjsewell]

Currently, the pip check is commented out, because it fails:

quantum-mobile/playbook-build.yml

Lines 123 to 132 in b2632be

	# TODO this is currently failing due to:
	# aiidalab 20.7.0b2 has requirement numpy==1.17.5, but you have numpy 1.17.4.
	# post_tasks:
	# - name: Check package version integrity in virtualenv
	# command: "{{ aiida_venv }}/bin/pip check --no-color"
	# become: true
	# become_user: "{{ vm_user }}"
	# changed_when: false
	# tags: [aiida, aiidalab, pipcheck]

This should be fixed in aiida and aiidalab roles then uncommented.

Holistically, it would perhaps be ideal to have a better solution for installing packages into the virtualenv.
Creating a virtualenv in a single step, e.g. with pip install -r requirements.txt, is usually fine; correctly solving compatible dependencies or failing if it can't resolve.
The problem comes when you start having subsequent installs (pip install apackage), which pip seems to allow clobbering existing dependency requirements, which only shows up when you subsequently run pip check.

Currently, the process is to:

1. Create the virtualenv
2. Install aiidalab + deps
3. Install aiida-core + deps
4. Iteratively install each aiida plugin + deps

the other issue is also that, e.g. if the aiidalab role installs (and is tested against) aiida-core v1.3.0, then the aiida role "overwrites" this install with aiida-core v1.4.2.
At the moment you just have to be manually careful to add compatible aiida and aiidalab role versions.

even if the aiida and aiidlab roles pass their testing independently, this does not necessarily ensure that there will not be issues introduced in the full build.

Perhaps the pip 2020 resolver will improve this issue?
(also conda tends to handle this a lot better, but that would be a big effort to start including conda in the VM)

cc @ltalirz and @csadorf

[chrisjsewell]

I think I've solved this in the aiida role: marvel-nccr/ansible-role-aiida#48 (comment) 😄

Just need to do something similar for the aiidalab role

[chrisjsewell]

As suggested by @webknjaz in marvel-nccr/ansible-role-aiida#48, we may want to consider using pip-tools and "lock" files, to keep the virtualenv reproducible

[chrisjsewell]

Also, as I was just saying to @csadorf

with quantum mobile, we in essence have two competing goals:

• in one sense it is a tool for building VMs, in which case we want the user to have the flexibility to create a custom build to their own specification (with particular codes/plugins etc).
  In this case we probably don't want to constrain them to a fixed lock file

• On the other hand we offer the "pre-made" VM releases which are more like a service and the environment should be fully "validated". In which case a lock file may be better

For aiidalab, it is currently very focussed on the latter and "hard-codes" the whole environment, encapsulated in a docker image with a set version for the whole image (this version incorporates not just the python versions, but also the DFT code versions and other software like postres and rabbitmq)

For quantum-mobile we need to consider what the balance is