Installing A+ dependencies fails on the latest pip and setuptools versions

[markkuriekkinen]

Installing A+ fails (pip install -r requirements.txt) if the system uses the latest pip and setuptools version.

---

From Aplusguru tickets:
https://rt.cs.aalto.fi/Ticket/Display.html?id=20345

when trying to install new RC v1.12.0rc3 I run into situation where requirements.txt won't install with latest version of python pkg setuptools.

In the attached Docker config the pkg coming with the debian python 3.9 is version 52.0.0 and it works.

But if I run update on pip and setuptools before install, the version is 60.0.3 and then there's an error in
https://github.com/apluslms/js-jquery-toggle.git to commit d9c5219a74e0e0536fd40df2c7ba8ea842313d19:
File "/tmp/pip-install-wah_ndnm/js-jquery-toggle-django_316200f9e1904fd1bf29c2a9b3cc150f/django/setup.py", line 19, in
scheme['data'] = scheme['purelib']
KeyError: 'purelib'

Dockerfile to show the error:

FROM debian:11.2-slim
RUN apt-get update &&
export DEBIAN_FRONTEND="noninteractive" &&
apt-get install -y --no-install-recommends
locales
bash git python3.9 python3-pip gettext

RUN python3 -m pip install --upgrade pip setuptools

RUN python3 -c "import setuptools; print('VERSION', setuptools.version)"

WORKDIR /srv/aplus

ENV BRANCH=v1.12.0rc3

RUN git clone --single-branch --branch $BRANCH https://github.com/apluslms/a-plus.git
&& cd a-plus
&& git describe

WORKDIR /srv/aplus/a-plus

RUN pip3 --no-cache-dir install -r requirements.txt

---

Python setuptools v60 was released Dec 20th 2021 and it seems to produce problems with aplus installation.

When trying out aplus v1.11.4 I run into situation where install goes fine but at runtime will generate an error:
Django Version: 3.2.10
Exception Type: ModuleNotFoundError
Exception Value:

No module named 'rest_framework_extensions'

Exception Location: ./api/urls_v2.py, line 5, in

I don't have exact details but changing my pip and setuptools upgrade to:
python3 -m pip install --upgrade pip "setuptools<60"
(e.g. don't use setuptools 60) the problem goes away.

[markkuriekkinen]

Celery is at least one cause. Its dependency click-didyoumean does not support setuptools >= 59.7:

plus_1        | Requirement already satisfied: click-didyoumean>=0.0.3 in /usr/local/lib/python3.9/dist-packages (from celery<6,>=5.2.1->-r /srv/aplus/requirements.txt (line 31)) (0.3.0)
plus_1        | Collecting setuptools<59.7.0,>=59.1.1
plus_1        |   Downloading setuptools-59.6.0-py3-none-any.whl (952 kB)

The most recent update in click-didyoumean was made in Sep 2021.

I'm not sure if these are related to this (kombu is part of celery):

• prevent KeyError: 'purelib'  celery/kombu#1467
• KeyError when scheme hacks encounter missing purelib. pypa/setuptools#2936
• https://stackoverflow.com/questions/70417812/python-distutils-keyerror

[annirytkonen]

the schedule depends on Tampere, and I've asked for it. I add a preliminary milestone now to proceed with overview planning and let's return to the case when we know more.

[markkuriekkinen]

I took a quick look at this again. This is not related to Tampere, but the changes in the pip package manager and setuptools. Many frameworks and libraries were (at least in the beginning) incompatible with the latest pip version. A+ depends on Celery and Celery was having issues with pip/setuptools. However, a new patch has been released in Celery and that seems to fix it: https://github.com/celery/celery/releases/tag/v5.2.6

After quick testing, it looks like that rest_framework_extensions are still causing issues and thus do not work with the latest pip or setuptools. A+ uses our customized fork of rest_framework_extensions, thus we can not simply increase the version of the library. We need to port our changes in the library to the latest release of the library. Though, the rest_framework_extensions haven't been changed much since since there is no active development going on.

• https://github.com/apluslms/drf-extensions/tree/reimplement_nested_routes_v5
• the official library: https://github.com/chibisov/drf-extensions
• Jaakko's original and outdated fork: https://github.com/raphendyr/drf-extensions
• issue about updating drf-extensions: Update the version of the original drf-extensions and Django REST framework drf-extensions#1

Rest_framework_extensions depend on the Django REST framework, which we should update too assuming that rest_framework_extensions are still compatible with the latest Django REST framework.

• https://www.django-rest-framework.org/community/release-notes/

Related issues in the official library rest_framework_extensions:

• distutils has been deprecated in Python 3.10 in favour of setuptools chibisov/drf-extensions#318
• Reimplement nested routes v3 chibisov/drf-extensions#236
• Implementation of nested route framework using definitions in viewsets chibisov/drf-extensions#153

[markkuriekkinen]

I think this issue was resolved in the commit df64e84