Install python into a venv (#718)

- Continues to use python and venv from Ubuntu LTS repositories, so they
are supported as with everything else that is gotten from apt (see
#670 (comment))
- Doesn't currently change any permissions, so present behavior is
preserved. However, in the future, we should probably change ownership
so end users can install packages in there at runtime (see
#670 (comment))
- Sets the VIRTUAL_ENV environment variable to path of the venv we
create. This is what the `source activate` script does, and Reticulate
also looks for this to discover which python to use (see point 4 of
https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery)
- Sets up PATH appropriately, so python and python3 refer to what is in
our venv. This, along with the previous step, ensures same behavior as
users typing `source ${VIRTUAL_ENV}/bin/activate` without actually
having to do that, preserving end user behavioral semantics. See
#670 (comment)
- RStudio is also told about new `PATH` and `VIRTUAL_ENV`, using the
same pattern as `install_texlive.sh`
- Remove the explicit symlink of python3 -> python, as venv handles this
automatically.
- `install_python.sh` now needs to be `source`d, following same pattern
as `install_texlive.sh`

Decisions to be made:

- Where do we set the appropriate env variables (VIRTUAL_ENV and PATH)?
They need to be set for `install_python.sh` to work correctly. I've set
them in the binder image for now, but it should probably be set on a
more base image. This is a no-op if `install-python.sh` is not called
anywhere.

TODO:
- [x] Update `NEWS` (can be done once everything else is finalized)

Ref #670

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

NEWS.md

@@ -1,5 +1,20 @@
 # News
 
+## 2023-12
+
+### Changes in pre-built images
+
+- For images that ship with a python environment installed, the default python
+  environment will now be a [virtual environment](https://docs.python.org/3/library/venv.html)
+  created at `/opt/venv`. ([#718](https://github.com/rocker-org/rocker-versioned2/pull/718))
+
+### Changes in rocker_scripts
+
+- `install_python.sh` will now setup a [virtual environment](https://docs.python.org/3/library/venv.html)
+  under `/opt/venv`, and this will be the default python environment (accomplished via setting
+  the `$PATH` and `$VIRTUAL_ENV` variables). `/opt/venv` will be owned by the `staff` group so non-root
+  users can safely install python packages. ([#718](https://github.com/rocker-org/rocker-versioned2/pull/718))
+
 ## 2023-11
 
 ### Changes in pre-built images

dockerfiles/binder_devel.Dockerfile

@@ -6,6 +6,8 @@ LABEL org.opencontainers.image.licenses="GPL-2.0-or-later" \
       org.opencontainers.image.authors="Carl Boettiger <[email protected]>"
 
 ENV NB_USER=rstudio
+ENV VIRTUAL_ENV=/opt/venv
+ENV PATH=${VIRTUAL_ENV}/bin:${PATH}
 
 RUN /rocker_scripts/install_jupyter.sh
 

scripts/install_jupyter.sh

@@ -26,8 +26,9 @@ if ! id -u "${NB_USER}" >/dev/null 2>&1; then
     /rocker_scripts/default_user.sh "${NB_USER}"
 fi
 
-# install python
-/rocker_scripts/install_python.sh
+# install python & setup venv
+# shellcheck source=/dev/null
+source /rocker_scripts/install_python.sh
 
 python3 -m pip install --no-cache-dir jupyter-rsession-proxy notebook jupyterlab jupyterhub
 

scripts/install_python.sh

@@ -22,13 +22,26 @@ apt_install \
     python3-venv \
     swig
 
+# Setup a virtualenv to install things into
+
+# Put things under /opt/venv, if nothing else is specified
+export VIRTUAL_ENV="${VIRTUAL_ENV:=/opt/venv}"
+export PATH="${VIRTUAL_ENV}/bin:${PATH}"
+
+# Make sure that Rstudio sees these env vars too
+echo "PATH=${PATH}" >>"${R_HOME}/etc/Renviron.site"
+echo "VIRTUAL_ENV=${VIRTUAL_ENV}" >>"${R_HOME}/etc/Renviron.site"
+
+python3 -m venv "${VIRTUAL_ENV}"
+
+# Upgrade version of pip inside the virtualenv
 python3 -m pip --no-cache-dir install --upgrade \
     pip
 
-# Some TF tools expect a "python" binary
-if [ ! -e /usr/local/bin/python ]; then
-    ln -s "$(which python3)" /usr/local/bin/python
-fi
+# Make the venv owned by the staff group, so users can install packages
+# without having to be root
+chown -R root:staff "${VIRTUAL_ENV}"
+chmod g+ws "${VIRTUAL_ENV}"
 
 install2.r --error --skipmissing --skipinstalled -n "$NCPUS" reticulate
 
@@ -52,6 +65,12 @@ if [ "${UBUNTU_CODENAME}" == "focal" ]; then
     fi
 fi
 
+# Check that python and python3 point to correct places
+echo "Check python, python3 and pip executables point to the correct place..."
+echo "python -> $(which python)"
+echo "python3 -> $(which python3)"
+echo "pip -> $(which pip)"
+
 # Check Python version
 echo -e "Check the Python to use with reticulate...\n"
 

stacks/devel.json

@@ -127,7 +127,9 @@
       },
       "FROM": "rocker/geospatial:devel",
       "ENV": {
-        "NB_USER": "rstudio"
+        "NB_USER": "rstudio",
+        "VIRTUAL_ENV": "/opt/venv",
+        "PATH": "${VIRTUAL_ENV}/bin:${PATH}"
       },
       "RUN": [
         "/rocker_scripts/install_jupyter.sh"