Install python into a venv #718

yuvipanda · 2023-11-02T09:36:35Z

Continues to use python and venv from Ubuntu LTS repositories, so they are supported as with everything else that is gotten from apt (see Using pip without venv or conda env will stop working soon #670 (comment))
Doesn't currently change any permissions, so present behavior is preserved. However, in the future, we should probably change ownership so end users can install packages in there at runtime (see Using pip without venv or conda env will stop working soon #670 (comment))
Sets the VIRTUAL_ENV environment variable to path of the venv we create. This is what the source activate script does, and Reticulate also looks for this to discover which python to use (see point 4 of https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery)
Sets up PATH appropriately, so python and python3 refer to what is in our venv. This, along with the previous step, ensures same behavior as users typing source ${VIRTUAL_ENV}/bin/activate without actually having to do that, preserving end user behavioral semantics. See Using pip without venv or conda env will stop working soon #670 (comment)
RStudio is also told about new PATH and VIRTUAL_ENV, using the same pattern as install_texlive.sh
Remove the explicit symlink of python3 -> python, as venv handles this automatically.
install_python.sh now needs to be sourced, following same pattern as install_texlive.sh

Decisions to be made:

Where do we set the appropriate env variables (VIRTUAL_ENV and PATH)? They need to be set for install_python.sh to work correctly. I've set them in the binder image for now, but it should probably be set on a more base image. This is a no-op if install-python.sh is not called anywhere.

TODO:

Update NEWS (can be done once everything else is finalized)

Ref #670

yuvipanda · 2023-11-02T09:36:47Z

Created this draft mostly to see what CI says!

Discovered while working on #718

yuvipanda · 2023-11-02T18:19:19Z

ok, this is ready for a first round of review :)

dockerfiles/binder_devel.Dockerfile

scripts/install_python.sh

stacks/devel.json

- Continues to use python and venv from Ubuntu LTS repositories, so they are supported as with everything else that is gotten from apt (see rocker-org#670 (comment)) - Doesn't currently change any permissions, so present behavior is preserved. However, in the future, we should probably change ownership so end users can install packages in there at runtime (see rocker-org#670 (comment)) - Sets the VIRTUAL_ENV environment variable to path of the venv we create. This is what the `source activate` script does, and Reticulate also looks for this to discover which python to use (see point 4 of https://rstudio.github.io/reticulate/articles/versions.html#order-of-discovery) - Sets up PATH appropriately, so python and python3 refer to what is in our venv. This, along with the previous step, ensures same behavior as users typing `source ${VIRTUAL_ENV}/bin/activate` without actually having to do that, preserving end user behavioral semantics. - Remove the explicit symlink of python3 -> python, as venv handles this automatically. Decisions to be made: - Where do we set the appropriate env variables (VIRTUAL_ENV and PATH)? They need to be set for `install_python.sh` to work correctly. I've set them in the binder image for now, but it should probably be set on a more base image. This is a no-op if `install-python.sh` is not called anywhere. Ref rocker-org#670

So it sets PATH appropriately

eitsupi · 2023-11-28T10:10:49Z

What is the current status of this?

yuvipanda · 2023-11-29T21:47:01Z

@cboettig gave me some feedback via slack that I need to dig into. @cboettig can you put them here too?

cboettig · 2023-12-02T01:59:31Z

Thanks @yuvipanda !

Yup, I think the main thing is considering changing the permissions on /opt/venv from root to something else. (In the R site library we put these in the staff group, allowing any user added to that group to update or add packages to the shared site library. Not sure how nicely pip plays with group-based vs user-based permissions though? In my use/testing of this image I've just been making /opt/venv owned by the default user, https://github.com/boettiger-lab/nasa-topst-env-justice/blob/main/.devcontainer/Dockerfile#L16

A second more minor issue is that under this setup, the default python now points to /opt/venv/bin/python as intended, but if a user calls pip install instead of python -m pip install, they still get /usr/bin/pip instead of /opt/venv/bin/pip, which creates a gotcha that it would be better to avoid if possible??

yuvipanda · 2023-12-20T18:34:10Z

A second more minor issue is that under this setup, the default python now points to /opt/venv/bin/python as intended, but if a user calls pip install instead of python -m pip install, they still get /usr/bin/pip instead of /opt/venv/bin/pip, which creates a gotcha that it would be better to avoid if possible??

Hmm, since there should be a pip in /opt/venv/bin/pip, if /opt/venv/bin is in $PATH, that should mean pip should call that pip by default, right? Is that not what's happening?

yuvipanda · 2023-12-20T18:35:08Z

Making /opt/venv be owned by staff seems reasonable to me. If it works for R, my intuition is that it will also work for Python!

- Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

cboettig · 2023-12-20T20:58:10Z

@eitsupi ok just tested and this looks ready to merge to me!

stacks/devel.json

yuvipanda · 2023-12-20T21:28:09Z

Yay that's awesome, thank you @cboettig!

eitsupi · 2023-12-20T22:50:44Z

scripts/install_python.sh

-fi
+# Make the venv owned by the staff group, so users can install packages
+# without having to be root
+chown -R rstudio:staff /opt/venv


Don't hard-code the username. There is no guarantee that the user exists.

@eitsupi in install_texlive.sh I see:

NON_ROOT_USER=$(getent passwd "1000" | cut -d: -f1)

and this is used as username. Should I do that?

Is there a bad reason not to be the root user?
If so, please do so.

However, in that case, I don't think it will work if the uid is anything other than 1000

Ok, given I was trying to replicate what happens in R, I've set this to match what is done to $R_HOME/site-library

eitsupi

The tests seem passed. Thanks!

eitsupi · 2023-12-20T23:20:56Z

Could you update the NEWS file?

yuvipanda · 2023-12-21T04:57:36Z

I always seem to struggle with updating NEWS but gave it a shot!

eitsupi · 2023-12-21T05:04:40Z

Looks good.

The CI failure is notthing to do with changes in this PR, so merging.

yuvipanda · 2023-12-21T05:07:10Z

Awesome! Thanks a lot, @eitsupi! Will this trigger a rebuild of the images?

eitsupi · 2023-12-23T00:34:34Z

Sorry, my review was not thorough enough (i.e. this should not have been merged yet), the fix for the stack file is completely incomplete here.
We had to update it manually for all versions.
See the issue #739.

yuvipanda mentioned this pull request Nov 2, 2023

Document R dependencies of make setup #719

Merged

eitsupi pushed a commit that referenced this pull request Nov 2, 2023

Document R dependencies of make setup (#719)

485c8bb

Discovered while working on #718

yuvipanda force-pushed the venv branch 3 times, most recently from 07af31c to 484342f Compare November 2, 2023 18:16

yuvipanda marked this pull request as ready for review November 2, 2023 18:18

cboettig requested changes Nov 6, 2023

View reviewed changes

dockerfiles/binder_devel.Dockerfile Show resolved Hide resolved

scripts/install_python.sh Show resolved Hide resolved

stacks/devel.json Show resolved Hide resolved

yuvipanda added 10 commits November 6, 2023 14:23

Use default value for VIRTUAL_ENV if not provided

83a48b1

source install_python in install_jupyter.sh

dcd8b03

So it sets PATH appropriately

Satisfy shellcheck

7497edc

Don't try to lint install_python.sh

63c823e

Check that python & python3 point to the correct place

dbb8453

Prett-print python and python3 paths

ee5b9a8

Fix PATH export in install_python.sh

0bb2798

Rely on echo putting a newline at the end automatically

541b676

Make sure RStudio sees new env vars we set

c1a4724

yuvipanda force-pushed the venv branch from 484342f to c1a4724 Compare November 6, 2023 08:53

cboettig mentioned this pull request Nov 8, 2023

Refactor images, start from same base geocompx/docker#40

Open

github-actions bot and others added 4 commits December 20, 2023 11:05

Automatic update of container definition files (rocker-org#723)

ca9d697

- Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Swap the order of ${VIRTUAL_ENV}/bin

aa601a2

Check that pip points to the correct place too

c33f53c

Make the venv be owned by non-root user

96cdc36

cboettig self-requested a review December 20, 2023 20:46

Merge remote-tracking branch 'upstream/master' into venv

ed57f02

cboettig approved these changes Dec 20, 2023

View reviewed changes

stacks/devel.json Show resolved Hide resolved

eitsupi requested changes Dec 20, 2023

View reviewed changes

Set permissions + ownership correctly for venv dir

9162281

eitsupi approved these changes Dec 20, 2023

View reviewed changes

Add NEWS entry

bda44e6

eitsupi merged commit 28085cc into rocker-org:master Dec 21, 2023
31 of 33 checks passed

yuvipanda mentioned this pull request Dec 22, 2023

rocker/binder broken (jupyter not in $PATH) #739

Closed

This was referenced Jan 13, 2024

2i2c Access Policy Hackdays: every-2-months Mondays 12-1pm PT NASA-Openscapes/2i2cAccessPolicies#7

Open

Using pip without venv or conda env will stop working soon #670

Closed

eitsupi mentioned this pull request Dec 5, 2024

rocker/ml:latest fails to install tensorflow due to not finding its own venv #880

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install python into a venv #718

Install python into a venv #718

yuvipanda commented Nov 2, 2023 •

edited by eitsupi

Loading

yuvipanda commented Nov 2, 2023

yuvipanda commented Nov 2, 2023

eitsupi commented Nov 28, 2023

yuvipanda commented Nov 29, 2023

cboettig commented Dec 2, 2023

yuvipanda commented Dec 20, 2023

yuvipanda commented Dec 20, 2023

cboettig commented Dec 20, 2023

yuvipanda commented Dec 20, 2023

eitsupi Dec 20, 2023

yuvipanda Dec 20, 2023

eitsupi Dec 20, 2023

yuvipanda Dec 20, 2023

eitsupi left a comment

eitsupi commented Dec 20, 2023

yuvipanda commented Dec 21, 2023

eitsupi commented Dec 21, 2023

yuvipanda commented Dec 21, 2023

eitsupi commented Dec 23, 2023

Install python into a venv #718

Install python into a venv #718

Conversation

yuvipanda commented Nov 2, 2023 • edited by eitsupi Loading

yuvipanda commented Nov 2, 2023

yuvipanda commented Nov 2, 2023

eitsupi commented Nov 28, 2023

yuvipanda commented Nov 29, 2023

cboettig commented Dec 2, 2023

yuvipanda commented Dec 20, 2023

yuvipanda commented Dec 20, 2023

cboettig commented Dec 20, 2023

yuvipanda commented Dec 20, 2023

eitsupi Dec 20, 2023

Choose a reason for hiding this comment

yuvipanda Dec 20, 2023

Choose a reason for hiding this comment

eitsupi Dec 20, 2023

Choose a reason for hiding this comment

yuvipanda Dec 20, 2023

Choose a reason for hiding this comment

eitsupi left a comment

Choose a reason for hiding this comment

eitsupi commented Dec 20, 2023

yuvipanda commented Dec 21, 2023

eitsupi commented Dec 21, 2023

yuvipanda commented Dec 21, 2023

eitsupi commented Dec 23, 2023

yuvipanda commented Nov 2, 2023 •

edited by eitsupi

Loading