Skip to content
This repository has been archived by the owner on Dec 11, 2024. It is now read-only.

Create environment from environment.yml #73

Open
matthewfeickert opened this issue Oct 24, 2022 · 5 comments
Open

Create environment from environment.yml #73

matthewfeickert opened this issue Oct 24, 2022 · 5 comments

Comments

@matthewfeickert
Copy link

matthewfeickert commented Oct 24, 2022

To build ontop of these images in an easier manner, it would be quite useful if the images were constructed from an environment.yml file that was copied into the image and then built from. So replacing

RUN mamba install --yes \

with

RUN mamba install --yes --file /environment.yml

This would also make it easier to be able to create a lock file for the environment with conda-lock and then to build on top of these images by being able to extend the environment.yml and conda-lock files as needed.

Is this in scope?

@oshadura
Copy link
Member

oshadura commented Oct 26, 2022

@lgray do you have any preferences?

(for reference Matthew is working on the generic IRIS-HEP Analysis System docker image and we had an idea to use docker-coffea-base as a baseline image for it)

@nsmith-
Copy link
Member

nsmith- commented Oct 25, 2023

I am trying this now as part of making a coffea2023 image, and of course there is an issue in that any conda-installable pytorch conflicts with nomkl and if we pip-install it then we cannot at the same time pip-install torch-scatter, etc. since their setup seems to require the ability to import torch. Perhaps we just take the hit and start using Intel MKL?

@matthewfeickert
Copy link
Author

matthewfeickert commented Oct 26, 2023

I am trying this now as part of making a coffea2023 image, and of course there is an issue in that any conda-installable pytorch conflicts with nomkl and if we pip-install it then we cannot at the same time pip-install torch-scatter, etc. since their setup seems to require the ability to import torch. Perhaps we just take the hit and start using Intel MKL?

@nsmith- I don't know the specifics of why you do/don't want PyTorch with nomkl, but if you haven't already I would make an Issue with torch-scatter as requiring torch to be available at install time and not using a PEP 518 build-system to provide it at build time should be considered a bad bug these days. (edit: c.f. rusty1s/pytorch_scatter#265)

@nsmith-
Copy link
Member

nsmith- commented Oct 26, 2023

The original impetus was to reduce the image size, but now that our image has ballooned to 1.4GB, saving the 200MB for MKL seems less important. In https://github.com/CoffeaTeam/docker-coffea-base/blob/coffea2023/base/environment.yml I managed to build a first image and manually uploaded it to dockerhub.

@nsmith-
Copy link
Member

nsmith- commented Feb 1, 2024

FYI we're consolidating the images/environments in CoffeaTeam/af-images#1

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants