[Packaging] Can the environment and packaging management be more unified?

[marshallmcdonnell]

This is part of my openjournals/joss-reviews#4494
NOTE: Only an optional suggestion, definitely not a requested change for the review

Description

The packaging and environment management seems to be duplicating lists that appear unmatched.

Specifically, for "dev" setup, I am comparing requirements.txt to conda/environment-dev.yaml

For the "production" setup, I am comparing setup.py to conda/meta.yaml.

I see where the PyPi package publishing occurs using setup.py here as a CD pipeline: https://github.com/RECETOX/MSMetaEnhancer/blob/main/.github/workflows/publish_pypi.yaml

Yet, I don't see the conda package publishing in CD. Maybe I am missing something?

Suggestion
Since this is a "pure" Python package, maybe you can re-use a "dev" and "prod" file for both conda setup and the non-conda python setup (for virtual environments or such)

You could have a requirements.txt and a requirements-dev.txt for "production" and "development", respectively.

Then, setup.py reads in these two files for the install_requires and test_require, like following:

def read_requirements_from_file(filepath):
    with open(filepath, 'rU') as req_file:
        return req_file.readlines()
...
setup_args = dict(
    install_requires=read_requirements_from_file('requirements.txt')),
    tests_require=read_requirements_from_file('requirements-dev.txt')
)
...
setuptools.setup(
    ...
    install_requires=setup_args['install_requires'],
    tests_require=setup_args['install_requires'] + setup_args['tests_require'],
    ...
)

Then, you can probably have your meta.yml conda recipe simplified to something like:

build:
  noarch: python
  number: 0
  script: python -m pip install --no-deps --ignore-installed .

requirements:
  host:
    - python
    - pip
  run:
    - python

NOTE: Just copied this from here

And you can do away with the conda/environment-dev.yml
Instead just have in the README under "developer docs" that you can use:

conda create --name MSMetaEnhancer-dev --file requirements.txt
conda activate MSMetaEnhancer-dev
pip install -r requirements-dev.txt

Overall,Just throwing the idea out there since it looks like this is becoming an issue in your project. I haven't completely fleshed out the idea but hopefully provided enough context for a possible strategy. Feel free to close if not helpful.

[xtrojak]

Thank you for suggestions @marshallmcdonnell.

Unfortunately, it is not possible to unify the environment and packaging management completely, conda and pypi package managers are just too separated (see details in this discussion).

Still we were able to improve the local environment and requirements files in #121 as you suggested, so thank you for that!

---

Yet, I don't see the conda package publishing in CD. Maybe I am missing something?

Conda package publishing is done automatically by a bioconda bot which checks for releases in out repository and creates an update PR in bioconda recipes repository.

[marshallmcdonnell]

Apologies for the delay in response and thanks for explaining! Completely understand and thanks for closing!
👍