Missing CONDA_PREFIX_1 env var to run cnmfe notebook on macOS

[rly]

In https://github.com/nel-lab/mesmerize-core/blob/master/notebooks/cnmfe.ipynb under the header "Run a single batch item"

process = row.caiman.run()

raises an error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[21], line 4
      1 # run the first "batch item"
      2 # this will run in a subprocess by default on Linux & Mac
      3 # on windows it will run locally
----> 4 process = row.caiman.run()
      6 # reload dataframe from disk when done
      7 df = df.caiman.reload_from_disk()

File ~/mambaforge/envs/mescore/lib/python3.11/site-packages/mesmerize_core/caiman_extensions/cache.py:214, in Cache.invalidate.<locals>._invalidate.<locals>.__invalidate(instance, *args, **kwargs)
    208 if pre:
    209     self.cache.drop(
    210         self.cache.loc[self.cache["uuid"] == u].index,
    211         inplace=True
    212     )
--> 214 rval = func(instance, *args, **kwargs)
    216 if post:
    217     self.cache.drop(
    218         self.cache.loc[self.cache["uuid"] == u].index,
    219         inplace=True
    220     )

File ~/mambaforge/envs/mescore/lib/python3.11/site-packages/mesmerize_core/caiman_extensions/common.py:507, in CaimanSeriesExtensions.run(self, backend, wait, **kwargs)
    504     args_str += f" --data-path {get_parent_raw_data_path()}"
    506 # make the runfile
--> 507 runfile_path = make_runfile(
    508     module_path=os.path.abspath(
    509         ALGO_MODULES[self._series["algo"]].__file__
    510     ),  # caiman algorithm
    511     filename=runfile_path,  # path to create runfile
    512     args_str=args_str,
    513 )
    514 try:
    515     self.process = getattr(self, f"_run_{backend}")(
    516         runfile_path, wait=wait, **kwargs
    517     )

File ~/mambaforge/envs/mescore/lib/python3.11/site-packages/mesmerize_core/utils.py:110, in make_runfile(module_path, args_str, filename)
    104 f.write(f"#!/bin/bash\n")
    106 if "CONDA_PREFIX" in os.environ.keys():
    107     f.write(
    108         f'export CONDA_PREFIX={os.environ["CONDA_PREFIX"]}\n'
    109         f'export CONDA_PYTHON_EXE={os.environ["CONDA_PYTHON_EXE"]}\n'
--> 110         f'export CONDA_PREFIX_1={os.environ["CONDA_PREFIX_1"]}\n'
    111     )
    113 elif "VIRTUAL_ENV" in os.environ.keys():
    114     f.write(
    115         f'export PATH={os.environ["PATH"]}\n'
    116         f'export VIRTUAL_ENV={os.environ["VIRTUAL_ENV"]}\n'
    117         f'export LD_LIBRARY_PATH={os.environ["LD_LIBRARY_PATH"]}\n'
    118     )

File <frozen os>:679, in __getitem__(self, key)

KeyError: 'CONDA_PREFIX_1'

It seems like this environment variable is assumed to have been set but I do not have it set by default?

I am on a Mac M1 on a zsh shell with:

mamba 1.5.1
conda 23.7.2

In case it makes a difference, I had installed conda and mamba using mambaforge. My conda-related environment vars in this mescore env are:

CONDA_CHANGEPS1=no
CONDA_EXE=/Users/rly/mambaforge/bin/conda
_CE_M=
_CE_CONDA=
CONDA_PYTHON_EXE=/Users/rly/mambaforge/bin/python
CONDA_SHLVL=1
VIRTUAL_ENV_DISABLE_PROMPT=12
CONDA_PREFIX=/Users/rly/mambaforge/envs/mescore
CONDA_DEFAULT_ENV=mescore
CONDA_PROMPT_MODIFIER=

I installed mesmerize using the current instructions in the README after the modification in #228.

[rly]

My .condarc file is:

channels: [conda-forge]
auto_activate_base: false

I don't really know what CONDA_PREFIX_1 is but perhaps CONDA_PREFIX and CONDA_PREFIX_1 maintain a stack of the last activated conda environments. I think because I do not have any conda environment activated by default in my shell, when I activate mescore, CONDA_PREFIX_1 is not set. Sure enough, if I activate some other environment dev and then activate mescore, then I have the env vars:

CONDA_PREFIX=/Users/rly/mambaforge/envs/mescore
CONDA_PREFIX_1=/Users/rly/mambaforge/envs/dev

[kushalkolar]

Thanks for looking into this! What happens when you activate a third env, does a new env variable with "2" show up?

Is the current environment with or without a number?

[rly]

If I run:

conda activate dev
conda activate mescore
conda activate spyglass
conda activate base
conda activate test

Then the related env vars are:

CONDA_CHANGEPS1=no
CONDA_EXE=/Users/rly/mambaforge/bin/conda
_CE_M=
_CE_CONDA=
CONDA_PYTHON_EXE=/Users/rly/mambaforge/bin/python
CONDA_SHLVL=5
CONDA_PREFIX=/Users/rly/mambaforge/envs/test
CONDA_DEFAULT_ENV=test
CONDA_PROMPT_MODIFIER=
CONDA_PREFIX_1=/Users/rly/mambaforge/envs/dev
CONDA_PREFIX_2=/Users/rly/mambaforge/envs/mescore
CONDA_PREFIX_3=/Users/rly/mambaforge/envs/spyglass
CONDA_PREFIX_4=/Users/rly/mambaforge

CONDA_SHLVL also goes up. I think you should be able to use CONDA_PREFIX to get the path to the current environment. I'm not sure why mesmerize needs to save CONDA_PREFIX_1.

[kushalkolar]

Thanks! Do you want to make a PR for this so it uses only CONDA_PREFIX?

I'm not very familiar with conda, I use only pip for my development. Mesmerize-core is offered on conda mainly because caiman is only on conda.

I probably put both env vars into it because I saw that both referenced the current env when I wrote that.

[ethanbb]

I just ran into this bug as well, seems easy to fix, can I help push this forward?

[kushalkolar]

I just ran into this bug as well, seems easy to fix, can I help push this forward?

rly did a PR here, if that fix works for you as well I'll merge it #232