SLEAP 1.3.1 (#1373)

* Disable data caching by default for SingleImageVideos (#1243) * Disable data caching by default for SingleImageVideos * Remove a couple things that shouldn't be in this PR :) * Centralize video extensions (#1244) * Add video extension support lists to io.video module * Use centralized extension definitions * Remove some unused code * Add some coverage for indirect coverage reduction * Fix single frame GUI increment (#1254) * Fix frame increment for single frame videos * Add test and lint * Add video search to "robustify" (excessively) fragile test (#1262) Add video search to fix fragile test * Organize docks (#1265) * Add `DockWidget` class and subclasses * Create docks in `MainWindow` using `DockWidget` classes * Remove unused imports * Fix references in existing tests * Fix intermittent bug (file-corruption) that has snuck into the tests (#1267) * Remove corrupted test file * Add non-corrupted copy of test file * Rename the file so tests run * Layout docks in a tab configuration (instead of stacked) (#1289) * Add tab with to docks (and use `self.model` in table) * Use `self.model_type` in `create_models` * Increase GUI crop size range from 512 to 832 (#1295) * Fix conversion to numpy array when last frame(s) do not have labels (#1307) * Ensure frames to predict list is unique (#1293) * Ensure frames to predict list is unique * Ensure frames to predict on are ordered correctly * Better frame sorting * Fix GUI resume training (#1314) * Do not choose `top_k` instances if `max_instances` < num centroids (#1313) * Use max_instances as a max without any minimum requirement * Create test (that fails) * Fix test by re-init inference model each call * Do not compute top k if max is greater * Add `--max_instances` to `sleap-track` and GUI (#1305) * Expose --max_instances to sleap-track and in GUI * Lint * Change shorthand to `-n` for both `sleap-export` and `sleap-track` * Add test for creating predictor from cli * Add max instances support to bottom up model (#1306) * Add max instances support to bottom up model * Remove unnecessary attribute setter in CLI parser * Edge case * Expose max_instances for BU to train/infer GUI --------- Co-authored-by: roomrys <[email protected]> * Update docs for `--max_instances` command * Add test for BU `--max_instances` --------- Co-authored-by: Talmo Pereira <[email protected]> * Remove `--labels` and redundant `data_path` (#1326) * Create copy of config info to modify (gui) (#1325) * Fixes GPU memory polling using environment variable filtering (#1272) * add cuda visible gpus to nvidia smi CLI * tests masked none, single and multiple GPUs * deal with comma separated lists and test 2 gpus * clean up system script and better test that actually works * add a case for cuda_visible_devices = [] * add test for gpu order and length in nvidia and tf * test nvidia smi indices and checking gpu memory * fix spaces * fix linting with Black * put skips in tests for git not having nvidia-smi * add doc strings to get_gpu_memory * set CUDA device order to PCI BUS ID * test with no nvidia smi + invalid visible devices * fixed linting again * remove pci env variable, move to other issue --------- Co-authored-by: Eric Leonardis <[email protected]> * Set `split_by_inds`, `test_labels`, and `validation_labels` to default (GUI) (#1331) * Allow returning PAF graph during low level inference (#1329) * allow returning the paf graph during low level inference * reformatted with black * Fix `SingleImageVideo` caching (#1330) * Set `SingleImageVideo.caching` as a class attribute * Modify tests for `SingleImageVideo.caching` * Add caching as a preference and menu checkbox * Test `SingleImageVideo.toggle_caching` * Remove GUI elements for `SingleImageVideo.CACHING` * Remove remaining prefs for `SingleImageVideo.CACHING` * Add depreciated comment * Clean-up * Update comments * Bump to 1.3.1 (#1335) * Bump to 1.3.1 * Test build to dev label * Update environment creation (#1366) * First 'working' env for 1.3.1, no GPU... * First working 1.3.1 w/gpu * Sorry, this is the first gpu working env * Fix `imgstore` version-dependent bug * Build and entry points work, but pip packages not installed * Add default channels in condarc.yaml * Rename environment.yaml to .yml * Working build no gpu (documented) * Env working w/gpu using tensorflow from pypi * Working build with GPU! And pip dep * Attempt to fix cannot find wheel ~= 0.35 issue * Run constrain everything * Use minimal conda req for install_requires, inc build number to get it working * Ubuntu build and environments working * Env creation working on M2 * Build and env working on windows (again) * Apple silicon (M2) build working * Pip package working (M2) * Exclude tests and docs from pypi wheel * Add comments to requirements * Get ready for manual build * Update os images, trigger dev build * Retry manual build win and mac * Retry mac manual build (to upload) * Require latest pynwb, remove setuptools * Remove old mac environment * Add twine as a build dependency * Add comments to manual build * Update build.yml * Rename "apple_silicon" to "mac" * Update installation docs (#1365) * Update installation docs * Trigger website build * Update quick install and build docs * Add Mambaforge links * Remove comment about experimental apple silicon support Co-authored-by: Talmo Pereira <[email protected]> * Fix minor typo * Change installation docs to reference universal mac env --------- Co-authored-by: Talmo Pereira <[email protected]> * More flexible with python version ~=3.7 * Pin py3.7.12 for host and run (good on win) * Add comments for why python ~=3.7 * Update ci workflow (#1371) * Update CI workflow to include mac * Trigger CI workflow * Remove verbose? python version * Migrate to micromamba * Explicitly state environment name * Remove environment chaching (for now) * Use different environment file for mac * Use correct syntax * Add `env_hash` and `bash -l {0}` * Remove env_file * Try different nested variable call via format * Use correct syntax * Update env file reference AS -> mac * Different path to environment file * Checkout repo * Use default shells * Remove unused comments * Fix caching attempt by changing caching key * Remove environment caching * Increase python in no_cuda to 3.7.17 * Less restrictive with python ~3.7 version * More direct installation docs 🤞 * Increase build numbers for develop branch build --------- Co-authored-by: Talmo Pereira <[email protected]> --------- Co-authored-by: Talmo Pereira <[email protected]> Co-authored-by: Eric Leonardis <[email protected]> Co-authored-by: Eric Leonardis <[email protected]> Co-authored-by: Caleb Weinreb <[email protected]>
talmolab · Jun 30, 2023 · c9113da · c9113da
1 parent 81b4342
commit c9113da
Show file tree

Hide file tree

Showing 67 changed files with 2,028 additions and 1,178 deletions.
diff --git a/.conda/README.md b/.conda/README.md
@@ -0,0 +1,21 @@
+This folder defines the conda package build for Linux and Windows. There are runners for both Linux and Windows on GitHub Actions, but it is faster to experiment with builds locally first.
+
+To build, first go to the base repo directory and install the build environment:
+
+```
+conda env create -f environment_build.yml -n sleap_build && conda activate sleap_build
+```
+
+And finally, run the build command pointing to this directory:
+
+```
+conda build .conda --output-folder build -c conda-forge -c nvidia -c https://conda.anaconda.org/sleap/ -c anaconda
+```
+
+To install the local package:
+
+```
+conda create -n sleap_0 -c conda-forge -c nvidia -c ./build -c https://conda.anaconda.org/sleap/ -c anaconda sleap=x.x.x
+```
+
+replacing x.x.x with the version of SLEAP that you just built.
diff --git a/.conda/bld.bat b/.conda/bld.bat
@@ -1,62 +1,14 @@
-@echo off
+@REM Install anything that didn't get conda installed via pip.
 
-rem # Install anything that didn't get conda installed via pip.
-rem # We need to turn pip index back on because Anaconda turns
-rem # it off for some reason. Just pip install -r requirements.txt
-rem # doesn't seem to work, tensorflow-gpu, jsonpickle, networkx,
-rem # all get installed twice if we do this. pip doesn't see the
-rem # conda install of the packages.
-
-rem # Install the pip dependencies and their dependencies. Conda turns of
-rem # pip index and dependencies by default so re-enable them. Had to figure
-rem # this out myself, ughhh.
+@REM We need to turn pip index back on because Anaconda turns it off for some reason.
 set PIP_NO_INDEX=False
 set PIP_NO_DEPENDENCIES=False
 set PIP_IGNORE_INSTALLED=False
 
-pip install numpy==1.19.5
-pip install six==1.15.0
-pip install imageio==2.15.0
-pip install attrs==21.2.0
-pip install cattrs==1.1.1
-pip install jsonpickle==1.2
-pip install networkx
-pip install nixio>=1.5.3
-@REM pip install tensorflow>=2.6.3,<=2.7.1
-@REM pip install h5py>=3.1.0,<=3.6.0
-pip install python-rapidjson
-@REM pip install opencv-python-headless>=4.2.0.34,<=4.5.5.62
-@REM pip install opencv-python @ git+https://github.com/talmolab/wrap_opencv-python-headless.git@ede49f6a23a73033216339f29515e59d594ba921
-@REM pip install pandas
-pip install psutil
-@REM pip install PySide2>=5.13.2,<=5.14.1
-pip install pyzmq
-pip install pyyaml
-pip install imgaug==0.4.0
-@REM pip install scipy>=1.4.1,<=1.7.3
-pip install scikit-image
-pip install scikit-learn==1.0.*
-pip install scikit-video
-pip install tensorflow-hub
-pip install imgstore==0.2.9
-pip install qimage2ndarray==1.9.0
-pip install jsmin
-pip install seaborn
-pip install pykalman==0.9.5
-pip install segmentation-models==1.0.1
-pip install rich==10.16.1
-pip install certifi==2021.10.8
-pip install pynwb
-pip install ndx-pose
-
-rem # Use and update environment.yml call to install pip dependencies. This is slick.
-rem # While environment.yml contains the non pip dependencies, the only thing left
-rem # uninstalled should be the pip stuff because that is left out of meta.yml
-rem conda env update -f=environment.yml
-
-rem # Install requires setuptools-scm
-pip install setuptools-scm
+@REM Install the pip dependencies. Note: Using urls to wheels might be better: 
+@REM https://docs.conda.io/projects/conda-build/en/stable/user-guide/wheel-files.html)
+pip install -r .\requirements.txt
 
-rem # Install sleap itself.
-rem # NOTE: This is the recommended way to install packages
+@REM Install sleap itself. This does not install the requirements, but will list which 
+@REM requirements are missing (see "install_requires") when user attempts to install.
 python setup.py install --single-version-externally-managed --record=record.txt
diff --git a/.conda/build.sh b/.conda/build.sh
@@ -1,51 +1,15 @@
-#!/usr/bin/env bash
-
 # Install anything that didn't get conda installed via pip.
-# We need to turn pip index back on because Anaconda turns
-# it off for some reason. Just pip install -r requirements.txt
-# doesn't seem to work, tensorflow-gpu, jsonpickle, networkx,
-# all get installed twice if we do this. pip doesn't see the
-# conda install of the packages.
 
+# We need to turn pip index back on because Anaconda turns it off for some reason.
 export PIP_NO_INDEX=False
 export PIP_NO_DEPENDENCIES=False
 export PIP_IGNORE_INSTALLED=False
 
-pip install numpy==1.19.5
-pip install six==1.15.0
-pip install imageio==2.15.0
-pip install attrs==21.2.0
-pip install cattrs==1.1.1
-pip install jsonpickle==1.2
-pip install networkx
-pip install nixio>=1.5.3
-# pip install tensorflow>=2.6.3,<=2.7.1
-# pip install h5py>=3.1.0,<=3.6.0
-pip install python-rapidjson
-# pip install opencv-python-headless==4.5.5.62
-# pip install git+https://github.com/talmolab/wrap_opencv-python-headless.git@ede49f6a23a73033216339f29515e59d594ba921
-# pip install pandas
-pip install psutil
-# pip install PySide2>=5.13.2,<=5.14.1
-pip install pyzmq
-pip install pyyaml
-pip install imgaug==0.4.0
-# pip install scipy>=1.4.1,<=1.7.3
-pip install scikit-image
-pip install scikit-learn==1.0.*
-pip install scikit-video
-pip install tensorflow-hub
-pip install imgstore==0.2.9
-pip install qimage2ndarray==1.9.0
-pip install jsmin
-pip install seaborn
-pip install pykalman==0.9.5
-pip install segmentation-models==1.0.1
-pip install rich==10.16.1
-pip install certifi==2021.10.8
-pip install pynwb
-pip install ndx-pose
+# Install the pip dependencies. Note: Using urls to wheels might be better: 
+# https://docs.conda.io/projects/conda-build/en/stable/user-guide/wheel-files.html)
+pip install -r ./requirements.txt
 
-pip install setuptools-scm
 
+# Install sleap itself. This does not install the requirements, but will list which 
+# requirements are missing (see "install_requires") when user attempts to install.
 python setup.py install --single-version-externally-managed --record=record.txt
diff --git a/.conda/conda_build_config.yaml b/.conda/conda_build_config.yaml
diff --git a/.conda/condarc.yaml b/.conda/condarc.yaml
@@ -0,0 +1,5 @@
+channels:
+  - conda-forge
+  - nvidia
+  - sleap
+  - anaconda
diff --git a/.conda/meta.yaml b/.conda/meta.yaml
@@ -1,6 +1,3 @@
-# Ref: https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
-
-# Jinja template: Process setup.py to obtain version and metadata
 {% set data = load_setup_py_data() %}
 
 
@@ -15,50 +12,79 @@ about:
   license: {{ data.get('license') }}
   summary: {{ data.get('description') }}
 
-build:
-  number: 4
-
 source:
   path: ../
 
+build:
+  number: 9
+
 requirements:
   host:
-    - python=3.7
-    # - sleap::pyside2=5.14.1
-    - conda-forge::numpy=1.19.5
-    - sleap::tensorflow=2.6.3
-    - conda-forge::pyside2=5.13.2
-    - conda-forge::h5py=3.1.0
-    - conda-forge::scipy=1.7.3
-    - conda-forge::six=1.15.0
-    - pillow=8.4.0
-    - shapely=1.7.1
-    - conda-forge::pandas
-    - ffmpeg
-    - qtpy>=2.0.1
-    - conda-forge::pip!=22.0.4
+    - conda-forge::python ==3.7.12  # Run into _MAX_WINDOWS_WORKERS not found if <
+    - numpy >=1.19.5,<1.23.0  # Linux likes anaconda, windows likes conda-forge
+    - conda-forge::cudatoolkit ==11.3.1
+    - conda-forge::cudnn=8.2.1
+    - nvidia::cuda-nvcc=11.3
+    - conda-forge::setuptools
+    - conda-forge::pip
 
-  run:
-    - python=3.7
-    - conda-forge::numpy~=1.19.5
-    - sleap::tensorflow=2.6.3
-    - conda-forge::pyside2>=5.13.2,<=5.14.1
-    - conda-forge::h5py~=3.1.0
-    - conda-forge::scipy>=1.4.1,<=1.7.3
-    - conda-forge::six~=1.15.0
-    - pillow=8.4.0
-    - shapely=1.7.1
+    # Only the packages above are required to build, but listing them all ensures no 
+    # unnecessary pypi packages are installed via the build script (bld.bat, build.sh)
+    - conda-forge::attrs ==21.4.0
+    - conda-forge::cattrs ==1.1.1
+    - conda-forge::h5py ==3.1  # [not win]
+    - conda-forge::imgaug ==0.4.0
+    - conda-forge::jsmin
+    - conda-forge::jsonpickle ==1.2
+    - conda-forge::networkx
+    - conda-forge::opencv
     - conda-forge::pandas
-    - ffmpeg
-    - qtpy>=2.0.1
-    - cudatoolkit=11.3.1
-    - cudnn=8.2.1
+    - conda-forge::pillow >=8.3.2
+    - conda-forge::psutil
+    - conda-forge::pykalman
+    - conda-forge::pyside2 >=5.12,<5.14
+    - conda-forge::python-rapidjson
+    - conda-forge::pyyaml
+    - conda-forge::pyzmq
+    - conda-forge::qtpy >=2.0.1
+    - conda-forge::rich
+    - conda-forge::scipy >=1.4.1,<=1.9.0
+    - conda-forge::scikit-image
+    - conda-forge::scikit-learn ==1.0
+    - conda-forge::scikit-video
+    - conda-forge::seaborn
+  run:
+    - conda-forge::python ==3.7.12  # Run into _MAX_WINDOWS_WORKERS not found if <
+    - conda-forge::attrs ==21.4.0
+    - conda-forge::cattrs ==1.1.1
+    - conda-forge::cudatoolkit ==11.3.1
+    - conda-forge::cudnn=8.2.1
     - nvidia::cuda-nvcc=11.3
-    - conda-forge::pip!=22.0.4
-
-  run_constrained:
-    - pyqt==9999999999
+    - conda-forge::h5py ==3.1  # [not win]
+    - conda-forge::imgaug ==0.4.0
+    - conda-forge::jsmin
+    - conda-forge::jsonpickle ==1.2
+    - conda-forge::networkx
+    - numpy >=1.19.5,<1.23.0  # Linux likes anaconda, windows likes conda-forge
+    - conda-forge::opencv
+    - conda-forge::pandas
+    - conda-forge::pillow >=8.3.2
+    - conda-forge::psutil
+    - conda-forge::pykalman
+    - conda-forge::pyside2 >=5.12,<5.14  # To ensure works correctly with QtPy.
+    - conda-forge::python-rapidjson
+    - conda-forge::pyyaml
+    - conda-forge::pyzmq
+    - conda-forge::qtpy >=2.0.1
+    - conda-forge::rich
+    - conda-forge::scipy >=1.4.1,<=1.9.0
+    - conda-forge::scikit-image
+    - conda-forge::scikit-learn ==1.0
+    - conda-forge::scikit-video
+    - conda-forge::seaborn
+    - sleap::tensorflow >=2.6.3,<2.11  # No windows GPU support for >2.10, sleap channel has 2.6.3
+    - conda-forge::tensorflow-hub
 
 test:
   imports:
-    - sleap
+    - sleap
diff --git a/.conda_apple_silicon/README.md b/.conda_apple_silicon/README.md
diff --git a/.conda_apple_silicon/build.sh b/.conda_apple_silicon/build.sh
diff --git a/.conda_apple_silicon/conda_build_config.yaml b/.conda_apple_silicon/conda_build_config.yaml