Backsub rework (#14)

* Reworked backsub, Palom pyramid writer, parallelization

* Rework finalization

* Updated changelog

* Updated README

* Updated cosign for github action

---------

Co-authored-by: Kresimir Bestak <[email protected]>
Co-authored-by: Kresimir Bestak <[email protected]>

.github/workflows/docker-publish.yml

@@ -35,11 +35,11 @@ jobs:
 
       # Install the cosign tool except on PR
       # https://github.com/sigstore/cosign-installer
-      - name: Install cosign
+      - name: Install Cosign
         if: github.event_name != 'pull_request'
-        uses: sigstore/cosign-installer@f3c664df7af409cb4873aa5068053ba9d61a57b6 #v2.6.0
+        uses: sigstore/cosign-installer@v3.1.1
         with:
-          cosign-release: 'v1.11.0'
+          cosign-release: 'v2.2.1'
 
       # Workaround: https://github.com/docker/build-push-action/issues/461
       - name: Setup Docker buildx

CHANGELOG.md

@@ -0,0 +1,23 @@
+# schapirolabor/background_subtraction: Changelog
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## v0.4.1 - [2023.11.21]
+
+Complete rework of Backsub to include Palom's pyramid writer (https://github.com/labsyspharm/palom).
+Added dask array chunking and delayed execution for subtraction that happenes while the output pyramidal `ome.tif` is being created.
+Added `CHANGELOG.md`.
+
+### `Added`
+- `--chunk-size` parameter
+- Palom's pyramid writer
+
+### `Fixed`
+- Fixed issue with RAM inefficiency - reworked Backsub.
+
+### `Removed`
+- `--pyramid` tag introduced in v0.3.4, for smaller images, a smaller tile size should be specified now.
+
+
+I did not keep a changelog before version v0.4.1.

Dockerfile

@@ -1,7 +1,13 @@
-FROM mambaorg/micromamba:0.26.0
-COPY --chown=$MAMBA_USER:$MAMBA_USER environment.yml /tmp/env.yaml
-RUN micromamba install -y -n base -f /tmp/env.yaml && \
-    micromamba clean --all --yes
-ENV PATH="${PATH}:/opt/conda/bin"
+FROM continuumio/miniconda3
+
+COPY environment.yml .
+RUN apt-get update -qq && apt-get install -y \
+    build-essential \
+    ffmpeg \
+    libsm6 \
+    libxext6
+
+RUN conda env create -f environment.yml
+ENV PATH="/opt/conda/envs/backsub/bin:$PATH"
 WORKDIR /background_subtraction
-COPY . .
+COPY . .

README.md

@@ -13,13 +13,13 @@ Marker<sub>*corrected*</sub> = Marker<sub>*raw*</sub> - Background / Exposure<su
 
 ## Usage 
 
-### Markers file
-
-#### Versions v0.3.0 and newer:
 The `markers.csv` file which gives details about the channels needs to contain the following columns: "marker_name", "background" and "exposure". An exemplary [markers.csv](https://github.com/SchapiroLabor/Background_subtraction/blob/main/example/markers.csv) file is given. The "marker_name" column should indicate the marker for the acquired channel and all values should be unique. The "background" column should indicate the marker name of the channel which needs to be subtracted. This value must match the "marker_name" value of the background channel. The "exposure" column should contain the exposure time used for channel acquisition, and the measure unit should be consistent across the column. Exposure time is used for scaling the value of the background to be comparable to the processed channel. The "remove" column should contain logical `TRUE` values for channels which should be exluded in the output image.
 
+### Versions v0.4.1 and newer:
+The script has been rewritten to perform channel subtraction in a RAM-efficient manner - updating is highly recommended. If the output file is much bigger than expected, adjust the `--tile-size` parameter to a smaller value (e.g `512`). Changing the `--chunk-size` parameter may affect performance (lower values increase execution time, higher values increase RAM usage).
+
 
-#### Versions v0.2.0 and older:
+### Versions v0.2.0 and older:
 The `markers.csv` file which gives details about the channels needs to contain the following columns: "Filter", "background" and "exposure". An exemplary [markers_old.csv](https://github.com/SchapiroLabor/Background_subtraction/files/9549686/markers.csv) file is given. The "Filter" column should specify the Filter used when acquiring images. If different stains are aquired with the same filter, the *exact same value* needs to be written (including background) as it is used for determining which background channel should be subtracted. The "background" column should contain logical `TRUE` values for channels which represent autofluorescence. The "exposure" column should contain the exposure time used for channel acquisition, and the measure unit should be consistent across the column. Exposure time is used for scaling the value of the background to be comparable to the processed channel.
 
 
@@ -33,8 +33,8 @@ The script requires four inputs:
 Optional inputs:
 * `--pixel-size` to specify the pixel size of the input image (default: `1.0`), if not specified, the pixel size will be read from the metadata of the input image.
 * `--version` to print version and exit (added in v0.3.4)
-* `--pyramid` to create a pyramidal output image (default: `True`) (added in v0.3.4)
 * `--tile-size` to specify the tile size for the pyramidal output image (default: `1024`) (added in v0.3.4)
+* `--chunk-size` to specify chunk size for delayed calculation execution (default: `5000`) - lower values result in higher execution time, higher values in higher RAM usage
 
 
 ### Output