Julia build in container

Dockerfile

@@ -5,17 +5,19 @@ ENV JULIA_PROJECT=/opt/ice-floe-tracker-pipeline
 ENV JULIA_DEPOT_PATH=/opt/julia
 ENV JULIA_PKGDIR=/opt/julia
 
-RUN apt-get clean && apt-get update && \
-    apt-get install -y wget python3-pip git python3.10 && \
-    rm -rf /var/lib/apt/list/*
+RUN apt-get -y update && \
+    apt-get install -y git python3.10 && \
+    rm -rf /var/lib/apt/list/* 
 
 WORKDIR /opt
 
 RUN git clone https://github.com/WilhelmusLab/ice-floe-tracker-pipeline.git
 
-RUN julia --project="/opt/ice-floe-tracker-pipeline" -e 'ENV["PYTHON"]=""; using Pkg; Pkg.instantiate(); Pkg.precompile(); Pkg.build("PyCall")' 
+RUN /usr/local/julia/bin/julia --project="/opt/ice-floe-tracker-pipeline" -e 'ENV["PYTHON"]=""; using Pkg; Pkg.build()' 
 
-RUN chmod a+x /opt/ice-floe-tracker-pipeline/workflow/scripts/ice-floe-tracker.jl
+COPY workflow/scripts/ice-floe-tracker.jl /usr/local/bin/ice-floe-tracker.jl
+
+RUN chmod a+x /usr/local/bin/ice-floe-tracker.jl
 
 ENV JULIA_DEPOT_PATH="$HOME/.julia:$JULIA_DEPOT_PATH"
 

Project.toml

@@ -13,3 +13,14 @@ LoggingExtras = "e6f89c97-d47a-5376-807f-9c37f3926c36"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
 TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
+
+[compat]
+ArgParse = "1.1.4"
+Folds = "0.2.8"  
+HDF5 = "0.16.15"
+IceFloeTracker = "0.2.1"
+LoggingExtras = "1.0.1"
+PyCall = "1.96.1"
+Pkg = "1.9.0"
+TOML = "1.0.3"
+

README.md

@@ -22,25 +22,22 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th
     * this will start a compute session for 1 day with 32 GB memory and 20 cores
     * see [here](https://docs.ccv.brown.edu/oscar/submitting-jobs/interact) for more options
 
-3. Load the Julia module
-   - [ ] `module load julia/1.9.0`
-
-4. Build a virtual environment and install Cylc
+3. Build a virtual environment and install Cylc
    - [ ] `cd <your-project-path>/ice-floe-tracker-pipeline`
    - [ ] `conda env create -f ./config/ift-env.yaml`
    - [ ] `conda activate ift-env`
 
-5. Register an account with [space-track.org](https://www.space-track.org/) for SOIT
+4. Register an account with [space-track.org](https://www.space-track.org/) for SOIT
 
-6. Export SOIT username/password to environment variable
+5. Export SOIT username/password to environment variable
    - [ ] From your home directory `nano .bash_profile`
    - [ ] add `export HISTCONTROL=ignoreboth` to the bottom of your .bash_profile
         * this will ensure that your username/password are not stored in history
         * when exporting the following environment variables, there __must__ be a space in front of each command
    - [ ] ` export SPACEUSER=<firstname>_<lastname>@brown.edu`
    - [ ] ` export SPACEPSWD=<password>`
 
-7. Prepare the runtime environment 
+6. Prepare the runtime environment 
 
     Cylc will use software dependencies inside a Singularity container to fetch images and satellite times from external APIs. 
    - [ ] It is a good idea to reset the Singularity cache dir as specified [here](https://docs.ccv.brown.edu/oscar/singularity-containers/building-images)
@@ -56,10 +53,7 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th
      - maxfloearea
      - project_dir
      **Note:** bounding box format = top_left_x top_left_y bottom_right_x bottom_right_y (x = lat(wgs84) or easting(epsg3413),  y = lon(wgs84) or northing(epsg3413))
-
-   - [ ] run `singularity build fetchdata.simg docker://brownccv/icefloetracker-fetchdata:main`
-        * This will pull the image containing all the depencies and make them accessible to Cylc
-   - [ ] then, build the workflow, run it, and open the terminal-based user interface (TUI) to monitor the progress of each task. 
+   - [ ] then, build the workflow, run it, and open the Terminal-based User Interface (TUI) to monitor the progress of each task. 
     ![TUI example](./tui-example.png)
 
     ```
@@ -83,9 +77,9 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th
 
 ### Running the Cylc pipeline locally
 
-When running locally, make sure you have at least Julia 1.9.0 installed with the correct architecture for your local machine. (https://julialang.org/downloads/)
-To use the Cylc pipeline locally, also make sure Docker Desktop client is running in the background. (https://www.docker.com/products/docker-desktop/)
-
+#### Prerequisites
+__Julia:__ When running locally, make sure you have at least Julia 1.9.0 installed with the correct architecture for your local machine. (https://julialang.org/downloads/)
+__Docker Desktop:__ Also make sure Docker Desktop client is running in the background to use the Cylc pipeline locally. (https://www.docker.com/products/docker-desktop/)
 
 1. Build a virtual environment and install Cylc
    - [ ] `cd <your-project-path>/ice-floe-tracker-pipeline`
@@ -123,7 +117,7 @@ To use the Cylc pipeline locally, also make sure Docker Desktop client is runnin
    - [ ] `cylc play <workflow-name>`
    - [ ] `cylc tui <workflow-name>`
 
-The Terminal-based user interface provides a simple way to watch the status of each task called in the `flow.cylc` workflow. Use arrow keys to investigate each task (see more [here](https://cylc.github.io/cylc-doc/latest/html/7-to-8/major-changes/ui.html#cylc-tui).
+The Terminal-based User Interface provides a simple way to watch the status of each task called in the `flow.cylc` workflow. Use arrow keys to investigate each task (see more [here](https://cylc.github.io/cylc-doc/latest/html/7-to-8/major-changes/ui.html#cylc-tui).
 ![TUI](tui-example.png)).
 
 If you need to change parameters and re-run a workflow, first do:
@@ -151,10 +145,10 @@ Open a Julia REPL and build the package
 Enter Pkg mode and precompile
 - [ ] `]`
 - [ ] `activate .`
-- [ ] `precompile`
+- [ ] `build`
 
 Use the backspace to go back to the Julia REPL and start running Julia code!
 
 __Note__ Use the help for wrapper scripts to learn about available options in each wrapper function
 For example, from a bash prompt: 
-`julia --project=. ice-floe-tracker-pipeline/workflow/scripts/ice-floe-tracker.jl extractfeatures --help`
+`julia --project=. ./workflow/scripts/ice-floe-tracker.jl extractfeatures --help`

config/cylc_hpc/flow.cylc

@@ -5,7 +5,7 @@
     initial cycle point = 1
     [[graph]]
         R1 = """
-            mkpaths => fetchdata => soit & landmask => preprocess => extractfeatures => tracking & exportH5
+            mkpaths & pullfetchimage & pulljuliaimage => fetchdata & soit => landmask => preprocess => extractfeatures => tracking & exportH5
              """
 [runtime]
     [[root]]
@@ -22,6 +22,8 @@
             project_dir = "~/ice-floe-tracker-pipeline"
 
             # Recommend using these default paths for output
+            julia_exec = "/usr/local/julia/bin/julia"
+            report_dir = $project_dir/"workflow/report"
             results_dir = $project_dir/"results"
             fetchdata_dir = $project_dir/"resources"
             truecolor_dir = $fetchdata_dir/"truecolor"
@@ -30,13 +32,19 @@
             preprocess_dir = $results_dir/"preprocess"
             soit_dir = $results_dir/"soit"
             tracker_dir = $results_dir/"tracker"
+            h5_dir = $preprocess_dir/"hdf5-files"
 
     [[mkpaths]]
         script = """
             mkdir -p $soit_dir
             mkdir -p $landmask_dir
             mkdir -p $preprocess_dir
             mkdir -p $tracker_dir
+            mkdir -p $h5_dir
+        """
+    [[pullfetchimage]]
+        script = """
+            apptainer build --force $project_dir/fetchdata.simg docker://brownccv/icefloetracker-fetchdata:main
         """
     [[fetchdata]]
         script = """
@@ -46,23 +54,28 @@
         script = """
             singularity exec --bind $soit_dir:/tmp $project_dir/fetchdata.simg python3 /usr/local/bin/pass_time_cylc.py --startdate $startdate --enddate $enddate --csvoutpath /tmp --centroid_x $centroid_x --centroid_y $centroid_y --SPACEUSER $SPACEUSER --SPACEPSWD $SPACEPSWD
         """
+    [[pulljuliaimage]]
+        script = """
+            apptainer build --force $project_dir/icefloetracker-julia.simg docker://brownccv/icefloetracker-julia:pr-45
+        """
     [[landmask]]
         script = """
-            singularity exec $project_dir/icefloetracker-julia.simg julia --project="/opt/ice-floe-tracker-pipeline" -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl landmask $fetchdata_dir $landmask_dir
+            singularity exec --bind $landmask_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl landmask $fetchdata_dir /tmp
         """
     [[preprocess]]
         script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl preprocess -t $fetchdata_dir/truecolor -r $fetchdata_dir/reflectance -l $landmask_dir -p $soit_dir -o $preprocess_dir
+            singularity exec --bind $preprocess_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl preprocess -t $fetchdata_dir/truecolor -r $fetchdata_dir/reflectance -l $landmask_dir -p $soit_dir -o /tmp
         """
     [[extractfeatures]]
         script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl extractfeatures -i $preprocess_dir -o $preprocess_dir --minarea $minfloearea --maxarea $maxfloearea
+            singularity exec --bind $preprocess_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl extractfeatures -i $preprocess_dir -o /tmp --minarea $minfloearea --maxarea $maxfloearea
         """
     [[tracking]]
         script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl track --imgs $preprocess_dir --props $preprocess_dir --deltat $preprocess_dir --output $tracker_dir
+            singularity exec --bind $tracker_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl track --imgs $preprocess_dir --props $preprocess_dir --deltat $preprocess_dir --output /tmp
         """
     [[exportH5]]
         script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl makeh5files --pathtosampleimg $fetchdata_dir/truecolor/$(ls $fetchdata_dir/truecolor | head -1) --resdir $preprocess_dir
-        """
+            singularity exec --bind $preprocess_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl makeh5files --pathtosampleimg $fetchdata_dir/truecolor/$(ls $fetchdata_dir/truecolor | head -1) --resdir /tmp
+        """
+

config/cylc_local/flow.cylc

@@ -5,14 +5,14 @@
     initial cycle point = 1
     [[graph]]
         R1 = """
-            mkpaths => fetchdata => soit & landmask => preprocess => extractfeatures => tracking & exportH5
+            mkpaths => fetchdata & soit => landmask => preprocess => extractfeatures => tracking & exportH5
              """
 [runtime]
     [[root]]
         [[[environment]]]
             # Update these variables with your run parameters
             startdate = "2022-05-04"
-            enddate = "2022-05-08"
+            enddate = "2022-05-06"
             crs = "wgs84" #epsg3413 for polar stereographic
             bounding_box = "78.186394 38.250605 70.749318 15.32373"
             centroid_x = "75"
@@ -30,25 +30,25 @@
             preprocess_dir = $results_dir/"preprocess"
             soit_dir = $results_dir/"soit"
             tracker_dir = $results_dir/"tracker"
+            h5_dir = $preprocess_dir/"hdf5-files"
 
     [[mkpaths]]
         script = """
             mkdir -p $soit_dir
             mkdir -p $landmask_dir
             mkdir -p $preprocess_dir
             mkdir -p $tracker_dir
+            mkdir -p $h5_dir
         """
 
     [[fetchdata]]
         script = """
-            docker run --mount type=bind,source=$fetchdata_dir,target=/tmp brownccv/icefloetracker-fetchdata:main fetchdata.sh -o /tmp -s $startdate -e $enddate -c $crs $bounding_box
+            docker run --mount type=bind,source=$fetchdata_dir,target=/tmp brownccv/icefloetracker-fetchdata:main /usr/local/bin/fetchdata.sh -o /tmp -s $startdate -e $enddate -c $crs $bounding_box
         """
-
     [[soit]]
         script = """
             docker run --env SPACEUSER --env SPACEPSWD --mount type=bind,source=$soit_dir,target=/tmp brownccv/icefloetracker-fetchdata:main python3 /usr/local/bin/pass_time_cylc.py --startdate $startdate --enddate $enddate --csvoutpath /tmp --centroid_x $centroid_x --centroid_y $centroid_y --SPACEUSER $SPACEUSER --SPACEPSWD $SPACEPSWD
         """
-
     [[landmask]]
         script = """
             julia --project=$project_dir -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl landmask $fetchdata_dir $landmask_dir

config/ift-env.yaml

@@ -4,4 +4,4 @@ channels:
   - conda-forge
 
 dependencies:
-  - cylc-flow=8.2
+  - cylc-flow=8.3

resources/README.md

@@ -1,3 +1,3 @@
 # Resources Directory
 
-This directory contains any retrieved resources used in the pipeline. This includes the initial satellite images and land mask.
+This directory contains any retrieved resources used in the pipeline. This includes the initial satellite images and land mask.

src/h5.jl

@@ -67,6 +67,7 @@ This function expects the following files to be present in `resdir`: `filenames.
 
   * `pathtosampleimg`: Path to a sample image in the truecolor resource folder. This is used to extract the coordinate reference system (CRS) and the latitude and longitude coordinates of the image pixels.
   * `resdir`: Path to the directory containing the results of the IceFloeTracker pipeline.
+  * `iftversion`: This is automatically pulled into the function from an environment variable.
 
 # File structure
 Each HDF5 file has the following structure: