Julia build in container #45
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,25 +22,22 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th | |
| * this will start a compute session for 1 day with 32 GB memory and 20 cores | ||
| * see [here](https://docs.ccv.brown.edu/oscar/submitting-jobs/interact) for more options | ||
|
|
||
| 3. Build a virtual environment and install Cylc | ||
| - [ ] `cd <your-project-path>/ice-floe-tracker-pipeline` | ||
| - [ ] `conda env create -f ./config/ift-env.yaml` | ||
| - [ ] `conda activate ift-env` | ||
|
|
||
| 4. Register an account with [space-track.org](https://www.space-track.org/) for SOIT | ||
|
|
||
| 5. Export SOIT username/password to environment variable | ||
| - [ ] From your home directory `nano .bash_profile` | ||
| - [ ] add `export HISTCONTROL=ignoreboth` to the bottom of your .bash_profile | ||
| * this will ensure that your username/password are not stored in history | ||
| * when exporting the following environment variables, there __must__ be a space in front of each command | ||
| - [ ] ` export SPACEUSER=<firstname>_<lastname>@brown.edu` | ||
| - [ ] ` export SPACEPSWD=<password>` | ||
|
Comment on lines
+32
to
38
There was a problem hiding this comment. From Josh's talk yesterday, I was thinking maybe secrets could be used for this. Not sure if it's possible. There was a problem hiding this comment. I'll make an issue for looking into this. That might be a good way to do it. Talking with @broarr, it seemed like this use case was best to not store anything in a file. |
||
|
|
||
| 6. Prepare the runtime environment | ||
|
|
||
| Cylc will use software dependencies inside a Singularity container to fetch images and satellite times from external APIs. | ||
| - [ ] It is a good idea to reset the Singularity cache dir as specified [here](https://docs.ccv.brown.edu/oscar/singularity-containers/building-images) | ||
|
|
@@ -56,10 +53,7 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th | |
| - maxfloearea | ||
| - project_dir | ||
| **Note:** bounding box format = top_left_x top_left_y bottom_right_x bottom_right_y (x = lat(wgs84) or easting(epsg3413), y = lon(wgs84) or northing(epsg3413)) | ||
| - [ ] then, build the workflow, run it, and open the Terminal-based User Interface (TUI) to monitor the progress of each task. | ||
|  | ||
|
|
||
| ``` | ||
|
|
@@ -83,9 +77,9 @@ Cylc is used to encode the entire pipeline from start to finish and relies on th | |
|
|
||
| ### Running the Cylc pipeline locally | ||
|
|
||
| #### Prerequisites | ||
| __Julia:__ When running locally, make sure you have at least Julia 1.9.0 installed with the correct architecture for your local machine. (https://julialang.org/downloads/) | ||
| __Docker Desktop:__ Also make sure Docker Desktop client is running in the background to use the Cylc pipeline locally. (https://www.docker.com/products/docker-desktop/) | ||
cpaniaguam marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| 1. Build a virtual environment and install Cylc | ||
| - [ ] `cd <your-project-path>/ice-floe-tracker-pipeline` | ||
|
|
@@ -123,7 +117,7 @@ To use the Cylc pipeline locally, also make sure Docker Desktop client is runnin | |
| - [ ] `cylc play <workflow-name>` | ||
| - [ ] `cylc tui <workflow-name>` | ||
|
|
||
| The Terminal-based User Interface provides a simple way to watch the status of each task called in the `flow.cylc` workflow. Use arrow keys to investigate each task (see more [here](https://cylc.github.io/cylc-doc/latest/html/7-to-8/major-changes/ui.html#cylc-tui). | ||
| ). | ||
|
|
||
| If you need to change parameters and re-run a workflow, first do: | ||
|
|
@@ -151,10 +145,10 @@ Open a Julia REPL and build the package | |
| Enter Pkg mode and precompile | ||
| - [ ] `]` | ||
| - [ ] `activate .` | ||
| - [ ] `build` | ||
|
|
||
| Use the backspace to go back to the Julia REPL and start running Julia code! | ||
|
|
||
| __Note__ Use the help for wrapper scripts to learn about available options in each wrapper function | ||
| For example, from a bash prompt: | ||
| `julia --project=. ./workflow/scripts/ice-floe-tracker.jl extractfeatures --help` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,7 +5,7 @@ | |
| initial cycle point = 1 | ||
| [[graph]] | ||
| R1 = """ | ||
| mkpaths & pullfetchimage & pulljuliaimage => fetchdata & soit => landmask => preprocess => extractfeatures => tracking & exportH5 | ||
| """ | ||
| [runtime] | ||
| [[root]] | ||
|
|
@@ -22,6 +22,7 @@ | |
| project_dir = "~/ice-floe-tracker-pipeline" | ||
|
|
||
| # Recommend using these default paths for output | ||
| julia_exec = "/usr/local/julia/bin/julia" | ||
| results_dir = $project_dir/"results" | ||
| fetchdata_dir = $project_dir/"resources" | ||
| truecolor_dir = $fetchdata_dir/"truecolor" | ||
|
|
@@ -30,13 +31,19 @@ | |
| preprocess_dir = $results_dir/"preprocess" | ||
| soit_dir = $results_dir/"soit" | ||
| tracker_dir = $results_dir/"tracker" | ||
| h5_dir = $preprocess_dir/"hdf5-files" | ||
|
|
||
| [[mkpaths]] | ||
| script = """ | ||
| mkdir -p $soit_dir | ||
| mkdir -p $landmask_dir | ||
| mkdir -p $preprocess_dir | ||
| mkdir -p $tracker_dir | ||
| mkdir -p $h5_dir | ||
| """ | ||
| [[pullfetchimage]] | ||
| script = """ | ||
| apptainer build --force $project_dir/fetchdata.simg docker://brownccv/icefloetracker-fetchdata:main | ||
| """ | ||
| [[fetchdata]] | ||
| script = """ | ||
|
|
@@ -46,23 +53,27 @@ | |
| script = """ | ||
| singularity exec --bind $soit_dir:/tmp $project_dir/fetchdata.simg python3 /usr/local/bin/pass_time_cylc.py --startdate $startdate --enddate $enddate --csvoutpath /tmp --centroid_x $centroid_x --centroid_y $centroid_y --SPACEUSER $SPACEUSER --SPACEPSWD $SPACEPSWD | ||
| """ | ||
| [[pulljuliaimage]] | ||
| script = """ | ||
| apptainer build --force $project_dir/icefloetracker-julia.simg docker://brownccv/icefloetracker-julia:pr-45 | ||
| """ | ||
|
There was a problem hiding this comment. After merging, I'll have to start a new PR and change the image tag to |
||
| [[landmask]] | ||
| script = """ | ||
| singularity exec --bind $landmask_dir:/tmp $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl landmask $fetchdata_dir /tmp | ||
| """ | ||
| [[preprocess]] | ||
| script = """ | ||
| singularity exec --bind $preprocess_dir:/tmp $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl preprocess -t $fetchdata_dir/truecolor -r $fetchdata_dir/reflectance -l $landmask_dir -p $soit_dir -o /tmp | ||
| """ | ||
| [[extractfeatures]] | ||
| script = """ | ||
| singularity exec --bind $preprocess_dir:/tmp $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl extractfeatures -i $preprocess_dir -o /tmp --minarea $minfloearea --maxarea $maxfloearea | ||
| """ | ||
| [[tracking]] | ||
| script = """ | ||
| singularity exec --bind $tracker_dir:/tmp $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl track --imgs $preprocess_dir --props $preprocess_dir --deltat $preprocess_dir --output /tmp | ||
| """ | ||
| [[exportH5]] | ||
| script = """ | ||
| singularity exec --bind $preprocess_dir:/tmp $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl makeh5files --pathtosampleimg $fetchdata_dir/truecolor/$(ls $fetchdata_dir/truecolor | head -1) --resdir /tmp | ||
| """ | ||
tdivoll marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,14 +5,14 @@ | |
| initial cycle point = 1 | ||
| [[graph]] | ||
| R1 = """ | ||
| mkpaths => fetchdata & soit => landmask => preprocess => extractfeatures => tracking & exportH5 | ||
| """ | ||
| [runtime] | ||
| [[root]] | ||
| [[[environment]]] | ||
| # Update these variables with your run parameters | ||
| startdate = "2022-05-04" | ||
| enddate = "2022-05-06" | ||
| crs = "wgs84" #epsg3413 for polar stereographic | ||
| bounding_box = "78.186394 38.250605 70.749318 15.32373" | ||
| centroid_x = "75" | ||
|
|
@@ -30,25 +30,25 @@ | |
| preprocess_dir = $results_dir/"preprocess" | ||
| soit_dir = $results_dir/"soit" | ||
| tracker_dir = $results_dir/"tracker" | ||
| h5_dir = $preprocess_dir/"hdf5-files" | ||
|
|
||
| [[mkpaths]] | ||
| script = """ | ||
| mkdir -p $soit_dir | ||
| mkdir -p $landmask_dir | ||
| mkdir -p $preprocess_dir | ||
| mkdir -p $tracker_dir | ||
| mkdir -p $h5_dir | ||
| """ | ||
|
|
||
| [[fetchdata]] | ||
| script = """ | ||
| docker run --mount type=bind,source=$fetchdata_dir,target=/tmp brownccv/icefloetracker-fetchdata:main /usr/local/bin/fetchdata.sh -o /tmp -s $startdate -e $enddate -c $crs $bounding_box | ||
| """ | ||
| [[soit]] | ||
| script = """ | ||
| docker run --env SPACEUSER --env SPACEPSWD --mount type=bind,source=$soit_dir,target=/tmp brownccv/icefloetracker-fetchdata:main python3 /usr/local/bin/pass_time_cylc.py --startdate $startdate --enddate $enddate --csvoutpath /tmp --centroid_x $centroid_x --centroid_y $centroid_y --SPACEUSER $SPACEUSER --SPACEPSWD $SPACEPSWD | ||
| """ | ||
| [[landmask]] | ||
| script = """ | ||
| julia --project=$project_dir -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl landmask $fetchdata_dir $landmask_dir | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,4 +4,4 @@ channels: | |
| - conda-forge | ||
|
|
||
| dependencies: | ||
| - cylc-flow=8.3 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -67,6 +67,7 @@ This function expects the following files to be present in `resdir`: `filenames. | |
|
|
||
| * `pathtosampleimg`: Path to a sample image in the truecolor resource folder. This is used to extract the coordinate reference system (CRS) and the latitude and longitude coordinates of the image pixels. | ||
| * `resdir`: Path to the directory containing the results of the IceFloeTracker pipeline. | ||
| * `iftversion`: This is automatically pulled into the function from an environment variable. | ||
|
|
||
| # File structure | ||
| Each HDF5 file has the following structure: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -79,11 +79,11 @@ function main(args) | |
| # delete log option from command_args so it doesn't get passed to command_func | ||
| delete!(command_args, :log) | ||
|
|
||
| # logger = setuplogger(logoption, command) | ||
|
|
||
| # with_logger(logger) do | ||
| @time command_func(; command_args...) | ||
| # end | ||
|
There was a problem hiding this comment. Why comment out the logger? If it's not needed, some deps (LoggingExtras) could also be dropped. There was a problem hiding this comment. I think it was the issue with the logger trying to write in the read-only container. I can retest it and see if it will work now that we've updated some other code. It's possible that we have the right bindings now and it might work. There was a problem hiding this comment. I was able to add a binding so that it will write the logs to |
||
| return nothing | ||
| end | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this really required? I see that the command used to run the
ice-floe-tracker.jlscript includes 'julia' which is probably required to pass the-t autoflag to enable multithreading.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it has to do with the permissions in a singularity container. Things work for container permissions when we copy to
/usr/local/bin. I was running into issues with running it directly from the repo.