Julia build in container #45

tdivoll · 2023-07-21T00:29:29Z

This PR works out the precompiling and building of IFTPipeline in the container.

codecov · 2023-07-21T00:41:07Z

Codecov Report

Patch and project coverage have no change.

Comparison is base (4ef5326) 77.92% compared to head (2596540) 77.92%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #45   +/-   ##
=======================================
  Coverage   77.92%   77.92%           
=======================================
  Files           8        8           
  Lines         299      299           
=======================================
  Hits          233      233           
  Misses         66       66

Files Changed	Coverage Δ
src/h5.jl	`97.18% <ø> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Conflicts: src/h5.jl

… feat-docker-build Conflicts: src/h5.jl

cpaniaguam · 2023-08-22T21:27:06Z

Project.toml

+LoggingExtras = "1.0.1"
+PyCall = "1.96.1"
+Pkg = "1.9.0"
+TOML = "1.0.3"


Suggested change

TOML = "1.0.3"

TOML = "1.0.3"

README.md

cpaniaguam · 2023-08-22T21:34:34Z

resources/README.md

Why delete this file?

Good catch, I think it just got inadvertently deleted when I was testing (wiping the downloaded resources).

cpaniaguam · 2023-08-22T21:36:41Z

workflow/scripts/ice-floe-tracker.jl

+    # logger = setuplogger(logoption, command)

-    with_logger(logger) do
-        @time command_func(; command_args...)
-    end
+    # with_logger(logger) do
+         @time command_func(; command_args...)
+    # end


Why comment out the logger? If it's not needed, some deps (LoggingExtras) could also be dropped.

I think it was the issue with the logger trying to write in the read-only container. I can retest it and see if it will work now that we've updated some other code. It's possible that we have the right bindings now and it might work.

I was able to add a binding so that it will write the logs to workflow/report!

tdivoll · 2023-08-22T22:10:51Z

config/cylc_hpc/flow.cylc

+    [[pulljuliaimage]]
+        script = """
+            apptainer build --force $project_dir/icefloetracker-julia.simg docker://brownccv/icefloetracker-julia:pr-45
+        """


After merging, I'll have to start a new PR and change the image tag to main rather than pr-45.

cpaniaguam

👍
Added a few things to consider.

cpaniaguam · 2023-08-23T15:03:47Z

Dockerfile

-RUN chmod a+x /opt/ice-floe-tracker-pipeline/workflow/scripts/ice-floe-tracker.jl
+COPY workflow/scripts/ice-floe-tracker.jl /usr/local/bin/ice-floe-tracker.jl
+
+RUN chmod a+x /usr/local/bin/ice-floe-tracker.jl


Is this really required? I see that the command used to run the ice-floe-tracker.jl script includes 'julia' which is probably required to pass the -t auto flag to enable multithreading.

I think it has to do with the permissions in a singularity container. Things work for container permissions when we copy to/usr/local/bin. I was running into issues with running it directly from the repo.

cpaniaguam · 2023-08-23T15:04:24Z

Project.toml

+[compat]
+ArgParse = "1.1.4"
+Folds = "0.2.8"  
+HDF5 = "0.16.15"
+IceFloeTracker = "0.2.1"
+LoggingExtras = "1.0.1"
+PyCall = "1.96.1"
+Pkg = "1.9.0"
+TOML = "1.0.3"
+


cpaniaguam · 2023-08-23T15:06:17Z

README.md

+5. Export SOIT username/password to environment variable
   - [ ] From your home directory `nano .bash_profile`
   - [ ] add `export HISTCONTROL=ignoreboth` to the bottom of your .bash_profile
        * this will ensure that your username/password are not stored in history
        * when exporting the following environment variables, there __must__ be a space in front of each command
   - [ ] ` export SPACEUSER=<firstname>_<lastname>@brown.edu`
   - [ ] ` export SPACEPSWD=<password>`


From Josh's talk yesterday, I was thinking maybe secrets could be used for this. Not sure if it's possible.

I'll make an issue for looking into this. That might be a good way to do it. Talking with @broarr, it seemed like this use case was best to not store anything in a file.

cpaniaguam · 2023-08-23T15:10:59Z

config/cylc_hpc/flow.cylc

+            singularity exec --bind $landmask_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl landmask $fetchdata_dir /tmp
        """
    [[preprocess]]
        script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl preprocess -t $fetchdata_dir/truecolor -r $fetchdata_dir/reflectance -l $landmask_dir -p $soit_dir -o $preprocess_dir
+            singularity exec --bind $preprocess_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl preprocess -t $fetchdata_dir/truecolor -r $fetchdata_dir/reflectance -l $landmask_dir -p $soit_dir -o /tmp
        """
    [[extractfeatures]]
        script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl extractfeatures -i $preprocess_dir -o $preprocess_dir --minarea $minfloearea --maxarea $maxfloearea
+            singularity exec --bind $preprocess_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl extractfeatures -i $preprocess_dir -o /tmp --minarea $minfloearea --maxarea $maxfloearea
        """
    [[tracking]]
        script = """
-            julia -t auto $project_dir/workflow/scripts/ice-floe-tracker.jl track --imgs $preprocess_dir --props $preprocess_dir --deltat $preprocess_dir --output $tracker_dir
+            singularity exec --bind $tracker_dir:/tmp,$report_dir:/usr/local/bin/../report $project_dir/icefloetracker-julia.simg $julia_exec -t auto /usr/local/bin/ice-floe-tracker.jl track --imgs $preprocess_dir --props $preprocess_dir --deltat $preprocess_dir --output /tmp
        """
    [[exportH5]]
        script = """


Great work figuring out the bindings!

config/cylc_hpc/flow.cylc

Add line ending Co-authored-by: Carlos Paniagua <[email protected]>

test: try build rather than instantiate

219fafc

tdivoll added the enhancement New feature or request label Jul 21, 2023

tdivoll self-assigned this Jul 21, 2023

tdivoll marked this pull request as ready for review July 21, 2023 00:30

tdivoll changed the base branch from feat-docker-julia to main July 21, 2023 00:31

tdivoll and others added 24 commits July 21, 2023 14:52

Merge branch 'main' into feat-docker-build

66b816c

test: activate IFTPipeline module

b6b11c2

test: activate IFTPipeline module

f35aa64

test: add quotes to module call

c316020

test: add quotes to module call

44acad7

test: julia import IFTP

974473c

test: using IFTP

a59913b

test: rearrange order of calls

dd6a02d

test: add deliberate build for PyCall

93044c1

test: copy script to opt

9d85834

fix: path to copied script

f036ccd

fix: path to copied script

026560a

test: try cp to usr local bin

79d675a

test: try cp to usr local bin

717ef03

test: try cp to usr local bin

d92e28d

test: try cp to usr local bin

b225460

test: try full path for cp to usr local bin

8e30f68

test: switch order of cp

f0b65e6

test: remove cp

9ed5fee

fix: chmod path

4819e40

fix: chmod path

3139a54

feat: add arm64 platform to build

d96ba54

test: arm64 only build

719146c

fix: revert to amd64 only

8c8510d

update python path in Dockerfile

36f7972

tdivoll mentioned this pull request Aug 17, 2023

54 use getlatlon from ift #55

Merged

Timothy Divoll and others added 6 commits August 17, 2023 15:26

revert python path to default

2698fe8

Merge branch 'main' into feat-docker-build

4db6b29

Conflicts: src/h5.jl

test: make h5 dir outside rather than in container

a262133

test: rm py deps from container after IFT refactor

7d0b467

Merge branch '62-deprecate-getiftversion-in-favor-of-iftversion' into…

b60aae4

… feat-docker-build Conflicts: src/h5.jl

reset h5 script

86cba59

tdivoll mentioned this pull request Aug 21, 2023

62 deprecate getiftversion in favor of iftversion #63

Merged

Timothy Divoll and others added 8 commits August 21, 2023 13:28

Merge branch 'main' into feat-docker-build

1aca8b9

feat: final cylc pipeline test in hpc

82e3802

fix: indentation of tasks in flow file

9b48017

fix: filename paths in makefilenames

c5b722e

fix: last path to fix for makefilenames

9e45a97

fix: revert julia commands to call from local julia install

51ed16b

fix: add project activation to last two cylc tasks

a8aa62b

feat: add compat to project toml

4e156a3

cpaniaguam requested changes Aug 22, 2023

View reviewed changes

docs: clean up line ending and missing readme

7663c42

tdivoll commented Aug 22, 2023

View reviewed changes

tdivoll and others added 4 commits August 22, 2023 18:14

docs: add line ending

a6fffad

test: try logger now that binding paths are working

9a5e6e9

feat: add binding for logging report directory

0b5944e

fix: indentation error

b4c124b

tdivoll requested a review from cpaniaguam August 23, 2023 14:52

tdivoll mentioned this pull request Aug 23, 2023

Make option to toggle logging in CLI #47

Closed

cpaniaguam approved these changes Aug 23, 2023

View reviewed changes

Update config/cylc_hpc/flow.cylc

2596540

Add line ending Co-authored-by: Carlos Paniagua <[email protected]>

tdivoll merged commit 806d516 into main Aug 23, 2023

tdivoll mentioned this pull request Aug 23, 2023

update: latest cylc version #65

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Julia build in container #45

Julia build in container #45

tdivoll commented Jul 21, 2023

codecov bot commented Jul 21, 2023 •

edited

Loading

cpaniaguam Aug 22, 2023

cpaniaguam Aug 22, 2023

tdivoll Aug 22, 2023

cpaniaguam Aug 22, 2023

tdivoll Aug 22, 2023

tdivoll Aug 23, 2023

tdivoll Aug 22, 2023

cpaniaguam left a comment

cpaniaguam Aug 23, 2023

tdivoll Aug 23, 2023

cpaniaguam Aug 23, 2023

cpaniaguam Aug 23, 2023

tdivoll Aug 23, 2023

cpaniaguam Aug 23, 2023

Julia build in container #45

Julia build in container #45

Conversation

tdivoll commented Jul 21, 2023

codecov bot commented Jul 21, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpaniaguam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 21, 2023 •

edited

Loading