[1pt] WIP PR: Create inputs for selected HUC(s)

[mluck]

Summary

Contains files to generate data to run and evaluate FIM (fim_pipeline.sh and synthesize_test_cases.py) for specified HUC(s) as well as update code to generate pre-clip data so that WBD for Alaska contains only one layer. NOTE: this PR requires wbd.gpkg to be created by the updated generate_pre_clip_fim_huc8.py to be copied to the pre-clip HUC folders to remove a warning in synthesize_test_case.py.

Resolves #1265 and resolves #1296.

Usage

python /foss_fim/data/sandbox/get_sample_data.py -u 03100204 -i /data -o /foss_fim/data/sample-data

Additions

• data/
  • sandbox/ [archived]
    • Dockerfile: A copy of the root Dockerfile that also pulls code and data into the build image [archived]
    • fim-in-a-box.ipynb: Jupyter notebook to run and evaluate an example HUC [archived]
    • README.md [archived]
  • get_sample_data.py: Copies relevant data (e.g., inputs and test_cases) from the FIM data folder for specified HUC(s) and saves it to a separate location
  • wbd/generate_pre_clip_fim_huc8.py: Fix file paths and layers

Testing

fim_pipeline.sh and synthesize_test_cases.py ran on HUCs 12090301 and 19020302 without errors.

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?
• If applicable, has a deployment plan be created with the deployment person/team?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Reviewers requested
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[mluck]

Here are some example steps to add data to a Docker image and save it as a file using the command line

Run Docker container

docker run --rm -it --name sandbox fim_4:4.5.2.1

Add data to a container

docker cp ~/sample-data sandbox:data

Save container to an image

docker commit sandbox fim_4:4.5.2.1-sandbox

Export image to an image file

docker save fim_4:4.5.2.1-sandbox | gzip > ~/Downloads/fim-in-a-box.tar.gz

Load image file to Docker image

docker load < ~/Downloads/fim-in-a-box.tar.gz

[RobHanna-NOAA]

We are putting this on hold for a bit. As of one or two days ago the Whitebox installer is failing. They have been making updates, and even one hotfix last night, so they may still be actively fixing it. We will wait until next week and try again on our Docker builds tests. If worse comes to worse, we can see if we can extract Whitebox's old zip file from one of our current Docker images, put it in our Repo and use that to build new Docker image. We will see how this plays out over the next few weeks.

[mluck]

We are putting this on hold for a bit. As of one or two days ago the Whitebox installer is failing. They have been making updates, and even one hotfix last night, so they may still be actively fixing it. We will wait until next week and try again on our Docker builds tests. If worse comes to worse, we can see if we can extract Whitebox's old zip file from one of our current Docker images, put it in our Repo and use that to build new Docker image. We will see how this plays out over the next few weeks.

The Docker build error is now a separate issue #1181.

[mluck]

As of facca2e, the sandbox (FIM in a Box) code in this feature branch will be archived. Refer to commit 4e6ae2e for the latest version of the sandbox code.

This pull request will contain scripts to create a new inputs folder with data for specified HUC(s) as well as related patches to replace absolute file paths with relative file paths, a fix for evaluating when IFC data are not available, and update code to generate pre-clip data so that WBD for Alaska contains only one layer. NOTE: this PR requires wbd.gpkg to be created by the updated generate_pre_clip_fim_huc8.py to be copied to the pre-clip HUC folders to remove a warning in synthesize_test_case.py.

[RobHanna-NOAA]

With opening the s3 input option, we will need to update the readme.md (and/or others) to tell them it exists and how to either use it or a pointer for instructions. I personally think it is better to add a some text inside the README.md on how to do it. Pointing to the PR seems loose. Thoughts?

[mluck]

Note: The get_sample_data.py script should be checked any time new input data are changed, especially if locations are moved. One suggestion would be to update the PR checkbox under Deployment Plan.

[mluck]

This PR now depends on #1387.