Archived until a team picks back up this project.
Using Convolutional Neural Networks to generate water masks from SAR data.
- python3.8
- GDAL
First install the GDAL dev libraries:
$ sudo apt install libgdal-dev
We are using pipenv to track dependicies instead of a requirements.txt file.
So pipenv will have to be installed.
$ pip3 install pipenv
Then create a new virtual environment inside of the AI_Water directory.
$ virtualenv .venv --python=python3.8
Then enter the new virtual environment:
$ pipenv shell
Install the python packages from the pipfile:
$ pipenv install --dev
Specifying the --dev flag will also install dependencies you will need to run
the training and unit tests.
GDAL can take some troubleshooting to get working properly in a virtual environment. This is why we use the pygdal wrapper.
NOTE: pygdal must have the same version as GDAL to work!
To find GDALs package version run the following command:
$ gdal-config --version
Next run this command, inputing your version of GDAL:
$ pip3 install pygdal==<Your GDAL version>
or you can also try this:
$ pip3 install pygdal=="`gdal-config --version`.*"
To run the container version with proper volume mounting run the make command:
$ make container
All the scripts for the project can be found in aiwater.py. Make sure to run these scripts in a virtual enviroment.
Run setup to create the file structure for working with the date.
$ python aiwater.py setup
This will create the folling file structure: """ data/ -input/ -products -aoi -working/ -datasets -output/ -models -mask -tensorboard """
These default directory's can be altered by modifying src/config.py.
ASF allows you to download product files from them using a .metalink file. You can choose the specific products to download from their site and then will be provided with a 'products.metalink' file.
$ python aiwater.py download-metalink [metalink path] [output directory]
You can download products directly from a hyp3 subscription to create your mask with. This functions allows you to limit your products to those which are of minimaly required to cover the given shape file. Also, you can also set a date bounds.
Date boudry option must be entered in format yyyy-mm-dd.
This script will require you to enter your earthdata credentials.
$ python aiwater.py download-sub [name] --id [Subscription ID] --date-start [yyyy-mm-dd] --date-end [yyy-mm-dd] --aoi [Shape file path to area of interest] --min-cover [Sets mincover algorithm to true] --display [graphical shows final coverage] --output-dir [Ouput dir path]
To create a single mask run:
$ python aiwater.py create-mask [model name] [vv path] [vh path] [name]
Use this script to create a mask for every product in a directory:
$ python aiwater.py mask-directory [model name] [source directory] [output directory] [mask name]
To train a new model run:
$ python aiwater.py train [model name] [dataset name]
$ python aiwater.py identify-water [vv path] [vh path] [mask name]
$ python aiwater.py tile-image [image path] [tile size]
Divide dataset into test and traid directories.
$ python aiwater.py divide-dataset [directory] [test/train split (0.1-0.9)]
Groom images to remove inaccurate masks from the dataset.
$ python aiwater.py groom-images [directory] [holdout]
Basically runs download-sub and then mask-direcotry. Deletes directory when finished, leaving just the mask.
$ python aiwater.py mask-sub [model name] [mask name]
The project is organized into directories as follows.
AI_Water
├── aiwater.py # Main Script with all comands
├── data
│ ├── input
│ │ ├── products # unaltered product files
│ │ └── aoi # AOI shape files
│ ├── working
│ │ └── datasets # Tiled/groomed Datasets for training
│ └── output
│ ├── models
│ └── mask
├── src # source code
├── tests # Unit and integration tests
│ └── unit_tests
This project uses pytest for unit testing. The easiest way to run the tests is
with this make command:
$ make test