uavsar-download.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "standard-worse",
   "metadata": {},
   "source": [
    "# UAVSAR Data Access and Conversion\n",
    "\n",
    "```{admonition} Learning Objectives\n",
    "- overview of UAVSAR data\n",
    "- demonstrate how to access and transform data to Geotiffs\n",
    "```\n",
    "\n",
    "There are multiple ways to access UAVSAR data. Also the SQL database.\n",
    "\n",
    "* [Alaska Satellite Facility Vertex Portal](https://search.asf.alaska.edu/#/?dataset=UAVSAR)\n",
    "* [NASA Earthdata Suborbital Search](https://search.earthdata.nasa.gov/portal/suborbital/search?fi=UAVSAR&as[instrument][0]=UAVSAR)\n",
    "* [JPL UAVSAR Data Search](https://uavsar.jpl.nasa.gov/cgi-bin/data.pl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "above-diabetes",
   "metadata": {},
   "source": [
    "```{admonition} InSAR Data Types\n",
    ":class: InSAR Data Types\n",
    "- ANN file (.ann): a text annotation file with metadata\n",
    "- AMP files (.amp1 and .amp2): calibrated multi-looked amplitude products\n",
    "- INT files (.int): interferogram product, complex number format (we won't be using these here)\n",
    "- COR files (.cor): interferometric correlation product, a measure of the noise level of the phase\n",
    "- GRD files (.grd): interferometric products projected to the ground in simple geographic coordinates (latitude, longitude)\n",
    "- HGT file  (.hgt): the DEM that was used in the InSAR processing\n",
    "- KML and KMZ files (.kml or .kmz): format for viewing files in Google Earth (can't be used for analysis)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "prospective-score",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import re\n",
    "import zipfile\n",
    "import getpass\n",
    "from osgeo import gdal \n",
    "import os  # for chdir, getcwd, path.basename, path.exists\n",
    "import pandas as pd # for DatetimeIndex\n",
    "import codecs # for text parsing code\n",
    "import netrc\n",
    "import rasterio as rio\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blocked-sandwich",
   "metadata": {},
   "source": [
    "### Data Download\n",
    "\n",
    "We will use our NASA EarthData credentials and ASF Vertex to download an InSAR pair data into our notebook directly. For this tutorial, we will be working with UAVSAR data from February of 2020. If you want to use different data in the future, change the links in the files variable. The screengrab below shows how I generated these download links from the ASF site."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "introductory-liabilities",
   "metadata": {},
   "source": [
    ":::{figure-md} vertex\n",
    "<img src=\"../../img/asf_vertex.png\" alt=\"asf vertex\" width=\"800px\">\n",
    "\n",
    "Screenshot of ASF Vertex interface\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "urban-latter",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter Username: jacktarricone\n",
      "Enter Password: ········\n"
     ]
    }
   ],
   "source": [
    "# Get NASA EARTHDATA Credentials from ~/.netrc or manual input\n",
    "try:\n",
    "    os.chmod('/home/jovyan/.netrc', 0o600) #only necessary on jupyterhub\n",
    "    (ASF_USER, account, ASF_PASS) = netrc.netrc().authenticators(\"urs.earthdata.nasa.gov\")\n",
    "except:\n",
    "    ASF_USER = input(\"Enter Username: \")\n",
    "    ASF_PASS = getpass.getpass(\"Enter Password: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "focused-fellow",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Notebook directory:  /Users/jacktarricone/uavsar_scripts\n",
      "/Users/\n"
     ]
    }
   ],
   "source": [
    "# directory in which the notebook resides\n",
    "if 'tutorial_home_dir' not in globals():\n",
    "     tutorial_home_dir = os.getcwd()\n",
    "print(\"Notebook directory: \", tutorial_home_dir)\n",
    "\n",
    "if not os.path.exists('/Users/'):\n",
    "    os.chdir('/Users/')\n",
    "   \n",
    "# directory for data downloads\n",
    "\n",
    "data_dir = os.path.join('/Users/')\n",
    "os.makedirs(data_dir, exist_ok=True)\n",
    "print(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "alternative-factory",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading https://datapool.asf.alaska.edu/INTERFEROMETRY_GRD/UA/uticam_21003_21004-002_21013-003_0034d_s01_L090_01_int_grd.zip...\n",
      "done\n",
      "CPU times: user 6.75 ms, sys: 3.86 ms, total: 10.6 ms\n",
      "Wall time: 6min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "files = ['https://datapool.asf.alaska.edu/INTERFEROMETRY_GRD/UA/uticam_21003_21004-002_21013-003_0034d_s01_L090_01_int_grd.zip']\n",
    "    \n",
    "for file in files:\n",
    "    print(f'downloading {file}...')\n",
    "    filename = os.path.basename(file)\n",
    "    \n",
    "    if not os.path.exists(os.path.join(data_dir,filename)):\n",
    "        cmd = \"wget -q {0} --user={1} --password={2} -P {3} -nc\".format(file, ASF_USER, ASF_PASS, data_dir)\n",
    "        os.system(cmd)\n",
    "    else:\n",
    "        print(filename + \" already exists. Skipping download ..\")\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "danish-background",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/tmp/drivefs_ipc.501_shell', '/tmp/com.google.Keystone', '/tmp/drivefs_ipc.501', '/tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090_01_amp_grd.zip', '/tmp/com.apple.launchd.8Lb1KV6AN2', '/tmp/com.apple.launchd.suhywqvpge', '/tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090_01_int_grd.zip', '/tmp/uticam_21003_21004-002_21013-003_0034d_s01_L090_01_int_grd.zip']\n"
     ]
    }
   ],
   "source": [
    "# check to see if downloaded\n",
    "# the *.* syntax means print all files in the directory\n",
    "\n",
    "print(glob.glob(\"/tmp/*.*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "successful-cable",
   "metadata": {},
   "outputs": [],
   "source": [
    "#use for deleting contents of temp directory if needed. DO NOT uncomment, this will cause the notebook to fail.\n",
    "\n",
    "#files = glob.glob(\"/tmp/*.*\")\n",
    "#for f in files:\n",
    "#    os.remove(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alien-visibility",
   "metadata": {},
   "source": [
    "### Unzipping the files we just downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "sublime-moore",
   "metadata": {},
   "outputs": [],
   "source": [
    "## unzip files just downloaded\n",
    "\n",
    "# define file path for both files\n",
    "int_zip = '/tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090_01_int_grd.zip'\n",
    "#amp_zip = '/tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090_01_amp_grd.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "expired-sailing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File Name                                             Modified             Size\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.cor.grd 2020-04-18 06:21:56    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.hgt.grd 2020-04-18 06:25:52    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.int.grd 2020-04-18 06:25:58    267542016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.unw.grd 2020-04-18 06:26:10    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HV_01.cor.grd 2020-04-18 06:26:16    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HV_01.hgt.grd 2020-04-18 06:26:22    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HV_01.int.grd 2020-04-18 06:26:30    267542016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HV_01.unw.grd 2020-04-18 06:26:42    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VH_01.cor.grd 2020-04-18 06:26:48    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VH_01.hgt.grd 2020-04-18 06:26:54    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VH_01.int.grd 2020-04-18 06:27:02    267542016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VH_01.unw.grd 2020-04-18 06:27:12    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VV_01.cor.grd 2020-04-18 06:27:18    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VV_01.hgt.grd 2020-04-18 06:27:26    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VV_01.int.grd 2020-04-18 06:27:32    267542016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VV_01.unw.grd 2020-04-18 06:27:44    133771008\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VV_01.ann 2020-04-18 06:27:50        30016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HV_01.ann 2020-04-18 06:27:50        30016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.ann 2020-04-18 06:27:50        30016\n",
      "grmesa_27416_20003-028_20005-007_0011d_s01_L090VH_01.ann 2020-04-18 06:27:50        30016\n",
      "Extracting all the files now...\n",
      "done\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'amp_zip' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-b4f8f74d15ef>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;31m# amp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mzipfile\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mZipFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamp_zip\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"r\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mzip_ref\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     \u001b[0mzip_ref\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprintdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Extracting all the files now...'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'amp_zip' is not defined"
     ]
    }
   ],
   "source": [
    "# unzip\n",
    "\n",
    "# int\n",
    "with zipfile.ZipFile(int_zip, \"r\") as zip_ref:\n",
    "    zip_ref.printdir()\n",
    "    print('Extracting all the files now...')\n",
    "    zip_ref.extractall('/tmp')\n",
    "    print(\"done\")\n",
    "    \n",
    "# amp\n",
    "with zipfile.ZipFile(amp_zip, \"r\") as zip_ref:\n",
    "    zip_ref.printdir()\n",
    "    print('Extracting all the files now...')\n",
    "    zip_ref.extractall('/tmp')\n",
    "    print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regular-disposal",
   "metadata": {},
   "source": [
    "### Removing unwanted data\n",
    "For simplicity, we'll only work with HH polarization. The three other polarizations (VV, VH, HV) provide additional information about the surface properties and can be utilized in further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "naval-direction",
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean up unwanted data from what we just downloaded\n",
    "\n",
    "directory = '/tmp'\n",
    "os.chdir(directory)\n",
    "HV_files = glob.glob('*HV_01*') #define all HV\n",
    "VV_files = glob.glob('*VV_01*') #define all VV\n",
    "VH_files = glob.glob('*VH_01*') #define all VH\n",
    "zips = glob.glob('*.zip') # define the zip files\n",
    "\n",
    "# loops to remove them\n",
    "\n",
    "for f in HV_files:\n",
    "    os.remove(f)\n",
    "    \n",
    "for f in VV_files:\n",
    "    os.remove(f)\n",
    "    \n",
    "for f in VH_files:\n",
    "    os.remove(f)\n",
    "    \n",
    "for f in zips:\n",
    "    os.remove(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "jewish-gazette",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check to see what files are left in the directory\n",
    "\n",
    "print(glob.glob(\"/tmp/*.*\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chemical-comparison",
   "metadata": {},
   "source": [
    "Now we only have the HH polarization, the annotation file, and the 6 .grd files!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "middle-chile",
   "metadata": {},
   "source": [
    "## Converting Data to GeoTiffs\n",
    "\n",
    "The downloadable UAVSAR data comes in a flat binary format (.grd), which is not readable by GDAL (Geospatial Data Abstraction Library). Therefore it needs to be transformed for use in standard spatial analysis software (ArcGIS, QGIS, Python, R, MATLAB, etc.). To do this, we will use the uavsar_tiff_convert function, which takes information (latitude, longitude, number of lines and samples, data type, pixel size) from the annotation file to create an ENVI header (.hdr). Once the ENVI header is created, the files can be read into Python and converted to GeoTiffs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "enormous-burke",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's print the annotation file to get a look at it's content\n",
    "# these file contain a lot of information and can be very intimidating and hard to understand, but being able to read them is vital to working this UAVSAR data\n",
    "\n",
    "#f = open('/tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.ann', 'r')\n",
    "#file_contents = f.read()\n",
    "#print (file_contents)\n",
    "!head -n 15 /tmp/grmesa_27416_20003-028_20005-007_0011d_s01_L090HH_01.ann"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extended-lecture",
   "metadata": {},
   "source": [
    "This function pulls out information from the annotation file, builds and ENVI header, and then converts the data to GeoTIFFS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "centered-developer",
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder is path to a folder with an .ann (or .txt) and .grd files (.amp1, .amp2, .cor, .unw, .int)\n",
    "\n",
    "def uavsar_tiff_convert(folder):\n",
    "    \"\"\"\n",
    "    Builds a header file for the input UAVSAR .grd file,\n",
    "    allowing the data to be read as a raster dataset.\n",
    "    :param folder:   the folder containing the UAVSAR .grd and .ann files\n",
    "    \"\"\"\n",
    "\n",
    "    os.chdir(folder)\n",
    "    int_file = glob.glob(os.path.join(folder, 'int.grd'))\n",
    "\n",
    "    # Empty lists to put information that will be recalled later.\n",
    "    Lines_list = []\n",
    "    Samples_list = []\n",
    "    Latitude_list = []\n",
    "    Longitude_list = []\n",
    "    Files_list = []\n",
    "\n",
    "    # Step 1: Look through folder and determine how many different flights there are\n",
    "    # by looking at the HDR files.\n",
    "    for files in os.listdir(folder):\n",
    "        if files [-4:] == \".grd\":\n",
    "            newfile = open(files[0:-4] + \".hdr\", 'w')\n",
    "            newfile.write(\"\"\"ENVI\n",
    "description = {DESCFIELD}\n",
    "samples = NSAMP\n",
    "lines = NLINE\n",
    "bands = 1\n",
    "header offset = 0\n",
    "data type = DATTYPE\n",
    "interleave = bsq\n",
    "sensor type = UAVSAR L-Band\n",
    "byte order = 0\n",
    "map info = {Geographic Lat/Lon, \n",
    "            1.000, \n",
    "            1.000, \n",
    "            LON, \n",
    "            LAT,  \n",
    "            0.0000555600000000, \n",
    "            0.0000555600000000, \n",
    "            WGS-84, units=Degrees}\n",
    "wavelength units = Unknown\n",
    "                \"\"\"\n",
    "                          )\n",
    "            newfile.close()\n",
    "            if files[0:18] not in Files_list:\n",
    "                Files_list.append(files[0:18])\n",
    "\n",
    "    #Variables used to recall indexed values.\n",
    "    var1 = 0\n",
    "\n",
    "    #Step 2: Look through the folder and locate the annotation file(s).\n",
    "    # These can be in either .txt or .ann file types.\n",
    "    for files in os.listdir(folder):\n",
    "        if Files_list[var1] and files[-4:] == \".txt\" or files[-4:] == \".ann\":\n",
    "            #Step 3: Once located, find the info we are interested in and append it to\n",
    "            # the appropriate list. We limit the variables to <=1 so that they only\n",
    "            # return two values (one for each polarization of\n",
    "            searchfile = codecs.open(files, encoding = 'windows-1252', errors='ignore')\n",
    "            for line in searchfile:\n",
    "                if \"Ground Range Data Latitude Lines\" in line:\n",
    "                    Lines = line[65:70]\n",
    "                    print(f\"Number of Lines: {Lines}\")\n",
    "                    if Lines not in Lines_list:\n",
    "                        Lines_list.append(Lines)\n",
    "\n",
    "                elif \"Ground Range Data Longitude Samples\" in line:\n",
    "                    Samples = line[65:70]\n",
    "                    print(f\"Number of Samples: {Samples}\")\n",
    "                    if Samples not in Samples_list:\n",
    "                        Samples_list.append(Samples)\n",
    "\n",
    "                elif \"Ground Range Data Starting Latitude\" in line:\n",
    "                    Latitude = line[65:85]\n",
    "                    print(f\"Top left lat: {Latitude}\")\n",
    "                    if Latitude not in Latitude_list:\n",
    "                        Latitude_list.append(Latitude)\n",
    "\n",
    "                elif \"Ground Range Data Starting Longitude\" in line:\n",
    "                    Longitude = line[65:85]\n",
    "                    print(f\"Top left Lon: {Longitude}\")\n",
    "                    if Longitude not in Longitude_list:\n",
    "                        Longitude_list.append(Longitude)\n",
    "    \n",
    "                        \n",
    "                 \n",
    "            #Reset the variables to zero for each different flight date.\n",
    "            var1 = 0\n",
    "            searchfile.close()\n",
    "\n",
    "\n",
    "    # Step 3: Open .hdr file and replace data for all type 4 (real numbers) data\n",
    "    # this all the .grd files expect for .int\n",
    "    for files in os.listdir(folder):\n",
    "        if files[-4:] == \".hdr\":\n",
    "            with open(files, \"r\") as sources:\n",
    "                lines = sources.readlines()\n",
    "            with open(files, \"w\") as sources:\n",
    "                for line in lines:\n",
    "                    if \"data type = DATTYPE\" in line:\n",
    "                        sources.write(re.sub(line[12:19], \"4\", line))\n",
    "                    elif \"DESCFIELD\" in line:\n",
    "                        sources.write(re.sub(line[15:24], folder, line))\n",
    "                    elif \"lines\" in line:\n",
    "                        sources.write(re.sub(line[8:13], Lines, line))\n",
    "                    elif \"samples\" in line:\n",
    "                        sources.write(re.sub(line[10:15], Samples, line))\n",
    "                    elif \"LAT\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Latitude, line))\n",
    "                    elif \"LON\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Longitude, line))\n",
    "                    else:\n",
    "                        sources.write(re.sub(line, line, line))\n",
    "    \n",
    "    # Step 3: Open .hdr file and replace data for .int file date type 6 (complex)                 \n",
    "    for files in os.listdir(folder):\n",
    "        if files[-8:] == \".int.hdr\":\n",
    "            with open(files, \"r\") as sources:\n",
    "                lines = sources.readlines()\n",
    "            with open(files, \"w\") as sources:\n",
    "                for line in lines:\n",
    "                    if \"data type = 4\" in line:\n",
    "                        sources.write(re.sub(line[12:13], \"6\", line))\n",
    "                    elif \"DESCFIELD\" in line:\n",
    "                        sources.write(re.sub(line[15:24], folder, line))\n",
    "                    elif \"lines\" in line:\n",
    "                        sources.write(re.sub(line[8:13], Lines, line))\n",
    "                    elif \"samples\" in line:\n",
    "                        sources.write(re.sub(line[10:15], Samples, line))\n",
    "                    elif \"LAT\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Latitude, line))\n",
    "                    elif \"LON\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Longitude, line))\n",
    "                    else:\n",
    "                        sources.write(re.sub(line, line, line))\n",
    "                        \n",
    "    \n",
    "    # Step 4: Now we have an .hdr file, the data is geocoded and can be loaded into python with rasterio\n",
    "    # once loaded in we use gdal.Translate to convert and save as a .tiff\n",
    "    \n",
    "    data_to_process = glob.glob(os.path.join(folder, '*.grd')) # list all .grd files\n",
    "    for data_path in data_to_process: # loop to open and translate .grd to .tiff, and save .tiffs using gdal\n",
    "        raster_dataset = gdal.Open(data_path, gdal.GA_ReadOnly)\n",
    "        raster = gdal.Translate(os.path.join(folder, os.path.basename(data_path) + '.tiff'), raster_dataset, format = 'Gtiff', outputType = gdal.GDT_Float32)\n",
    "    \n",
    "    # Step 5: Save the .int raster, needs separate save because of the complex format\n",
    "    data_to_process = glob.glob(os.path.join(folder, '*.int.grd')) # list all .int.grd files (only 1)\n",
    "    for data_path in data_to_process:\n",
    "        raster_dataset = gdal.Open(data_path, gdal.GA_ReadOnly)\n",
    "        raster = gdal.Translate(os.path.join(folder, os.path.basename(data_path) + '.tiff'), raster_dataset, format = 'Gtiff', outputType = gdal.GDT_CFloat32)\n",
    "\n",
    "    print(\".tiffs have been created\")\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "interior-samba",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder = '/tmp' # define folder where the .grd and .ann files are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "infrared-industry",
   "metadata": {},
   "outputs": [],
   "source": [
    "uavsar_tiff_convert(data_folder) # call the tiff convert function, and it will print the information it extracted from the .ann file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immune-ribbon",
   "metadata": {},
   "source": [
    "Now we'll delete the unneeded .grd and .hdr files that our tiffs have been created. If you're using this code on your local machine, this probably isn't absolutely necessary. The JupyterHub cloud we're working in has limited space, so deletion is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sublime-disease",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(data_folder)\n",
    "grd = glob.glob('*.grd') #define .grd\n",
    "hdr = glob.glob('*.hdr*') #define .hdr\n",
    "\n",
    "# remove both\n",
    "for f in grd:\n",
    "    os.remove(f)\n",
    "    \n",
    "for f in hdr:\n",
    "    os.remove(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sticky-dynamics",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check what's in the directory, only .tiffs and our annotation file!\n",
    "print(glob.glob(\"*.*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "portuguese-cornell",
   "metadata": {},
   "outputs": [],
   "source": [
    "### inspect our newly created .tiffs, and create named objects for each data type. We'll use these new obects in the next step\n",
    "\n",
    "# amplitude from the first acquisition\n",
    "for amp1 in glob.glob(\"*amp1.grd.tiff\"):\n",
    "    print(amp1)\n",
    "    \n",
    "# amplitude from the second acquisition\n",
    "for amp2 in glob.glob(\"*amp2.grd.tiff\"):\n",
    "    print(amp2)\n",
    "\n",
    "# coherence\n",
    "for cor in glob.glob(\"*cor.grd.tiff\"):\n",
    "    print(cor)\n",
    "\n",
    "# unwrapped phase\n",
    "for unw in glob.glob(\"*unw.grd.tiff\"):\n",
    "    print(unw)\n",
    "\n",
    "# dem used in processing\n",
    "for dem in glob.glob(\"*hgt.grd.tiff\"):\n",
    "    print(dem)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "multiple-heather",
   "metadata": {},
   "source": [
    "Inspect the meta data the rasters using the rio (shorthand for rasterio) ```profile``` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "metric-poverty",
   "metadata": {},
   "outputs": [],
   "source": [
    "unw_rast = rio.open(unw)\n",
    "meta_data = unw_rast.profile\n",
    "print(meta_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wireless-feeling",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:test_env]",
   "language": "python",
   "name": "conda-env-test_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}