Skip to content

Latest commit

 

History

History
344 lines (258 loc) · 38.7 KB

Resources.md

File metadata and controls

344 lines (258 loc) · 38.7 KB

Resources and tutorials

All-in-one tutorials for earth or geospatial data science

Introductions to remote sensing from NASA ARSET

GIS and geospatial concepts

Learning programming

  1. How to Think Like a Computer Scientist: Learning with Python 3: An online, html textbook for an introduction to programming (in Python 3). Referenced in MIT 6.00.
  2. DataCamp courses. These require a paid membership, but I think it's well worth the money: there are no ads, the lessons have built-in code editors with hints and auto-"grading", and the content is pared down to the essentials.

Learning python

Python and its numerical array programming package, numpy, have become so ubiquitous in the sciences that they were the topic of a 2020 review article in Nature!

  1. A tutorial provided by the developers of python- not recommended for beginners because it is quite formal and technical. Good for reference, though.

  2. An introduction to Python for scientific computing (M. Scott Shell / UCSB) - To the point and written by a biochemical modeling professor

  3. Python Like You Mean It: this looks good! Written by a former physics grad student who wished something like this had existed when they started. Described as "a free resource for learning the basics of Python & NumPy, and moreover, becoming a competent Python user," and "a lean, one-stop resource for learning the essentials of Python from scratch."

  4. Basic, interactive tutorial from W3Schools.com with advertisements.

  5. A bare-bones, interactive, 10-minute-long tutorial in python basics, with a more detailed version available for purchase at DataCamp.

  6. MIT 6.00: Introduction to Computer Science and Programming: How Ethan learned programming and python. Not recommended because it is quite verbose! It is also taught in python 2.xx, while python 3.xx is now current.

  7. An Introduction to Numpy and Scipy (M. Scott Shell / UCSB) - A great introduction to the basic functions available in numpy. Can also be used as a reference guide.

  8. Matplotlib plotting style guide

  9. Python best practices course - Material for the course "Python best practices", Scientific Software Center, Heidelberg University, by Inga Ulusoy, October 2022. Focused on advanced code formatting topics, like linting and PEP style recommendations.

  10. One-page "cheat sheets" for common python data science packages:

  • Numpy
  • Many others...just google cheat sheets for numpy, pandas, matplotlib, jupyter, xarray
  1. Old school? Here are some textbooks:

Python for tabular data analysis: pandas package

  1. A list of tutorials from pydata.org.

  2. Geohackweek and Intro to Earth Data Science online textbook chapters

Introduction to the Linux operating system

The following tutorials are recommended by the NASA Center for Climate Simulation:

  • Introduction to Linux Tutorial
  • More Advanced Linux Tutorial
  • SATERN Linux shell-scripting course code: os_doss_a01_it_enus (requires NASA affiliation)
  • Linux Handbook on file-permissions and ownership.

Shell scripting in bash

Bash is the most commonly used shell scripting language in minutes systems, and is often the default. It is the go-to for scientific processing workflows, and can be used to run other programs: python, matlab, etc. It is delightfully old-school, yet highly powerful.

Step one: install the windows subsystem for linux

...or Docker, Virtual Box, or git-bash. WSL is available from the microsoft store for free and I find it the most convenient way to run Linux on a Windows machine in a fast, low-resource way. Bonus: WSL integrates with the text editor Visual Studio Code.

  1. Lynda tutorial on "Linux: Bash Shell and Scripts": requires LinkedIn Learning membership, which Brown subscribes to. Information should be fairly easy to understand, but will take some time to watch the videos.

  2. The linux documentation project: lots of useful, if dated, guides and code snippets to learn with. For a quick crash course, or info on a specific topic, try the Bash Guide for Beginners by Machtelt Garrels and Advanced Bash-Scripting Guide by Mendel Cooper.

Geocomputation workflows

  1. Geohackweek: An up-to-date resource for geoprocessing workflows in Python, introduced through a series of python Jupiter notebook tutorials. Thanks to The University of Washington organizers for offering these workshops and archiving the tutorials!

  2. Spatial-ecology.net workshops and tutorials: Now preserved on Youtube, this site gives introductions to a variety of geospatial processing, including command line programs like gdal, and GIS software like QGIS and GRASS. Includes some introduction to linux, bash, and python as well. Thank you to Giuseppe Amatulli at Yale for hosting this site.

Remote sensing tutorials

  1. NASA ARSET has dozens of tutorials on a number of topics, from sensors like optical and SAR to applications such as wildfires and hydrology.

Synthetic Aperture Radar (SAR) tutorials

  1. NASA SERVIR SAR Handbook: Comprehensive Methodologies for Forest Monitoring and Biomass Estimation - "The SAR Handbook: Comprehensive Methodologies for Forest Monitoring and Biomass Estimation is the culmination of a two-year collaboration between NASA SERVIR and SilvaCarbon. Five trainings, led by six SAR subject matter experts, were held at hubs throughout the SERVIR network. The topics of these trainings included SAR basics, SAR for forest change detection, forest height estimation, biomass estimation, mangrove monitoring, and sampling design."
  2. SERVIR Global: SAR Handbook: Comprehensive Methodologies for Forest Monitoring and Biomass Estimation
  3. UAF SAR Capacity Building Center - Free, self-paced archived courses via EdX. As of September 2022, only SAR for hazards is available.
  4. NRCAN SAR tutorial
  5. Sisters of SAR resources
  6. EdX class: Synthetic Aperture Radar: Hazards by Franz Meyer - EdX is free unless you want a certificate of completion

SAR Software

  1. PolSAR Pro (ESA)
  2. SNAP (ESA)
  3. Sentinel Playground (web software for visualization)
  4. ASF Vertex (download scenes, apply terrain correction, order inSAR processing)
  5. ASF Mapready
  6. ASF OpenSARLab
  7. RSCube - Landcover classification using SAR from Marc Simard's group
  8. SnowEx UAVSAR tools on Github
  9. Many others on Awesome SAR github curated list - similar to Awesome Python, etc.

Data science

  1. Introduction to Data Science A Python Approach to Concepts, Techniques and Applications (Laura Igual and Santi Seguí, Springer 2017)

  2. Practical Data Science - Duke University Data Science Masters Program class available publically - covers everything beyond numpy and pandas

  3. ORNL DAAC learning resources: Dozens of quick tutorials on specific topcs such as downloading and analyzing multi-spectral AVIRIS data to using their WMS web maps in GIS tools.

  4. Pangeo gallery, a set of example scripts and workflows for various earth science topics, using the open – source pangeo Python environment

  5. Resources from Karianne Bergen's Machine Learning for Earth & Environment class at Brown (EEPS1960d)

  6. ... Many others!

Machine learning

  1. Artificial Intelligence for Earth System Sciences workshop (https://www2.cisl.ucar.edu/events/summer-school/ai4ess/2020/artificial-intelligence-earth-system-science-ai4ess-summer-school): includes archived workshop tutorials and jupyter notebooks, organized by NCAR.

  2. DELTA - A NASA toolbox (run from command line or python) for turnkey machine learning workflows- it tiles raster into chips, runs a classification algorithm on it (defaults are various neural networks), and restitches results. You give directories of images, training data as categorical rasters, and a yml file with desired algorithm parameters and it trains and runs the model!

  3. Solaris - An open source ML pipeline for overhead imagery by defense intelligence contractor CosmiQ Works. Tiling datasets, raster/vector/ML format conversion, segmentation/object detection/deep learning, evaluation

Statistics

  1. An Introduction to Statistics with Python (Thomas Haslwanter, Springer, 2016)

  2. An Introduction to Statistical Learning with Applications in R (James, G., Witten, D., Hastie, T., Tibshirani, R., Springer, 2013) - recommended by Ethan, 2017 edition available in print

  3. Earth Data Science online textbook linear regression overview - basic intro, using the latest plotting/data packages

  4. Which Statistical Test? from UCLA Statistical Consulting Group

  5. Python Spatial Analysis Library (pysal)- a series of packages for spatial statistics, including exploring, modeling, and visualizing spatial data

Google Earth Engine (GEE)

  1. Straight from the source: guides and tutorials. Lots of links to resources during Geo For Good 2020.
  2. Also covered in geohackweek.
  3. Examples scripts in the default code editor
  4. Useful python toolboxes: geemap with lots of documentation and tutorials.
  5. Google Earth Engine with Python (EEwPython) - by Cesar Aybar Camacho and Roy Yali Samaniego - A series of Colab notebooks teaching the EE python toolbox, with folium plots embedded. Adapted from Google Earth Engine Guides, linked above.

Note: Be sure to check out community earth engine datasets, such as as https://gee-community-catalog.org/, for a much more comprehensive collection of datasets!

Visualization

  1. Pydeck/Google Earth Engine
  2. ColorMoves: An interactive color map picker tool, downloadable color maps, and a script to import them into python or matlab
  3. Additional tools listed on sciviscolor:

GIS software

  1. QGIS
  2. ArcGIS Desktop
  3. CloudCompare: open-source GIS-like software for all 3D formats, especially point clouds. Can build surfaces, subsample points, make videos and has some rendering features such as illumination angle.

Geospatial plotting and map packages for python

Although python is excellent at managing and processing geospatial data, visualizing it beyond simple matplotlib plots is less seamless. Packages that require enabling jupyterlab extensions seem to be more prone to dependency conflicts. See this article for some inspo. Dr. Qiusheng Wu sums up the need for versatile geospatial packages in the "Statement of Need" for leafmap.

  1. leafmap by Qiusheng Wu. Amazing package, and my favorite among these! It does require enabling jupyterlab widgets, but I didn't experience any glitches. I installed in a new environmnt, initially, as recommended. I think you can get away with just conda install -c conda-forge leafmap and enable the corresponding jupyterlab extension. Requires xarray-leaflet for loading locally-stored rasters.
  2. cartopy (since 2016, originated from matplotlib + basemap, which is now deprecated) - "originally developed at the UK Met Office" - simple tutorial. Another tutorial in Google Colab from EEPS1960d. I don't believe maps are interactive unless you install the ipympl interactive widget and enable with %matplotlib widget Basemaps need to be downloaded with a line of code.
  3. plotly (via mapbox + dash) - examples
  4. folium - good basemap functionality with one line of code. To plot rasters, you must reproject to match rest of map, convert to 8bit and remove NaNs (requires rioxarray). Tutorial from earthdatascience.org
  5. ipyleaflet - recommended in geohackweek, but I had dependency issues with other packages and jupyterlab extensions
  6. geoviews - recommended in geohackweek, and it works successfully using bokeh for visualizations, although slowly.
  7. whitebox-python by Qiusheng Wu
  8. geemap by Qiusheng Wu. Interface to GEE. Very well documented with huge user community.
  9. GeoJSON - a simple way of mapping geojson files or text strings with a basemap in a new jupyter tab or notebook cell. Not designed for viewing multiple layers. Requires enabling jupyterlab extensions
  10. ggplot for python?
  11. mapbox-gl - I've never used, but it's also in the tutorial from earthdatascience.org. Requires free MapBox Access Token to view basemaps, and is supposedly less well-documented than folium.
  12. keplergl. Built around mapbox-gl and optimized for big data and 3d visualizations. Looks complicated to install. Requires enabling jupyterlab extensions
  13. contextlily - Used for basemaps in plots - "is a small Python 3 (3.7 and above) package to retrieve tile maps from the internet. It can add those tiles as basemap to matplotlib figures or write tile maps to disk into geospatial raster files. Bounding boxes can be passed in both WGS84 (EPSG:4326) and Spheric Mercator (EPSG:3857)."

Deep Learning

  1. satellite-image-deep-learning - Robin Cole - An amazing compilation of tools for every stage in the process for running neural networks on satellite images. Links to dozens of other repos with descriptions of what they are used for.
  2. Fast.ai - Making deep learning training accessible. "Deep learning is transforming the world. We are making deep learning easier to use and getting more people from all backgrounds involved through our: free courses for coders, software library, cutting-edge research, community" Written about in The Economnist, NY Times, MIT Tech Review.

Python environments

Not all of these packages play together nicely, and I have spent a lot of time trying to fix dependency conflicts within my geospatial conda environments. This is completely typical for open-source software, but the benefits outweight the cost, in my opinion! In general, I have found the best ways of fixing conflicts is starting from scratch and installing as few packages as possible. You can re-create an environment from only the packages you explicitly installed, and not from whatever dependencies were necessary at the time using the conda command: conda env export --from-history. I have found this command helpful in these scenarios.

Fortunately, there are some great resources for existing geospatial python environments. Check out the environment files in:

  • Geohackweek tutorial environments conda environment.yml files for the various exercises. As far as I can tell, they don't provide a combined environment that works for everything.
  • Earth Analytics python environment - developed for the Earth Data Analytics Online Certificate courses

My new favorite method is a package that automatically installs a very complete geospatial environment when you install it:

Other compilations of python and GIS tools

  • Awesome GIS - Awesome GIS is a collection of geospatial related sources, including cartographic tools, geoanalysis tools, developer tools, data, conference & communities, news, massive open online course, some amazing map sites, and more.
  • Awesome Python - A curated list of awesome Python frameworks, libraries, software and resources.

DEM tools

  1. RichDEM: High-performance Terrain and Hydrology Analysis
  2. TauDEM
  3. Topotoolbox (Matlab): super popular File Exchange package that runs smoothly and has great tutorials.
  4. Mod-WET: (Matlab) accompanying matlab package for Steven Margulis' online hydrology textbook.
  5. Arcmap standard hydrology toolbox and optimized pit removal tool
  6. QGIS hydrology toolboxes (includes GDAL, GRASS and SAGA algorithms)
  7. Whitebox GAT: geomorphology package Jessica likes: GUI and CLI. Includes citations for the algorithms.
  8. STARS stream modeling software as a (buggy) ArcGIS package. Designed to interface with an R toolbox called SSN that computes spatial stats, such as semivariogram, but calculated along stream distance. Includes hydrology tools such as reach-contributing area and others. Popular with aquatic biologists studying things like salmon density vs temperature along streams. Annual tutorial workshops. Ethan attended one in 2019.
  9. PCRaster - python toolbox for "spatio-temporal environmental models", including calculting watersheds and reach contributing area.
  10. phoREAL from IceSat-2 Science Team
  11. icepyx - haven't tried it

Hydrology open-source/free textbooks

  1. Introduction to Hydrology - Margulis - Great textbook on fundamentals of surface, subsurface, and atmospheric hydrology.
  2. Physical Hydrology course (Utah_State_University) on Hydrolearn - Lane/Garousi-Nejad - for Junior/Senior Hydrology and Water Resources Engineering Students
  3. The Groundwater Project books
  4. Rain or Shine - Ochsner/Howerton/Ellis - This text is an introduction to soil physical properties and processes.
  5. Hydrological Drought – 1st Edition (2004) - Tallaksen/van Lanen- from the European Drought Centre
  6. Techniques of Water-Resources Investigations Reports - a massive collection from the USGS Publications Warehouse

Climate

GDAL/OGR command line reference

Git

Git is software for code version control for individuals who want to be able to revert to checkpoints. It also offers ways to seamlessly sync scripts between collaborators using websites like github.

  • LinkedIn Learning course on git - how Ethan learned
  • Much shorter all-in-one tutorial through the online school, W3docs
  • Git tutorial from the developers of github
  • Learning through a series of turorials on github
  • Another jupyter notebook-based tutorial for learning git, from Patrick Gray.

Other compilations

  • Resources lists: Introduction to Earth Observation and Introduction to Machine Learning/Deep Learning by Hannah Kerner

Matlab

...

R

Best data management practices

  • This article (Borer et al., 2007) has aged well and outlines specific principles for working with and archiving datasets (e.g. don't put spaces in column names).

Free cloud computing resources

  • Google Earth Engine, of course
  • Google Colab - cloud-hosted jupyter notebooks with a distinct Google flavor
  • CryoCloud - from IceSAT-2 team
  • Pangeo Cloud - for using Pangeo tools, funded through the end of 2023
  • OpenSARlab - from the Alaska Satellite Facility, free to use for the community
  • Binder - use any github repo or jupyter notebook interactively, spins up a small server without persistent storage, like Google Colab

Getting help

  • GIS stack exchange will be your best friend!
  • The Spatial Community runs a slack workspace with over 8k people on it, many of them adept programmers and web developers in various geospatial-related industries. It is very friendly and welcoming community.

Books on career development

  • Navigating Graduate School and Beyond: A Career Guide for Graduate Students and a Must Read For Every Advisor, Sundar A. Christopher (AGU Publications, 2011)
  • The Professor is In - Karen Kelsky - a bleak set of observations on the academic job market and practical advice for pursuing (or not pursuing) an academic career- focused on social sciences. The author also writes a blog on the topic.
  • Navigating an Academic Career: A Brief Guide for PhD Students, Postdocs, and New Faculty - Jeffrey J. McDonnell (AGU Publications, 2019)
  • ThinkWell (books, workshops, resources - I highly recommend!) - Maria Gardiner and Hugh Kearns have worked as an award winning team for the past fifteen years. They are well known as leading practitioners and researchers in cognitive behavioural coaching.
    • Book: 52 Ways to Stay Well: During your PhD, post-doc or research career, by Hugh Kearns

Networking and promoting your work

Undergraduate REUs and internships

  • NSF REU programs
  • Lamont Summer Intern Program
  • GSA
  • Summer undergraduate program for planetary research (SUPPR)
  • NASA
    • NASA SARP
    • NASA JPL internships
    • MA Space Grant fellowships
    • The NASA L'SPACE Program is a free, online, interactive experience open to undergraduate STEM students interested in pursuing a career with NASA or other space organizations.
    • NASA DEVELOP
  • DOE
  • NOAA internships
  • USGS student opportunities (through USAJobs)
  • National Park Service internships and young adulat programs
  • Earthscope seismology internships
  • Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) Resourceshttps://www.cuahsi.org/resources-for-undergraduate-students for Undergraduate Students
  • NASA L'SPACE Program - a free, online, interactive experience open to undergraduate STEM students interested in pursuing a career with NASA or other space - includes Mission Concept Academy and Proposal Writing and Evaluation Experience Academy (with $10k funding opportunity)

Summer and/or field schools (registration and payment needed)