methane-gapfill-ml

Python codebase for our manuscript "Gap-filling eddy covariance methane fluxes: Comparison of machine learning model predictions and uncertainties at FLUXNET-CH4 wetlands".

This codebase implements the following features:

• Data preprocessing
• Model training
• Model testing
• Uncertainty estimation, calibration/sharpness, and post-processing calibration
• Gap-filling a dataset with a trained model

Jupyter Notebook Tutorial

https://github.com/stanfordmlgroup/methane-gapfill-ml/blob/master/fluxgapfill_tutorial.ipynb

Installation

pip install fluxgapfill

Usage

Prepare data in a CSV and include the following headers:

• TIMESTAMP_END: Format YYYYMMDDHHmm (e.g. 201312060030)
• FCH4: Methane flux in nmol m^-2 s^-1 All other headers will be treated as input predictors.

Create a folder called data/ and make another folder in data/ using the site ID (data/{SiteID}/). This is where all of the processed data for the site will be written.

Place the CSV in this folder and name it raw.csv, so the full path to the CSV should be data/{SiteID}/raw.csv, where {SiteID} should be replaced with the actual ID of the site.

Preprocess the data

python main.py preprocess

Train models

python main.py train

Evaluate a trained model

python main.py test

Gapfill using a trained model

python main.py gapfill

Run all steps, including data preprocessing, model training, model evaluation, and gapfilling

python main.py run_all

Run python main.py {preprocess,train,test,gapfill} --help for descriptions of all of the command-line arguments.

Example commands using the sample data in the repository:

python main.py preprocess --sites NZKop --eval_frac 0.1 --n_train 10
python main.py train --sites NZKop --models [lasso,rf] --predictors_paths predictors/meteorlogical.txt

When specifying multiple values for a parameter, you can either use a comma-separated string or list syntax like in the above command.

Contributions

This tool was developed by Jeremy Irvin, Yulun Zhou, Fred Lu, Vincent Liu, and Sharon Zhou.

License

Apache License 2.0.

Citing

If you're using this codebase, please cite:

1. The Gapfilling Algorithm: Irvin, J., Zhou, S., McNicol, G., Lu, F., Liu, V., Fluet-Chouinard, E., ... & Jackson, R. B. (2021). Gap-filling eddy covariance methane fluxes: Comparison of machine learning model predictions and uncertainties at FLUXNET-CH4 wetlands. Agricultural and Forest Meteorology, 308, 108528.

2. The Python-Toolkit: In text: "We used the FluxGapfill python toolkit (Version 0.2.0; Irvin, et al, 2021) to complete our work."

References

Irvin, J., Zhou, Y., Lu, F., Liu, V., Zhou, S., McNicol, G., and Liu, J. (2021). FluxGapfill: A Python Interface for Machine-learning Driven Methane Gap-filling. Version 0.2.0. Zenodo. https://doi.org/10.5281/zenodo.5515761. Accessed 2021-09-19.