Automate Geoglam & NO2 dataset ingestion

[smohiudd]

Description

NO2 (#89) and Geoglam (#167, #173) datasets requires monthly ingestion as new assets are created. This is currently a manual process however should be automated. veda-data-airflow has a feature that allows scheduled ingestion by creating dataset specific DAGs. The file must still be transferred to the collection s3 bucket. A json file must be uploaded to the airflow event bucket. Here is an example json:

{
    "collection": "emit-ch4plume-v1",
    "bucket": "lp-prod-protected",
    "prefix": "EMITL2BCH4PLM.001/",
    "filename_regex": ".*.tif$",
    "schedule": "00 05 * * *",
    "assets": {
        "ch4-plume-emissions": {
            "title": "EMIT Methane Point Source Plume Complexes",
            "description": "Methane plume complexes from point source emitters.",
            "regex": ".*.tif$"
        }
    }
}

Acceptance Criteria

• transfer no2-monthly and no2-monthly-diff from s3://covid-eo-data bucket to s3://veda-data-store-staging and s3://veda-data-store using MWAA transfer dag
• scheduled ingestion (bi weekly) json files are created and uploaded for NO2 (no2-monthly, no2-monthly-diff) in mwaa event bucket for staging (UAH) and production (MCP)
• scheduled ingestion (bi weekly) json files are created and uploaded for Geoglam (geoglam) in mwaa event bucket for staging (UAH) and production (MCP)
• Validate that new ingestions are initiated in staging and production. Add new geoglam files from GEOGLAM August 2024 #167, GEOGLAM September & October 2024 #173 into veda-data-store-staging bucket.
• transfer, automated ingestion json configs are in veda-data

[slesaad]

Putting the discovery-items config within s3://<EVENT_BUCKET>/collections/ in the following format: https://github.com/US-GHG-Center/ghgc-data/blob/add/lpdaac-dataset-scheduled-config/ingestion-data/discovery-items/scheduled/emit-ch4plume-v1-items.json will trigger the discovery and subsequent ingestion of the collection items based on the schedule attribute

[smohiudd]

mcp-prod will need a new release of airflow to include automated ingestion

[smohiudd]

aws s3 ls s3://covid-eo-data/OMNO2d_HRMDifference/

aws s3 ls s3://covid-eo-data/OMNO2d_HRM/

[sandrahoang686]

Update: We have decided to run these weekly instead of bi-weekly

[anayeaye]

I added the scheduled collection configs from veda-data #177 to mcp-test and mcp-production

[anayeaye]

It looks like the uah-staging DAG has run for geoglam and discovered no files (expected). And the DAGs for the NO2 collections are visible but have not run (I need to revisit the configs to see if this is also expected). In mcp-prod the DAGs are present but have not yet run.

Question: Do we expect the scheduled ingest setup for SM2A to be the same as it was for MWAA?
2 files

[anayeaye]

UPDATE: Let's keep this open until I get a chance to add the config to the SM2A /collections bucket because we will be deprecating MWAA

[anayeaye]

Configs now in SM2A. We may later update the scheduled job regex to be more restrictive to address a recurring filename pattern change for the geoglam collection #213