index wv2 files from extra_data/wv2 ?

[7yl4r]

Files currently living in extra_data/wv2 have not been "indexed" by the imars-product-metadata db. What I mean by that is that there is no entry for them in the database. This is the first step to processing them through the airflow I have set up.

They can be easily added using imars-etl load --storage_driver no_upload --directory ..., but without additional metadata about each image (region coverage or other) the metadata db wouldn't serve much purpose other than to mark which files have been processed and allow tasks to fetch files by datetime.

[7yl4r]

I am indexing a few to test with. (NOTE that status_id=5 is just a temporary placeholder I am using)

2014-04/*M1BS*ntf

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2014.04/ -p 11 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.ntf"
...
INFO:imars_etl.load._load_dir:43 files loaded, 387 skipped, 0 duplicates.

[7yl4r]

2014-04/*M1BS*xml

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2014.04/ -p 14 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.xml"

INFO:imars_etl.load._load_dir:43 files loaded, 387 skipped, 0 duplicates.

[7yl4r]

2014.06/*M1BS*xml:

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2014.06/ -p 14 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.xml"
INFO:imars_etl.load._load_dir:62 files loaded, 558 skipped, 0 duplicates.

2014.06/*M1BS*.ntf:

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2014.06/ -p 11 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.ntf"
INFO:imars_etl.load._load_dir:62 files loaded, 558 skipped, 0 duplicates.

2016.06 M1BS ntf:

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2016.06/ -p 11 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.ntf"
INFO:imars_etl.load._load_dir:13 files loaded, 107 skipped, 0 duplicates.

2016.06 M1BS xml:

tylar@tylardesk:~/imars-etl$ python3 -m imars_etl -vvv load -d /srv/imars-objects/extra_data/WV02/2016.06/ -p 14 -j '{"status_id":5, "area_id":5}' --storage_driver no_upload -l "WV02_%Y%m%d%H%M%S_{junk}-M1BS-{idNumber:12}_{otherNum:2}_P{passNumber:0>3d}.xml"
INFO:imars_etl.load._load_dir:13 files loaded, 107 skipped, 0 duplicates.