From db8dc7698b1ee740a1a76ac681b140e3afce2ccf Mon Sep 17 00:00:00 2001 From: Pete Gadomski Date: Tue, 10 Oct 2023 08:13:43 -0600 Subject: [PATCH] docs: s/veda-data-pipelines/veda-data-airflow/ I can't update the diagram b/c it's draw.io. --- .readme/veda-backend.drawio.svg | 4 +- .readme/veda-backend.drawio.xml | 107 +++++++++++++++++++++++++++++++- README.md | 4 +- 3 files changed, 109 insertions(+), 6 deletions(-) diff --git a/.readme/veda-backend.drawio.svg b/.readme/veda-backend.drawio.svg index 53c7f05f..0283d570 100644 --- a/.readme/veda-backend.drawio.svg +++ b/.readme/veda-backend.drawio.svg @@ -1,4 +1,4 @@ - + -
Data Store
Data Store
PgSTAC
PgSTAC
TiTiler
TiTiler
STAC-API
STAC-API
Scientists &
Public
Scientis...
veda-stac-ingestor
veda-stac-ingestor
veda-data-pipelines
veda-data-pipelines
 
Analysis
Platform
...
VPC
VPC
Scientists &
Data Providers
Scientis...
Dashboard (veda-ui & veda-config)
Dashboard (veda-...
Text is not SVG - cannot display
\ No newline at end of file +
Data Store
Data Store
PgSTAC
PgSTAC
TiTiler
TiTiler
STAC-API
STAC-API
Scientists &
Public
Scientis...
veda-stac-ingestor
veda-stac-ingestor
veda-data-airflow
veda-data-airflow
 
Analysis
Platform
...
VPC
VPC
Scientists &
Data Providers
Scientis...
Dashboard (veda-ui & veda-config)
Dashboard (veda-...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/.readme/veda-backend.drawio.xml b/.readme/veda-backend.drawio.xml index 0310b577..c684f973 100644 --- a/.readme/veda-backend.drawio.xml +++ b/.readme/veda-backend.drawio.xml @@ -1,2 +1,105 @@ - -7Vxtc+MoEv41qbr7YJXeZX+0neRuq/auXJupvdtPU1giNhdZ6BCOnf31BwJsCZBfYtmZ3GQykzENQgiefrppWr4Lpqvt3wgol//AGczvfDfb3gX3d74/ShL2mwvehCAKXSFYEJQJkbcXPKE/oRSqZmuUwarVkGKcU1S2hSkuCpjSlgwQgjftZs84b9+1BAtoCJ5SkJvSf6GMLoV0GLl7+d8hWizVnT1X1sxB+rIgeF3I+935wUP9R1SvgOpLtq+WIMObhih4uAumBGMqPq22U5jzqVXTJq577KjdjZvAgp5ygS8ueAX5GqoRxzm7dPKMWQ9s6kAqKuL/rvmgJlO8JggSVvVPuNmL2aeF/L++fK4LZH8VfcvbHfKKQVVDYMwaeEG5FZdpPd8DClj9E8UEqk7Zg4l+2/diYuP+RlO/NRofr2mOCjjdIcqVY5viHJO6TeCN+Q+XozxvyB+j8STi7StK8AtUNQUuWOeTDFRLmMkeXyGhiAHtVzCH+QxXiCJcsLo5phSvGg3GOVrwCopLJgWylLJ1ZZMfTJZ0lbOyJwcpVcjzVVk+GL8lqErxRM9oy8cxKTHivTy8ss4q2QkDYskvWG0XXKMdsKlCZ75OXyCfrRrRu2cQT6kUw9vNJh863HYi0dvhm9EGxCtIyRtrIi9IvFBcIikjcH0nEpLNXgUTKVo2tC8ZSsWXSr/Y9b0HPvsgsW/Xg6BTDzL0yvuW87+HZQ6faTf8L0D75So3Wzx9G0/PUoaduH7cDhWpcVNPcjRhfxmzTMW/iNVOucTxI4vQJktMoWc2Y/95tjvoQpssMYWe2YyX1KjbQpssicwR61d7lqs97Wr2N5icxThMKzPEVOkeEdZYcEaBCdeLE9mI1Yxc/tMzJdWKcBkhWcmHwIrhPoW/pHw0E1YUn7RWWWXwUw9sFOlspDyaJhe5Fi5yL+ei0FC89uPxVccLXLAVw3wl6jn/D6T0TU46WFPcXhEDCd7Qf+B378RbzsEw2XkzjSbSn+GdAkLH3NvihikHVYVSJX5Eubo3LDLVSBpEJpH1+9WCmeaRicUXoli6gIAsIG05LiesKIE5oOi13fsl6xP1vz7sGcjbv+X1deEPXuDUIsv322bt/VuzNIMEsafgfsG9Ve8//WoHH7fa8TEP+WNsfG9u9Tf0DeX8xh/rUXfbqi7r9sO51RdYsRys5hn4/rwupG3v36IFkd+2aFFgWLQ4MS2akl2iQ8ntGDP5GRlz+EMx5vDzxBT4Pmkwnv3yxX4/GfuFoenPx5bYgpJdog8jAxtcmdVkSp3vkRCdHQX+0azr4MNPTn0qYtzaGww/jvzUeLoiSVYCu5gQn1IeG0AVre54wDcGq1q/699CItqTdzGbPTh09Yearec5g03vYazqBdJ0qTTtK+xrsPa6qu/VA+16btCmXdd0OiNLGCXqIYzieRZNvAERHybUR7BCOZ+NtjZ0Y24HI+806j2FM/2b+Ivy0hmHXSOyFrUja6EeMhOjkldpy70bxmkIuP7p1ivMwICZtXSAigWsKCZn8ZUGx80SUfhUikFtmEK2kWXniCYpSQwfcCffEV5uIVBnmatHXiNvZDkH8iLPZI0g6IE1uk+CesVMBigYlKiE3PpUX6C5EDSBBhpFKjeBjBmwv7KDn/xUHn70g3n40TkeflOdhT9ezKv3uuWn+d+7K8cMXm8Vqm5wqxmb5GdMVuf0qulMA+p2p/wcyjpETe30Bd/iBReQbjB5qZw0x+vs0IbgmEenHPRAp0RemAHK/PGilvhuT3kUseZxjyKb+QyudHbpxca6/uyHlyr+0+Sv+APp6wrB+E++QoElhvSRBsYWQP/kG1e4RVT4LqNwKMvcdxm4jhurBnv/hRfeGgX9vKanffDtNr2jwAkD1428oReEeqpbFGvMKwZpbIGNbhOFXEX23vX20iMDlL/PpgYurSliRnqYnhpmpIW1U8KMdCs91cpIs2onfBk5YXrimJFd1k5AM3LA9EQxI5vsYIpXQ+Vse7djcb2UlQHrlMg+9LCeaJPnoKzQfL/lgumaVAzBv0FxLia2BrbwH9epsvahbIc1de331zI9vGU0nbWTssgqNheoWPxal+6DQ7vPVny1B8fJD7WUL8/cQ4a2YGWga917SN+3nRv835B+GIVt0vfc6GzSf+8JV7/GoukkqADj9Q1I7B40ILrrfqoBiRItGDu6mgHZzemtoiRNiJyQF9KzIvQDOkUCLdB9YGqHGs/5sY9LIqiXnG6ekc5xZsjjwoeSr7PMCH5FGSRVD2PW1OvrnPNm55yhO2rxaByYvsO1zjmt765oUFAuXdeCd06BfMEOzFVP7rlTo8Wj7K/1xJ4l89DXjdG7Zic8PjtHQXv0OMXAr1VTOtVKB+efawKdFK/KtejNhuizl7K5ZAIz5po1Q4Re4oyiaDjkuVOj0LOt0LCH0xNLOv2VT0/8K5+eNGIdiXKDpRMyUrGPXkMd142UmbjRfJTQ9FFE+OwWDkkP6fl+R3Lo5WmnZs/vcBOq5RwDkrEGf6lPjtfI6gXVdYyun9Hir+/zJT7glFkR4yUHzMqRufCAuYsS1RW+E/qJu+dDv23wDZM2GjlDt8GeFvrsxfonx+2bsi9oVb/2fnzdjhrEuqexcrysXpi82f2SUv7S/pg/kf+YZoXroBqmbIEIN3Lci6194UcuZ37wI2T3rz8P/LDcDl5xPuATwonmkVtFUA08f+iUxcLGz620Bd1xPObtSkDWg5803+VvVWCSwUbrXvDlMXxFLUypBLamUY6dINhBKg4toOpochHGTjiQUBhjj0wRyH/jTmOx4LXWIGcDggpUdYHIB6mT3UQQkH+sYdl1CsyWiKJiMa19L254ByJAdH5CC8EUSKiPeiMON3YS/t0R2oLsFnnoeGZSpOcOnQZ5xGFsWemhk/SQnu6bkX1zdd+9lTQWgZEi+zngDZ+0n1yhLMttUeUdRR3aUh4Lu+smpd5Bisg0KwVd+8mS4GydUvmyQ2kNn4MSfV8ACjfgrZ+tp6+lPgUqv7K5vQpN9CjZRd+acELQ+gs614HOVb6EQwfTFV+TYcX919yIePL+q4SCh/8B \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/README.md b/README.md index 888e1c1c..fcd4bc11 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ docker compose down > **Warning** PgSTAC records should be loaded in the database using [pypgstac](https://github.com/stac-utils/pgstac#pypgstac) for proper indexing and partitioning. -The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-pipelines](https://github.com/NASA-IMPACT/veda-data-pipelines) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor). +The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-airflow](https://github.com/NASA-IMPACT/veda-data-airflow) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor). ## Support scripts Support scripts are provided for manual system operations. @@ -139,7 +139,7 @@ Support scripts are provided for manual system operations. | [**veda-config**](https://github.com/NASA-IMPACT/veda-config) | Configuration for viewing VEDA assets in dashboard UI | | [**veda-ui**](https://github.com/NASA-IMPACT/veda-ui) | Dashboard UI for viewing and analysing VEDA assets | | [**veda-stac-ingestor**](https://github.com/NASA-IMPACT/veda-stac-ingestor) | Entry-point for users/services to add new records to database | -| [**veda-data-pipelines**](https://github.com/NASA-IMPACT/veda-data-pipelines) | Cloud optimize data assets and submit records for publication to veda-stac-ingestor | +| [**veda-data-airflow**](https://github.com/NASA-IMPACT/veda-data-airflow) | Cloud optimize data assets and submit records for publication to veda-stac-ingestor | | [**veda-docs**](https://github.com/NASA-IMPACT/veda-docs) | Documentation repository for end users of VEDA ecosystem data and tools | ## VEDA usage examples