Rename ntd api tables #3546
Rename ntd api tables #3546
Conversation
|
Warehouse report 📦 Checks/potential follow-upsChecks indicate the following action items may be necessary.
New models 🌱calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__breakdowns calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__breakdowns_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__capital_expenses_by_capital_use calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__capital_expenses_by_mode calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__capital_expenses_for_existing_service calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__capital_expenses_for_expansion_of_service calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__employees_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__employees_by_mode calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__employees_by_mode_and_employee_type calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__fuel_and_energy calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__fuel_and_energy_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_by_expense_type calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_directly_generated calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_federal calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_local calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_state calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__funding_sources_taxes_levied_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__maintenance_facilities calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__maintenance_facilities_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__metrics calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__operating_expenses_by_function calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__operating_expenses_by_function_and_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__operating_expenses_by_type calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__operating_expenses_by_type_and_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__service_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__service_by_mode calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__service_by_mode_and_time_period calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__stations_and_facilities_by_agency_and_facility_type calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__stations_by_mode_and_age calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__track_and_roadway_by_agency calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__track_and_roadway_by_mode calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__track_and_roadway_guideway_age_distribution calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__vehicles_age_distribution calitp_warehouse.staging.ntd_annual_data_tables.stg_ntd_annual_data__vehicles_type_count_by_agency DAGLegend (in order of precedence)
|
There was a problem hiding this comment.
Maybe Github got confused here. Can you check if metrics table still exists after this change?
There was a problem hiding this comment.
Found it.
It is here, but inverted.
There was a problem hiding this comment.
It is nicer when it points as a replaced by though. Helps when you need to find the history, but since these models are new and not much to show before that, just having the old removed and a new add sounds good.
There was a problem hiding this comment.
ah yeah it was such a bulk action and I generated all the new stuff with scripts so it was way easier to delete and add but that definitely is not as nice as when it points as replace by, sorry about that. hopefully because it is new it's not too much of an issue
charlie-costanzo
requested review from
evansiroky,
vevetron and
tiffanychu90
as code owners
charlie-costanzo
force-pushed
the
rename-ntd-api-tables
branch
2 times, most recently
from
4a538a4
to
282b6a2
Compare
…r-agnostic naming convention
…lude references to the year 2022 specifically
charlie-costanzo
force-pushed
the
rename-ntd-api-tables
branch
from
282b6a2
to
0bc53d0
Compare
* make ntd table API sync for annual data year-agnostic * revise sync dag ymls * rename paths and destination tables in external table config, rename files * replace all staging tables with new year-agnostic naming * replace the staging tables referenced in ntd mart tables with new year-agnostic naming convention * revise testing for year values to include the year 2023 * revise ntd annual reporting mart table documentation to no longer include references to the year 2022 specifically
charlie-costanzo
added
the
data-pipeline-ingestion-and-modeling
Description
Recently, we created a new data pipeline to ingest NTD annual reporting data from the DOT data portal API. The data portal API is new, so the only annual reporting data that existed within it at the time was for the year 2022.
Documentation showed no indication that future years' data would be appended to the same endpoint, so it was assumed that future years' data endpoints would be added as new Airflow DAG tasks. When 2023 reporting data was released, DOT appended the data to the 2022 data endpoint and adjusted the documentation to reflect data from the years 2022-2023.
When ingesting the initial 2022 endpoints, the naming conventions used were very 2022-centric. This PR replaces the naming conventions used in the data ingestion and external, source, and staging tables to be year-agnostic. It also replaces the tables referenced in mart tables to utilize the new table names.
Resolves #3523
Type of change
How has this been tested?
locally with dbt
Post-merge follow-ups
observe post-merge for expected behavior, change any outstanding/new models to use these new schemas and communicate out new schema.