Scrape geoportal for shn, add SHN flag on stops table #3529

charlie-costanzo · 2024-11-05T17:53:17Z

Description

As requested in #3397, this PR creates:

An airflow operator to scrape the state geoportal for the state highway network geometry, clean the data and save it to a GCS bucket
A dag task for that operator
A dag task for creating a new external table for the SHN geometry
New source, staging, and docs in dbt for this data
- Geometry operations in the dim_stops_latest table and a new flag column on the table to indicate whether it exists on the state highway network

Resolves #3397

Type of change

New feature

How has this been tested?

locally using airflow/dbt

Post-merge follow-ups

Actions required (specified below)
GCS bucket needs to be added to Composer variables, new airflow operator needs to be turned on

github-actions · 2024-11-12T16:50:03Z

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__breakdowns

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__breakdowns_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_by_capital_use

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_for_existing_service

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__capital_expenses_for_expansion_of_service

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__employees_by_mode_and_employee_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__fuel_and_energy

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__fuel_and_energy_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_by_expense_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_directly_generated

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_federal

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_local

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_state

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__funding_sources_taxes_levied_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__maintenance_facilities

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__maintenance_facilities_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__metrics

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_function

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_function_and_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__operating_expenses_by_type_and_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__service_by_mode_and_time_period

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__stations_and_facilities_by_agency_and_facility_type

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__stations_by_mode_and_age

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_by_agency

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_by_mode

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__track_and_roadway_guideway_age_distribution

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__vehicles_age_distribution

calitp_warehouse.mart.ntd_fct_annual.fct_ntd_annual_data__vehicles_type_count_by_agency

calitp_warehouse.staging.state_geoportal.stg_state_geoportal__state_highway_network_stops

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

mjumbewu

Ok, I know it looks like a lot of comments, but most of it pertains to the options on the geoportal operator. Overall, I really dig this work. Also note that I did not run any of the suggested changes, so 🙏 please make sure they make sense. Happy to hop on a call tomorrow to discuss.

airflow/dags/scrape_state_geoportal/state_highway_network.yml

mjumbewu · 2024-11-14T22:58:42Z

airflow/plugins/operators/scrape_state_geoportal.py

+    product: str
+    where: str
+    outFields: str
+    f: str


suggestion: We probably always want the format that comes from the API to be the same (e.g. GeoJSON) if this operator is going to convert it to JSON-L, so maybe we don't include an f option.

airflow/plugins/operators/scrape_state_geoportal.py

mjumbewu · 2024-11-14T23:39:10Z

airflow/plugins/operators/scrape_state_geoportal.py

+        if self.product == "state_highway_network":
+            # Select and rename columns
+            columns = {
+                "properties.Route": "Route",
+                "properties.County": "County",
+                "properties.District": "District",
+                "properties.RouteType": "RouteType",
+                "properties.Direction": "Direction",
+                "geometry.type": "type",
+                "geometry.coordinates": "coordinates",
+            }
+            df = df[list(columns.keys())].rename(columns=columns)


suggestion (non-blocking): I don't think this is something that needs to be addressed right now, but it seems like we could do this without hard-coding the product here. We're really just cleaning up column names -- removing properties. and geometry. prefixes that come from the GeoJSON.

yes very good point, refactored

mjumbewu · 2024-11-14T23:40:19Z

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

+
+stg_state_geoportal__state_highway_network_stops AS (
+SELECT *
+-- FROM `cal-itp-data-infra-staging.external_state_geoportal.stg_state_geoportal__state_highway_network_stops`


Suggested change

-- FROM `cal-itp-data-infra-staging.external_state_geoportal.stg_state_geoportal__state_highway_network_stops`

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

evansiroky · 2024-11-19T19:16:51Z

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

+    IF(stops_on_shn.stop_id IS NOT NULL, TRUE, FALSE) AS exists_in_dim_stops_latest
+FROM
+    dim_stops_latest
+LEFT JOIN
+    stops_on_shn
+ON
+    dim_stops_latest.stop_id = stops_on_shn.stop_id


Is joining on stop_id going to potentially cause some conflicts here? There may be two feeds that have the same stop_id that this could match on. Would a join on dim_stops_latest._gtfs_key be more appropriate?

Hey @evansiroky – you're completely right. I must have left this in here after some comparative testing, apologies! I just made the swap and am re-requesting your review

mjumbewu

Ok, just two more small notes from me, but otherwise I'm ready to approve! Lemme know if you have any questions.

mjumbewu · 2024-11-21T22:14:27Z

airflow/dags/scrape_state_geoportal/state_highway_network.yml

+where: "1=1"  # You can change this to filter data
+outFields: "*"  # Specify the fields to return
+f: "geojson"  # Format of the response


suggestion: These are no longer necessary with the new defaults, right? Should probably remove them (but verify that it still works with them removed).

removed and still ran successfully!

mjumbewu · 2024-11-21T22:37:44Z

airflow/plugins/operators/scrape_state_geoportal.py

+                params["resultOffset"] = offset
+
+                # Make the request
+                response = requests.get(validated_url, params=params)


question: What should happen if we get a non-200 response from the server? I would think we should raise an exception so that the task fails and has to retry. Does that seem right? You could add .raise_for_status() at the end of the line if so. See https://3.python-requests.org/user/quickstart/#response-status-codes

added! great suggestion

evansiroky

Approved pending your discussion with @mjumbewu.

…oordinates

…hen ready

Co-authored-by: Mjumbe Poe <[email protected]>

…uffer distance

charlie-costanzo self-assigned this Nov 5, 2024

charlie-costanzo requested review from evansiroky, vevetron and tiffanychu90 as code owners November 5, 2024 17:53

charlie-costanzo marked this pull request as draft November 5, 2024 18:00

charlie-costanzo force-pushed the shn-stops-attribute branch from 61a59c9 to 4a914c0 Compare November 12, 2024 17:37

mjumbewu self-requested a review November 12, 2024 20:41

charlie-costanzo force-pushed the shn-stops-attribute branch from 6278fde to c7384bc Compare November 13, 2024 16:35

charlie-costanzo mentioned this pull request Nov 13, 2024

airflow: PythonOperator to scrape state highway network stop locations #3485

Closed

6 tasks

mjumbewu reviewed Nov 14, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch 2 times, most recently from fd5e5d4 to 4365a6c Compare November 18, 2024 20:45

charlie-costanzo marked this pull request as ready for review November 18, 2024 20:45

charlie-costanzo added the data-pipeline-ingestion-and-modeling Ingesting, parsing and modeling data. Evan Siroky is product owner. label Nov 18, 2024

evansiroky reviewed Nov 19, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch from 4365a6c to 4e6ecbb Compare November 20, 2024 18:34

charlie-costanzo requested review from evansiroky and mjumbewu November 20, 2024 18:45

mjumbewu approved these changes Nov 21, 2024

View reviewed changes

evansiroky approved these changes Nov 22, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch 2 times, most recently from 65689f8 to ac1cd87 Compare November 22, 2024 20:50

charlie-costanzo added 7 commits November 22, 2024 17:48

scrape geoportal for shn, add flag on stops table

3b8ad97

successful initial scrape of geoportal endpoint

9c47850

functioning json format and external table creation, need to modify c…

f305060

…oordinates

shn external tables

61acfe9

add initial source, staging, and docs for state geoportal scrape

4bb445a

add flag for shn to dim_stops_latest

f6f2bd4

readd templating for buckets and remove hardcode schema

84574ce

charlie-costanzo and others added 13 commits November 22, 2024 17:48

refactor operator to be more agnostic for data inputs

f1b6e0b

add handling for state_highway_network table

f079625

fix comment

284ed97

remove local variables and comments to allow for a production merge w…

95fb806

…hen ready

use stop_id for calculations

f85538b

Apply suggestions from code review

0202bf2

Co-authored-by: Mjumbe Poe <[email protected]>

restore source table

a0a8ec3

remove hardcoded source, remove old comments, add comment decribing b…

7507c99

…uffer distance

remove hardcoded bucket

57eb225

refactor how we keep final columns, dynamically rename columns

3162f1b

restore _gtfs_key over stop_id in shn geo calculation

8c985ed

checking before refactor

a81b205

revisions based on Mjumbe's review

f0247fe

charlie-costanzo force-pushed the shn-stops-attribute branch from ac1cd87 to f0247fe Compare November 22, 2024 22:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scrape geoportal for shn, add SHN flag on stops table #3529

Scrape geoportal for shn, add SHN flag on stops table #3529

charlie-costanzo commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024 •

edited

Loading

mjumbewu left a comment

mjumbewu Nov 14, 2024

mjumbewu Nov 14, 2024

charlie-costanzo Nov 18, 2024

mjumbewu Nov 14, 2024

evansiroky Nov 19, 2024

charlie-costanzo Nov 20, 2024

mjumbewu left a comment

mjumbewu Nov 21, 2024

charlie-costanzo Nov 22, 2024

mjumbewu Nov 21, 2024

charlie-costanzo Nov 22, 2024

evansiroky left a comment •

edited

Loading

Scrape geoportal for shn, add SHN flag on stops table #3529

Are you sure you want to change the base?

Scrape geoportal for shn, add SHN flag on stops table #3529

Conversation

charlie-costanzo commented Nov 5, 2024 • edited Loading

Description

Type of change

How has this been tested?

Post-merge follow-ups

github-actions bot commented Nov 12, 2024 • edited Loading

Checks/potential follow-ups

New models 🌱

DAG

mjumbewu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjumbewu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evansiroky left a comment • edited Loading

Choose a reason for hiding this comment

charlie-costanzo commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024 •

edited

Loading

evansiroky left a comment •

edited

Loading