(somewhat) unify OTP and Valhalla api behind "travelmux" service (#326)

Much of the API surface of Valhalla (bike/ped/walk directions) and OTP (transit directions) is very similar, but it's different enough that it requires redundant client implementations - e.g. drawing the routes on the maps is different because the Valhalla and OTP have different conventions for geometry among other things. I initially thought it'd be "real easy" to completely unify the APIs, but there is a lot of information for transit that just doesn't have a great corollary in the non-transit world. Rather than add a bunch of fields that are blank half the time depending on whether you are looking for transit vs. non-transit directions, for now I'm simply including the entirety of the original otp or valhalla request in a subfield for the data which wasn't easy to unify. --- * [transit] version api * DO NOT MERGE: OTP on local port 9001 * WIP: Valhalla integration into transitmux Currently this returns a "one of" response, but it did allow me to plumb through all the configuration we need to talk between OTP and Valhalla. And I did unify the "query" api - though it is limited to the subset of features we currently use. The big remaining thing will be unifying the output so that a transitmux client can render trips from opentripplanner or valhalla with more or less the same code. * [www] nbd: better typing for valhalla client * [www] use transitmux /v2/plan in preparation for unifying valhalla api * [www] nbd: dedupe decode polyline code. I sort of regret it * make raw fields private * WIP: unifying API * wip: unifying clients Currently only `duration` is properly unified but much of the plumbing is in place to gradually merge more fields in. TODO: Basically move the fields in [www] TravelmuxPlan one by one from inner to a locally stored state, updating transitmux as necessary to keep things working. * DO NOT MERGE: dev setup * unify trip.mode * transit displays proper units for walk length * [www] respect units * unify trip overall distance API * unit test against stubbed response * start to unify Legs * unify leg geometry * update deps * unify `start` * Unify mode and line style rendering * keep transit and route specific stuff in their respective Itinerary/Route rather than try to unify everything * handle units Valhalla lets you pick miles/kilometers OTP only ever returns meters At first I thought it would be smart to just always request meters, but then realized that valhalla embeds distance calculations into the localized narrative turn by turn directions, so we can't simply convert it on the front end. Maybe a better approach would be to transform the OTP output to respect the requested units - at least OTP doesn't produce any narrative directions (that we're using anyway). In any case that would be insufficient at this point because we actually still use the raw OTP/Valhalla response for rendering some of the Transit vs. Non-Transit UI. * Cleanup * moving code around * rename transitmux -> travelmux * [docker] fixup terrain.mbtiles in transit compose * travelmux vis a vis docker compose * fixup! rename transitmux -> travelmux * mark integration tests as such * dedupe code * deduping * deduping * sort transit itinerary by arrival time * minimize diff * fixup! rename transitmux -> travelmux * k8s vis a vis travelmux * wip: server side bounds * shoveling code around * fixup! wip: server side bounds * DO NOT MERGE: dev setup * fixup! fixup! wip: server side bounds * rename back to Trip * no integration test setup yet * rename modules since they no longer contain clients * undo some dev changes * hitting this assert - fix it up. * fixup githubworkflow - i disabled features for the wrong crate =/ * fix typo * Surface trip planner errors Before this commit, we weren't surfacing any user visible indication when an error occurs while fetching trips from the OTP or Valhalla API. At some point long ago, we surfaced specific error cases "outside of transit region" but that was broken before travelmux. * simplify and test decodePolyline * remove redundant code * round out method even though we dont use these units... seem like a footgun otherwise * return error rather than panic
headwaymaps · Mar 5, 2024 · 454985b · 454985b
1 parent 3fa01b8
commit 454985b
Show file tree

Hide file tree

Showing 73 changed files with 8,757 additions and 1,104 deletions.
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
@@ -35,10 +35,10 @@ jobs:
       - run: yarn format-check
       - run: yarn tsc
       - run: yarn test
-  transitmux_checks:
+  travelmux_checks:
     defaults:
       run:
-        working-directory: services/transitmux
+        working-directory: services/travelmux
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
@@ -52,11 +52,11 @@ jobs:
             ~/.cargo/registry/index/
             ~/.cargo/registry/cache/
             ~/.cargo/git/db/
-            services/transitmux/target/
+            services/travelmux/target/
           key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
       - run: cargo fmt --all --check
       - run: cargo clippy --all-features
-      - run: cargo test --all-features
+      - run: cargo test --all-targets
       - run: cargo build --release
   gtfout:
     defaults:
@@ -75,7 +75,7 @@ jobs:
             ~/.cargo/registry/index/
             ~/.cargo/registry/cache/
             ~/.cargo/git/db/
-            services/transitmux/target/
+            services/travelmux/target/
           key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
       - run: cargo fmt --all --check
       - run: cargo clippy --all-targets --all-features

diff --git a/Earthfile b/Earthfile
@@ -180,7 +180,7 @@ images:
     FROM debian:bookworm-slim
     ARG tags="dev"
     ARG branding
-    BUILD +transitmux-serve-image --tags=${tags}
+    BUILD +travelmux-serve-image --tags=${tags}
     BUILD +otp-serve-image --tags=${tags}
     BUILD +valhalla-serve-image --tags=${tags}
     BUILD +web-serve-image --tags=${tags} --branding=${branding}
@@ -585,47 +585,47 @@ otp-serve-image:
         SAVE IMAGE --push ghcr.io/headwaymaps/opentripplanner:${tag}
     END
 
-build-transitmux:
+build-travelmux:
     FROM rust:bookworm
 
-    WORKDIR transitmux
+    WORKDIR travelmux
 
     # This speeds up rebuilds of rust projectst by caching the prebuilt
     # dependencies in a separate docker layer. Without this, every change to
     # the source requires re-downloading and re-building all the project deps,
     # which takes a while.
-    COPY ./services/transitmux/Cargo.toml .
-    COPY ./services/transitmux/Cargo.lock .
+    COPY ./services/travelmux/Cargo.toml .
+    COPY ./services/travelmux/Cargo.lock .
     RUN mkdir src
     RUN echo 'fn main() { /* dummy main to get cargo to build deps */ }' > src/main.rs
     RUN cargo build --release
     RUN rm src/main.rs
 
-    COPY ./services/transitmux .
+    COPY ./services/travelmux .
     RUN cargo build --release
-    SAVE ARTIFACT target/release/transitmux-server /transitmux-server
+    SAVE ARTIFACT target/release/travelmux-server /travelmux-server
 
-transitmux-serve-image:
+travelmux-serve-image:
     FROM debian:bookworm-slim
 
     RUN apt-get update \
         && apt-get install -y --no-install-recommends libssl3 \
         && rm -rf /var/lib/apt/lists/*
 
-    RUN adduser --disabled-login transitmux --gecos ""
-    USER transitmux
+    RUN adduser --disabled-login travelmux --gecos ""
+    USER travelmux
 
-    WORKDIR /home/transitmux
-    COPY +build-transitmux/transitmux-server transitmux-server
+    WORKDIR /home/travelmux
+    COPY +build-travelmux/travelmux-server travelmux-server
 
     EXPOSE 8000
     ENV RUST_LOG=info
-    ENTRYPOINT ["/home/transitmux/transitmux-server"]
-    CMD ["http://opentripplanner:8000/otp/routers"]
+    ENTRYPOINT ["/home/travelmux/travelmux-server"]
+    CMD ["http://valhalla:8002", "http://opentripplanner:8000/otp/routers"]
 
     ARG --required tags
     FOR tag IN ${tags}
-        SAVE IMAGE --push ghcr.io/headwaymaps/transitmux:${tag}
+        SAVE IMAGE --push ghcr.io/headwaymaps/travelmux:${tag}
     END
 
 ##############################
@@ -825,8 +825,7 @@ web-serve-image:
     ENV HEADWAY_SHARED_VOL=/data
     ENV HEADWAY_HTTP_PORT=8080
     ENV HEADWAY_RESOLVER=127.0.0.11
-    ENV HEADWAY_TRANSITMUX_URL=http://transitmux:8000
-    ENV HEADWAY_VALHALLA_URL=http://valhalla:8002
+    ENV HEADWAY_TRAVELMUX_URL=http://travelmux:8000
     ENV HEADWAY_TILESERVER_URL=http://tileserver:8000
     ENV HEADWAY_PELIAS_URL=http://pelias-api:8080
     # for escaping $ in nginx template

diff --git a/bin/_headway_version.sh b/bin/_headway_version.sh
@@ -25,6 +25,10 @@ export HEADWAY_DATA_TAG=0.7.0
 
 # # Schema change Log
 #
+# ## DATA v0.7.0, CONTAINER v0.9.0
+#
+# BREAKING CONTAINER: transitmux is now travelmux, and additionally fronts Valhalla, not just OTP
+#
 # ## DATA v0.7.0, CONTAINER v0.8.0
 #
 # BREAKING DATA: Updated Planetiler

diff --git a/bin/publish-images b/bin/publish-images
@@ -35,7 +35,7 @@ images=(
   pelias-init
   tileserver
   tileserver-init
-  transitmux
+  travelmux
   valhalla
   valhalla-init
 )

diff --git a/bin/tag-published-images b/bin/tag-published-images
@@ -22,7 +22,7 @@ NEW_TAG="$2"
 
 registry_root=ghcr.io/headwaymaps
 
-images=( headway headway-init opentripplanner opentripplanner-init pelias-init tileserver tileserver-init transitmux valhalla valhalla-init )
+images=( headway headway-init opentripplanner opentripplanner-init pelias-init tileserver tileserver-init travelmux valhalla valhalla-init )
 
 for image in "${images[@]}"; do
   docker pull "${registry_root}/${image}:${SRC_TAG}"

diff --git a/docker-compose-with-transit.yaml b/docker-compose-with-transit.yaml
@@ -6,8 +6,8 @@ services:
     environment:
       AREAMAP_ARTIFACT_DEST: /data/${HEADWAY_AREA}.mbtiles
       AREAMAP_ARTIFACT_SOURCE: /bootstrap/${HEADWAY_AREA}.mbtiles
-      TERRAIN_ARTIFACT_DEST_PATH: /data/terrain.mbtiles
-      TERRAIN_ARTIFACT_SOURCE_PATH: /bootstrap/terrain.mbtiles
+      TERRAIN_ARTIFACT_DEST: /data/terrain.mbtiles
+      TERRAIN_ARTIFACT_SOURCE: /bootstrap/terrain.mbtiles
       LANDCOVER_ARTIFACT_DEST: /data/landcover.mbtiles
       LANDCOVER_ARTIFACT_SOURCE: /bootstrap/landcover.mbtiles
     volumes:
@@ -46,17 +46,21 @@ services:
       opentripplanner-init:
         condition: service_completed_successfully
     networks:
-      - transit_backend
-  transitmux:
-    image: ghcr.io/headwaymaps/transitmux:latest
+      - otp_backend
+    # ports:
+    #   - "9001:8000"
+  travelmux:
+    image: ghcr.io/headwaymaps/travelmux:latest
     restart: always
     env_file: .env
     depends_on:
       opentripplanner:
         condition: service_healthy
     networks:
-      - transit_backend
-      - transit_frontend
+      - travel_frontend
+      - otp_backend
+      - valhalla_backend
+    command: ["http://valhalla:8002", "http://opentripplanner:8000/otp/routers"]
   valhalla-init:
     image: ghcr.io/headwaymaps/valhalla-init:latest
     env_file: .env
@@ -74,7 +78,7 @@ services:
     restart: always
     env_file: .env
     networks:
-      - valhalla_frontend
+      - valhalla_backend
     volumes:
       - "valhalla_data:/data/:ro"
     ulimits:
@@ -84,6 +88,8 @@ services:
     depends_on:
       valhalla-init:
         condition: service_completed_successfully
+    # ports:
+    #   - "9002:8002"
   frontend-init:
     image: ghcr.io/headwaymaps/headway-init:latest
     env_file: .env
@@ -98,21 +104,19 @@ services:
       HEADWAY_RESOLVER: 127.0.0.11
       HEADWAY_PELIAS_URL: http://pelias-api:4000
       HEADWAY_TILESERVER_URL: http://tileserver:8000
-      HEADWAY_TRANSITMUX_URL: http://transitmux:8000
-      HEADWAY_VALHALLA_URL: http://valhalla:8002
+      HEADWAY_TRAVELMUX_URL: http://travelmux:8000
     ports:
       - "8080:8080"
     networks:
       - pelias_frontend
       - tileserver_frontend
-      - transit_frontend
-      - valhalla_frontend
+      - travel_frontend
     volumes:
       - "frontend_data:/data/:ro"
     depends_on:
       - "pelias-api"
       - "tileserver"
-      - "transitmux"
+      - "travelmux"
       - "valhalla"
   pelias-config-init:
     image: ghcr.io/headwaymaps/pelias-init:latest
@@ -162,6 +166,8 @@ services:
         condition: service_healthy
       pelias-config-init:
         condition: service_completed_successfully
+    # ports:
+    #   - "4000:4000"
   pelias-placeholder:
     image: pelias/placeholder:master
     restart: always
@@ -205,9 +211,9 @@ networks:
   tileserver_frontend:
   pelias_frontend:
   pelias_backend:
-  valhalla_frontend:
-  transit_frontend:
-  transit_backend:
+  travel_frontend:
+  valhalla_backend:
+  otp_backend:
 volumes:
   pelias_config_data:
   pelias_placeholder_data:

diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -28,6 +28,14 @@ services:
       - "8000"
     networks:
       - tileserver_frontend
+  travelmux:
+    image: ghcr.io/headwaymaps/travelmux:latest
+    restart: always
+    env_file: .env
+    networks:
+      - travel_frontend
+      - valhalla_backend
+    command: ["http://valhalla:8002"]
   valhalla-init:
     image: ghcr.io/headwaymaps/valhalla-init:latest
     env_file: .env
@@ -45,7 +53,7 @@ services:
     restart: always
     env_file: .env
     networks:
-      - valhalla_frontend
+      - valhalla_backend
     volumes:
       - "valhalla_data:/data/:ro"
     ulimits:
@@ -55,6 +63,8 @@ services:
     depends_on:
       valhalla-init:
         condition: service_completed_successfully
+    # ports:
+    #   - "9002:8002"
   frontend-init:
     image: ghcr.io/headwaymaps/headway-init:latest
     env_file: .env
@@ -69,21 +79,19 @@ services:
       HEADWAY_RESOLVER: 127.0.0.11
       HEADWAY_PELIAS_URL: http://pelias-api:4000
       HEADWAY_TILESERVER_URL: http://tileserver:8000
-      HEADWAY_TRANSITMUX_URL: http://transitmux:8000
-      HEADWAY_VALHALLA_URL: http://valhalla:8002
+      HEADWAY_TRAVELMUX_URL: http://travelmux:8000
     ports:
       - "8080:8080"
     networks:
       - pelias_frontend
       - tileserver_frontend
-      - valhalla_frontend
+      - travel_frontend
     volumes:
       - "frontend_data:/data/:ro"
-    extra_hosts:
-      - "transitmux:0.0.0.0"
     depends_on:
-      - "tileserver"
       - "pelias-api"
+      - "tileserver"
+      - "travelmux"
       - "valhalla"
   pelias-config-init:
     image: ghcr.io/headwaymaps/pelias-init:latest
@@ -176,7 +184,8 @@ networks:
   tileserver_frontend:
   pelias_frontend:
   pelias_backend:
-  valhalla_frontend:
+  travel_frontend:
+  valhalla_backend:
 volumes:
   pelias_config_data:
   pelias_placeholder_data:

diff --git a/git-hooks/pre-commit b/git-hooks/pre-commit
@@ -15,10 +15,10 @@ set -e
   yarn lint &&
   yarn build)
 
-(cd services/transitmux &&
+(cd services/travelmux &&
     cargo fmt &&
     cargo clippy --all-features -- -D warnings &&
-    cargo test --all-features)
+    cargo test)
 
 (cd services/gtfs/gtfout &&
     cargo fmt --all &&

diff --git a/k8s/_template/frontend-deployment.yaml.tpl b/k8s/_template/frontend-deployment.yaml.tpl
@@ -83,8 +83,6 @@ spec:
               value: http://tileserver:8000
             - name: HEADWAY_PELIAS_URL
               value: http://pelias-api:4000
-            - name: HEADWAY_VALHALLA_URL
-              value: http://valhalla:8002
             - name: HEADWAY_PUBLIC_URL
               valueFrom:
                 configMapKeyRef:

diff --git a/k8s/_template/transitmux-deployment.yaml.tpl → k8s/_template/travelmux-deployment.yaml.tpl b/k8s/_template/transitmux-deployment.yaml.tpl → k8s/_template/travelmux-deployment.yaml.tpl
@@ -1,20 +1,20 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: transitmux
+  name: travelmux
 spec:
   selector:
     matchLabels:
-      app: transitmux
+      app: travelmux
   replicas: 1
   template:
     metadata:
       labels:
-        app: transitmux
+        app: travelmux
     spec:
       containers:
         - name: main
-          image: ghcr.io/headwaymaps/transitmux:${HEADWAY_CONTAINER_TAG}
+          image: ghcr.io/headwaymaps/travelmux:${HEADWAY_CONTAINER_TAG}
           imagePullPolicy: Always
           args: ${OTP_ENDPOINTS_JSON}
           ports:

diff --git a/k8s/configs/planet/transitmux-service.yaml → k8s/_template/travelmux-service.yaml.tpl b/k8s/configs/planet/transitmux-service.yaml → k8s/_template/travelmux-service.yaml.tpl
@@ -1,10 +1,10 @@
 apiVersion: v1
 kind: Service
 metadata:
-  name: transitmux
+  name: travelmux
 spec:
   selector:
-    app: transitmux
+    app: travelmux
   ports:
     - protocol: TCP
       name: service