feat: feeds operations API function #838

davidgamez · 2024-11-28T14:34:18Z

Summary:

This function introduces the Operations API. This API is intended to be used by MobilityData internal team. The endpoints are protected by Google Oauth2.

Expected behavior:

The endpoints for updating GTFS and GTFS-RT are implemented.

Testing tips:

Via retool

Go to https://mobilitydata.retool.com/
Select development environment in the bottom left corner of the application
Open Feeds Dashboard application
List feeds and re-authenticate if necessary
Select a feed and open the feed screen
Make changes to the feed and save
Revert feed update changes so integration tests pointing to DEV are not affected

Via curl:
Replace the data with the desired values and populate the access token:

curl --request PUT \
  --url https://api-dev.mobilitydatabase.org//v1/operations/feeds/gtfs \
  --header 'Content-Type: ' \
--header 'Authorization: Bearer <YOUR_ACCESS_TOKEN_HERE> 
  --data '{
  "id": "mdb-1",
  "data_type": "gtfs",
  "status": "active",
  "created_at": "2023-07-10T22:06:00Z",
  "external_ids": [
    {
      "external_id": "10",
      "source": "mdb"
    }
  ],
  "provider": "Los Angeles Department of Transportation (LADOT, DASH, Commuter Express)",
  "feed_name": "Bus",
  "note": "A note to clarify complex use cases for consumers.",
  "feed_contact_email": "[email protected]",
  "source_info": {
    "producer_url": "https://ladotbus.com/gtfs",
    "authentication_type": 1,
    "authentication_info_url": "https://apidevelopers.ladottransit.com",
    "api_key_parameter_name": "Ocp-Apim-Subscription-Key",
    "license_url": "https://www.ladottransit.com/dla.html"
  },
  "redirects": [
    {
      "target_id": "mdb-100",
      "comment": "Redirected because of a change of URL."
    }
  ]
}'

From our AI friend

This pull request includes a variety of changes to the API deployment workflows, the feeds API implementation, and the documentation. The most important changes include the addition of OAuth2 client ID handling, the generation and upload of Operations API code, and the inclusion of logging in the feeds API implementation.

Workflow Enhancements:

Added OPERATIONS_OAUTH2_CLIENT_ID_1PASSWORD to the environment variables in .github/workflows/api-deployer.yml and .github/workflows/api-dev.yml to handle OAuth2 client ID for the operations API. [1] [2]
Included steps to generate and upload Operations API code in .github/workflows/build-test.yml. [1] [2]

Feeds API Implementation:

Introduced a logger in api/src/feeds/impl/feeds_api_impl.py and added logging for user email restriction checks. [1] [2]
Updated methods in api/src/feeds/impl/feeds_api_impl.py to use the logger and refactored email restriction checks. [1] [2] [3] [4]

Documentation:

Added docs/OperationsAPI.yaml to document the Operations API, including endpoints for updating GTFS and GTFS-RT feeds.

Miscellaneous:

Updated api/src/middleware/request_context.py to correct the domain check in is_user_email_restricted function.
Modified the test for is_user_email_restricted in api/tests/unittest/middleware/test_request_context.py to reflect the corrected domain check.
Updated .flake8 and added .gcloudignore to exclude new directories and files from linting and Google Cloud uploads respectively. [1] [2]

Please make sure these boxes are checked before submitting your pull request - thanks!

Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
Add or update any needed documentation to the repo
Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
Linked all relevant issues
Include screenshot(s) showing how this pull request works and fixes the issue(s)

functions-python/helpers/database.py

davidgamez · 2024-12-04T17:00:46Z

api/src/middleware/request_context.py

-    unrestricted_domains = ["@mobilitydata.org"]
+    unrestricted_domains = ["mobilitydata.org"]
    return not email or not any(email.endswith(f"@{domain}") for domain in unrestricted_domains)


This fixes the unrestricted logic as it was adding two times the @ character

davidgamez · 2024-12-05T16:51:08Z

api/src/feeds/impl/search_api_impl.py

@@ -42,7 +42,7 @@ def add_search_query_filters(query, search_query, data_type, feed_id, status) ->
            or_(
                t_feedsearch.c.operational_status == None,  # noqa: E711
                t_feedsearch.c.operational_status != "wip",
-                is_user_email_restricted(),
+                not is_user_email_restricted(),


This logic gave access to wip feeds to restricted users.

davidgamez · 2024-12-05T16:52:22Z

api/src/middleware/request_context.py

@@ -108,8 +111,8 @@ def is_user_email_restricted() -> bool:
    Check if an email's domain is restricted (e.g., for WIP visibility).
    """
    request_context = get_request_context()
-    if not isinstance(request_context, RequestContext):


The request_context is actually a dictionary rather than a RequestContext class. FYI, the request_context is storage per request.

davidgamez · 2024-12-05T16:57:26Z

api/tests/unittest/middleware/test_request_context.py

@@ -54,45 +54,3 @@ def test_get_request_context(self):
        request_context = RequestContext(MagicMock())
        _request_context.set(request_context)
        self.assertEqual(request_context, get_request_context())
-
-    def test_is_user_email_restricted(self):


This test just needs more time to be fixed, so I'm creating an issue for it and de-scoping it from this PR. Follow up issue, #849

davidgamez · 2024-12-05T16:59:30Z

functions-python/helpers/database.py

 DB_REUSE_SESSION: Final[str] = "DB_REUSE_SESSION"
 lock = threading.Lock()
 global_session = None


+def configure_polymorphic_mappers():


These functions should be moved to a shared folder once we have the ability to share code between API and Python functions. Pending completion of #779 and #826

This reverts commit eeb2c7b.

qcdyx

LGTM

functions-python/operations_api/src/feeds_operations/impl/feeds_operations_impl.py

jcpitre · 2024-12-06T03:36:16Z

docs/OperationsAPI.yaml

+          description: A comment explaining the redirect.
+          type: string
+          example: Redirected because of a change of URL.
+    BasicFeed:


Could it refer to the BasicFeed in DatabaseCatalogAPI.yml?
Maybe it could be a future improvement

It will be tricky to do an import from DatabaseCatalogAPI.ym as it will impact the development process and the generator. However, a good improvement when we implement the getFeeds endpoint, add a BasicFeed entity to this schema.

.github/workflows/api-deployer.yml

jcpitre

It's just too much to do a deep review in a reasonable amount of time.
I did part of it, but for the rest I will just trust your changes based on your track record :-)

Co-authored-by: jcpitre <[email protected]>

davidgamez added 6 commits November 28, 2024 08:10

add feeds operations api function and deploy script

2ab431f

Merge branch 'main' into feat/operations_api

7fec9f3

increase test coverage

0d32e82

add missing api generation script call

0c99b8d

fix local test imports

6614855

open api schema cleanup

662b3b4

jcpitre reviewed Dec 2, 2024

View reviewed changes

functions-python/helpers/database.py Outdated Show resolved Hide resolved

davidgamez added 18 commits December 2, 2024 15:50

add operational status temporary support

c016098

Merge branch 'main' into feat/operations_api

4ad0194

add unit tests

78dfb30

add feed api endpoint unit tests

ab808f7

add infra code

510099e

fix ci failing test

35941ed

add missing configuration

88c2c63

add missing configuration

8afc2df

fix ingress settings

53d54a2

add missing generated code

7c4935f

fix operations generated folder reference

8b1660c

update oauth2 client id from 1password

2c89a5b

fix var name

5bed790

add update gtfs_rt feed

5656197

add update gtfs-rt fixes and unit test coverage

3fd13f4

allow operations api to run unauthenticated

a28edc7

Fix GTFS-RT update endpoint when static reference is changed

d125ddd

fix email restricted function

dbb0221

davidgamez commented Dec 4, 2024

View reviewed changes

add logging

f308820

davidgamez marked this pull request as ready for review December 4, 2024 18:33

davidgamez added 2 commits December 4, 2024 14:03

fix request context email verification

fef4bcd

add missing variables

3a45595

davidgamez added 5 commits December 4, 2024 18:17

fix materialized view refresh

a2cb3f9

waiting for the view to be refreshed

b440494

revert materialized view changes

839f1e5

fix search wip filter

6a49786

unifying oauth2 creds

3ffaeae

davidgamez commented Dec 5, 2024

View reviewed changes

davidgamez mentioned this pull request Dec 5, 2024

Add back test_is_user_email_restricted #849

Open

davidgamez commented Dec 5, 2024

View reviewed changes

davidgamez added 3 commits December 5, 2024 12:10

adding login

eeb2c7b

Revert "adding login"

6db8ae4

This reverts commit eeb2c7b.

fix upper case issue for entity type

6d9094f

qcdyx approved these changes Dec 5, 2024

View reviewed changes

functions-python/operations_api/src/feeds_operations/impl/feeds_operations_impl.py Show resolved Hide resolved

davidgamez added 2 commits December 5, 2024 21:11

fix operational status update

d0ee801

fix failing test

0c17397

jcpitre reviewed Dec 6, 2024

View reviewed changes

.github/workflows/api-deployer.yml Outdated Show resolved Hide resolved

jcpitre approved these changes Dec 6, 2024

View reviewed changes

Update .github/workflows/api-deployer.yml

9ce3f6c

Co-authored-by: jcpitre <[email protected]>

davidgamez merged commit dc5700c into main Dec 6, 2024
3 checks passed

davidgamez deleted the feat/operations_api branch December 6, 2024 17:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: feeds operations API function #838

feat: feeds operations API function #838

davidgamez commented Nov 28, 2024 •

edited

Loading

davidgamez Dec 4, 2024

davidgamez Dec 5, 2024

davidgamez Dec 5, 2024

davidgamez Dec 5, 2024

davidgamez Dec 5, 2024

qcdyx left a comment

jcpitre Dec 6, 2024 •

edited

Loading

davidgamez Dec 6, 2024

jcpitre left a comment •

edited

Loading

feat: feeds operations API function #838

feat: feeds operations API function #838

Conversation

davidgamez commented Nov 28, 2024 • edited Loading

Summary:

Expected behavior:

Testing tips:

From our AI friend

Workflow Enhancements:

Feeds API Implementation:

Documentation:

Miscellaneous:

davidgamez Dec 4, 2024

Choose a reason for hiding this comment

davidgamez Dec 5, 2024

Choose a reason for hiding this comment

davidgamez Dec 5, 2024

Choose a reason for hiding this comment

davidgamez Dec 5, 2024

Choose a reason for hiding this comment

davidgamez Dec 5, 2024

Choose a reason for hiding this comment

qcdyx left a comment

Choose a reason for hiding this comment

jcpitre Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

davidgamez Dec 6, 2024

Choose a reason for hiding this comment

jcpitre left a comment • edited Loading

Choose a reason for hiding this comment

davidgamez commented Nov 28, 2024 •

edited

Loading

jcpitre Dec 6, 2024 •

edited

Loading

jcpitre left a comment •

edited

Loading