promote PLUTO along with ZTL if specified, use dev buckets #1176

fvankrieken · 2024-10-04T14:14:14Z

Lot of testing for a pretty simple feature! Ended up spending most of time working on kinks around my data setup in the dev bucket. Overall changes are simple enough to follow! Plus I fix some things in sequential commits, so just look at overall

Two main things

when we both build pluto as part of ztl dataloading action and promote ztl to draft in the same action, also promote pluto
make use of dev buckets an option for ztl dataloading, promote to draft, and publish actions

Ztl dataloading action runs

Dev buckets working for dataloading

Automated promotion to draft

Job promoting to publish in dev bucket here. Important note that I added logic to not create a github tag for dev runs of this. Below, 2024-10-01 was promoted from draft folder in dev bucket, not copied from edm-publsihing.

fvankrieken · 2024-10-07T21:22:47Z

Oops sorry - need to let some data finish being loaded to my dev bucket before I can rerun the the ztl dataloading job one more time

sf-dcp · 2024-10-08T22:37:37Z

I think there is an issue with the logging table when you run this code (found the issue by accident while investing the logging table)...

This is the latest run I see for today.

And here is the logging table for ztl (looks same for PLUTO):

It seems like build events with dev flag don't get logged while promote_to_draft do (not expected behavior). Not sure whether logging happens with the publish event, with dev flag set to true.

I don't think we want to log dev stuff; unless we indicate a presence of dev flag in the custom_fields column

fvankrieken · 2024-10-09T13:53:09Z

I think there is an issue with the logging table when you run this code (found the issue by accident while investing the logging table)...

This is the latest run I see for today.

And here is the logging table for ztl (looks same for PLUTO):

It seems like build events with dev flag don't get logged while promote_to_draft do (not expected behavior). Not sure whether logging happens with the publish event, with dev flag set to true.

I don't think we want to log dev stuff; unless we indicate a presence of dev flag in the custom_fields column

Ooh good catch. I'll look into that

fvankrieken · 2024-10-09T14:00:48Z

Do we want to be logging db-template events? I would lean no

Also, I deleted the rows I generated through this in the logging table

fixed publish action here

codecov · 2024-10-09T14:14:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.87%. Comparing base (f69651e) to head (8683822).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1176      +/-   ##
==========================================
+ Coverage   67.80%   67.87%   +0.06%     
==========================================
  Files         109      109              
  Lines        5746     5746              
  Branches      846      846              
==========================================
+ Hits         3896     3900       +4     
+ Misses       1721     1718       -3     
+ Partials      129      128       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fvankrieken · 2024-10-09T14:17:56Z

.github/workflows/promote_to_draft.yml

@@ -69,6 +69,7 @@ jobs:
      image: nycplanning/build-base:latest
    env:
      PUBLISHING_BUCKET: ${{ inputs.dev_bucket && format('de-dev-{0}', inputs.dev_bucket) || 'edm-publishing' }}
+      DEV_FLAG: ${{ inputs.dev_bucket && 'true' || 'false' }}


Long term this doesn't feel the most sustainable way of handling this, ideally we have a single value somewhere setting the environment and we have a script or something handle translating that to buckets, db connection strings (if we had different db envs), etc... but this is what we currently have in other places and I think it's simplest for now

maybe we could also add a condition, not necessarily here, to only run a job if one of the following conditions is true?

(DEV_FLAG = false and dev_bucket is an empty string)
or
(DEV_FLAG = true and dev_bucket is not empty)

You are already addressing the second condition in the DEV_FLAG

or maybe I'm tripping and we don't need it?

I would lean with no - it's just adding another thing that needs to be manually specified. Right now, the dev bucket is really the only flag we have, and I think it makes sense to set a dev flag this way. And I'd rather put effort into consolidating logic to generate the various variables related to some sort of overall env flag than to try to add logic to validate that they're consistent

sf-dcp · 2024-10-09T14:24:11Z

Do we want to be logging db-template events? I would lean no

Also, I deleted the rows I generated through this in the logging table

fixed publish action here

I think we originally decided to log db-template events. I'm personally indifferent. Perhaps I'm leaning towards not logging them because every triggered test logs db-template, resulting in many records in the logging table

sf-dcp · 2024-10-09T15:24:44Z

.github/workflows/publish.yml

@@ -55,7 +59,8 @@ jobs:
    container:
      image: nycplanning/build-base:latest
    env:
-      PUBLISHING_BUCKET: edm-publishing
+      PUBLISHING_BUCKET: ${{ inputs.dev_bucket && format('de-dev-{0}', inputs.dev_bucket) || 'edm-publishing' }}
+      DEV_FLAG: ${{ inputs.dev_bucket && 'true' || 'false' }}


same comment as for promote_to_draft.yml

sf-dcp · 2024-10-09T15:32:06Z

dcpy/utils/versions.py

@@ -121,7 +121,7 @@ def label(self) -> str:
                if self.patch == 0:
                    return self.date.strftime("%Y-%m-%d")
                else:
-                    return f"{self.date.strftime('%Y-%m')}.{self.patch}"
+                    return f"{self.date.strftime('%Y-%m-%d')}.{self.patch}"


sf-dcp

LGTM! Thanks for fixing a silly error with patching.

I tripped on GH syntax around setting up a dev flag. I thought dev flag is a user input variable in addition to the bucket

fvankrieken force-pushed the fvk-patch branch 6 times, most recently from 361d9ca to f30a03b Compare October 7, 2024 21:07

fvankrieken assigned damonmcc, alexrichey and sf-dcp Oct 7, 2024

fvankrieken force-pushed the fvk-patch branch from f30a03b to dd895b1 Compare October 7, 2024 21:18

fvankrieken marked this pull request as draft October 8, 2024 13:51

fvankrieken unassigned damonmcc, alexrichey and sf-dcp Oct 8, 2024

fvankrieken force-pushed the fvk-patch branch from dd895b1 to 86e87fb Compare October 8, 2024 14:19

fvankrieken marked this pull request as ready for review October 8, 2024 19:27

fvankrieken assigned damonmcc, alexrichey and sf-dcp Oct 8, 2024

fvankrieken added 7 commits October 9, 2024 09:56

promote any built dataset in ztl action

80961a8

use dev buckets in ztl dataloading, promote

92b8fa7

fix broken function

3463579

appease mypy

141d242

allow publishing in dev bucket, clean up input descriptions

3493aa8

oops - fix dependencies for promote jobs

71a5d41

specify DEV_FLAG in promote/publish actions

d3a4fa8

fvankrieken force-pushed the fvk-patch branch from 3e84327 to d3a4fa8 Compare October 9, 2024 14:07

fix date patching bug

8683822

fvankrieken requested a review from sf-dcp October 9, 2024 14:13

fvankrieken commented Oct 9, 2024

View reviewed changes

sf-dcp reviewed Oct 9, 2024

View reviewed changes

sf-dcp approved these changes Oct 9, 2024

View reviewed changes

fvankrieken merged commit 3f3eb1a into main Oct 9, 2024
21 checks passed

fvankrieken deleted the fvk-patch branch October 9, 2024 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

promote PLUTO along with ZTL if specified, use dev buckets #1176

promote PLUTO along with ZTL if specified, use dev buckets #1176

fvankrieken commented Oct 4, 2024 •

edited

Loading

fvankrieken commented Oct 7, 2024

sf-dcp commented Oct 8, 2024 •

edited

Loading

fvankrieken commented Oct 9, 2024

fvankrieken commented Oct 9, 2024 •

edited

Loading

codecov bot commented Oct 9, 2024 •

edited

Loading

fvankrieken Oct 9, 2024

sf-dcp Oct 9, 2024

sf-dcp Oct 9, 2024

sf-dcp Oct 9, 2024

fvankrieken Oct 9, 2024

sf-dcp commented Oct 9, 2024

sf-dcp Oct 9, 2024

sf-dcp Oct 9, 2024

sf-dcp left a comment

promote PLUTO along with ZTL if specified, use dev buckets #1176

promote PLUTO along with ZTL if specified, use dev buckets #1176

Conversation

fvankrieken commented Oct 4, 2024 • edited Loading

fvankrieken commented Oct 7, 2024

sf-dcp commented Oct 8, 2024 • edited Loading

fvankrieken commented Oct 9, 2024

fvankrieken commented Oct 9, 2024 • edited Loading

codecov bot commented Oct 9, 2024 • edited Loading

Codecov Report

fvankrieken Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp Oct 9, 2024

Choose a reason for hiding this comment

fvankrieken Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp commented Oct 9, 2024

sf-dcp Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp Oct 9, 2024

Choose a reason for hiding this comment

sf-dcp left a comment

Choose a reason for hiding this comment

fvankrieken commented Oct 4, 2024 •

edited

Loading

sf-dcp commented Oct 8, 2024 •

edited

Loading

fvankrieken commented Oct 9, 2024 •

edited

Loading

codecov bot commented Oct 9, 2024 •

edited

Loading