Connector publishing automation (#197)

# What

This PR is the first PR towards adding automating the publishing of a
new connector version. The scope of this PR is limited to the following
and further changes will be made in future PRs. More on the automation
process
[here](https://docs.google.com/document/d/1QL37EkCQI-Ze1spNhUllher4AYcaibqq8T3Na1cOKvs/edit#heading=h.bwydohmrjusx)

The scope of the PR is limited to the following:

1. On opening a new PR, any new connector versions added to the
`registry` folder according to the format specified in the doc will
**only** be deployed to staging. This is basically to test the whole
worklow. After that, we will enable the publishing process to prod after
a PR is merged to the `main` branch.
2. Only new connector versions added are published automatically
(creating a new folder under the `registry/<connector_name>/releases`),
changes to an existing version are not covered (I'm not sure if this is
useful to do anyways, since a release that has already been made never
changes).
3. The `README.md` and `logo.png` changes will be made in a separate PR.


# How does the automation work?

1. Whenever any PR is opened against the `main` branch, we get the list
of files that have been added, modified and deleted. In this PR, we only
use the list of added files.
2. We get the name of the connector and the version from the file path.
For example: if the `ndc-postgres` releases `v2.0.0`, then one of the
added files would be
`registry/postgres/releases/v2.0.0/connector-packaging.json` , the name
and version of the connector are parsed from the file path.
3. Parse the `connector-packaging.json` file to get the URI of the
connector metadata.
4. Download the file from the URL obtained in step 3.
5. Upload the file to GCP registry and also parse the file.
6. Once all uploads are done, then insert rows into the connector
versions table in the connector registry DB.
7. Connector published! 🎉 . You should be able to access the added
connector version using the staging CLI.

# Future work

1. Create an RFC about the file structure for connector version releases
and ask the connector authors to make these changes. Once the connector
authors make the changes, the connector versions will be deployed to the
staging, this will be a good opportunity to test out the automation
workflow.
2. Once all the connectors are updated in the `registry` folder
according to the proposed structure in the RFC, we can enable the
workflow in prod after a PR is merged to the `main` branch.
3. Handle changes made in the `README.md` and the `logo.png`.

---------

Co-authored-by: Lyndon Maydwell <[email protected]>

.github/workflows/registry-updates.yaml

@@ -0,0 +1,61 @@
+name: Update Hub DB from GH Registry
+
+on:
+  pull_request:
+    branches:
+      - main
+    types: [opened, synchronize, reopened]
+    paths:
+      - registry/**
+
+jobs:
+  update_registry_db:
+    runs-on: ubuntu-latest
+    environment: staging
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+        with:
+          fetch_depth: 1
+
+      - name: Get all connector version package changes
+        id: connector-version-changed-files
+        uses: tj-actions/changed-files@v44
+        with:
+          json: true
+          escape_json: false
+          files: |
+            registry/**
+
+
+      - name: Print out all the changed filse
+        env:
+          ADDED_FILES: ${{ steps.connector-version-changed-files.outputs.added_files }}
+          MODIFIED_FILES: ${{ steps.connector-version-changed-files.outputs.modified_files }}
+          DELETED_FILES: ${{ steps.connector-version-changed-files.outputs.deleted_files }}
+        run: |
+          echo "{\"added_files\": $ADDED_FILES, \"modified_files\": $MODIFIED_FILES, \"deleted_files\": $DELETED_FILES}" > changed_files.json
+
+      - name: List changed files
+        id: list_files
+        run: |
+          cat changed_files.json
+
+      - name: Setup Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.21.x
+
+      - name: Run registry automation program
+        env:
+          CHANGED_FILES_PATH: "changed_files.json"
+          PUBLICATION_ENV: "staging"
+          CONNECTOR_REGISTRY_GQL_URL: ${{ secrets.CONNECTOR_REGISTRY_GQL_URL }}
+          GCP_BUCKET_NAME: dev-connector-platform-registry
+          GCP_SERVICE_ACCOUNT_DETAILS: ${{ secrets.GCP_SERVICE_ACCOUNT_DETAILS }}
+          CONNECTOR_PUBLICATION_KEY: ${{ secrets.CONNECTOR_PUBLICATION_KEY }}
+        run: |
+          mv changed_files.json registry-automation/changed_files.json
+          cd registry-automation
+          go run main.go ci

.gitignore

@@ -10,3 +10,4 @@
 
 # testing
 /tmp/empty
+registry-automation/extracted_tgz

README.md

@@ -40,3 +40,4 @@ implementation][NDC reference].
 
 [NDC specification]: http://hasura.github.io/ndc-spec/
 [NDC reference]: https://github.com/hasura/ndc-spec/tree/main/ndc-reference
+