Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector publishing automation #197

Merged
merged 121 commits into from
Aug 13, 2024
Merged

Conversation

codingkarthik
Copy link
Collaborator

@codingkarthik codingkarthik commented Jul 11, 2024

What

This PR is the first PR towards adding automating the publishing of a new connector version. The scope of this PR is limited to the following and further changes will be made in future PRs. More on the automation process here

The scope of the PR is limited to the following:

  1. On opening a new PR, any new connector versions added to the registry folder according to the format specified in the doc will only be deployed to staging. This is basically to test the whole worklow. After that, we will enable the publishing process to prod after a PR is merged to the main branch.
  2. Only new connector versions added are published automatically (creating a new folder under the registry/<connector_name>/releases), changes to an existing version are not covered (I'm not sure if this is useful to do anyways, since a release that has already been made never changes).
  3. The README.md and logo.png changes will be made in a separate PR.

How does the automation work?

  1. Whenever any PR is opened against the main branch, we get the list of files that have been added, modified and deleted. In this PR, we only use the list of added files.
  2. We get the name of the connector and the version from the file path. For example: if the ndc-postgres releases v2.0.0, then one of the added files would be registry/postgres/releases/v2.0.0/connector-packaging.json , the name and version of the connector are parsed from the file path.
  3. Parse the connector-packaging.json file to get the URI of the connector metadata.
  4. Download the file from the URL obtained in step 3.
  5. Upload the file to GCP registry and also parse the file.
  6. Once all uploads are done, then insert rows into the connector versions table in the connector registry DB.
  7. Connector published! 🎉 . You should be able to access the added connector version using the staging CLI.

Future work

  1. Create an RFC about the file structure for connector version releases and ask the connector authors to make these changes. Once the connector authors make the changes, the connector versions will be deployed to the staging, this will be a good opportunity to test out the automation workflow.
  2. Once all the connectors are updated in the registry folder according to the proposed structure in the RFC, we can enable the workflow in prod after a PR is merged to the main branch.
  3. Handle changes made in the README.md and the logo.png.

@@ -0,0 +1,104 @@
# Connector registry Github packaging

This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feel free to remove "Work-In-Progress"

@codingkarthik codingkarthik merged commit 8b02282 into main Aug 13, 2024
@codingkarthik codingkarthik deleted the kc/test-registry-automation branch August 13, 2024 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants