From 8b022824031d4811f6acfa994f8cdfa8158f709d Mon Sep 17 00:00:00 2001 From: Karthikeyan Chinnakonda Date: Tue, 13 Aug 2024 12:51:09 +0530 Subject: [PATCH] Connector publishing automation (#197) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # What This PR is the first PR towards adding automating the publishing of a new connector version. The scope of this PR is limited to the following and further changes will be made in future PRs. More on the automation process [here](https://docs.google.com/document/d/1QL37EkCQI-Ze1spNhUllher4AYcaibqq8T3Na1cOKvs/edit#heading=h.bwydohmrjusx) The scope of the PR is limited to the following: 1. On opening a new PR, any new connector versions added to the `registry` folder according to the format specified in the doc will **only** be deployed to staging. This is basically to test the whole worklow. After that, we will enable the publishing process to prod after a PR is merged to the `main` branch. 2. Only new connector versions added are published automatically (creating a new folder under the `registry//releases`), changes to an existing version are not covered (I'm not sure if this is useful to do anyways, since a release that has already been made never changes). 3. The `README.md` and `logo.png` changes will be made in a separate PR. # How does the automation work? 1. Whenever any PR is opened against the `main` branch, we get the list of files that have been added, modified and deleted. In this PR, we only use the list of added files. 2. We get the name of the connector and the version from the file path. For example: if the `ndc-postgres` releases `v2.0.0`, then one of the added files would be `registry/postgres/releases/v2.0.0/connector-packaging.json` , the name and version of the connector are parsed from the file path. 3. Parse the `connector-packaging.json` file to get the URI of the connector metadata. 4. Download the file from the URL obtained in step 3. 5. Upload the file to GCP registry and also parse the file. 6. Once all uploads are done, then insert rows into the connector versions table in the connector registry DB. 7. Connector published! 🎉 . You should be able to access the added connector version using the staging CLI. # Future work 1. Create an RFC about the file structure for connector version releases and ask the connector authors to make these changes. Once the connector authors make the changes, the connector versions will be deployed to the staging, this will be a good opportunity to test out the automation workflow. 2. Once all the connectors are updated in the `registry` folder according to the proposed structure in the RFC, we can enable the workflow in prod after a PR is merged to the `main` branch. 3. Handle changes made in the `README.md` and the `logo.png`. --------- Co-authored-by: Lyndon Maydwell --- .github/workflows/registry-updates.yaml | 61 +++ .gitignore | 1 + LICENSE | 201 ------- README.md | 1 + registry-automation/cmd/ci.go | 525 +++++++++++++++++++ registry-automation/cmd/gcp.go | 46 ++ registry-automation/cmd/root.go | 29 + registry-automation/cmd/utils.go | 108 ++++ registry-automation/go.mod | 53 ++ registry-automation/go.sum | 204 +++++++ registry-automation/main.go | 7 + rfcs/0002-distribution-gh.md | 6 +- rfcs/0006-connector-publishing-automation.md | 105 ++++ 13 files changed, 1145 insertions(+), 202 deletions(-) create mode 100644 .github/workflows/registry-updates.yaml delete mode 100644 LICENSE create mode 100644 registry-automation/cmd/ci.go create mode 100644 registry-automation/cmd/gcp.go create mode 100644 registry-automation/cmd/root.go create mode 100644 registry-automation/cmd/utils.go create mode 100644 registry-automation/go.mod create mode 100644 registry-automation/go.sum create mode 100644 registry-automation/main.go create mode 100644 rfcs/0006-connector-publishing-automation.md diff --git a/.github/workflows/registry-updates.yaml b/.github/workflows/registry-updates.yaml new file mode 100644 index 00000000..56a6af14 --- /dev/null +++ b/.github/workflows/registry-updates.yaml @@ -0,0 +1,61 @@ +name: Update Hub DB from GH Registry + +on: + pull_request: + branches: + - main + types: [opened, synchronize, reopened] + paths: + - registry/** + +jobs: + update_registry_db: + runs-on: ubuntu-latest + environment: staging + + steps: + - name: Checkout repository + uses: actions/checkout@v2 + with: + fetch_depth: 1 + + - name: Get all connector version package changes + id: connector-version-changed-files + uses: tj-actions/changed-files@v44 + with: + json: true + escape_json: false + files: | + registry/** + + + - name: Print out all the changed filse + env: + ADDED_FILES: ${{ steps.connector-version-changed-files.outputs.added_files }} + MODIFIED_FILES: ${{ steps.connector-version-changed-files.outputs.modified_files }} + DELETED_FILES: ${{ steps.connector-version-changed-files.outputs.deleted_files }} + run: | + echo "{\"added_files\": $ADDED_FILES, \"modified_files\": $MODIFIED_FILES, \"deleted_files\": $DELETED_FILES}" > changed_files.json + + - name: List changed files + id: list_files + run: | + cat changed_files.json + + - name: Setup Go + uses: actions/setup-go@v4 + with: + go-version: 1.21.x + + - name: Run registry automation program + env: + CHANGED_FILES_PATH: "changed_files.json" + PUBLICATION_ENV: "staging" + CONNECTOR_REGISTRY_GQL_URL: ${{ secrets.CONNECTOR_REGISTRY_GQL_URL }} + GCP_BUCKET_NAME: dev-connector-platform-registry + GCP_SERVICE_ACCOUNT_DETAILS: ${{ secrets.GCP_SERVICE_ACCOUNT_DETAILS }} + CONNECTOR_PUBLICATION_KEY: ${{ secrets.CONNECTOR_PUBLICATION_KEY }} + run: | + mv changed_files.json registry-automation/changed_files.json + cd registry-automation + go run main.go ci diff --git a/.gitignore b/.gitignore index e15b2a27..0d856277 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,4 @@ # testing /tmp/empty +registry-automation/extracted_tgz diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 261eeb9e..00000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/README.md b/README.md index 8e798367..c617d1ef 100644 --- a/README.md +++ b/README.md @@ -40,3 +40,4 @@ implementation][NDC reference]. [NDC specification]: http://hasura.github.io/ndc-spec/ [NDC reference]: https://github.com/hasura/ndc-spec/tree/main/ndc-reference + diff --git a/registry-automation/cmd/ci.go b/registry-automation/cmd/ci.go new file mode 100644 index 00000000..5f701888 --- /dev/null +++ b/registry-automation/cmd/ci.go @@ -0,0 +1,525 @@ +package cmd + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log" + "os" + "regexp" + + "cloud.google.com/go/storage" + "github.com/machinebox/graphql" + "github.com/spf13/cobra" + "google.golang.org/api/option" + "gopkg.in/yaml.v2" +) + +// ciCmd represents the ci command +var ciCmd = &cobra.Command{ + Use: "ci", + Short: "Run the CI workflow for hub registry publication", + Run: runCI, +} + +type ChangedFiles struct { + Added []string `json:"added_files"` + Modified []string `json:"modified_files"` + Deleted []string `json:"deleted_files"` +} + +// ConnectorVersion represents a version of a connector, this type is +// used to insert a new version of a connector in the registry. +type ConnectorVersion struct { + // Namespace of the connector, e.g. "hasura" + Namespace string `json:"namespace"` + // Name of the connector, e.g. "mongodb" + Name string `json:"name"` + // Semantic version of the connector version, e.g. "v1.0.0" + Version string `json:"version"` + // Docker image of the connector version (optional) + // This field is only required if the connector version is of type `PrebuiltDockerImage` + Image *string `json:"image,omitempty"` + // URL to the connector's metadata + PackageDefinitionURL string `json:"package_definition_url"` + // Is the connector version multitenant? + IsMultitenant bool `json:"is_multitenant"` + // Type of the connector packaging `PrebuiltDockerImage`/`ManagedDockerBuild` + Type string `json:"type"` +} + +// Create a struct with the following fields: +// type string +// image *string (optional) +type ConnectionVersionMetadata struct { + Type string `yaml:"type"` + Image *string `yaml:"image,omitempty"` +} + +const ( + ManagedDockerBuild = "ManagedDockerBuild" + PrebuiltDockerImage = "PrebuiltDockerImage" +) + +// Make a struct with the fields expected in the command line arguments +type ConnectorRegistryArgs struct { + ChangedFilesPath string + PublicationEnv string + ConnectorRegistryGQLUrl string + ConnectorPublicationKey string + GCPServiceAccountDetails string + GCPBucketName string +} + +var ciCmdArgs ConnectorRegistryArgs + +func init() { + rootCmd.AddCommand(ciCmd) + + // Path for the changed files in the PR + var changedFilesPathEnv = os.Getenv("CHANGED_FILES_PATH") + ciCmd.PersistentFlags().StringVar(&ciCmdArgs.ChangedFilesPath, "changed-files-path", changedFilesPathEnv, "path to a line-separated list of changed files in the PR") + if changedFilesPathEnv == "" { + ciCmd.MarkPersistentFlagRequired("changed-files-path") + } + + // Publication environment + var publicationEnv = os.Getenv("PUBLICATION_ENV") + ciCmd.PersistentFlags().StringVar(&ciCmdArgs.PublicationEnv, "publication-env", publicationEnv, "publication environment (staging/prod). Default: staging") + // default publicationEnv to "staging" + if publicationEnv == "" { + ciCmd.PersistentFlags().Set("publication-env", "staging") + } + +} + +func buildContext() { + // Connector registry Hasura GraphQL URL + registryGQLURL := os.Getenv("CONNECTOR_REGISTRY_GQL_URL") + if registryGQLURL == "" { + log.Fatalf("CONNECTOR_REGISTRY_GQL_URL is not set") + } else { + ciCmdArgs.ConnectorRegistryGQLUrl = registryGQLURL + } + + // Connector publication key + connectorPublicationKey := os.Getenv("CONNECTOR_PUBLICATION_KEY") + if connectorPublicationKey == "" { + log.Fatalf("CONNECTOR_PUBLICATION_KEY is not set") + } else { + ciCmdArgs.ConnectorPublicationKey = connectorPublicationKey + } + + // GCP service account details + gcpServiceAccountDetails := os.Getenv("GCP_SERVICE_ACCOUNT_DETAILS") + if gcpServiceAccountDetails == "" { + log.Fatalf("GCP_SERVICE_ACCOUNT_DETAILS is not set") + } else { + ciCmdArgs.GCPServiceAccountDetails = gcpServiceAccountDetails + } + + // GCP bucket name + gcpBucketName := os.Getenv("GCP_BUCKET_NAME") + if gcpBucketName == "" { + log.Fatalf("GCP_BUCKET_NAME is not set") + } else { + ciCmdArgs.GCPBucketName = gcpBucketName + } +} + +// processChangedFiles processes the files in the PR and extracts the connector name and version +// This function checks for the following things: +// 1. If a new connector version is added, it adds the connector version to the `newlyAddedConnectorVersions` map. +// 2. If the logo file is modified, it adds the connector name and the path to the modified logo to the `modifiedLogos` map. +// 3. If the README file is modified, it adds the connector name and the path to the modified README to the `modifiedReadmes` map. +func processChangedFiles(changedFiles ChangedFiles) NewConnectorVersions { + + newlyAddedConnectorVersions := make(map[Connector]map[string]string) + + var connectorVersionPackageRegex = regexp.MustCompile(`^registry/([^/]+)/([^/]+)/releases/([^/]+)/connector-packaging\.json$`) + + files := append(changedFiles.Added, changedFiles.Modified...) + + for _, file := range files { + // Extract the connector name and version from the file path + if connectorVersionPackageRegex.MatchString(file) { + + matches := connectorVersionPackageRegex.FindStringSubmatch(file) + if len(matches) == 4 { + connectorNamespace := matches[1] + connectorName := matches[2] + connectorVersion := matches[3] + + connector := Connector{ + Name: connectorName, + Namespace: connectorNamespace, + } + + if _, exists := newlyAddedConnectorVersions[connector]; !exists { + newlyAddedConnectorVersions[connector] = make(map[string]string) + } + + newlyAddedConnectorVersions[connector][connectorVersion] = file + } + + } else { + fmt.Println("Skipping file: ", file) + } + } + + return newlyAddedConnectorVersions +} + +// runCI is the main function that runs the CI workflow +func runCI(cmd *cobra.Command, args []string) { + buildContext() + changedFilesContent, err := os.Open(ciCmdArgs.ChangedFilesPath) + if err != nil { + log.Fatalf("Failed to open the file: %v, err: %v", ciCmdArgs.ChangedFilesPath, err) + } + defer changedFilesContent.Close() + + client, err := storage.NewClient(context.Background(), option.WithCredentialsJSON([]byte(ciCmdArgs.GCPServiceAccountDetails))) + if err != nil { + log.Fatalf("Failed to create Google bucket client: %v", err) + } + defer client.Close() + + // Read the changed file's contents. This file contains all the changed files in the PR + changedFilesByteValue, err := io.ReadAll(changedFilesContent) + if err != nil { + log.Fatalf("Failed to read the changed files JSON file: %v", err) + } + + var changedFiles ChangedFiles + err = json.Unmarshal(changedFilesByteValue, &changedFiles) + if err != nil { + log.Fatalf("Failed to unmarshal the changed files content: %v", err) + + } + + // Collect the added or modified connectors + addedOrModifiedConnectorVersions := processChangedFiles(changedFiles) + // check if the map is empty + if len(addedOrModifiedConnectorVersions) == 0 { + fmt.Println("No connector versions found in the changed files.") + return + } else { + processNewlyAddedConnectorVersions(client, addedOrModifiedConnectorVersions) + } +} + +func processNewlyAddedConnectorVersions(client *storage.Client, newlyAddedConnectorVersions NewConnectorVersions) { + // Iterate over the added or modified connectors and upload the connector versions + var connectorVersions []ConnectorVersion + var uploadConnectorVersionErr error + encounteredError := false + + for connectorName, versions := range newlyAddedConnectorVersions { + for version, connectorVersionPath := range versions { + var connectorVersion ConnectorVersion + connectorVersion, uploadConnectorVersionErr = uploadConnectorVersionPackage(client, connectorName, version, connectorVersionPath) + + if uploadConnectorVersionErr != nil { + encounteredError = true + break + + } else { + connectorVersions = append(connectorVersions, connectorVersion) + } + + } + + if encounteredError { + // attempt to cleanup the uploaded connector versions + _ = cleanupUploadedConnectorVersions(client, connectorVersions) // ignore errors while cleaning up + // delete the uploaded connector versions from the registry + log.Fatalf("Failed to upload the connector version: %v", uploadConnectorVersionErr) + + } else { + fmt.Printf("Connector versions to be added to the registry: %+v\n", connectorVersions) + err := updateRegistryGQL(connectorVersions) + if err != nil { + // attempt to cleanup the uploaded connector versions + _ = cleanupUploadedConnectorVersions(client, connectorVersions) // ignore errors while cleaning up + log.Fatalf("Failed to update the registry: %v", err) + } + } + + fmt.Println("Successfully added connector versions to the registry.") + } +} + +func cleanupUploadedConnectorVersions(client *storage.Client, connectorVersions []ConnectorVersion) error { + // Iterate over the connector versions and delete the uploaded files + // from the google bucket + fmt.Println("Cleaning up the uploaded connector versions") + + for _, connectorVersion := range connectorVersions { + objectName := generateGCPObjectName(connectorVersion.Namespace, connectorVersion.Name, connectorVersion.Version) + err := deleteFile(client, ciCmdArgs.GCPBucketName, objectName) + if err != nil { + return err + } + } + return nil +} + +// Type that uniquely identifies a connector +type Connector struct { + Name string `json:"name"` + Namespace string `json:"namespace"` +} + +type NewConnectorVersions map[Connector]map[string]string + +// uploadConnectorVersionPackage uploads the connector version package to the registry +func uploadConnectorVersionPackage(client *storage.Client, connector Connector, version string, changedConnectorVersionPath string) (ConnectorVersion, error) { + + var connectorVersion ConnectorVersion + + // connector version's metadata, `registry/mongodb/releases/v1.0.0/connector-packaging.json` + connectorVersionPackagingInfo, err := readJSONFile[map[string]interface{}](changedConnectorVersionPath) // Read metadata file + if err != nil { + return connectorVersion, fmt.Errorf("failed to read the connector packaging file: %v", err) + } + // Fetch, parse, and reupload the TGZ + tgzUrl, ok := connectorVersionPackagingInfo["uri"].(string) + + // Check if the TGZ URL is valid + if !ok || tgzUrl == "" { + return connectorVersion, fmt.Errorf("invalid or undefined TGZ URL: %v", tgzUrl) + } + + connectorVersionMetadata, connectorMetadataTgzPath, err := getConnectorVersionMetadata(tgzUrl, connector, version) + if err != nil { + return connectorVersion, err + } + + uploadedTgzUrl, err := uploadConnectorVersionDefinition(client, connector.Name, connector.Namespace, version, connectorMetadataTgzPath) + if err != nil { + return connectorVersion, fmt.Errorf("failed to upload the connector version definition - connector: %v version:%v - err: %v", connector.Name, version, err) + } else { + // print success message with the name of the connector and the version + fmt.Printf("Successfully uploaded the connector version definition in google cloud registry for the connector: %v version: %v\n", connector.Name, version) + } + + // Build payload for registry upsert + return buildRegistryPayload(connector.Namespace, connector.Name, version, connectorVersionMetadata, uploadedTgzUrl) +} + +func uploadConnectorVersionDefinition(client *storage.Client, connectorNamespace, connectorName string, connectorVersion string, connectorMetadataTgzPath string) (string, error) { + bucketName := ciCmdArgs.GCPBucketName + objectName := generateGCPObjectName(connectorNamespace, connectorName, connectorVersion) + uploadedTgzUrl, err := uploadFile(client, bucketName, objectName, connectorMetadataTgzPath) + + if err != nil { + return "", err + } + return uploadedTgzUrl, nil +} + +// Downloads the TGZ File from the URL specified by `tgzUrl`, extracts the TGZ file and returns the content of the +// connector-definition.yaml present in the .hasura-connector folder. +func getConnectorVersionMetadata(tgzUrl string, connector Connector, connectorVersion string) (map[string]interface{}, string, error) { + var connectorVersionMetadata map[string]interface{} + tgzPath, err := getTempFilePath("extracted_tgz") + if err != nil { + return connectorVersionMetadata, "", fmt.Errorf("failed to get the temp file path: %v", err) + } + err = downloadFile(tgzUrl, tgzPath, map[string]string{}) + if err != nil { + return connectorVersionMetadata, "", fmt.Errorf("failed to download the connector version metadata file from the URL: %v - err: %v", tgzUrl, err) + } + + extractedTgzFolderPath := "extracted_tgz" + + if _, err := os.Stat(extractedTgzFolderPath); os.IsNotExist(err) { + err := os.Mkdir(extractedTgzFolderPath, 0755) + if err != nil { + return connectorVersionMetadata, "", fmt.Errorf("failed to read the connector version metadata file: %v", err) + } + } + + connectorVersionMetadataYamlFilePath, err := extractTarGz(tgzPath, extractedTgzFolderPath+"/"+connector.Namespace+"/"+connector.Name+"/"+connectorVersion) + if err != nil { + return connectorVersionMetadata, "", fmt.Errorf("failed to read the connector version metadata file: %v", err) + } else { + fmt.Println("Extracted metadata file at :", connectorVersionMetadataYamlFilePath) + } + + connectorVersionMetadata, err = readYAMLFile(connectorVersionMetadataYamlFilePath) + if err != nil { + return connectorVersionMetadata, "", fmt.Errorf("failed to read the connector version metadata file: %v", err) + } + return connectorVersionMetadata, tgzPath, nil +} + +// Write a function that accepts a file path to a YAML file and returns +// the contents of the file as a map[string]interface{}. +// readYAMLFile accepts a file path to a YAML file and returns the contents of the file as a map[string]interface{}. +func readYAMLFile(filePath string) (map[string]interface{}, error) { + // Open the file + file, err := os.Open(filePath) + if err != nil { + return nil, fmt.Errorf("failed to open file: %w", err) + } + defer file.Close() + + // Read the file contents + data, err := io.ReadAll(file) + if err != nil { + return nil, fmt.Errorf("failed to read file: %w", err) + } + + // Unmarshal the YAML contents into a map + var result map[string]interface{} + err = yaml.Unmarshal(data, &result) + if err != nil { + return nil, fmt.Errorf("failed to unmarshal YAML: %w", err) + } + + return result, nil +} + +func getConnectorNamespace(connectorMetadata map[string]interface{}) (string, error) { + connectorOverview, ok := connectorMetadata["overview"].(map[string]interface{}) + if !ok { + return "", fmt.Errorf("could not find connector overview in the connector's metadata") + } + connectorNamespace, ok := connectorOverview["namespace"].(string) + if !ok { + return "", fmt.Errorf("could not find the 'namespace' of the connector in the connector's overview in the connector's metadata.json") + } + return connectorNamespace, nil +} + +// struct to store the response of teh GetConnectorInfo query +type GetConnectorInfoResponse struct { + HubRegistryConnector []struct { + Name string `json:"name"` + MultitenantConnector *struct { + ID string `json:"id"` + } `json:"multitenant_connector"` + } `json:"hub_registry_connector"` +} + +func getConnectorInfoFromRegistry(connectorNamespace string, connectorName string) (GetConnectorInfoResponse, error) { + var respData GetConnectorInfoResponse + client := graphql.NewClient(ciCmdArgs.ConnectorRegistryGQLUrl) + ctx := context.Background() + + req := graphql.NewRequest(` +query GetConnectorInfo ($name: String!, $namespace: String!) { + hub_registry_connector(where: {_and: [{name: {_eq: $name}}, {namespace: {_eq: $namespace}}]}) { + name + multitenant_connector { + id + } + } +}`) + req.Var("name", connectorName) + req.Var("namespace", connectorNamespace) + + req.Header.Set("x-hasura-role", "connector_publishing_automation") + req.Header.Set("x-connector-publication-key", ciCmdArgs.ConnectorPublicationKey) + + // Execute the GraphQL query and check the response. + if err := client.Run(ctx, req, &respData); err != nil { + return respData, err + } else { + if len(respData.HubRegistryConnector) == 0 { + return respData, nil + } + } + + return respData, nil +} + +// buildRegistryPayload builds the payload for the registry upsert API +func buildRegistryPayload( + connectorNamespace string, + connectorName string, + version string, + connectorVersionMetadata map[string]interface{}, + uploadedConnectorDefinitionTgzUrl string, +) (ConnectorVersion, error) { + var connectorVersion ConnectorVersion + var connectorVersionDockerImage string + connectorVersionPackagingDefinition, ok := connectorVersionMetadata["packagingDefinition"].(map[interface{}]interface{}) + if !ok { + return connectorVersion, fmt.Errorf("could not find the 'packagingDefinition' of the connector %s version %s in the connector's metadata", connectorName, version) + } + connectorVersionPackagingType, ok := connectorVersionPackagingDefinition["type"].(string) + + if !ok && (connectorVersionPackagingType == ManagedDockerBuild || connectorVersionPackagingType == PrebuiltDockerImage) { + return connectorVersion, fmt.Errorf("invalid or undefined connector type: %v", connectorVersionPackagingDefinition) + } else if connectorVersionPackagingType == PrebuiltDockerImage { + connectorVersionDockerImage, ok = connectorVersionPackagingDefinition["dockerImage"].(string) + if !ok { + return connectorVersion, fmt.Errorf("could not find the 'dockerImage' of the PrebuiltDockerImage connector %s version %s in the connector's metadata", connectorName, version) + } + + } + + connectorInfo, err := getConnectorInfoFromRegistry(connectorNamespace, connectorName) + + if err != nil { + return connectorVersion, err + } + + // Check if the connector exists in the registry first + if len(connectorInfo.HubRegistryConnector) == 0 { + return connectorVersion, fmt.Errorf("Inserting a new connector is not supported yet") + } + + var connectorVersionType string + + if connectorVersionPackagingType == PrebuiltDockerImage { + // Note: The connector version type is set to `PreBuiltDockerImage` if the connector version is of type `PrebuiltDockerImage`, this is a HACK because this value might be removed in the future and we might not even need to insert new connector versions in the `hub_registry_connector_version` table. + connectorVersionType = "PreBuiltDockerImage" + } else { + connectorVersionType = ManagedDockerBuild + } + + connectorVersion = ConnectorVersion{ + Namespace: connectorNamespace, + Name: connectorName, + Version: version, + Image: &connectorVersionDockerImage, + PackageDefinitionURL: uploadedConnectorDefinitionTgzUrl, + IsMultitenant: connectorInfo.HubRegistryConnector[0].MultitenantConnector != nil, + Type: connectorVersionType, + } + + return connectorVersion, nil +} + +func updateRegistryGQL(payload []ConnectorVersion) error { + var respData map[string]interface{} + client := graphql.NewClient(ciCmdArgs.ConnectorRegistryGQLUrl) + ctx := context.Background() + + req := graphql.NewRequest(` +mutation InsertConnectorVersion($connectorVersion: [hub_registry_connector_version_insert_input!]!) { + insert_hub_registry_connector_version(objects: $connectorVersion, on_conflict: {constraint: connector_version_namespace_name_version_key, update_columns: [image, package_definition_url]}) { + affected_rows + returning { + id + } + } +}`) + // add the payload to the request + req.Var("connectorVersion", payload) + + req.Header.Set("x-hasura-role", "connector_publishing_automation") + req.Header.Set("x-connector-publication-key", ciCmdArgs.ConnectorPublicationKey) + + // Execute the GraphQL query and check the response. + if err := client.Run(ctx, req, &respData); err != nil { + return err + } + + return nil +} diff --git a/registry-automation/cmd/gcp.go b/registry-automation/cmd/gcp.go new file mode 100644 index 00000000..40d41d62 --- /dev/null +++ b/registry-automation/cmd/gcp.go @@ -0,0 +1,46 @@ +// Description: This file contains the functions to interact with Google Cloud Storage. +package cmd + +import ( + "cloud.google.com/go/storage" + "context" + "fmt" + "io" + "os" +) + +// deleteFile deletes a file from Google Cloud Storage +func deleteFile(client *storage.Client, bucketName, objectName string) error { + bucket := client.Bucket(bucketName) + object := bucket.Object(objectName) + + return object.Delete(context.Background()) +} + +// uploadFile uploads a file to Google Cloud Storage +// document this function with comments +func uploadFile(client *storage.Client, bucketName, objectName, filePath string) (string, error) { + bucket := client.Bucket(bucketName) + object := bucket.Object(objectName) + newCtx := context.Background() + wc := object.NewWriter(newCtx) + + file, err := os.Open(filePath) + if err != nil { + return "", fmt.Errorf("failed to open file: %v", err) + } + defer file.Close() + + if _, err := io.Copy(wc, file); err != nil { + return "", fmt.Errorf("failed to upload file: %w", err) + } + if err := wc.Close(); err != nil { + return "", fmt.Errorf("failed to close writer: %w", err) + } + + // Return the public URL of the uploaded object. + publicURL := fmt.Sprintf("https://storage.googleapis.com/%s/%s", bucketName, objectName) + + fmt.Printf("File %s uploaded to bucket %s as %s and is available at %s.\n", filePath, bucketName, objectName, publicURL) + return publicURL, nil +} diff --git a/registry-automation/cmd/root.go b/registry-automation/cmd/root.go new file mode 100644 index 00000000..9cf49a5d --- /dev/null +++ b/registry-automation/cmd/root.go @@ -0,0 +1,29 @@ +package cmd + +import ( + "os" + + "github.com/spf13/cobra" +) + +// rootCmd represents the base command when called without any subcommands +var rootCmd = &cobra.Command{ + Use: "registry-automation", + Short: "Commands associated with automation for the hub registry", +} + +// Execute adds all child commands to the root command and sets flags appropriately. +// This is called by main.main(). It only needs to happen once to the rootCmd. +func Execute() { + err := rootCmd.Execute() + if err != nil { + os.Exit(1) + } +} + +func init() { + // Here you will define your flags and configuration settings. + // Cobra supports persistent flags, which, if defined here, + // will be global for your application. + +} diff --git a/registry-automation/cmd/utils.go b/registry-automation/cmd/utils.go new file mode 100644 index 00000000..15357b07 --- /dev/null +++ b/registry-automation/cmd/utils.go @@ -0,0 +1,108 @@ +package cmd + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "os/exec" +) + +func generateGCPObjectName(namespace, connectorName, version string) string { + return fmt.Sprintf("packages/%s/%s/%s/package.tgz", namespace, connectorName, version) +} + +func downloadFile(sourceURL, destination string, headers map[string]string) error { + // Create a new HTTP client + client := &http.Client{} + + // Create a new GET request + req, err := http.NewRequest("GET", sourceURL, nil) + if err != nil { + return fmt.Errorf("error creating request: %v", err) + } + + // Add headers + for key, value := range headers { + req.Header.Set(key, value) + } + + // Send the request + resp, err := client.Do(req) + if err != nil { + return fmt.Errorf("error sending request: %v", err) + } + defer resp.Body.Close() + + // Create the destination file + outFile, err := os.Create(destination) + if err != nil { + return fmt.Errorf("error creating destination file: %v", err) + } + defer outFile.Close() + + // Write the response body to the file + _, err = io.Copy(outFile, resp.Body) + if err != nil { + return fmt.Errorf("error writing to file: %v", err) + } + return nil +} + +// Reads a JSON file and attempts to parse the content of the file +// into the type T. +// Note: The location is relative to the root of the repository +func readJSONFile[T any](location string) (T, error) { + // Read the file + var result T + fileBytes, err := os.ReadFile("../" + location) + if err != nil { + return result, fmt.Errorf("error reading file at location: %s %v", location, err) + } + + if err := json.Unmarshal(fileBytes, &result); err != nil { + return result, fmt.Errorf("error parsing JSON: %v", err) + } + + return result, nil +} + +// getTempFilePath generates a random file name in the specified directory. +func getTempFilePath(directory string) (string, error) { + // Ensure the directory exists + err := os.MkdirAll(directory, os.ModePerm) + if err != nil { + panic(fmt.Errorf("error creating directory: %v", err)) + } + + // Generate a random file name + tempFile, err := os.CreateTemp(directory, "connector-*.tar.gz") + if err != nil { + return "", fmt.Errorf("error creating temp file: %v", err) + } + defer tempFile.Close() + + return tempFile.Name(), nil + +} + +func extractTarGz(src, dest string) (string, error) { + // Create the destination directory + // Get the present working directory + pwd := os.Getenv("PWD") + filepath := pwd + "/" + dest + + if err := os.MkdirAll(filepath, 0755); err != nil { + return "", fmt.Errorf("error creating destination directory: %v", err) + } + // Run the tar command with the -xvzf options + cmd := exec.Command("tar", "-xvzf", src, "-C", dest) + + // Execute the command + if err := cmd.Run(); err != nil { + return "", fmt.Errorf("error extracting tar.gz file: %v", err) + } + + return fmt.Sprintf("%s/.hasura-connector/connector-metadata.yaml", filepath), nil +} diff --git a/registry-automation/go.mod b/registry-automation/go.mod new file mode 100644 index 00000000..59b7ef16 --- /dev/null +++ b/registry-automation/go.mod @@ -0,0 +1,53 @@ +module github.com/hasura/ndc-hub/registry-automation + +go 1.21.4 + +require github.com/spf13/cobra v1.8.0 + +require ( + cloud.google.com/go v0.115.0 // indirect + cloud.google.com/go/auth v0.7.1 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.3 // indirect + cloud.google.com/go/compute/metadata v0.5.0 // indirect + cloud.google.com/go/iam v1.1.11 // indirect + cloud.google.com/go/storage v1.43.0 // indirect + github.com/andybalholm/brotli v1.0.1 // indirect + github.com/felixge/httpsnoop v1.0.4 // indirect + github.com/go-logr/logr v1.4.2 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect + github.com/golang/protobuf v1.5.4 // indirect + github.com/golang/snappy v0.0.4 // indirect + github.com/google/s2a-go v0.1.7 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect + github.com/googleapis/gax-go/v2 v2.12.5 // indirect + github.com/inconshreveable/mousetrap v1.1.0 // indirect + github.com/klauspost/pgzip v1.2.5 // indirect + github.com/machinebox/graphql v0.2.2 // indirect + github.com/nwaples/rardecode v1.1.0 // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/spf13/pflag v1.0.5 // indirect + github.com/ulikunitz/xz v0.5.9 // indirect + github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect + go.opencensus.io v0.24.0 // indirect + go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 // indirect + go.opentelemetry.io/otel v1.28.0 // indirect + go.opentelemetry.io/otel/metric v1.28.0 // indirect + go.opentelemetry.io/otel/trace v1.28.0 // indirect + golang.org/x/crypto v0.25.0 // indirect + golang.org/x/net v0.27.0 // indirect + golang.org/x/oauth2 v0.21.0 // indirect + golang.org/x/sync v0.7.0 // indirect + golang.org/x/sys v0.22.0 // indirect + golang.org/x/text v0.16.0 // indirect + golang.org/x/time v0.5.0 // indirect + google.golang.org/api v0.188.0 // indirect + google.golang.org/genproto v0.0.0-20240711142825-46eb208f015d // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20240711142825-46eb208f015d // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20240711142825-46eb208f015d // indirect + google.golang.org/grpc v1.65.0 // indirect + google.golang.org/protobuf v1.34.2 // indirect + gopkg.in/yaml.v2 v2.4.0 +) diff --git a/registry-automation/go.sum b/registry-automation/go.sum new file mode 100644 index 00000000..5d0aac40 --- /dev/null +++ b/registry-automation/go.sum @@ -0,0 +1,204 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.115.0 h1:CnFSK6Xo3lDYRoBKEcAtia6VSC837/ZkJuRduSFnr14= +cloud.google.com/go v0.115.0/go.mod h1:8jIM5vVgoAEoiVxQ/O4BFTfHqulPZgs/ufEzMcFMdWU= +cloud.google.com/go/auth v0.7.1 h1:Iv1bbpzJ2OIg16m94XI9/tlzZZl3cdeR3nGVGj78N7s= +cloud.google.com/go/auth v0.7.1/go.mod h1:VEc4p5NNxycWQTMQEDQF0bd6aTMb6VgYDXEwiJJQAbs= +cloud.google.com/go/auth/oauth2adapt v0.2.3 h1:MlxF+Pd3OmSudg/b1yZ5lJwoXCEaeedAguodky1PcKI= +cloud.google.com/go/auth/oauth2adapt v0.2.3/go.mod h1:tMQXOfZzFuNuUxOypHlQEXgdfX5cuhwU+ffUuXRJE8I= +cloud.google.com/go/compute/metadata v0.5.0 h1:Zr0eK8JbFv6+Wi4ilXAR8FJ3wyNdpxHKJNPos6LTZOY= +cloud.google.com/go/compute/metadata v0.5.0/go.mod h1:aHnloV2TPI38yx4s9+wAZhHykWvVCfu7hQbF+9CWoiY= +cloud.google.com/go/iam v1.1.11 h1:0mQ8UKSfdHLut6pH9FM3bI55KWR46ketn0PuXleDyxw= +cloud.google.com/go/iam v1.1.11/go.mod h1:biXoiLWYIKntto2joP+62sd9uW5EpkZmKIvfNcTWlnQ= +cloud.google.com/go/storage v1.43.0 h1:CcxnSohZwizt4LCzQHWvBf1/kvtHUn7gk9QERXPyXFs= +cloud.google.com/go/storage v1.43.0/go.mod h1:ajvxEa7WmZS1PxvKRq4bq0tFT3vMd502JwstCcYv0Q0= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/andybalholm/brotli v1.0.1 h1:KqhlKozYbRtJvsPrrEeXcO+N2l6NYT5A2QAFmSULpEc= +github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= +github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 h1:iFaUwBSo5Svw6L7HYpRu/0lE3e0BaElwnNO1qkNQxBY= +github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s= +github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= +github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/golang/snappy v0.0.2 h1:aeE13tS0IiQgFjYdoL8qN3K1N2bXXtI6Vi51/y7BpMw= +github.com/golang/snappy v0.0.2/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= +github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/s2a-go v0.1.7 h1:60BLSyTrOV4/haCDW4zb1guZItoSq8foHCXrAnjBo/o= +github.com/google/s2a-go v0.1.7/go.mod h1:50CgR4k1jNlWBu4UfS4AcfhVe1r6pdZPygJ3R8F0Qdw= +github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.2 h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs= +github.com/googleapis/enterprise-certificate-proxy v0.3.2/go.mod h1:VLSiSSBs/ksPL8kq3OBOQ6WRI2QnaFynd1DCjZ62+V0= +github.com/googleapis/gax-go/v2 v2.12.5 h1:8gw9KZK8TiVKB6q3zHY3SBzLnrGp6HQjyfYBYGmXdxA= +github.com/googleapis/gax-go/v2 v2.12.5/go.mod h1:BUDKcWo+RaKq5SC9vVYL0wLADa3VcfswbOMMRmB9H3E= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= +github.com/klauspost/compress v1.11.4 h1:kz40R/YWls3iqT9zX9AHN3WoVsrAWVyui5sxuLqiXqU= +github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= +github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= +github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE= +github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs= +github.com/machinebox/graphql v0.2.2 h1:dWKpJligYKhYKO5A2gvNhkJdQMNZeChZYyBbrZkBZfo= +github.com/machinebox/graphql v0.2.2/go.mod h1:F+kbVMHuwrQ5tYgU9JXlnskM8nOaFxCAEolaQybkjWA= +github.com/mholt/archiver v3.1.1+incompatible h1:1dCVxuqs0dJseYEhi5pl7MYPH9zDa1wBi7mF09cbNkU= +github.com/mholt/archiver v3.1.1+incompatible/go.mod h1:Dh2dOXnSdiLxRiPoVfIr/fI1TwETms9B8CTWfeh7ROU= +github.com/mholt/archiver/v3 v3.5.1 h1:rDjOBX9JSF5BvoJGvjqK479aL70qh9DIpZCl+k7Clwo= +github.com/mholt/archiver/v3 v3.5.1/go.mod h1:e3dqJ7H78uzsRSEACH1joayhuSyhnonssnDhppzS1L4= +github.com/nwaples/rardecode v1.1.0 h1:vSxaY8vQhOcVr4mm5e8XllHWTiM4JF507A0Katqw7MQ= +github.com/nwaples/rardecode v1.1.0/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0= +github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM= +github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= +github.com/pierrec/lz4/v4 v4.1.2 h1:qvY3YFXRQE/XB8MlLzJH7mSzBs74eA2gg52YTk6jUPM= +github.com/pierrec/lz4/v4 v4.1.2/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= +github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/ulikunitz/xz v0.5.9 h1:RsKRIA2MO8x56wkkcd3LbtcE/uMszhb6DpRf+3uwa3I= +github.com/ulikunitz/xz v0.5.9/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo= +github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos= +go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= +go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0 h1:9G6E0TXzGFVfTnawRzrPl83iHOAV7L8NJiR8RSGYV1g= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0/go.mod h1:azvtTADFQJA8mX80jIH/akaE7h+dbm/sVuaHqN13w74= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 h1:4K4tsIXefpVJtvA/8srF4V4y0akAoPHkIslgAkjixJA= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0/go.mod h1:jjdQuTGVsXV4vSs+CJ2qYDeDPf9yIJV23qlIzBm73Vg= +go.opentelemetry.io/otel v1.28.0 h1:/SqNcYk+idO0CxKEUOtKQClMK/MimZihKYMruSMViUo= +go.opentelemetry.io/otel v1.28.0/go.mod h1:q68ijF8Fc8CnMHKyzqL6akLO46ePnjkgfIMIjUIX9z4= +go.opentelemetry.io/otel/metric v1.28.0 h1:f0HGvSl1KRAU1DLgLGFjrwVyismPlnuU6JD6bOeuA5Q= +go.opentelemetry.io/otel/metric v1.28.0/go.mod h1:Fb1eVBFZmLVTMb6PPohq3TO9IIhUisDsbJoL/+uQW4s= +go.opentelemetry.io/otel/trace v1.28.0 h1:GhQ9cUuQGmNDd5BTCP2dAvv75RdMxEfTmYejp+lkx9g= +go.opentelemetry.io/otel/trace v1.28.0/go.mod h1:jPyXzNPg6da9+38HEwElrQiHlVMTnVfM3/yv2OlIHaI= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.25.0 h1:ypSNr+bnYL2YhwoMt2zPxHFmbAN1KZs/njMG3hxUp30= +golang.org/x/crypto v0.25.0/go.mod h1:T+wALwcMOSE0kXgUAnPAHqTLW+XHgcELELW8VaDgm/M= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.27.0 h1:5K3Njcw06/l2y9vpGCSdcxWOYHOUk3dVNGDXN+FvAys= +golang.org/x/net v0.27.0/go.mod h1:dDi0PyhWNoiUOrAS8uXv/vnScO4wnHQO4mj9fn/RytE= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs= +golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M= +golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI= +golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4= +golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI= +golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= +golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/api v0.188.0 h1:51y8fJ/b1AaaBRJr4yWm96fPcuxSo0JcegXE3DaHQHw= +google.golang.org/api v0.188.0/go.mod h1:VR0d+2SIiWOYG3r/jdm7adPW9hI2aRv9ETOSCQ9Beag= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/genproto v0.0.0-20240711142825-46eb208f015d h1:/hmn0Ku5kWij/kjGsrcJeC1T/MrJi2iNWwgAqrihFwc= +google.golang.org/genproto v0.0.0-20240711142825-46eb208f015d/go.mod h1:FfBgJBJg9GcpPvKIuHSZ/aE1g2ecGL74upMzGZjiGEY= +google.golang.org/genproto/googleapis/api v0.0.0-20240711142825-46eb208f015d h1:kHjw/5UfflP/L5EbledDrcG4C2597RtymmGRZvHiCuY= +google.golang.org/genproto/googleapis/api v0.0.0-20240711142825-46eb208f015d/go.mod h1:mw8MG/Qz5wfgYr6VqVCiZcHe/GJEfI+oGGDCohaVgB0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240711142825-46eb208f015d h1:JU0iKnSg02Gmb5ZdV8nYsKEKsP6o/FGVWTrw4i1DA9A= +google.golang.org/genproto/googleapis/rpc v0.0.0-20240711142825-46eb208f015d/go.mod h1:Ue6ibwXGpU+dqIcODieyLOcgj7z8+IcskoNIgZxtrFY= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= +google.golang.org/grpc v1.65.0 h1:bs/cUb4lp1G5iImFFd3u5ixQzweKizoZJAwBNLR42lc= +google.golang.org/grpc v1.65.0/go.mod h1:WgYC2ypjlB0EiQi6wdKixMqukr6lBc0Vo+oOgjrM5ZQ= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= +google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= +google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/registry-automation/main.go b/registry-automation/main.go new file mode 100644 index 00000000..4f168b4b --- /dev/null +++ b/registry-automation/main.go @@ -0,0 +1,7 @@ +package main + +import "github.com/hasura/ndc-hub/registry-automation/cmd" + +func main() { + cmd.Execute() +} diff --git a/rfcs/0002-distribution-gh.md b/rfcs/0002-distribution-gh.md index e1b39817..2933022a 100644 --- a/rfcs/0002-distribution-gh.md +++ b/rfcs/0002-distribution-gh.md @@ -1,8 +1,13 @@ # Connector Package Distribution RFC - Milestone 1 +> [!NOTE] +> This RFC has since been extended by the [Connector publishing automation](./0006-connectors-publishing-automation.md) + This is a Work-In-Progress document. Please provide any feedback you wish to contribute via Github comments and suggestions. + + ## Purpose Connector API, definition and packaging are specified respectively by: @@ -271,4 +276,3 @@ Any publicly accessible APIs with publication capabilities have the potential to * Recycling of content * Unintentional mistakes * Spam / Reflection - diff --git a/rfcs/0006-connector-publishing-automation.md b/rfcs/0006-connector-publishing-automation.md new file mode 100644 index 00000000..ef4081ea --- /dev/null +++ b/rfcs/0006-connector-publishing-automation.md @@ -0,0 +1,105 @@ +# Connector registry Github packaging + +> [!NOTE] +> This RFC is an update on the [Connector Package Distribution RFC](0002-distribution-gh.md). + +## Introduction + +This RFC proposes how a new connector version should be added to the `registry` folder to automatically be published. Publishing in this context means that the connector version will be available for public use in Hasura's DDN. + +## File structure of the connectors `registry` + +The packages field in the `metadata.json` file will be removed and replaced by a releases folder within the connector directory. + +The releases folder will house a separate folder for each version of the connector, with each version folder containing a `connector-packaging.json` file. + +This `connector-packaging.json` file will include all the necessary information to access the package definition. + +The following directory structure for connector versions is proposed: + +``` +registry// +├── logo.png +├── metadata.json +├── README.md +└── releases + ├── v0.0.1 + │ └── connector-packaging.json + ├── v0.0.2 + │ └── connector-packaging.json + ├── v0.0.3 + │ └── connector-packaging.json + ├── v0.0.4 + │ └── connector-packaging.json + ├── v0.0.5 + │ └── connector-packaging.json + ├── v0.0.6 + │ └── connector-packaging.json + ├── v0.1.0 + │ └── connector-packaging.json + └── v1.0.0 + └── connector-packaging.json +``` + +The `registry` folder will contain a folder for each connector. Each connector folder will contain the following files: + +- `logo.png`: The logo of the connector. The logo should be in PNG format. +- `metadata.json`: The metadata of the connector. Metadata format is described in the [Github Distribution RFC](./0002-distribution-gh.md). +- `README.md`: The README file of the connector. The README file should contain information about the connector, how to use it, and any other relevant information. The contents of the README file would be displayed in the landing page of the connector in the Hasura. +- `releases`: The releases folder will contain a folder for each version of the connector. Each version folder will contain a `connector-packaging.json` file. More details about the `connector-packaging.json` file are provided below. + +NOTE: The `releases` folder should only be added for Hub connectors. +For example, `postgres-azure` connector should not have a `releases` folder as it is not a Hub connector. + +### `connector-packaging.json` + +Every connector version should have a package definition. The `connector-packaging.json` +file should contain the relevant information to access the package definition. + +```json +{ + "version": "0.0.1", + "uri": "https://github.com/hasura/ndc-mongodb/releases/download/v0.0.1/connector-definition.tgz", + "checksum": { + "type": "sha256", + "value": "2cd3584557be7e2870f3488a30cac6219924b3f7accd9f5f473285323843a0f4" + }, + "source": { + "hash": "c32adbde478147518f65ff465c40a0703239288a" + } +} +``` + +The following fields are required: + +- `version`: The version of the connector. +- `uri`: The URI to download the connector package. The package should be a tarball containing the connector package definition and the URL should be accessible without any authentication. +- `checksum`: The checksum of the connector package. The checksum should be calculated using the `sha256` algorithm. +- `source`: The source of the connector package. The `hash` field should contain the commit hash of the source code that was used to build the connector package. + + +## Publishing a new connector version + +To publish a new connector version, follow these steps: + +1. Create a new folder with the version number in the `releases` folder of the connector. +2. Create a `connector-packaging.json` file in the new folder with the relevant information. +3. Open a PR against the `main` branch of the repository. +4. You should see the `registry-update` workflow run on the PR. This workflow will validate the connector-packaging.json file and publish the new version to the registry if the validation is successful. +5. Once the workflow is successful, the new version of the connector will be available in the **Staging** Hasura DDN. Every new commit will overwrite the previous version of that connector in the staging DDN. So, feel free to push new commits to the PR to update the connector version in the staging DDN. +6. Once the PR is merged, the new version of the connector will be available in the **Production** Hasura DDN. + +> [!NOTE] +> The `registry-update` workflow will only run on the PRs against the `main` branch of the repository. + +> [!NOTE] +> Multiple connector versions can be published in the same PR. The `registry-update` workflow will publish all the versions in the PR to the registry. + + +## Updates to logo and README + +If you want to update the logo or README of the connector, you can do so by opening a PR against the `main` branch of the repository. + +The `registry-update` workflow will run on the PR and update the logo and README in the staging DDN. + +Once the PR is merged, the logo and README will be updated in the production DDN.