Skip to content

Commit

Permalink
Update with feedback from meeting
Browse files Browse the repository at this point in the history
  • Loading branch information
@daniel-chambers
daniel-chambers committed Feb 7, 2024
1 parent d567ea7 commit f3b48d4
Showing 1 changed file with 107 additions and 63 deletions.
170 changes: 107 additions & 63 deletions rfcs/0001-packaging.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Connector Packaging
# Connector Packaging and Hub Connector Definition

## Purpose

Expand All @@ -13,62 +13,46 @@ Hasura Hub data connectors are packaged as Docker images which follow the [deplo
- Connectors that do not require a build step can be described using a name and version, which should correspond to an entry in the connector hub registry.
- Connectors that require a build step (performed over some build inputs before they can execute correctly) can be described using a Dockerfile bundled with any additional build inputs.

### Connector Configuration

_TODO_: describe the layout of files on disk for prebuilt connectors, and for connectors which require a build step. Is this the same as the layout described in "Hasura Hub Connector Definition" below?

### Hasura Hub Connector Definition
In order for the Hasura tooling to understand a connector, know how to interact with it and know how it is packaged, a connector will need a definition that contains this information.

The connector definition package is a .tgz (tar file, gzipped) that contains the following files:
The connector definition is a file structure that contains the following files and directories:

```
/
.hasura/
connector-metadata.json # See ConnectorMetadataDefinition type
docker-compose.yaml # If DockerComposeWatchMode is used
### Start: If Docker packaging requires a build step
connector-metadata.json # See ConnectorMetadataDefinition type
docker-compose.yaml
build-files/ # Connector specific build/configuration files
src/
functions.ts
package.json
package-lock.json
tsconfig.json
docker/ # If ManagedDockerBuildPackaging is used
.dockerignore
Dockerfile
### End: If Docker packaging requires a build step
### Start: Connector specific configuration files
src/
functions.ts
package.json
package-lock.json
tsconfig.json
### End: Connector specific configuration files
```

```typescript
type ConnectorMetadataDefinition = {
packagingDefinition: PackagingDefinition
supportedEnvironmentVariables: EnvironmentVariableDefinition[]
cliPlugin?: CliPluginDefinition
}

type PackagingDefinition = PrebuiltDockerImagePackaging | DockerBuildPackaging
type PackagingDefinition = PrebuiltDockerImagePackaging | ManagedDockerBuildPackaging

type PrebuiltDockerImagePackaging = {
type: "PrebuiltDockerImage"
dockerImage: string // eg "hasura/postgres-data-connector:1.0.0"
}

type DockerBuildPackaging = {
type: "DockerBuild"
watchMode: WatchMode
}

type WatchMode = DockerRebuildWatchMode | DockerComposeWatchMode | ShellCommandWatchMode

type DockerRebuildWatchMode = {
type: "DockerRebuild"
}

type DockerComposeWatchMode = {
type: "DockerCompose"
type ManagedDockerBuildPackaging = {
type: "ManagedDockerBuild"
additionalWatchModes: ShellCommandWatchMode[]
}

type ShellCommandWatchMode = {
Expand All @@ -81,44 +65,47 @@ type EnvironmentVariableDefinition = {
description: string
defaultValue?: string
}

type CliPluginDefinition = {
pluginPackageUrl: string // https://url/to/download/plugin/to/install.tgz
pluginPackageSHA256: string // To ensure plugin integrity and also to identify which version to use out of those installed
watchSubcommand?: string // If the connector wants to add to the watch mode, it can define what subcommand to use "watch --whatever --flags"
}
```
The `.hasura/connector-metadata.json` contains JSON that describes:
The `connector-metadata.json` contains JSON that describes:
- The environment variables the connector supports to configure it (`supportedEnvironmentVariables`)
- The packaging definition, which can be either `PrebuiltDockerImagePackaging` (a connector that does not require a build step), or `DockerBuildPackaging` (a connector that requires a build step).
- The packaging definition, which can be either `PrebuiltDockerImagePackaging` (a connector that does not require a build step), or `ManagedDockerBuildPackaging` (a connector that requires a build step).
- `PrebuiltDockerImagePackaging` defines the prebuilt `dockerImage` used to run the connector (`dockerImage`)
- If `DockerBuildPackaging` is used, a Dockerfile must be in the `.hasura` directory (and optionally, a `.dockerignore`). It will be used to build the connector. A `watchMode` must be specified, to tell the Hasura tooling how to perform watching for this connector (see [Watch Mode section](#watch-mode)).
- If `ManagedDockerBuildPackaging` is used, a Dockerfile must be in the `.hasura` directory (and optionally, a `.dockerignore`). It will be used to build the connector.
- An optional `CLIPluginDefinition` that describes where to acquire the CLI plugin for this connector that can be used to enhance watch mode with connector-specific configuration updates
#### Watch Mode
### Watch Mode
When developing locally using the connector, the user may utilize the connector in a "watch mode" that automatically reloads the connector with new configuration as they change the configuration.
#### Connectors that do not require a build step (`PrebuiltDockerImagePackaging`)
If any configuration file changes, the connector image is restarted with the new version of the configuration file mounted.
Every connector must specify a `docker-compose.yaml` in their connector definition that defines how to watch the container.
#### Connectors that require a build step (`DockerBuildPackaging`)
There are three options a connector can choose from in their `connector-metadata.json`:
##### `DockerRebuildWatchMode`
In this mode, if any configuration file changes, the Docker container is rebuilt and restarted. This is the simplest mode, but the least performant, unless your Docker builds are extremely quick.
#### Connectors that do not require a build step (`PrebuiltDockerImagePackaging`)
The `docker-compose.yaml` should contain something like this:
##### `DockerComposeWatchMode`
In this mode, connectors can provide a `.hasura/docker-compose.yaml` file. This file should:
```yaml
services:
connector:
develop:
watch:
- path: ./
target: /etc/connector
action: sync+restart
```
- Define a single service, which represents the connector
- Build that service using the `.hasura/Dockerfile`
- Define a port mapping that uses the environment variable `PORT` for the host port and 8080 for the container port. This allows the watch tooling to control the port used by the connector and exposes the port on the host.
- Provide a [docker compose watch configuration](https://docs.docker.com/compose/compose-file/develop/#watch) that defines how to watch the configuration files and how to react to different files changing. It can react by causing a container rebuild and restart, or by copying the new file inside the existing container and optionally restarting the container. This allows the connector to optimise its hot reload capability by only performing a rebuild where necessary.
The `connector` service must be defined, and note that no `image` or `build` property is defined. The CLI tooling will [extend](https://docs.docker.com/compose/compose-file/05-services/#extends) from this to add the necessary pre-built image. This exists solely to provide the watch configuration.
For example, a NodeJS-based connector that needs to perform an npm package restore as a part of its Docker build may define the following `.hasura/docker.compose.yaml`:
#### Connectors that require a build step (`ManagedDockerBuildPackaging`)
The `docker-compose.yaml` should contain something like this (specific to the watch requirements of the connector):
```yaml
services:
connector:
build:
context: ../ # The build context is the parent directory of .hasura/
dockerfile: ./Dockerfile
ports:
- ${PORT}:8080 # Required so that the watch tooling can control the port used
develop:
watch:
# Rebuild the container if a new package restore is required because package[-lock].json changed
Expand All @@ -132,7 +119,11 @@ services:
- path: ./
target: /etc/connector
action: sync+restart
```
```
The `connector` service must be defined, and note that no `image` or `build` property is defined. The CLI tooling will [extend](https://docs.docker.com/compose/compose-file/05-services/#extends) from this to add the necessary `build` property based on the build context from `ConnectorManifest` and the Dockerfile included in the Connector Definition. This exists solely to provide the watch configuration.
`ManagedDockerBuildPackaging` connectors can also optionally provide additional watch modes. This is intended to support connectors that can run using local developer machine tooling (for example, local NodeJS for debugging purposes). These additional watch modes may be chosen by the user to run _instead_ of the docker compose watch-based watch mode. They are configured via the `additionalWatchModes` property, which currently only supports `ShellCommandWatchMode`.
##### `ShellCommandWatchMode`
In this mode, connectors can provide some native way of performing a hot-reloading watch mode. Whatever this custom method is, it will be started by running the shell command defined. This can be useful to allow local tooling to be used to run the connector and perform hot reloading.
Expand All @@ -144,19 +135,72 @@ In this mode, connectors can provide some native way of performing a hot-reloadi
For example, for the NodeJS Lambda Connector, it could set the watch shell command to `npm run watch`, which would run the connector and activate its built-in hot-reloading functionality.
#### CLI Plugin Watch Mode
If connectors need to extend the watch process with their own functionality, they can implement a CLI plugin that contains a `watch` subcommand. If specified in the `connector-metadata.json` the tooling will use it in addition to existing watch functionality.
An example use case for this would be for a Postgres connector, where you may want to watch the database for schema changes, and when they occur, update the schema introspection configuration file in the connector's build directory with the latest schema details. The docker compose watch would then reload the connector with the updated introspection configuration file.
### Connector Layout in Hasura Project
When a new connector is added to a Hasura project using `hasura3 add ConnectorManifest --hub hasura/nodejs-lambda:1.0`, the CLI acquires the Hasura Hub Connector Definition for the specified Hub Connector. It then places this inside the `~/.hasura/hub-connectors/` directory. This is done to keep non-user editable files out of the user's source tree.
The CLI then puts the `build-files/` into the build directory for that connector and adds a `.build.hml` file with the `ConnectorManifest` metadata object.
```
# A copy of the `Hasura Hub Connector Definition` for a particular hub connector version
~/.hasura/hub-connectors/hasura/nodejs-lambda/1.0/
connector-metadata.json
docker-compose.yaml
build-files/
src/
functions.ts
package.json
package-lock.json
tsconfig.json
docker/
.dockerignore
Dockerfile

project/
subgraphs/default/dataconnectors/
my-dataconnector/
build/
### Start: from `build-files/`
src/
functions.ts
package.json
package-lock.json
tsconfig.json
### End: from `build-files/`

my-dataconnector.build.hml # Connector manifest, see below
```

Here's an example of what `my-dataconnector.build.hml` would contain (`ConnectorManifest` is not specified by this RFC):
```yaml
kind: ConnectorManifest
definition:
name: my-dataconnector
type: local
hub-connector: hasura/nodejs-lambda:1.0 # This is used to find the Hasura Hub Connector Definition in ~/.hasura/hub-connectors/
tunnel: true
instances:
- build:
context: .
env:
STRIPE_API_ENDPOINT: https://api.stripe.com/
```
## Open Questions
### Build Inputs vs Configuration
- Is there a difference between Docker build inputs and connector configuration files (currently the RFC does not distinguish these)? If so:
- Can connector configuration be optional (defaulted to zero files) where it is not used (ie NodeJS Lambda Connector)
- How is the difference represented on disk in the user's Hasura project? (Different directories?)
### Custom CLI Plugins
- Do connectors need to declare if they have a custom CLI plugin?
- If custom CLI plugins provide configuration update services (such as DB schema introspection), does this need to be integrated with watch mode, and if so, how is this declared?
- Do we need a `validate` subcommand to support the LSP/CLI?
- `watchSubcommand` seems redundant if we're going to have a CLI contract. We could just have `watch: boolean` instead. Or perhaps the CLI plugin can declare what it supports and this exists outside of the connector-metadata.json?

### Publishing the Hasura Hub Connector Definition
- How is this published to the Hub?
- How is this published to the Hub? Git? `tar.gz`?

### OpenTelemetry
- Do we want to reserve environment variables `OTEL_*` for possible future use of the [OTLP exporter spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md)?

0 comments on commit f3b48d4

Please sign in to comment.