From 6f8eeba891482ab162c7fd14aeb70da9b6088dcd Mon Sep 17 00:00:00 2001 From: Todd Baert Date: Wed, 27 Sep 2023 11:06:59 -0400 Subject: [PATCH] fixup: add caching to provider, k8s stuff Signed-off-by: Todd Baert --- mkdocs.yml | 4 + web-docs/deployment.md | 2 +- .../crds/featureflagconfiguration.md | 28 +++ .../crds/flagsourceconfiguration.md | 168 ++++++++++++++++++ .../openfeature-operator/installation.md | 132 ++++++++++++++ web-docs/reference/providers.md | 2 + .../reference/specifications/rpc-providers.md | 37 +++- 7 files changed, 370 insertions(+), 3 deletions(-) create mode 100644 web-docs/reference/openfeature-operator/crds/featureflagconfiguration.md create mode 100644 web-docs/reference/openfeature-operator/crds/flagsourceconfiguration.md create mode 100644 web-docs/reference/openfeature-operator/installation.md diff --git a/mkdocs.yml b/mkdocs.yml index 474ee5714..6ac96b833 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -81,6 +81,10 @@ nav: - 'Fractional Specification': 'reference/specifications/custom-operations/fractional-operation-spec.md' - 'Semantic Version Specification': 'reference/specifications/custom-operations/semver-operation-spec.md' - 'String Comparison Specification': 'reference/specifications/custom-operations/string-comparison-operation-spec.md' + - 'OpenFeature Operator': + - 'Installation': 'reference/openfeature-operator/installation.md' + - 'FeatureFlagConfiguration': 'reference/openfeature-operator/crds/featureflagconfiguration.md' + - 'FlagSourceConfiguration': 'reference/openfeature-operator/crds/flagsourceconfiguration.md' - 'Naming': 'reference/naming.md' - 'FAQ': 'faq.md' - 'Troubleshooting': 'troubleshooting.md' diff --git a/web-docs/deployment.md b/web-docs/deployment.md index cc8ec064c..e474eafa7 100644 --- a/web-docs/deployment.md +++ b/web-docs/deployment.md @@ -20,7 +20,7 @@ flagd was designed with cloud-native paradigms in mind. You can run it as a sidecar, or as a central service in your cluster. If you're interested in a full-featured solution for using flagd in Kubernetes, consider the [OpenFeature operator](https://github.com/open-feature/open-feature-operator). - +For more information, see [OpenFeature Operator](./reference/openfeature-operator/installation.md). --- diff --git a/web-docs/reference/openfeature-operator/crds/featureflagconfiguration.md b/web-docs/reference/openfeature-operator/crds/featureflagconfiguration.md new file mode 100644 index 000000000..43e21b983 --- /dev/null +++ b/web-docs/reference/openfeature-operator/crds/featureflagconfiguration.md @@ -0,0 +1,28 @@ +# Feature Flag Configuration + +The `FeatureFlagConfiguration` version `v1alpha2` CRD defines a CR with the following example structure: + +```yaml +apiVersion: core.openfeature.dev/v1alpha2 +kind: FeatureFlagConfiguration +metadata: + name: featureflagconfiguration-sample +spec: + featureFlagSpec: + flags: + foo: + state: "ENABLED" + variants: + bar: "BAR" + baz: "BAZ" + defaultVariant: "bar" +``` + +In the example above, we have defined a `String` type feature flag named `foo` and it is in the `ENABLED` state. +It has variants of `bar` and `baz`, referring to respected values of `BAR` and `BAZ`. +The default variant is set to`bar`. + +## featureFlagSpec + +The `featureFlagSpec` is an object representing the flag configurations themselves. +The documentation for this object can be found [here](https://github.com/open-feature/flagd/blob/main/docs/configuration/flag_configuration.md). diff --git a/web-docs/reference/openfeature-operator/crds/flagsourceconfiguration.md b/web-docs/reference/openfeature-operator/crds/flagsourceconfiguration.md new file mode 100644 index 000000000..dd7c89d74 --- /dev/null +++ b/web-docs/reference/openfeature-operator/crds/flagsourceconfiguration.md @@ -0,0 +1,168 @@ +# FlagSourceConfigurations + +`FlagSourceConfiguration` support multiple flag sources. Sources are configured as a list and given below are supported sources and their configurations, + +## kubernetes aka `FeatureFlagConfiguration` + +This is `FeatureFlagConfiguration` custom resource backed flagd feature flag definition. +Read more on the custom resource at the dedicated documentation of [FeatureFlagConfiguration](https://github.com/open-feature/open-feature-operator/blob/main/docs/feature_flag_configuration.md) + +To refer this custom resource in `FlagSourceConfiguration`, provider type `kubernetes` is used as below example, + +```yaml +sources: + - source: flags/sample-flags # FeatureFlagConfiguration - namespace/custom_resource_name + provider: kubernetes # kubernetes flag source backed by FeatureFlagConfiguration custom resource +``` + +## flagd-proxy + +`flagd-proxy` is an alternative to direct resource access on `FeatureFlagConfiguration` custom resources. +This source type is useful when there is a need for restricting workload permissions and/or to reduce k8s API load. + +Read more about proxy approach to access kubernetes resources: [flagd-proxy](https://github.com/open-feature/open-feature-operator/blob/main/docs/flagd_proxy.md) + +## filepath + +Injected sidecar can use volume mounted files as flag sources. +For this, provider type `filepath` is used as below example, + +```yaml +sources: + - source: /etc/flagd/config.json + provider: filepath +``` + +## http + +Feature flags can be sources from a http endpoint using provider type `http`, + +```yaml +sources: + - source: http://my-flag-source.json + provider: http + httpSyncBearerToken: token # optional bearer token for the http connection +``` + +## grpc + +Given below is an example configuration with provider type `grpc` and supported options, + +```yaml +sources: + - source: my-flag-source:8080 + provider: grpc + certPath: /certs/ca.cert # certificate for tls connectivity + tls: true # enforce tls connectivity + providerID: flagd-weatherapp-sidecar # identifier for this connection + selector: 'source=database,app=weatherapp' # flag filtering options +``` + +## Sidecar configurations + +`FlagSourceConfiguration` further allows to provide configurations to the injected flagd sidecar. +Table given below is non-exhaustive list of overriding options, + +| Configuration | Explanation | Default | +|---------------|-------------------------------|----------------------------| +| port | Flag evaluation endpoint port | 8013 | +| metricsPort | Metrics port | 8014 | +| evaluator | Evaluator to use | json | +| image | flagD image | ghcr.io/open-feature/flagd | +| tag | flagD image tag | Latest tag | +| probesEnabled | Enable/Disable health probes | true | + +## Merging of configurations + +The annotation value is a comma separated list of values following one of two patterns: `{NAME}` or `{NAMESPACE}/{NAME}`. +If no namespace is provided, it is assumed that the CR is within the same namespace as the deployed pod, for example: + +```yaml + metadata: + namespace: test-ns + annotations: + openfeature.dev/enabled: "true" + openfeature.dev/flagsourceconfiguration: "config-A, test-ns-2/config-B" +``` + +In this example, 2 CRs are being used to configure the injected container (by default the operator uses the `flagd:main` image), `config-A` (which is assumed to be in the namespace `test-ns`) and `config-B` from the `test-ns-2` namespace, with `config-B` taking precedence in the configuration merge. + +The `FlagSourceConfiguration` version `v1alpha3` CRD defines a CR with the following example structure, the documentation for this CRD can be found [here](https://github.com/open-feature/open-feature-operator/blob/main/docs/crds.md#flagsourceconfiguration): + +```yaml +apiVersion: core.openfeature.dev/v1alpha3 +kind: FlagSourceConfiguration +metadata: + name: flag-source-sample +spec: + metricsPort: 8080 + Port: 80 + evaluator: json + image: my-custom-sidecar-image + defaultSyncProvider: filepath + tag: main + sources: + - source: namespace/name + provider: kubernetes + - source: namespace/name2 + - source: not-a-real-host.com + provider: http + envVars: + - name: MY_ENV_VAR + value: my-env-value + probesEnabled: true + debugLogging: false +``` + +The relevant `FlagSourceConfigurations` are passed to the operator by setting the `openfeature.dev/flagsourceconfiguration` annotation, and is responsible for providing the full configuration of the injected sidecar. + +## Configuration Merging + +When multiple `FlagSourceConfigurations` are provided, the configurations are merged. The last `CR` takes precedence over the first, with any configuration from the deprecated `FlagDSpec` field of the `FeatureFlagConfiguration` CRD taking the lowest priority. + +```mermaid +flowchart LR + FlagSourceConfiguration-values -->|highest priority| environment-variables -->|lowest priority| defaults +``` + +An example of this behavior: + +```yaml + metadata: + annotations: + openfeature.dev/enabled: "true" + openfeature.dev/flagsourceconfiguration:"config-A, config-B" +``` + +Config-A: + +```yaml +apiVersion: core.openfeature.dev/v1alpha2 +kind: FlagSourceConfiguration +metadata: + name: config-A +spec: + metricsPort: 8080 + tag: latest +``` + +Config-B: + +```yaml +apiVersion: core.openfeature.dev/v1alpha2 +kind: FlagSourceConfiguration +metadata: + name: config-B +spec: + port: 8000 + tag: main +``` + +Results in the following configuration: + +```yaml +spec: + metricsPort: 8080 + port: 8000 + tag: main +``` diff --git a/web-docs/reference/openfeature-operator/installation.md b/web-docs/reference/openfeature-operator/installation.md new file mode 100644 index 000000000..d69c2e2e6 --- /dev/null +++ b/web-docs/reference/openfeature-operator/installation.md @@ -0,0 +1,132 @@ +# Installing the OpenFeature Operator + +Use the [OpenFeature Operator](https://github.com/open-feature/open-feature-operator) to install and run flagd on a Kubernetes cluster. + +The operator includes flagd (no need to install flagd seperately). + +## Installation + +1. Install [cert-manager](https://cert-manager.io/docs/installation/) if you don't already have it on the cluster +1. Install the OpenFeature Operator (see below) +1. Define CRDs which describe the feature flag source (flagd) and the feature flags themselves +1. Annotate your deployment to enable flagd feature flagging for that pod + +## Install OpenFeature operator + +```bash +helm repo add openfeature https://open-feature.github.io/open-feature-operator/ +helm repo update +helm upgrade --install openfeature openfeature/open-feature-operator +``` + +## Define feature flags + +Create a namespace to house your flags: + +```bash +kubectl create namespace flags +``` + +Next define your feature flag(s) using the [FeatureFlagConfiguration](./crds/featureflagconfiguration.md) CRD. + +This example specifies one flag called `foo` which has two variants `bar` and `baz`. The `defaultVariant` is `bar`. + +If this doesn't make sense, review the [concepts](../concepts/index.md) section. + +```bash +kubectl apply -n flags -f - < READY: initialize(), connection to flagd established, stream connected NOT_READY --> ERROR: initialize(), unable to connect or establish stream(retry) READY --> ERROR: stream or connection disconnected - READY --> READY: configuration_change (emit changed*) - ERROR --> READY: reconnect successful (emit ready*, changed*) + READY --> READY: configuration_change (emit changed*, invalidate cache) + ERROR --> READY: reconnect successful (emit ready*, changed*, invalidate cache) ERROR --> ERROR: maxSyncRetries reached ERROR --> [*]: shutdown(), stream disconnected ``` \* ready=`PROVIDER_READY`, changed=`PROVIDER_CONFIGURATION_CHANGED`, stale=`PROVIDER_STALE`, error=`PROVIDER_ERROR` +## Caching + +`flagd` has a caching strategy implementable by RPC providers that support server-to-client streaming. + +### Cacheable flags + +`flagd` sets the `reason` of a flag evaluation as `STATIC` when no targeting rules are configured for the flag. +A client can safely store the result of a static evaluation in its cache indefinitely (until the configuration of the flag changes, see [cache invalidation](#cache-invalidation)). + +Put simply in pseudocode: + +```pseudo +if reason == "STATIC" { + isFlagCacheable = true +} +``` + +### Cache invalidation + +`flagd` emits events to the server-to-client stream, among these is the `configuration_change` event. +The structure of this event is as such: + +```json +{ + "type": "delete", // ENUM:["delete","write","update"] + "source": "/flag-configuration.json", // the source of the flag configuration + "flagKey": "foo" +} +``` + +A client should invalidate the cache of any flag found in a `configuration_change` event to prevent stale data. +If the connection drops all cache values must be cleared (any number of events may have been missed). + ## Configuration Expose means to configure the provider aligned with the following priority system (highest to lowest).