feat!: rework gherkin files and structure (#161)

Signed-off-by: Simon Schrottner <[email protected]>

.gherkin-lintrc

@@ -50,8 +50,8 @@
     {
       "steps-length": {
         "Background": 3,
-        "Scenario": 15
+        "Scenario": 16
       }
     }
   ]
-}
+}

README.md

@@ -23,11 +23,11 @@ Combined with the _flagd-provider_ for the associated SDK and the flagd-testbed,
 
 Included suites:
 
-[flagd.feature](gherkin/flagd.feature) includes tests relevant to flagd and all flagd providers:
+[evaluation.feature](gherkin/evaluation.feature) includes tests relevant to flagd and all flagd providers:
 * default value (zero-value) handling (some proto3 implementations handle these inconsistently).
 * basic event handling
 
-[flagd-json-evaluator.feature](gherkin/flagd-json-evaluator.feature) includes tests relevant to flagd and in-process providers:
+[targeting.feature](gherkin/targeting.feature) includes tests relevant to flagd and in-process providers:
 * custom JSONLogic operators (`starts_with`, `ends_with`, `fractional`, `sem_ver`)
 * evaluator reuse via `$ref`
 

gherkin/config.feature

@@ -28,35 +28,35 @@ Feature: Configuration Test
   Scenario Outline: Default Config
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<default>"
-    Examples: Basic
+    Scenarios: Basic
       | option     | type         | default   |
       | resolver   | ResolverType | rpc       |
       | host       | String       | localhost |
       | port       | Integer      | 8013      |
       | tls        | Boolean      | false     |
       | deadlineMs | Integer      | 500       |
     @targetURI
-    Examples: Target URI
+    Scenarios: Target URI
       | option    | type   | default |
       | targetUri | String | null    |
     @customCert
-    Examples: Certificates
+    Scenarios: Certificates
       | option   | type   | default |
       | certPath | String | null    |
     @unixsocket
-    Examples: Unixsocket
+    Scenarios: Unixsocket
       | option     | type   | default |
       | socketPath | String | null    |
     @events
-    Examples: Events
+    Scenarios: Events
       | option            | type    | default |
       | streamDeadlineMs  | Integer | 600000  |
       | keepAliveTime     | Long    | 0       |
       | retryBackoffMs    | Integer | 1000    |
       | retryBackoffMaxMs | Integer | 120000  |
       | retryGracePeriod  | Integer | 5       |
     @sync
-    Examples: Sync
+    Scenarios: Sync
       | option            | type    | default |
       | streamDeadlineMs  | Integer | 600000  |
       | keepAliveTime     | Long    | 0       |
@@ -65,12 +65,12 @@ Feature: Configuration Test
       | retryGracePeriod  | Integer | 5       |
       | selector          | String  | null    |
     @caching
-    Examples: caching
+    Scenarios: caching
       | option       | type      | default |
       | cache        | CacheType | lru     |
       | maxCacheSize | Integer   | 1000    |
     @offline
-    Examples: offline
+    Scenarios: offline
       | option                | type    | default |
       | offlineFlagSourcePath | String  | null    |
       | offlinePollIntervalMs | Integer | 5000    |
@@ -80,7 +80,7 @@ Feature: Configuration Test
     Given an option "resolver" of type "ResolverType" with value "rpc"
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<default>"
-    Examples:
+    Scenarios:
       | option | type    | default |
       | port   | Integer | 8013    |
 
@@ -89,7 +89,7 @@ Feature: Configuration Test
     Given an option "resolver" of type "ResolverType" with value "in-process"
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<default>"
-    Examples:
+    Scenarios:
       | option | type    | default |
       | port   | Integer | 8015    |
 
@@ -98,35 +98,35 @@ Feature: Configuration Test
     Given an option "<option>" of type "<type>" with value "<value>"
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<value>"
-    Examples:
+    Scenarios:
       | option     | type         | value      |
       | resolver   | ResolverType | in-process |
       | host       | String       | local      |
       | tls        | Boolean      | True       |
       | port       | Integer      | 1234       |
       | deadlineMs | Integer      | 123        |
     @targetURI
-    Examples: Target URI
+    Scenarios: Target URI
       | option    | type   | value |
       | targetUri | String | path  |
     @customCert
-    Examples:
+    Scenarios: Custom Certificate
       | option   | type   | value |
       | certPath | String | path  |
     @unixsocket
-    Examples:
+    Scenarios: Unixsocket
       | option     | type   | value |
       | socketPath | String | path  |
     @events
-    Examples:
+    Scenarios: events
       | option            | type    | value  |
       | streamDeadlineMs  | Integer | 500000 |
       | keepAliveTime     | Long    | 5      |
       | retryBackoffMs    | Integer | 5000   |
       | retryBackoffMaxMs | Integer | 12000  |
       | retryGracePeriod  | Integer | 10     |
     @sync
-    Examples:
+    Scenarios: sync
       | option            | type    | value    |
       | streamDeadlineMs  | Integer | 500000   |
       | keepAliveTime     | Long    | 5        |
@@ -135,12 +135,12 @@ Feature: Configuration Test
       | retryGracePeriod  | Integer | 10       |
       | selector          | String  | selector |
     @caching
-    Examples:
+    Scenarios: caching
       | option       | type      | value    |
       | cache        | CacheType | disabled |
       | maxCacheSize | Integer   | 1236     |
     @offline
-    Examples:
+    Scenarios: offline
       | option                | type    | value |
       | offlineFlagSourcePath | String  | path  |
       | offlinePollIntervalMs | Integer | 1000  |
@@ -150,7 +150,7 @@ Feature: Configuration Test
     Given an environment variable "<env>" with value "<value>"
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<value>"
-    Examples:
+    Scenarios:
       | option     | env               | type         | value      |
       | resolver   | FLAGD_RESOLVER    | ResolverType | in-process |
       | resolver   | FLAGD_RESOLVER    | ResolverType | IN-PROCESS |
@@ -161,27 +161,27 @@ Feature: Configuration Test
       | port       | FLAGD_PORT        | Integer      | 1234       |
       | deadlineMs | FLAGD_DEADLINE_MS | Integer      | 123        |
     @targetURI
-    Examples: Target URI
+    Scenarios: Target URI
       | option    | env              | type   | value |
       | targetUri | FLAGD_TARGET_URI | String | path  |
     @customCert
-    Examples:
+    Scenarios: Custom Certificates
       | option   | env                    | type   | value |
       | certPath | FLAGD_SERVER_CERT_PATH | String | path  |
     @unixsocket
-    Examples:
+    Scenarios: Unixsocket
       | option     | env               | type   | value |
       | socketPath | FLAGD_SOCKET_PATH | String | path  |
     @events
-    Examples:
+    Scenarios: Events
       | option            | env                        | type    | value  |
       | streamDeadlineMs  | FLAGD_STREAM_DEADLINE_MS   | Integer | 500000 |
       | keepAliveTime     | FLAGD_KEEP_ALIVE_TIME_MS   | Long    | 5      |
       | retryBackoffMs    | FLAGD_RETRY_BACKOFF_MS     | Integer | 5000   |
       | retryBackoffMaxMs | FLAGD_RETRY_BACKOFF_MAX_MS | Integer | 12000  |
       | retryGracePeriod  | FLAGD_RETRY_GRACE_PERIOD   | Integer | 10     |
     @sync
-    Examples:
+    Scenarios: Sync
       | option            | env                        | type    | value    |
       | streamDeadlineMs  | FLAGD_STREAM_DEADLINE_MS   | Integer | 500000   |
       | keepAliveTime     | FLAGD_KEEP_ALIVE_TIME_MS   | Long    | 5        |
@@ -190,12 +190,12 @@ Feature: Configuration Test
       | retryGracePeriod  | FLAGD_RETRY_GRACE_PERIOD   | Integer | 10       |
       | selector          | FLAGD_SOURCE_SELECTOR      | String  | selector |
     @caching
-    Examples:
+    Scenarios: Caching
       | option       | env                  | type      | value    |
       | cache        | FLAGD_CACHE          | CacheType | disabled |
       | maxCacheSize | FLAGD_MAX_CACHE_SIZE | Integer   | 1236     |
     @offline
-    Examples:
+    Scenarios: Offline
       | option                | env                            | type    | value |
       | offlineFlagSourcePath | FLAGD_OFFLINE_FLAG_SOURCE_PATH | String  | path  |
       | offlinePollIntervalMs | FLAGD_OFFLINE_POLL_MS          | Integer | 1000  |
@@ -206,35 +206,35 @@ Feature: Configuration Test
     And an option "<option>" of type "<type>" with value "<value>"
     When a config was initialized
     Then the option "<option>" of type "<type>" should have the value "<value>"
-    Examples:
+    Scenarios:
       | option     | env               | type         | value      | env-value |
       | resolver   | FLAGD_RESOLVER    | ResolverType | in-process | rpc       |
       | host       | FLAGD_HOST        | String       | local      | l         |
       | tls        | FLAGD_TLS         | Boolean      | True       | False     |
       | port       | FLAGD_PORT        | Integer      | 1234       | 3456      |
       | deadlineMs | FLAGD_DEADLINE_MS | Integer      | 123        | 345       |
     @targetURI
-    Examples: Target URI
+    Scenarios: Target URI
       | option    | env              | type   | value | env-value |
       | targetUri | FLAGD_TARGET_URI | String | path  | fun       |
     @customCert
-    Examples:
+    Scenarios: Custom Certificates
       | option   | env                    | type   | value | env-value |
       | certPath | FLAGD_SERVER_CERT_PATH | String | path  | rpc       |
     @unixsocket
-    Examples:
+    Scenarios: Unixsocket
       | option     | env               | type   | value | env-value |
       | socketPath | FLAGD_SOCKET_PATH | String | path  | rpc       |
     @events
-    Examples:
+    Scenarios: Events
       | option            | env                        | type    | value  | env-value |
       | streamDeadlineMs  | FLAGD_STREAM_DEADLINE_MS   | Integer | 500000 | 400       |
       | keepAliveTime     | FLAGD_KEEP_ALIVE_TIME_MS   | Long    | 5      | 4         |
       | retryBackoffMs    | FLAGD_RETRY_BACKOFF_MS     | Integer | 5000   | 4         |
       | retryBackoffMaxMs | FLAGD_RETRY_BACKOFF_MAX_MS | Integer | 12000  | 4         |
       | retryGracePeriod  | FLAGD_RETRY_GRACE_PERIOD   | Integer | 10     | 4         |
     @sync
-    Examples:
+    Scenarios: Sync
       | option            | env                        | type    | value    | env-value |
       | streamDeadlineMs  | FLAGD_STREAM_DEADLINE_MS   | Integer | 500000   | 400       |
       | keepAliveTime     | FLAGD_KEEP_ALIVE_TIME_MS   | Long    | 5        | 4         |
@@ -243,12 +243,12 @@ Feature: Configuration Test
       | retryGracePeriod  | FLAGD_RETRY_GRACE_PERIOD   | Integer | 10       | 4         |
       | selector          | FLAGD_SOURCE_SELECTOR      | String  | selector | sele      |
     @caching
-    Examples:
+    Scenarios: Caching
       | option       | env                  | type      | value    | env-value |
       | cache        | FLAGD_CACHE          | CacheType | disabled | lru       |
       | maxCacheSize | FLAGD_MAX_CACHE_SIZE | Integer   | 1236     | 2345      |
     @offline
-    Examples:
+    Scenarios: Offline
       | option                | env                            | type    | value | env-value |
       | offlineFlagSourcePath | FLAGD_OFFLINE_FLAG_SOURCE_PATH | String  | path  | lll       |
       | offlinePollIntervalMs | FLAGD_OFFLINE_POLL_MS          | Integer | 1000  | 4         |

gherkin/connection.feature

@@ -0,0 +1,57 @@
+@rpc @in-process
+Feature: flagd provider disconnect and reconnect functionality
+
+  Scenario Outline: Connection
+    Given a <name> flagd provider
+    And a Boolean-flag with key "boolean-flag" and a default value "true"
+    When the flag was evaluated with details
+    Then the resolved details value should be "true"
+
+    Scenarios: Stable
+      | name   |
+      | stable |
+    @targetURI
+    Scenarios: Target URI
+      | name   |
+      | target |
+    @customCert
+    Scenarios: Certificates
+      | name |
+      | ssl  |
+    @unixsocket @os.linux
+    Scenarios: Unixsocket
+      | name   |
+      | socket |
+
+  @reconnect
+  # This test suite tests the reconnection functionality of flagd providers
+  Scenario Outline: Provider reconnection
+    Given a <name> flagd provider
+    And a ready event handler
+    And a error event handler
+    When a ready event was fired
+    When the connection is lost for 4s
+    Then the error event handler should have been executed
+    Then the ready event handler should have been executed
+
+    Scenarios: Stable
+      | name   |
+      | stable |
+    @targetURI
+    Scenarios: Target URI
+      | name   |
+      | socket |
+    @customCert
+    Scenarios: Certificates
+      | name |
+      | ssl  |
+    @unixsocket @os.linux
+    Scenarios: Unixsocket
+      | name   |
+      | socket |
+
+  Scenario: Provider unavailable
+    Given an option "deadlineMs" of type "Integer" with value "1000"
+    And a unavailable flagd provider
+    And a error event handler
+    Then the error event handler should have been executed within 1000ms

gherkin/evaluation.feature

@@ -0,0 +1,34 @@
+@rpc @in-process
+Feature: flagd evaluations
+
+  # This test suite contains scenarios to test flagd providers.
+  # It's associated with the flags configured in flags.
+  # It should be used in conjunction with the suites supplied by the OpenFeature specification.
+
+  Background:
+    Given an option "cache" of type "CacheType" with value "disabled"
+    And a stable flagd provider
+
+  Scenario Outline: Resolve values
+    Given a <type>-flag with key "<key>" and a default value "<default>"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+
+    Examples:
+      | key          | type    | default | resolved_value |
+      | boolean-flag | Boolean | false   | true           |
+      | string-flag  | String  | bye     | hi             |
+      | integer-flag | Integer | 1       | 10             |
+      | float-flag   | Float   | 0.1     | 0.5            |
+
+  Scenario Outline: Resolves zero value
+    Given a <type>-flag with key "<key>" and a default value "<default>"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+
+    Examples:
+      | key               | type    | default | resolved_value |
+      | boolean-zero-flag | Boolean | true    | false          |
+      | string-zero-flag  | String  | hi      |                |
+      | integer-zero-flag | Integer | 1       | 0              |
+      | float-zero-flag   | Float   | 0.1     | 0.0            |