Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ATO-1315] Flows yaml schema #12901

Merged
merged 30 commits into from
Oct 16, 2023
Merged

[ATO-1315] Flows yaml schema #12901

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
92a0e24
init flows_yaml_schema
Urkem Sep 26, 2023
9cb1b04
add basic schema
Urkem Oct 6, 2023
06839c5
Merge remote-tracking branch 'origin/dm2' into ATO-1315-improve-yaml-…
Urkem Oct 9, 2023
3a04dd8
add fail tests
Urkem Oct 9, 2023
bb4f625
add changelog
Urkem Oct 9, 2023
e628f4b
fix minProp
Urkem Oct 9, 2023
994ec5f
add next to set slot
Urkem Oct 9, 2023
d966b24
fix set_slots test
Urkem Oct 9, 2023
f685d8e
rename set_slot to set_slots
Urkem Oct 9, 2023
01372e6
add intent
Urkem Oct 9, 2023
2175ab6
Trigger CI
Urkem Oct 9, 2023
6351f72
add entry_prompt
Urkem Oct 10, 2023
f5f65dd
add null to slot values
Urkem Oct 10, 2023
5a3bf1c
remove docs link
Urkem Oct 10, 2023
6627fb7
fix lint
Urkem Oct 10, 2023
47c596c
fix read
Urkem Oct 12, 2023
bfea853
drop reset in set_slots
Urkem Oct 12, 2023
cddceab
add name to defaults
Urkem Oct 12, 2023
226d193
Merge remote-tracking branch 'origin/dm2' into ATO-1315-improve-yaml-…
Urkem Oct 12, 2023
8aebb5f
fix test
Urkem Oct 12, 2023
9492b81
add additional slot types
Urkem Oct 12, 2023
a49870d
fix lint
Urkem Oct 12, 2023
dbde8bf
typo
Urkem Oct 12, 2023
da63faf
remove just id step
Urkem Oct 13, 2023
54e613f
add next to schemas
Urkem Oct 13, 2023
2448325
enforce next to none for link steps
Urkem Oct 13, 2023
cd07f27
fix fail messages
Urkem Oct 13, 2023
61a9891
remove intent step
Urkem Oct 13, 2023
5b21911
fix tests
Urkem Oct 13, 2023
72d50dc
fix lint
Urkem Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/12901.improvement.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added Schema file and schema validation for flows.
304 changes: 304 additions & 0 deletions rasa/shared/core/flows/flows_yaml_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,304 @@
{
"type": "object",
"required": [
"flows"
],
"properties": {
"version": {
"type": "string"
},
"flows": {
"type": "object",
"patternProperties": {
"^[A-Za-z_][A-Za-z0-9_]*$": {
"$ref": "#$defs/flow"
}
}
}
},
"$defs": {
"steps": {
"type": "array",
"minContains": 1,
"items": {
"type": "object",
"oneOf": [
{
"required": [
"action"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"action": {
"type": "string"
},
"next": {
"$ref": "#$defs/next"
}
}
},
{
"required": [
"collect"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"description":{
"type": "string"
},
"collect": {
"type": "string"
},
"ask_before_filling": {
"type": "boolean"
},
"reset_after_flow_ends": {
"type": "boolean"
},
"utter": {
"type": "string"
},
"rejections": {
"type": "array",
"items": {
"type": "object",
"required": [
"if",
"utter"
],
"properties": {
"if": {
"type": "string"
},
"utter": {
"type": "string"
}
}
}
},
"next": {
"$ref": "#$defs/next"
}
}
},
{
"required": [
"link"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"link": {
"type": "string"
}
}
},
{
"required": [
"set_slots"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"set_slots": {
"$ref": "#$defs/set_slots"
},
"next": {
"$ref": "#$defs/next"
}
}
},
{
"required": [
"next"
ancalita marked this conversation as resolved.
Show resolved Hide resolved
],
"additionalProperties": false,
"properties": {
"next": {
"$ref": "#$defs/next"
},
"id": {
"type": "string"
}
}
},
{
"required": [
"generation_prompt"
],
"additionalProperties": false,
"properties": {
"generation_prompt": {
"type": "string"
},
"id": {
"type": "string"
}
}
},
{
"required": [
"id"
Urkem marked this conversation as resolved.
Show resolved Hide resolved
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
}
}
},
{
"required": [
"intent"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just double-checking if this is still in use yet, or if it's been discarded, i thought we removed the need for steps of type user message, unless this is something different and related to coexistence?

Copy link
Contributor Author

@Urkem Urkem Oct 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added because of the flow spec specifically 3. Flow Rules 3rd point it's not used anywhere (that I found) same with nlu_trigger maybe they are used for same thing...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, yeah I was also under the impression that we don't use the intent step any longer, but nlu_trigger might be useful in coexistence. Best to check with Daksh/Engine on Slack if this part of the schema is correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we use nlu_trigger now. Although that property hasn't been implemented yet. @twerkmeister would know the implementation details of this.

Copy link
Contributor Author

@Urkem Urkem Oct 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification @dakshvar22 removed the intent step from the schema, the nlu_trigger is already part of the schema here (structure is the same as in the flows spec)

],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"intent": {
"type": "string"
},
"next": {
"$ref": "#$defs/next"
}
}
},
{
"required": [
"entry_prompt"
],
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"entry_prompt": {
"type": "string"
}
}
}
]
}
},
"flow": {
"required": [
"steps"
],
"type": "object",
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"if": {
"type": "string"
},
"name": {
"type": "string"
},
"nlu_trigger": {
"type": "array",
"items": {
"required": [
"intent"
],
"type": "object",
"properties": {
"intent": {
"type": "object",
"properties": {
"confidence_threshold": {
"type": "number"
},
"name": {
"type": "string"
}
},
"required": [
"name"
]
}
}
}
},
"steps": {
"$ref": "#$defs/steps"
}
}
},
"next": {
"anyOf": [
{
"type": "array",
"minContains": 1,
"items": {
"type": "object",
"oneOf": [
ancalita marked this conversation as resolved.
Show resolved Hide resolved
{
"required": [
"if",
"then"
]
},
{
"required": [
"else"
]
}
],
"properties": {
"else": {
"oneOf": [
{
"type": "string"
},
{
"$ref": "#$defs/steps"
}
]
},
"if": {
"type": "string"
},
"then": {
"oneOf": [
{
"$ref": "#$defs/steps"
},
{
"type": "string"
}
]
}
}
}
},
{
"type": "string"
Urkem marked this conversation as resolved.
Show resolved Hide resolved
}
]
},
"set_slots": {
"type": "array",
"items": {
"type": "object",
"patternProperties": {
"^[A-Za-z_][A-Za-z0-9_]*$": {
"type": ["string", "null"]
Urkem marked this conversation as resolved.
Show resolved Hide resolved
}
}
}
}
}
}
8 changes: 0 additions & 8 deletions rasa/shared/core/flows/flows_yaml_schema.yml
View file
Open in desktop

This file was deleted.

7 changes: 5 additions & 2 deletions rasa/shared/core/flows/yaml_flows_io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import textwrap
from pathlib import Path
from typing import List, Text, Union

from rasa.shared.core.flows.utils import KEY_FLOWS

import rasa.shared.utils.io
Expand All @@ -9,7 +10,7 @@

from rasa.shared.core.flows.flow import Flow, FlowsList

FLOWS_SCHEMA_FILE = "/shared/core/flows/flows_yaml_schema.yml"
FLOWS_SCHEMA_FILE = "shared/core/flows/flows_yaml_schema.json"


class YAMLFlowsReader:
Expand Down Expand Up @@ -53,7 +54,9 @@ def read_from_string(cls, string: Text, skip_validation: bool = False) -> FlowsL
`Flow`s read from `string`.
"""
if not skip_validation:
rasa.shared.utils.validation.validate_yaml_schema(string, FLOWS_SCHEMA_FILE)
rasa.shared.utils.validation.validate_yaml_with_jsonschema(
string, FLOWS_SCHEMA_FILE
)

yaml_content = rasa.shared.utils.io.read_yaml(string)

Expand Down
41 changes: 41 additions & 0 deletions rasa/shared/utils/validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,3 +289,44 @@ def validate_training_data_format_version(
docs=DOCS_URL_TRAINING_DATA,
)
return False


def validate_yaml_with_jsonschema(
yaml_file_content: Text, schema_path: Text, package_name: Text = PACKAGE_NAME
) -> None:
"""Validate data format.

Args:
yaml_file_content: the content of the yaml file to be validated
schema_path: the schema of the yaml file
package_name: the name of the package the schema is located in. defaults
to `rasa`.

Raises:
YamlSyntaxException: if the yaml file is not valid.
SchemaValidationError: if validation fails.
"""
from jsonschema import validate, ValidationError
from ruamel.yaml import YAMLError
import pkg_resources

schema_file = pkg_resources.resource_filename(package_name, schema_path)
schema_content = rasa.shared.utils.io.read_json_file(schema_file)

try:
# we need "rt" since
# it will add meta information to the parsed output. this meta information
# will include e.g. at which line an object was parsed. this is very
# helpful when we validate files later on and want to point the user to the
# right line
source_data = rasa.shared.utils.io.read_yaml(
yaml_file_content, reader_type=["safe", "rt"]
)
except (YAMLError, DuplicateKeyError) as e:
raise YamlSyntaxException(underlying_yaml_exception=e)

try:
validate(source_data, schema_content)
except ValidationError as error:
error.message += ". Failed to validate data, make sure your data is valid."
raise SchemaValidationError.create_from(error) from error
Loading
Loading