diff --git a/CHANGELOG.mdx b/CHANGELOG.mdx index 8b4f6e9c9c41..8b4dd86799df 100644 --- a/CHANGELOG.mdx +++ b/CHANGELOG.mdx @@ -832,7 +832,7 @@ Yanked. ### Deprecations and Removals -- [#10989](https://github.com/rasahq/rasa/issues/10989): [NLU training data](./building-classic-assistants/nlu-training-data.mdx) in JSON format is deprecated and will be +- [#10989](https://github.com/rasahq/rasa/issues/10989): [NLU training data](./nlu-based-assistants/nlu-training-data.mdx) in JSON format is deprecated and will be removed in Rasa Open Source 4.0. Please use `rasa data convert nlu -f yaml --data ` to convert your NLU JSON data to YAML format before support for NLU JSON data is removed. @@ -1135,7 +1135,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u - [#8864](https://github.com/rasahq/rasa/issues/8864): Follow through on deprecation warnings for policies and policy ensembles. - [#8867](https://github.com/rasahq/rasa/issues/8867): Follow through on deprecation warnings for `rasa.shared.data`. - [#8868](https://github.com/rasahq/rasa/issues/8868): Follow through on deprecation warnings for the `Domain`. Most importantly this will - enforce the schema of the [`forms` section](./building-classic-assistants/forms.mdx) in the domain file. + enforce the schema of the [`forms` section](./nlu-based-assistants/forms.mdx) in the domain file. This further includes the removal of the `UnfeaturizedSlot` type. - [#8869](https://github.com/rasahq/rasa/issues/8869): Remove deprecated `change_form_to` and `set_form_validation` methods from `DialogueStateTracker`. - [#8870](https://github.com/rasahq/rasa/issues/8870): Remove the support of Markdown training data format. This includes: @@ -1233,7 +1233,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u `Markers` allow you to define points of interest in conversations as a set of conditions that need to be met. A new command `rasa evaluate markers` allows you to apply these conditions to your existing tracker stores and outputs the points at which the conditions were satisfied. -- [#9803](https://github.com/rasahq/rasa/issues/9803): Rasa Open Source now uses the [model configuration](./building-classic-assistants/model-configuration.mdx) to build a +- [#9803](https://github.com/rasahq/rasa/issues/9803): Rasa Open Source now uses the [model configuration](./nlu-based-assistants/model-configuration.mdx) to build a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph). This graph describes the dependencies between the items in your model configuration and @@ -1654,7 +1654,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u Therefore, `model_confidence=linear_norm` is now deprecated and will be removed in Rasa Open Source `3.0.0`. If you were using `model_confidence=linear_norm` for any of the mentioned components, we recommend to revert it back to `model_confidence=softmax` and re-train the assistant. After re-training, - we also recommend to [re-tune the thresholds for fallback components](./building-classic-assistants/fallback-handoff.mdx#fallbacks). + we also recommend to [re-tune the thresholds for fallback components](./nlu-based-assistants/fallback-handoff.mdx#fallbacks). - [#9091](https://github.com/rasahq/rasa/issues/9091): The fallback mechanism for spaCy models has now been removed in Rasa `3.0.0`. @@ -1669,7 +1669,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Features - [#8724](https://github.com/rasahq/rasa/issues/8724): Added `sasl_mechanism` as an optional configurable parameters for the [Kafka Producer](./production/event-brokers.mdx#kafka-event-broker). -- [#8913](https://github.com/rasahq/rasa/issues/8913): Introduces a new policy called [`UnexpecTEDIntentPolicy`](./building-classic-assistants/policies.mdx#unexpected-intent-policy). +- [#8913](https://github.com/rasahq/rasa/issues/8913): Introduces a new policy called [`UnexpecTEDIntentPolicy`](./nlu-based-assistants/policies.mdx#unexpected-intent-policy). `UnexpecTEDIntentPolicy` helps you review conversations and also allows your bot to react to unexpected user turns in conversations. @@ -1691,7 +1691,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u As part of the feature, it also introduces: - - [`IntentMaxHistoryTrackerFeaturizer`](./building-classic-assistants/policies.mdx#3-intent-max-history) + - [`IntentMaxHistoryTrackerFeaturizer`](./nlu-based-assistants/policies.mdx#3-intent-max-history) to featurize the trackers for `UnexpecTEDIntentPolicy`. - `MultiLabelDotProductLoss` to support `UnexpecTEDIntentPolicy`'s multi-label training objective. - A new default action called [`action_unlikely_intent`](./concepts/default-actions.mdx#action_unlikely_intent). @@ -1724,8 +1724,8 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u models - `DIETClassifier`, `ResponseSelector`. In other words, no extra buffer is created in advance for additional vocabulary items and space will be dynamically allocated for them inside the model. - This means there's no need to specify `additional_vocabulary_size` for [`CountVectorsFeaturizer`](./building-classic-assistants/components.mdx#countvectorsfeaturizer) or - `number_additional_patterns` for [`RegexFeaturizer`](./building-classic-assistants/components.mdx#regexfeaturizer). These parameters are now deprecated. + This means there's no need to specify `additional_vocabulary_size` for [`CountVectorsFeaturizer`](./nlu-based-assistants/components.mdx#countvectorsfeaturizer) or + `number_additional_patterns` for [`RegexFeaturizer`](./nlu-based-assistants/components.mdx#regexfeaturizer). These parameters are now deprecated. **Before** @@ -1750,9 +1750,9 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u Also, all custom layers specifically built for machine learning models - `RasaSequenceLayer`, `RasaFeatureCombiningLayer` and `ConcatenateSparseDenseFeatures` now inherit from `RasaCustomLayer` so that they support flexible incremental training out of the box. -- [#8295](https://github.com/rasahq/rasa/issues/8295): Speed up the contradiction check of the [`RulePolicy`](./building-classic-assistants/policies.mdx#rule-policy) +- [#8295](https://github.com/rasahq/rasa/issues/8295): Speed up the contradiction check of the [`RulePolicy`](./nlu-based-assistants/policies.mdx#rule-policy) by a factor of 3. -- [#8801](https://github.com/rasahq/rasa/issues/8801): Change the confidence score assigned by [`FallbackClassifier`](./building-classic-assistants/components.mdx#fallbackclassifier) to fallback intent to be the same as the fallback threshold. +- [#8801](https://github.com/rasahq/rasa/issues/8801): Change the confidence score assigned by [`FallbackClassifier`](./nlu-based-assistants/components.mdx#fallbackclassifier) to fallback intent to be the same as the fallback threshold. - [#8926](https://github.com/rasahq/rasa/issues/8926): Issue a UserWarning if a specified **domain folder** contains files that look like YML files but cannot be parsed successfully. Only invoked if user specifies a folder path in `--domain` paramater. Previously those invalid files in the specified folder were silently ignored. **Does not apply** to individually specified domain YAML files, e.g. `--domain /some/path/domain.yml`, those being invalid will still raise an exception. @@ -1827,10 +1827,10 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Bugfixes -- [#8364](https://github.com/rasahq/rasa/issues/8364): Fixed a bug where [`ListSlot`](./building-classic-assistants/domain.mdx#list-slot)s were filled with single items +- [#8364](https://github.com/rasahq/rasa/issues/8364): Fixed a bug where [`ListSlot`](./nlu-based-assistants/domain.mdx#list-slot)s were filled with single items in case only one matching entity was extracted for this slot. - Values applied to [`ListSlot`](./building-classic-assistants/domain.mdx#list-slot)s will be converted to a `List` + Values applied to [`ListSlot`](./nlu-based-assistants/domain.mdx#list-slot)s will be converted to a `List` in case they aren't one. - [#8581](https://github.com/rasahq/rasa/issues/8581): Fix bug with false rule conflicts @@ -1886,7 +1886,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u - [#261](https://github.com/rasahq/rasa/issues/261): Added an optional `ignored_intents` parameter in forms. - - To use it, add the `ignored_intents` parameter in your `domain.yml` file after the forms name and provide a list of intents to ignore. Please see [Forms](./building-classic-assistants/forms.mdx) for more information. + - To use it, add the `ignored_intents` parameter in your `domain.yml` file after the forms name and provide a list of intents to ignore. Please see [Forms](./nlu-based-assistants/forms.mdx) for more information. - This can be used in case the user never wants to fill any slots of a form with the specified intent, e.g. chitchat. - [#5786](https://github.com/rasahq/rasa/issues/5786): Add function to carry `max_history` to featurizer @@ -1992,7 +1992,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u specify a `model`. This fallback behavior is temporary and will be deprecated in Rasa 3.0.0. We've updated our docs to reflect these changes. All examples now show a direct link to the - correct spaCy model. We've also added a warning to the [SpaCyNLP](./building-classic-assistants/components.mdx#spacynlp) + correct spaCy model. We've also added a warning to the [SpaCyNLP](./nlu-based-assistants/components.mdx#spacynlp) docs that explains the fallback behavior. ### Improvements @@ -2104,8 +2104,8 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u Response Selector - - The field `template_name` will be deprecated in Rasa Open Source 3.0.0. Please use `utter_action` instead. Please see [here](./building-classic-assistants/components.mdx#selectors) for more details. - - The field `response_templates` will be deprecated in Rasa Open Source 3.0.0. Please use `responses` instead. Please see [here](./building-classic-assistants/components.mdx#selectors) for more details. + - The field `template_name` will be deprecated in Rasa Open Source 3.0.0. Please use `utter_action` instead. Please see [here](./nlu-based-assistants/components.mdx#selectors) for more details. + - The field `response_templates` will be deprecated in Rasa Open Source 3.0.0. Please use `responses` instead. Please see [here](./nlu-based-assistants/components.mdx#selectors) for more details. ### Improvements @@ -2244,11 +2244,11 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u constrain_similarities: True ``` - This should ease up [tuning fallback thresholds](./building-classic-assistants/fallback-handoff.mdx#fallbacks) as confidences for wrong predictions are better distributed across the range `[0, 1]`. + This should ease up [tuning fallback thresholds](./nlu-based-assistants/fallback-handoff.mdx#fallbacks) as confidences for wrong predictions are better distributed across the range `[0, 1]`. If you trained a model with `model_confidence=cosine` or `model_confidence=inner` setting using previous versions of Rasa Open Source, please re-train by either removing the `model_confidence` option from the configuration or setting it to `linear_norm`. - `model_confidence=cosine` is removed from the configuration generated by [auto-configuration](./building-classic-assistants/model-configuration.mdx#suggested-config). + `model_confidence=cosine` is removed from the configuration generated by [auto-configuration](./nlu-based-assistants/model-configuration.mdx#suggested-config). ## [2.3.3] - 2021-02-25 @@ -2327,7 +2327,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ``` - [#7579](https://github.com/rasahq/rasa/issues/7579): Add BILOU tagging schema for entity extraction in end-to-end TEDPolicy. -- [#7616](https://github.com/rasahq/rasa/issues/7616): Added two new parameters `constrain_similarities` and `model_confidence` to machine learning (ML) components - [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier), [ResponseSelector](./building-classic-assistants/components.mdx#dietclassifier) and [TEDPolicy](./building-classic-assistants/policies.mdx#ted-policy). +- [#7616](https://github.com/rasahq/rasa/issues/7616): Added two new parameters `constrain_similarities` and `model_confidence` to machine learning (ML) components - [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier), [ResponseSelector](./nlu-based-assistants/components.mdx#dietclassifier) and [TEDPolicy](./nlu-based-assistants/policies.mdx#ted-policy). Setting `constrain_similarities=True` adds a sigmoid cross-entropy loss on all similarity values to restrict them to an approximate range in `DotProductLoss`. This should help the models to perform better on real world test sets. By default, the parameter is set to `False` to preserve the old behaviour, but users are encouraged to set it to `True` and re-train their assistants as it will be set to `True` by default from Rasa Open Source 3.0.0 onwards. @@ -2353,7 +2353,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u Configuration option `loss_type=softmax` is now deprecated and will be removed in Rasa Open Source 3.0.0 . Use `loss_type=cross_entropy` instead. - The default [auto-configuration](./building-classic-assistants/model-configuration.mdx#suggested-config) is changed to use `constrain_similarities=True` and `model_confidence=cosine` in ML components so that new users start with the recommended configuration. + The default [auto-configuration](./nlu-based-assistants/model-configuration.mdx#suggested-config) is changed to use `constrain_similarities=True` and `model_confidence=cosine` in ML components so that new users start with the recommended configuration. **EDIT**: Some post-release experiments revealed that using `model_confidence=cosine` is wrong as it can change the order of predicted labels. That's why this option was removed in Rasa Open Source version `2.3.3`. `model_confidence=inner` is deprecated as it produces an unbounded range of confidences which can break the logic of assistants in various other places. Please use `model_confidence=linear_norm` which will produce a linearly normalized version of dot product similarities with each value in the range `[0,1]`. Please read more about this change under the notes for release `2.3.4`. @@ -2417,7 +2417,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Bugfixes -- [#7764](https://github.com/rasahq/rasa/issues/7764): Fixes a bug in [forms](./building-classic-assistants/forms.mdx) where the next slot asked was not consistent after returning to a form from an unhappy path. +- [#7764](https://github.com/rasahq/rasa/issues/7764): Fixes a bug in [forms](./nlu-based-assistants/forms.mdx) where the next slot asked was not consistent after returning to a form from an unhappy path. ## [2.2.7] - 2021-01-25 @@ -2454,7 +2454,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Improvements -- [#7520](https://github.com/rasahq/rasa/issues/7520): Improve the warning in case the [RulePolicy](./building-classic-assistants/policies.mdx#rule-policy) or the deprecated +- [#7520](https://github.com/rasahq/rasa/issues/7520): Improve the warning in case the [RulePolicy](./nlu-based-assistants/policies.mdx#rule-policy) or the deprecated [MappingPolicy](https://rasa.com/docs/rasa/2.x/policies#mapping-policy) are missing from the model's `policies` configuration. Changed the info log to a warning as one of this policies should be added to the model configuration. @@ -2491,7 +2491,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Bugfixes -- [#7557](https://github.com/rasahq/rasa/issues/7557): Fixed a problem where a [form](./building-classic-assistants/forms.mdx) wouldn't reject when the +- [#7557](https://github.com/rasahq/rasa/issues/7557): Fixed a problem where a [form](./nlu-based-assistants/forms.mdx) wouldn't reject when the `FormValidationAction` re-implemented `required_slots`. - [#7585](https://github.com/rasahq/rasa/issues/7585): Fixed an error when using the [SQLTrackerStore](./production/tracker-stores.mdx#sqltrackerstore) with a Postgres database and the parameter `login_db` specified. @@ -2550,8 +2550,8 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u training](./command-line-interface.mdx#incremental-training). Added a configuration parameter `additional_vocabulary_size` to - [`CountVectorsFeaturizer`](./building-classic-assistants/components.mdx#countvectorsfeaturizer) - and `number_additional_patterns` to [`RegexFeaturizer`](./building-classic-assistants/components.mdx#regexfeaturizer). + [`CountVectorsFeaturizer`](./nlu-based-assistants/components.mdx#countvectorsfeaturizer) + and `number_additional_patterns` to [`RegexFeaturizer`](./nlu-based-assistants/components.mdx#regexfeaturizer). These parameters are useful to configure when using incremental training for your pipelines. - [#7408](https://github.com/rasahq/rasa/issues/7408): Add the option to use cross-validation to the @@ -2567,7 +2567,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u a callback URL in the query parameter `callback_url` which Rasa Open Source should send the results to. This URL will also be called in case of errors. -- [#7496](https://github.com/rasahq/rasa/issues/7496): Make [TED Policy](./building-classic-assistants/policies.mdx#ted-policy) an end-to-end policy. Namely, make it possible to train TED on stories that contain +- [#7496](https://github.com/rasahq/rasa/issues/7496): Make [TED Policy](./nlu-based-assistants/policies.mdx#ted-policy) an end-to-end policy. Namely, make it possible to train TED on stories that contain intent and entities or user text and bot actions or bot text. If you don't have text in your stories, TED will behave the same way as before. Add possibility to predict entities using TED. @@ -2631,7 +2631,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ::: - [#7306](https://github.com/rasahq/rasa/issues/7306): Fix an erroneous attribute for Redis key prefix in `rasa.core.tracker_store.RedisTrackerStore`: 'RedisTrackerStore' object has no attribute 'prefix'. - [#7407](https://github.com/rasahq/rasa/issues/7407): Remove token when its text (for example, whitespace) can't be tokenized by LM tokenizer (from `LanguageModelFeaturizer`). -- [#7408](https://github.com/rasahq/rasa/issues/7408): Temporary directories which were created during requests to the [HTTP API](http-api.mdx) +- [#7408](https://github.com/rasahq/rasa/issues/7408): Temporary directories which were created during requests to the [HTTP API](./production/http-api.mdx) are now cleaned up correctly once the request was processed. - [#7422](https://github.com/rasahq/rasa/issues/7422): Add option `use_word_boundaries` for `RegexFeaturizer` and `RegexEntityExtractor`. To correctly process languages such as Chinese that don't use whitespace for word separation, the user needs to add the `use_word_boundaries: False` option to those two components. - [#7529](https://github.com/rasahq/rasa/issues/7529): Correctly fingerprint the default domain slots. Previously this led to the issue @@ -2692,7 +2692,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Deprecations and Removals -- [#7136](https://github.com/rasahq/rasa/issues/7136): The [`Policy`](./building-classic-assistants/policies.mdx) interface was changed to return a `PolicyPrediction` object when +- [#7136](https://github.com/rasahq/rasa/issues/7136): The [`Policy`](./nlu-based-assistants/policies.mdx) interface was changed to return a `PolicyPrediction` object when `predict_action_probabilities` is called. Returning a list of probabilities directly is deprecated and support for this will be removed in Rasa Open Source 3.0. @@ -2756,14 +2756,14 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ### Features -- [#7136](https://github.com/rasahq/rasa/issues/7136): [Policies](./building-classic-assistants/policies.mdx) can now return obligatory and optional events as part of their +- [#7136](https://github.com/rasahq/rasa/issues/7136): [Policies](./nlu-based-assistants/policies.mdx) can now return obligatory and optional events as part of their prediction. Obligatory events are always applied to the current conversation tracker. Optional events are only applied to the conversation tracker in case the policy wins. ### Improvements - [#4341](https://github.com/rasahq/rasa/issues/4341): Changed `Agent.load` method to support `pathlib` paths. -- [#5715](https://github.com/rasahq/rasa/issues/5715): If you are using the feature [Entity Roles and Groups](./building-classic-assistants/nlu-training-data.mdx#entities-roles-and-groups), you should now also list the roles and groups +- [#5715](https://github.com/rasahq/rasa/issues/5715): If you are using the feature [Entity Roles and Groups](./nlu-based-assistants/nlu-training-data.mdx#entities-roles-and-groups), you should now also list the roles and groups in your domain file if you want roles and groups to influence your conversations. For example: ```yaml-rasa @@ -2784,13 +2784,13 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u ``` Entity roles and groups can now influence dialogue predictions. For more information see the section - [Entity Roles and Groups influencing dialogue predictions](./building-classic-assistants/nlu-training-data.mdx#entity-roles-and-groups-influencing-dialogue-predictions). + [Entity Roles and Groups influencing dialogue predictions](./nlu-based-assistants/nlu-training-data.mdx#entity-roles-and-groups-influencing-dialogue-predictions). -- [#6285](https://github.com/rasahq/rasa/issues/6285): Predictions of the [`FallbackClassifier`](./building-classic-assistants/components.mdx#fallbackclassifier) are +- [#6285](https://github.com/rasahq/rasa/issues/6285): Predictions of the [`FallbackClassifier`](./nlu-based-assistants/components.mdx#fallbackclassifier) are ignored when - [evaluating the NLU model](./building-classic-assistants/testing-your-assistant.mdx#evaluating-an-nlu-model) + [evaluating the NLU model](./nlu-based-assistants/testing-your-assistant.mdx#evaluating-an-nlu-model) Note that the `FallbackClassifier` predictions still apply to - [test stories](./building-classic-assistants/testing-your-assistant.mdx#writing-test-stories). + [test stories](./nlu-based-assistants/testing-your-assistant.mdx#writing-test-stories). - [#6474](https://github.com/rasahq/rasa/issues/6474): Adapt the training data reader and emulator for wit.ai to their latest format. - [#6498](https://github.com/rasahq/rasa/issues/6498): Adding configurable prefixes to Redis [Tracker](./production/tracker-stores.mdx) and [Lock Stores](./production/lock-stores.mdx) so that a single Redis instance (and logical DB) can support multiple conversation trackers and locks. By default, conversations will be prefixed with `tracker:...` and all locks prefixed with `lock:...`. Additionally, you can add an alphanumeric-only `prefix: value` in `endpoints.yml` such that keys in redis will take the form `value:tracker:...` and `value:lock:...` respectively. @@ -2825,7 +2825,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u [conversation session](./concepts/domain.mdx#session-configuration). Added a new boolean query parameter `all_sessions` (default: `false`) to the - [HTTP API](./http-api.mdx) endpoint for fetching test stories + [HTTP API](./production/http-api.mdx) endpoint for fetching test stories (`GET /conversations//story`). When setting `?all_sessions=true`, the endpoint returns test stories for all @@ -2841,8 +2841,8 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u - [#6865](https://github.com/rasahq/rasa/issues/6865): Improve the `rasa data convert nlg` command and introduce the `rasa data convert responses` command to simplify the migration from pre-2.0 response selector format to the new format. -- [#6966](https://github.com/rasahq/rasa/issues/6966): Added warning for when an option is provided for a [component](./building-classic-assistants/components.mdx) that is not listed as a key in the defaults for that component. -- [#6977](https://github.com/rasahq/rasa/issues/6977): [Forms](./building-classic-assistants/forms.mdx) no longer reject their execution before a potential custom +- [#6966](https://github.com/rasahq/rasa/issues/6966): Added warning for when an option is provided for a [component](./nlu-based-assistants/components.mdx) that is not listed as a key in the defaults for that component. +- [#6977](https://github.com/rasahq/rasa/issues/6977): [Forms](./nlu-based-assistants/forms.mdx) no longer reject their execution before a potential custom action for validating / extracting slots was executed. Forms continue to reject in two cases automatically: @@ -2863,9 +2863,9 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u of the stack and can be used with any other `Tokenizer`. - [#7061](https://github.com/rasahq/rasa/issues/7061): Gray out "Download" button in Rasa Playground when the project is not yet ready to be downloaded. -- [#7068](https://github.com/rasahq/rasa/issues/7068): Slot mappings for [Forms](./building-classic-assistants/forms.mdx) in the domain are now optional. If you do not +- [#7068](https://github.com/rasahq/rasa/issues/7068): Slot mappings for [Forms](./nlu-based-assistants/forms.mdx) in the domain are now optional. If you do not provide any slot mappings as part of the domain, you need to provide - [custom slot mappings](./building-classic-assistants/forms.mdx#custom-slot-mappings) through a custom action. + [custom slot mappings](./nlu-based-assistants/forms.mdx#custom-slot-mappings) through a custom action. A form without slot mappings is specified as follows: ```rasa-yaml @@ -2874,7 +2874,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u # no mappings ``` - The action for [forms](./building-classic-assistants/forms.mdx) can now be overridden by defining a custom action + The action for [forms](./nlu-based-assistants/forms.mdx) can now be overridden by defining a custom action with the same name as the form. This can be used to keep using the deprecated Rasa Open Source `FormAction` which is implemented within the Rasa SDK. Note that it is **not** recommended to override the form action for anything else than using the @@ -2886,7 +2886,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u as the default model. These model weights should help improve performance on intent classification and response selection. -- [#7122](https://github.com/rasahq/rasa/issues/7122): Add validations for [slot mappings](./building-classic-assistants/forms.mdx#slot-mappings). +- [#7122](https://github.com/rasahq/rasa/issues/7122): Add validations for [slot mappings](./nlu-based-assistants/forms.mdx#slot-mappings). If a slot mapping is not valid, an `InvalidDomain` error is raised. - [#7132](https://github.com/rasahq/rasa/issues/7132): Adapt the training data reader and emulator for LUIS to [their latest format](https://westcentralus.dev.cognitive.microsoft.com/docs/services/luis-endpoint-api-v3-0/) @@ -2897,7 +2897,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u - [#7263](https://github.com/rasahq/rasa/issues/7263): The [Pika Event Broker](./production/event-brokers.mdx#pika-event-broker) was reimplemented with the `[aio-pika` library[(https://aio-pika.readthedocs.io/en/latest/). Messages will now be published to RabbitMQ asynchronously which improves the prediction performance. -- [#7278](https://github.com/rasahq/rasa/issues/7278): The confidence of the [`FallbackClassifier`](./building-classic-assistants/components.mdx#fallbackclassifier) +- [#7278](https://github.com/rasahq/rasa/issues/7278): The confidence of the [`FallbackClassifier`](./nlu-based-assistants/components.mdx#fallbackclassifier) predictions is set to `1 - top intent confidence`. ### Bugfixes @@ -2910,7 +2910,7 @@ Upgrade dependent libraries with security vulnerabilities (Pillow, TensorFlow, u - [#6721](https://github.com/rasahq/rasa/issues/6721): Fixed bug where slots with `influence_conversation=false` affected the action prediction if they were set manually using the `POST /conversations/`. This graph component is run during training and @@ -373,21 +373,21 @@ to specify the following details: retrieve the model which should be provided to dependent graph components. - `ComponentType.MESSAGE_TOKENIZER`: Component type for - [tokenizers](../../building-classic-assistants/components.mdx#tokenizers). This graph component is run during training and + [tokenizers](../../nlu-based-assistants/components.mdx#tokenizers). This graph component is run during training and inference. Rasa will use the graph component's `train` method if `is_trainable=True` is specified. Rasa will use `process_training_data` for tokenizing training data examples and `process` to tokenize messages during inference. - `ComponentType.MESSAGE_FEATURIZER`: Component type for - [featurizers](../../building-classic-assistants/components.mdx#featurizers). This graph component is run during training and + [featurizers](../../nlu-based-assistants/components.mdx#featurizers). This graph component is run during training and inference. Rasa will use the graph component's `train` method if `is_trainable=True` is specified. Rasa will use `process_training_data` for featurizing training data examples and `process` to featurize messages during inference. - `ComponentType.INTENT_CLASSIFIER`: Component type for - [intent classifiers](../../building-classic-assistants/components.mdx#intent-classifiers). This graph component is run only + [intent classifiers](../../nlu-based-assistants/components.mdx#intent-classifiers). This graph component is run only during training if `is_trainable=True`. The graph component is always run during inference. Rasa will use the graph component's `train` method if @@ -396,7 +396,7 @@ to specify the following details: inference. - `ComponentType.ENTITY_EXTRACTOR`: Component type for - [entity extractors](../../building-classic-assistants/components.mdx#entity-extractors). This graph component is run only + [entity extractors](../../nlu-based-assistants/components.mdx#entity-extractors). This graph component is run only during training if `is_trainable=True`. The graph component is always run during inference. Rasa will use the graph component's `train` method if @@ -405,7 +405,7 @@ to specify the following details: - `ComponentType.POLICY_WITHOUT_END_TO_END_SUPPORT`: Component type for [policies](../policies.mdx) which don't require additional end-to-end features - (see [end-to-end training](../../building-classic-assistants/stories.mdx#end-to-end-training) for more information). + (see [end-to-end training](../../nlu-based-assistants/stories.mdx#end-to-end-training) for more information). This graph component is run only during training if `is_trainable=True`. The graph component is always run during inference. Rasa will use the graph component's `train` method if @@ -415,7 +415,7 @@ to specify the following details: - `ComponentType.POLICY_WITH_END_TO_END_SUPPORT`: Component type for [policies](../policies.mdx) which require additional end-to-end features - (see [end-to-end training](../../building-classic-assistants/stories.mdx#end-to-end-training) for more information). + (see [end-to-end training](../../nlu-based-assistants/stories.mdx#end-to-end-training) for more information). The end-to-end features are passed into the graph component's `train` and `predict_action_probabilities` as `precomputations` parameter. This graph component is run only during training if `is_trainable=True`. @@ -430,17 +430,17 @@ to specify the following details: predictions - `model_from`: Specifies if a pretrained - [language model](../../building-classic-assistants/components.mdx#language-models) needs to be provided to the `train`, + [language model](../../nlu-based-assistants/components.mdx#language-models) needs to be provided to the `train`, `process_training_data` and `process` methods of the graph component. These methods have to support the parameter `model` to receive the language model. Note that you still need to make sure that the graph component which provides this model is part of your model configuration. A common use case for this is if you want to expose the - [SpacyNLP](../../building-classic-assistants/components.mdx#spacynlp) language model to your other NLU components. + [SpacyNLP](../../nlu-based-assistants/components.mdx#spacynlp) language model to your other NLU components. ## Using Custom Components in your Model Configuration You can use custom graph components like any other NLU component or policy within your -[model configuration](../../building-classic-assistants/model-configuration.mdx). The only change is that you have to specify +[model configuration](../../nlu-based-assistants/model-configuration.mdx). The only change is that you have to specify the full module name instead of the class name only. The full module name depends on your module's location in relation to the specified [PYTHONPATH](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH). @@ -468,7 +468,7 @@ policies: ### Message Metadata -When you [define metadata for your intent examples in your training data](../../building-classic-assistants/training-data-format.mdx#training-examples), +When you [define metadata for your intent examples in your training data](../../nlu-based-assistants/training-data-format.mdx#training-examples), your NLU component can access both the intent metadata and the intent example metadata during processing: ```python @@ -493,7 +493,7 @@ The sentence features are represented by a matrix of size `(1 x feature-dimensio ### Dense Message Featurizer -The following is the example of a dense [message featurizer](../../building-classic-assistants/components.mdx#featurizers) +The following is the example of a dense [message featurizer](../../nlu-based-assistants/components.mdx#featurizers) which uses a pretrained model: ```python (docs/sources/data/test_classes/custom_graph_components/nlu_dense.py) @@ -502,7 +502,7 @@ which uses a pretrained model: ### Sparse Message Featurizer -The following is the example of a dense [message featurizer](../../building-classic-assistants/components.mdx#featurizers) +The following is the example of a dense [message featurizer](../../nlu-based-assistants/components.mdx#featurizers) which trains a new model: ```python (docs/sources/data/test_classes/custom_graph_components/nlu_sparse.py) diff --git a/docs/docs/llms/llm-configuration.mdx b/docs/docs/concepts/components/llm-configuration.mdx similarity index 94% rename from docs/docs/llms/llm-configuration.mdx rename to docs/docs/concepts/components/llm-configuration.mdx index 747369ced6a5..f196969ea19b 100644 --- a/docs/docs/llms/llm-configuration.mdx +++ b/docs/docs/concepts/components/llm-configuration.mdx @@ -9,13 +9,6 @@ abstract: | efficiently with your specific use case. --- -import RasaLabsLabel from "@theme/RasaLabsLabel"; -import RasaLabsBanner from "@theme/RasaLabsBanner"; - - - - - ## Overview This guide will walk you through the process of configuring Rasa to connect to an @@ -49,10 +42,10 @@ nlg: Additional configuration parameters are explained in detail in the documentation pages for each of these components: -- [LLMCommandGenerator](../concepts/dialogue-understanding.mdx) -- [FlowPolicy](../concepts/policies.mdx#flow-policy) +- [LLMCommandGenerator](../dialogue-understanding.mdx) +- [FlowPolicy](../policies.mdx#flow-policy) - Docseacrh -- [ContextualResponseRephraser](../concepts/contextual-response-rephraser.mdx) +- [ContextualResponseRephraser](../contextual-response-rephraser.mdx) ## OpenAI Configuration @@ -115,9 +108,9 @@ you might use one model for intent classification and another for rephrasing. To configure models per component, follow these steps described on the pages for each component: -1. [intent classification instructions](../building-classic-assistants/components.mdx#llmintentclassifier) -2. [rephrasing instructions](../concepts/contextual-response-rephraser.mdx#llm-configuration) -3. [intentless policy instructions](../concepts/policies.mdx#flow-policy) +1. [intent classification instructions](../../nlu-based-assistants/components.mdx#llmintentclassifier) +2. [rephrasing instructions](../contextual-response-rephraser.mdx#llm-configuration) +3. [intentless policy instructions](../policies.mdx#flow-policy) 4. docsearch instructions ### Additional Configuration for Azure OpenAI Service @@ -338,13 +331,13 @@ pipeline: type: "cohere" ``` -The above configuration specifies that the [LLMIntentClassifier](../building-classic-assistants/components.mdx#llmintentclassifier) +The above configuration specifies that the [LLMIntentClassifier](../../nlu-based-assistants/components.mdx#llmintentclassifier) should use the [Cohere](https://cohere.ai/) embeddings provider rather than OpenAI. :::note Only Some Components need Embeddings Not every component uses embeddings. For example, the -[ContextualResponseRephraser](../concepts/contextual-response-rephraser.mdx) component does not use embeddings. +[ContextualResponseRephraser](../contextual-response-rephraser.mdx) component does not use embeddings. For these components, no `embeddings` property is needed. ::: @@ -438,7 +431,8 @@ embeddings: ## FAQ ### Does OpenAI use my data to train their models? + No. OpenAI does not use your data to train their models. From their [website](https://openai.com/security): -> Data submitted through the OpenAI API is not used to train OpenAI +> Data submitted through the OpenAI API is not used to train OpenAI > models or improve OpenAI's service offering. diff --git a/docs/docs/llms/llm-custom.mdx b/docs/docs/concepts/components/llm-custom.mdx similarity index 95% rename from docs/docs/llms/llm-custom.mdx rename to docs/docs/concepts/components/llm-custom.mdx index ea3d91baa2da..895eb55b342c 100644 --- a/docs/docs/llms/llm-custom.mdx +++ b/docs/docs/concepts/components/llm-custom.mdx @@ -7,10 +7,8 @@ abstract: import RasaLabsLabel from "@theme/RasaLabsLabel"; import RasaLabsBanner from "@theme/RasaLabsBanner"; - - - - +import RasaProLabel from "@theme/RasaProLabel"; +import RasaProBanner from "@theme/RasaProBanner"; The LLM components can be extended and modified with custom versions. This allows you to customize the behavior of the LLM components to your needs and @@ -19,10 +17,10 @@ experiment with different algorithms. ## Customizing a component The LLM components are implemented as a set of classes that can be extended -and modified. The following example shows how to extend the +and modified. The following example shows how to extend the `LLMIntentClassifier` component to add a custom behavior. -For example, we can change the logic that selects the intent labels that are +For example, we can change the logic that selects the intent labels that are included in the prompt to the LLM model. By default, we only include a selection of the available intents in the prompt. But we can also include all available intents in the prompt. This can be done by extending the `LLMIntentClassifier` @@ -44,7 +42,7 @@ class CustomLLMIntentClassifier(LLMIntentClassifier): Returns: The list of intent examples to use for the LLM training. """ - + # use all available intents for the LLM prompt return list(self.available_intents) ``` @@ -61,18 +59,23 @@ To reference a component in the Rasa configuration file, you need to use the full name of the component class. The full name of the component class is `.`. -All components are well documented in their source code. The code can -be found in your local installation of the `rasa_plus` python package. +All components are well documented in their source code. The code can +be found in your local installation of the `rasa_plus` python package. ## Common functions to be overridden + Below is a list of functions that could be overwritten to customize the LLM components: ### LLMIntentClassifier + + + + #### select_intent_examples -Selects the intent examples to use for the LLM prompt. The selected intent +Selects the intent examples to use for the LLM prompt. The selected intent labels are included in the generation prompt. By default, only the intent labels that are used in the few shot examples are included in the prompt. @@ -103,7 +106,8 @@ labels that are used in the few shot examples are included in the prompt. ``` #### closest_intent_from_training_data -The LLM generates an intent label which + +The LLM generates an intent label which might not always be part of the domain. This function can be used to map the generated intent label to an intent label that is part of the domain. @@ -126,7 +130,7 @@ labels from the domain and returns the closest intent label from the domain. Selects the NLU training examples that are included in the LLM prompt. The selected examples are included in the prompt to help the LLM to generate the -correct intent. By default, the most similar training examples are selected. +correct intent. By default, the most similar training examples are selected. The selection is based on the message that should be classified. The most similar examples are selected by embedding the incoming message, all training examples and doing a similarity search. @@ -149,6 +153,10 @@ examples and doing a similarity search. ### ContextualResponseRephraser + + + + #### rephrase Rephrases the response generated by the LLM. The default implementation @@ -176,11 +184,15 @@ replaced with the generated response. ### IntentlessPolicy + + + + #### select_response_examples Samples responses that fit the current conversation. The default implementation samples responses from the domain that fit the current conversation. -The selection is based on the conversation history, the history will be +The selection is based on the conversation history, the history will be embedded and the most similar responses will be selected. ```python @@ -229,4 +241,4 @@ embedded and the most similar conversations will be selected. Returns: The sampled conversation ordered by similarity decrease. """ -``` \ No newline at end of file +``` diff --git a/docs/docs/concepts/contextual-response-rephraser.mdx b/docs/docs/concepts/contextual-response-rephraser.mdx index 773147d99122..9bb3098d3f50 100644 --- a/docs/docs/concepts/contextual-response-rephraser.mdx +++ b/docs/docs/concepts/contextual-response-rephraser.mdx @@ -13,11 +13,11 @@ import RasaProBanner from "@theme/RasaProBanner"; - + :::info New in 3.7 -The *Contextual Respones Rephrasing* is part of Rasa's new +The _Contextual Response Rephrasing_ is part of Rasa's new Conversational AI with Language Models (CALM) approach and available starting with version `3.7.0`. ::: @@ -165,7 +165,7 @@ model using the completions API of If you want to use Azure OpenAI Service, you can configure the necessary parameters as described in the -[Azure OpenAI Service](../llms/llm-configuration.mdx#additional-configuration-for-azure-openai-service) +[Azure OpenAI Service](./components/llm-configuration.mdx#additional-configuration-for-azure-openai-service) section. :::info Using Other LLMs @@ -183,7 +183,7 @@ nlg: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings) +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings) ::: diff --git a/docs/docs/concepts/unhappy-paths.mdx b/docs/docs/concepts/conversation-repair.mdx similarity index 95% rename from docs/docs/concepts/unhappy-paths.mdx rename to docs/docs/concepts/conversation-repair.mdx index 519568848bba..687dcfd83573 100644 --- a/docs/docs/concepts/unhappy-paths.mdx +++ b/docs/docs/concepts/conversation-repair.mdx @@ -1,27 +1,23 @@ --- -id: unhappy-paths -sidebar_label: Unhappy Paths -title: Unhappy Paths +id: conversation-repair +sidebar_label: Conversation Repair +title: Conversation Repair abstract: | Discover how to navigate situations where users deviate from a flow's prescribed business logic. --- -import RasaDiscoveryBanner from "@theme/RasaDiscoveryBanner"; - - - ## Overview [Flows](../concepts/flows.mdx) offer structured way to design conversation-driven business logic. While -flow designers primarily focus on linear [happy paths](../building-classic-assistants/glossary.mdx#happy--unhappy-paths), -Rasa provides a robust way to manage deviations, simplifying design complexity. +flow designers primarily focus on linear [happy paths](../nlu-based-assistants/glossary.mdx#happy-path--conversation-repair), +Rasa provides a robust way to manage conversation repair, simplifying design complexity. ### Default Behaviors -Rasa ships a **default behavior for every unhappy path**. Override +Rasa ships a **default behavior for every conversation repair case**. Override defaults by creating a flows with the same name of a pattern, like `pattern_correction`. -## Unhappy Path Types +## Conversation Repair Cases ### 1. Digressions @@ -136,7 +132,7 @@ Errors arise from unexpected system or flow issues. ## Configurations -Each of these unhappy paths can be configured using Rasa's YAML-based +Each of these conversation repair cases can be configured using Rasa's YAML-based domain and flow system. You can overwrite a default behavior by creating a flow with the same name as the pattern: @@ -287,7 +283,7 @@ was interrupted is called `transfer_money`. ## Reference: Default Pattern Configuration -For reference, here is the complete default configuration for unhappy paths: +For reference, here is the complete default configuration for conversation repair: ```yaml title="Default Patterns" version: "3.1" diff --git a/docs/docs/concepts/default-actions.mdx b/docs/docs/concepts/default-actions.mdx index c90f531228c1..9e24fe8eb1ed 100644 --- a/docs/docs/concepts/default-actions.mdx +++ b/docs/docs/concepts/default-actions.mdx @@ -50,7 +50,7 @@ This action resets the whole conversation history, including any slots that were set during it. It can be triggered by the user in a conversation by sending a -"/restart" message, if the [RulePolicy](../building-classic-assistants/rules.mdx) is included in the model configuration. +"/restart" message, if the [RulePolicy](../nlu-based-assistants/rules.mdx) is included in the model configuration. If you define an `utter_restart` response in your domain, this will be sent to the user as well. ## action_session_start @@ -139,12 +139,12 @@ class ActionSessionStart(Action): ## action_default_fallback This action undoes the last user-bot interaction and sends the `utter_default` response if it is defined. -It is triggered by low action prediction confidence, if you have this [fallback mechanism](../building-classic-assistants/fallback-handoff.mdx) enabled. +It is triggered by low action prediction confidence, if you have this [fallback mechanism](../nlu-based-assistants/fallback-handoff.mdx) enabled. ## action_deactivate_loop This action deactivates the active loop and resets the requested slot. This is used when -[handling unhappy paths in forms](../building-classic-assistants/forms.mdx#writing-stories--rules-for-unhappy-form-paths). +[handling conversation repair in forms](../nlu-based-assistants/forms.mdx#writing-stories--rules-for-conversation-repair-cases). :::note If you wish to reset all slots, we recommend using a custom action @@ -154,7 +154,7 @@ that returns the [`AllSlotsReset`](https://rasa.com/docs/rasa/reference/rasa/sha ## action_two_stage_fallback This is a fallback loop that can be used to handle low NLU confidence. Read more about -[handling low NLU confidence](../building-classic-assistants/fallback-handoff.mdx#nlu-fallback). +[handling low NLU confidence](../nlu-based-assistants/fallback-handoff.mdx#nlu-fallback). ## action_default_ask_affirmation @@ -171,21 +171,21 @@ their message. ## action_back This action undoes the last user-bot interaction. It can be triggered by the user -by sending a "/back" message to the assistant if the [RulePolicy](../building-classic-assistants/policies.mdx#rule-policy) is configured. +by sending a "/back" message to the assistant if the [RulePolicy](../nlu-based-assistants/policies.mdx#rule-policy) is configured. | ## Form Action By default Rasa uses `FormAction` for processing any -[form logic](../building-classic-assistants/forms.mdx). You can override this default action with a custom action by +[form logic](../nlu-based-assistants/forms.mdx). You can override this default action with a custom action by adding a custom action with the form's name to the domain. Overriding the default action for forms should **only** be used during the process of migrating from Rasa 1.0 to 2.0. ## action_unlikely_intent -Rasa triggers `action_unlikely_intent` via [`UnexpecTEDIntentPolicy`](../building-classic-assistants/policies.mdx#unexpected-intent-policy). -You can control how often this action is predicted by tuning the [`tolerance`](../building-classic-assistants/policies.mdx#unexpected-intent-policy) +Rasa triggers `action_unlikely_intent` via [`UnexpecTEDIntentPolicy`](../nlu-based-assistants/policies.mdx#unexpected-intent-policy). +You can control how often this action is predicted by tuning the [`tolerance`](../nlu-based-assistants/policies.mdx#unexpected-intent-policy) parameter of `UnexpecTEDIntentPolicy`. ### Customization diff --git a/docs/docs/concepts/dialogue-understanding.mdx b/docs/docs/concepts/dialogue-understanding.mdx index 438df53972c4..cb6e3b7957db 100644 --- a/docs/docs/concepts/dialogue-understanding.mdx +++ b/docs/docs/concepts/dialogue-understanding.mdx @@ -3,25 +3,24 @@ id: dialogue-understanding sidebar_label: Dialogue Understanding title: Dialogue Understanding abstract: | - Dialogue Understanding aims to understand how the end user of an assistant wants + Dialogue Understanding aims to understand how the end user of an assistant wants to progress the conversation. --- :::info New in 3.7 -The *Command Generator* is part of Rasa's new +The _Command Generator_ is part of Rasa's new [Conversational AI with Language Models (CALM) approach](../calm.mdx) and available starting with version `3.7.0`. ::: ## CommandGenerator -The `CommandGenerator` component performs *Dialogue Understanding*. -In the CALM approach, *Dialogue Understanding* aims to represent how the end user -wants to progress the conversation. +The `CommandGenerator` component performs _Dialogue Understanding_. +In the CALM approach, _Dialogue Understanding_ aims to represent how the end user +wants to progress the conversation. There is currently only one command generator available: the `LLMCommandGenerator`. - ## Using the LLMCommandGenerator in Your Bot To use this component in your bot, you need to add the @@ -44,12 +43,12 @@ The job of a Command Generator is to ingest information about a conversation so and output a sequence of `commands` which represent _how the user wants to progress the conversation_. For example, if you have defined a flow called `transfer_money`, and a user starts a conversation -by saying "I need to transfer some money", then the correct command output would be `StartFlow("transfer_money")`. +by saying "I need to transfer some money", then the correct command output would be `StartFlow("transfer_money")`. If you have just asked the user a yes/no question (using a `collect` step) and they say _"yes."_ , the correct command output would be `SetSlot(slot_name, True)`. -If the user answers the question but also requests something new, like _"yes. Oh what's my balance?"_, +If the user answers the question but also requests something new, like _"yes. Oh what's my balance?"_, the command output might be `[SetSlot(slot_name, True), StartFlow("check_balance")]`. By generating a sequence of commands, Dialogue Understanding has a much richer way to represent @@ -57,12 +56,11 @@ what the user wants than a classification-based NLU system. The current implementation of the Command Generator uses _in-context learning_, and uses information about the current state of the conversation as well as the flows defined in your bot -to interpret the user's message in context. - +to interpret the user's message in context. ## Customization -You can customize the command generator as much as you wish. +You can customize the command generator as much as you wish. ### LLM configuration @@ -80,24 +78,24 @@ pipeline: # - ... ``` -The `model_name` defaults to `gpt-4`. The model name should be set +The `model_name` defaults to `gpt-4`. The model name should be set to a chat model of [OpenAI](https://platform.openai.com/docs/guides/gpt/chat-completions-api). -Similarly, you can specify the `request_timeout` and -`temperature` parameters for the LLM. The `request_timeout` +Similarly, you can specify the `request_timeout` and +`temperature` parameters for the LLM. The `request_timeout` defaults to `7` seconds and the `temperature` defaults to `0.0`. If you want to use Azure OpenAI Service, you can configure the necessary parameters as described in the -[Azure OpenAI Service](../llms/llm-configuration.mdx#additional-configuration-for-azure-openai-service) +[Azure OpenAI Service](./components/llm-configuration.mdx#additional-configuration-for-azure-openai-service) section. :::info Using Other LLMs -By default, OpenAI is used as the underlying LLM provider. +By default, OpenAI is used as the underlying LLM provider. The used LLM provider provider can be configured in the -`config.yml` file to use another provider, e.g. `cohere`: +`config.yml` file to use another provider, e.g. `cohere`: ```yaml-rasa title="config.yml" pipeline: @@ -109,24 +107,23 @@ pipeline: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings) +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings) ::: ### Customizing The Prompt Because the `LLMCommandGenerator` uses in-context learning, one of the primary ways -to tweak or improve performance is to customize the prompt. +to tweak or improve performance is to customize the prompt. In most cases, you should be able to achieve what you need to by customizing the `description` fields in your flows. Every flow has its own `description` field, and optionally every `step` in your flow can have one as well. If you notice that a flow is being triggered when it shouldn't, or that a slot is not being extracted correctly, -adding more detail to the description will often solve the issue. +adding more detail to the description will often solve the issue. For example if you have a `transfer_money` flow with a `collect` step for the slot `amount`, you might add a description -to extract the value more reliably: - +to extract the value more reliably: ```yaml-rasa title="flows.yml" flows: diff --git a/docs/docs/concepts/domain.mdx b/docs/docs/concepts/domain.mdx index 4a1851cfb642..e855372e2598 100644 --- a/docs/docs/concepts/domain.mdx +++ b/docs/docs/concepts/domain.mdx @@ -8,14 +8,14 @@ abstract: The domain defines the universe in which your assistant operates. In Rasa, your `domain` defines the universe in which your assistant operates. Specifically, it lists: -* The `responses` that can be used as templated messages to send to a user. -* The custom `actions` that can be predicted by dialogue policies. -* The `slots` that act as your assistant's memory throughout a conversation. -* Session configuration parameters including inactivity timeout. +- The `responses` that can be used as templated messages to send to a user. +- The custom `actions` that can be predicted by dialogue policies. +- The `slots` that act as your assistant's memory throughout a conversation. +- Session configuration parameters including inactivity timeout. -If you are building an NLU-based assistant, refer to [this page](../building-classic-assistants/domain.mdx) to see how -intents, entities, slot mappings, and slot featurization can be configured in -your domain. +If you are building an NLU-based assistant, refer to [this page](../nlu-based-assistants/domain.mdx) to see how +intents, entities, slot mappings, and slot featurization can be configured in +your domain. ## Multiple Domain Files @@ -29,13 +29,12 @@ you can train a model with split domain files by running: rasa train --domain path_to_domain_directory ``` - ## Responses -Responses are templated messages that your assistant can send to your user. -Responses can contain rich content like buttons, images, and custom json payloads. +Responses are templated messages that your assistant can send to your user. +Responses can contain rich content like buttons, images, and custom json payloads. Every responses is also an [action](#actions), meaning that it can be used directly in -an `action` step in a flow. +an `action` step in a flow. Responses can be defined directly in the domain file under the `responses` key. For more information on responses and how to define them, see [Responses](./responses.mdx). @@ -55,8 +54,8 @@ For example, an action could: All [custom actions](./custom-actions.mdx) should be listed in your domain. -Rasa also has [default actions](./default-actions.mdx) which you do not -need to list in your domain. +Rasa also has [default actions](./default-actions.mdx) which you do not +need to list in your domain. ## Slots @@ -82,6 +81,7 @@ Different slot types exist to restrict the possible values a slot can take. mappings: - type: custom ``` + - **Allowed Values** Any string @@ -95,8 +95,9 @@ Different slot types exist to restrict the possible values a slot can take. confirmation: type: bool mappings: - - type: custom + - type: custom ``` + - **Allowed Values** `true` or `false` @@ -110,7 +111,7 @@ Different slot types exist to restrict the possible values a slot can take. risk_level: type: categorical mappings: - - type: custom + - type: custom values: - low - medium @@ -126,7 +127,7 @@ Different slot types exist to restrict the possible values a slot can take. temperature: type: float mappings: - - type: custom + - type: custom ``` #### Any Slot @@ -138,10 +139,9 @@ Different slot types exist to restrict the possible values a slot can take. shopping_items: type: any mappings: - - type: custom + - type: custom ``` - ### Initial slot values You can provide an initial value for any slot in your domain file: @@ -152,7 +152,7 @@ slots: type: float initial_value: 0 mappings: - - type: custom + - type: custom ``` ### Slot Mappings @@ -206,7 +206,6 @@ call, or to start the conversation with a bot message. The docs on ::: - ## Select which actions should receive domain :::info New in 3.4.3 diff --git a/docs/docs/concepts/flows.mdx b/docs/docs/concepts/flows.mdx index 70df93a5cc91..5b3a24d1add5 100644 --- a/docs/docs/concepts/flows.mdx +++ b/docs/docs/concepts/flows.mdx @@ -8,10 +8,6 @@ abstract: | Flows within Rasa. --- -import RasaDiscoveryBanner from "@theme/RasaDiscoveryBanner"; - - - ## Overview **Flow** is a novel Rasa feature designed to build intuitive, @@ -297,7 +293,7 @@ step and another provider, e.g. `cohere` can be used: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings) +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings) ::: @@ -401,7 +397,7 @@ step and another provider, e.g. `cohere` can be used: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings) +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings) ::: diff --git a/docs/docs/concepts/policies.mdx b/docs/docs/concepts/policies.mdx index 53538ea88224..0cee35039d98 100644 --- a/docs/docs/concepts/policies.mdx +++ b/docs/docs/concepts/policies.mdx @@ -6,11 +6,12 @@ abstract: Policies decide the next action in a conversation. --- import useBaseUrl from "@docusaurus/useBaseUrl"; -import RasaProLabel from '@theme/RasaProLabel'; +import RasaProLabel from "@theme/RasaProLabel"; +import RasaProBanner from "@theme/RasaProBanner"; -In Rasa, policies are the components responsible for dialogue management. -In NLU-based assistants, dialogue policies play a larger role. -You can read more about those policies and other advanced topics [here](../building-classic-assistants/policies.mdx). +In Rasa, policies are the components responsible for dialogue management. +In NLU-based assistants, dialogue policies play a larger role. +You can read more about those policies and other advanced topics [here](../nlu-based-assistants/policies.mdx). In a CALM-based assistant, the `FlowPolicy` is responsible for executing your business logic. @@ -31,8 +32,6 @@ policies: - name: DocSearchPolicy ``` - - ## Action Selection At every turn, each policy defined in your configuration gets a chance to @@ -48,12 +47,11 @@ to the desired number of maximum predictions. ::: - ### Flow Policy :::info New in 3.7 -The *Flow Policy* is part of Rasa's new +The _Flow Policy_ is part of Rasa's new [Conversational AI with Language Models (CALM)](../calm.mdx) approach and available starting with version `3.7.0`. ::: @@ -67,8 +65,6 @@ triggers new flows when needed. The "Dialogue Stack" represents all started but incomplete flows and refers to steps defined in the [flows specification](../concepts/flows.mdx). - - Using the Flow Policy, you can encapsulate your logic into flows, thereby guiding the conversation based on user input and system response. Other classifiers adjust the underlying state machine to handle edge cases. @@ -78,7 +74,7 @@ For more detailed information on defining flows, refer to the ## Adding the FlowPolicy to your bot -The Flow Policy can be integrated like [any other policy](../building-classic-assistants/policies.mdx). +The Flow Policy can be integrated like [any other policy](../nlu-based-assistants/policies.mdx). It involves two steps: 1. Configure the `FlowPolicy` in your `config.yml` file: @@ -129,14 +125,13 @@ sorry I meant John"_, initiates a correction flow. This correction is not explicitly modeled in the `transfer_money` flow; it's handled by a conversational pattern: - The correction flow only executes when the user wants to correct a previous input. Once triggered, it's placed on top of the dialogue stack for execution. For the `transfer_money` flow, the correction flow is an interruption. Once completed, the `transfer_money` flow resumes. Before its res umption, a `resume_flow` flow is activated. You can customize what happens in `resume_flow` -and `correction` flows in [Handling Unhappy Paths](./unhappy-paths.mdx). +and `correction` flows in [Conversation Repair](./conversation-repair.mdx). Upon completion of the `resume_flow` flow, the `transfer_money` flow resumes, and the bot returns to the state before the correction flow started. The @@ -146,7 +141,7 @@ and the bot returns to the state before the correction flow started. The The `resume_flow` and `correction` flows are conversational patterns not explicitly modeled in the `transfer_money` flow. The `FlowPolicy` automatically triggers them when needed. For more details on defining conversational patterns, -refer to [Handling Unhappy Paths](./unhappy-paths.mdx). +refer to [Conversation Repair](./conversation-repair.mdx). ### Transitioning Between States @@ -175,7 +170,7 @@ Flows and their specifications should ideally model the happy path of a conversation. The `FlowPolicy` depends on other components to identify digressions and interruptions and interrupt the flow accordingly. For details on which digressions and interruptions the flow policy can handle, refer to -[Handling Unhappy Paths](./unhappy-paths.mdx). +[Conversation Repair](./conversation-repair.mdx). For example, the user message _"oh sorry I meant John"_ is a correction to a prior input. The `FlowPolicy` depends on another component, like the @@ -200,8 +195,7 @@ with the `FlowPolicy`. - -## Overview + The component uses an LLM to generate rephrased responses. The LLM is trained on a prompt that includes a transcript of the conversation with the user and a @@ -225,7 +219,7 @@ policies: ``` The `source` parameter specifies the directory containing your documentation. -The `DocsearchPolicy` will automatically index all files with a `.txt` +The `DocsearchPolicy` will automatically index all files with a `.txt` extension in this directory (recursively) and uses them to generate responses. ## Customization @@ -235,11 +229,11 @@ You can customize the LLM by modifying the following parameters in the ### Mapping to existing responses -Instead of generating new responses, you can also use the LLM to respond with -responses from your domain. This allows you to use the LLMs information +Instead of generating new responses, you can also use the LLM to respond with +responses from your domain. This allows you to use the LLMs information retrieval capabilities to find the best response from your domain. -The response generated by the document search is compared to the responses +The response generated by the document search is compared to the responses defined in your domain. If there is a response that is similar enough, it is used instead of the generated response. The `max_distance` parameter allows you to control this behavior: @@ -257,8 +251,6 @@ value of `0.0` means that the generated response needs to be identical to an existing response. A value of `1.0` means that the generated response will always be maped to an existing response. The default value is `0.2`. - - ### LLM / Embeddings You can choose the OpenAI model that is used for the LLM by adding the `llm.model_name` @@ -275,17 +267,17 @@ policies: Defaults to `gpt-3.5-turbo`. -If you want to use Azure OpenAI Service, you can configure the necessary -parameters as described in the -[Azure OpenAI Service](../llms/llm-configuration.mdx#additional-configuration-for-azure-openai-service) +If you want to use Azure OpenAI Service, you can configure the necessary +parameters as described in the +[Azure OpenAI Service](./components/llm-configuration.mdx#additional-configuration-for-azure-openai-service) section. :::info Using Other LLMs / Embeddings -By default, OpenAI is used as the underlying LLM and embedding provider. +By default, OpenAI is used as the underlying LLM and embedding provider. The used LLM provider and embeddings provider can be configured in the -`config.yml` file to use another provider, e.g. `cohere`: +`config.yml` file to use another provider, e.g. `cohere`: ```yaml-rasa title="config.yml" policies: @@ -299,13 +291,13 @@ policies: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings) +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings) ::: ### Prompt -You can change the prompt template used to generate a response based on +You can change the prompt template used to generate a response based on retrieved documents by setting the `prompt` property in the `config.yml`: ```yaml-rasa title="config.yml" @@ -313,9 +305,9 @@ policies: # - ... - name: rasa_plus.ml.DocsearchPolicy prompt: | - Given the following passages of relevant documents, and the - previous conversation, answer the question. If you don't - know the answer, just say that you don't know. Don't try + Given the following passages of relevant documents, and the + previous conversation, answer the question. If you don't + know the answer, just say that you don't know. Don't try to make up an answer. Relevant Passages: @@ -329,8 +321,8 @@ policies: === - Please formulate an answer for the question or request in - the user's last message. If you don't know the answer, just + Please formulate an answer for the question or request in + the user's last message. If you don't know the answer, just say that you don't know. Don't try to make up an answer. Your answer: @@ -353,16 +345,16 @@ The component uses an LLM to generate rephrased responses. The following threat vectors should be considered: -- **Privacy**: Most LLMs are run as remote services. The component sends - your bot's conversations to remote servers for prediction. By default, - the used prompt templates include a transcript of the conversation. +- **Privacy**: Most LLMs are run as remote services. The component sends + your bot's conversations to remote servers for prediction. By default, + the used prompt templates include a transcript of the conversation. Slot values are not included. -- **Hallucination**: When generating answers, it is possible that the LLM - changes your document content in a way that the meaning is no longer exactly - the same. The temperature parameter allows you to control this trade-off. - A low temperature will only allow for minor variations. A higher temperature - allows greater flexibility but with the risk of the meaning being - changed - but allows the model to better combine knowledge from +- **Hallucination**: When generating answers, it is possible that the LLM + changes your document content in a way that the meaning is no longer exactly + the same. The temperature parameter allows you to control this trade-off. + A low temperature will only allow for minor variations. A higher temperature + allows greater flexibility but with the risk of the meaning being + changed - but allows the model to better combine knowledge from different documents. - **Prompt Injection**: Messages sent by your end users to your bot will become part of the LLM prompt (see template above). That means a malicious user can @@ -376,14 +368,13 @@ The following threat vectors should be considered: More detailed information can be found in Rasa's webinar on [LLM Security in the Enterprise](https://info.rasa.com/webinars/llm-security-in-the-enterprise-replay). - ### Intentless Policy :::info New in 3.7 -The *Intentless Policy* is part of Rasa's new Conversational AI with +The _Intentless Policy_ is part of Rasa's new Conversational AI with Language Models (CALM) approach and available starting with version `3.7.0`. ::: @@ -399,7 +390,6 @@ Using the `IntentlessPolicy`, a question-answering bot can already understanding many different ways that users could phrase their questions - even across a series of user messages: - This only requires appropriate responses to be defined in the domain file. To eliminate hallucinations, the policy only chooses which response from @@ -498,7 +488,7 @@ Defaults to `text-davinci-003`. The model name needs to be set to an If you want to use Azure OpenAI Service, you can configure the necessary parameters as described in the -[Azure OpenAI Service](../llms/llm-configuration.mdx#additional-configuration-for-azure-openai-service) +[Azure OpenAI Service](./components/llm-configuration.mdx#additional-configuration-for-azure-openai-service) section. #### Other LLMs / Embeddings @@ -519,7 +509,7 @@ policies: ``` For more information, see the -[LLM setup page on llms and embeddings](../llms/llm-configuration.mdx#other-llmsembeddings). +[LLM setup page on llms and embeddings](./components/llm-configuration.mdx#other-llmsembeddings). ### Other Policies @@ -552,7 +542,6 @@ policies are uncertain: - If the `IntentlessPolicy` prediction has low confidence, the `RulePolicy` will trigger fallback based on the `core_fallback_threshold`. - **What about TED?** There is no reason why you can't also have TED in your configuration. However, @@ -588,7 +577,7 @@ utter_faq_6: Beyond having the `utter_` prefix, the naming of the utterances is not relevant. The second step is to add -[end-to-end stories](../building-classic-assistants/training-data-format.mdx#end-to-end-training) +[end-to-end stories](../nlu-based-assistants/training-data-format.mdx#end-to-end-training) to `data/e2e_stories.yml`. These stories teach the LLM about your domain, so it can figure out when to say what. @@ -692,7 +681,4 @@ At this point, the intentless policy can only predict utterances but not custom actions. Triggering custom actions needs to be done by traditional policies, such as the rule- or memoization policy. - - - ## Configuring Policies diff --git a/docs/docs/concepts/responses.mdx b/docs/docs/concepts/responses.mdx index cd3cdc6837ac..c7966fcfc66c 100644 --- a/docs/docs/concepts/responses.mdx +++ b/docs/docs/concepts/responses.mdx @@ -21,7 +21,7 @@ responses: - text: "See you!" ``` -If you are using [retrieval intents](../building-classic-assistants/glossary.mdx#retrieval-intent) in your assistant, you also need to add responses +If you are using [retrieval intents](../nlu-based-assistants/glossary.mdx#retrieval-intent) in your assistant, you also need to add responses for your assistant's replies to these intents: ```yaml-rasa {5-6,8-9} @@ -40,7 +40,7 @@ responses: Notice the special format of response names for retrieval intents. Each name starts with `utter_`, followed by the retrieval intent's name (here `chitchat`) and finally a suffix specifying the different response keys (here `ask_name` and `ask_weather`). See [the documentation for NLU -training examples](../building-classic-assistants/training-data-format.mdx#training-examples) to learn more. +training examples](../nlu-based-assistants/training-data-format.mdx#training-examples) to learn more. ::: ### Using Variables in Responses diff --git a/docs/docs/deploy/introduction.mdx b/docs/docs/deploy/introduction.mdx index c30d0ccb24f0..f9701b7ca02e 100644 --- a/docs/docs/deploy/introduction.mdx +++ b/docs/docs/deploy/introduction.mdx @@ -19,7 +19,7 @@ Are you unfamiliar with Docker, Kubernetes and Helm? Check out "[Understanding R ## When to Deploy Your Assistant The best time to deploy your assistant and make it available to test users is once it can handle the most -important happy paths or is what we call a [minimum viable assistant](../building-classic-assistants/glossary.mdx). Then you can use incoming +important happy paths or is what we call a [minimum viable assistant](../nlu-based-assistants/glossary.mdx). Then you can use incoming conversations to inform further development of your assistant. ## Recommended Deployment Method diff --git a/docs/docs/installation/environment-set-up.mdx b/docs/docs/installation/environment-set-up.mdx index 056902e1f92e..4ffb24d4ce5e 100644 --- a/docs/docs/installation/environment-set-up.mdx +++ b/docs/docs/installation/environment-set-up.mdx @@ -126,7 +126,7 @@ C:\> .\venv\Scripts\activate Rasa installations on Apple Silicon _don't_ use [Apple Metal](https://developer.apple.com/metal/) by default. We found that using the GPU on Apple Silicon increased training time for the -[`DIETClassifier`](../building-classic-assistants/components.mdx#dietclassifier) and [`TEDPolicy`](../building-classic-assistants/policies.mdx#ted-policy) dramatically. +[`DIETClassifier`](../nlu-based-assistants/components.mdx#dietclassifier) and [`TEDPolicy`](../nlu-based-assistants/policies.mdx#ted-policy) dramatically. You can, however, install the optional dependency to test it yourself or try it with other components using: `pip3 install 'rasa[metal]'`. @@ -134,10 +134,10 @@ Currently, not all of Rasa's dependencies support Apple Silicon natively. This l to the following restrictions: - You can not run Duckling as a docker container on Apple Silicon. If you want - to use the [duckling entity extractor](../building-classic-assistants/components.mdx#ducklingentityextractor) we recommend a cloud deployment of + to use the [duckling entity extractor](../nlu-based-assistants/components.mdx#ducklingentityextractor) we recommend a cloud deployment of duckling. Progress on this can be tracked on the [Duckling project](https://github.com/facebook/duckling/issues/695). -- Rasa on Apple Silicon does not support the [`ConveRTFeaturizer` component](../building-classic-assistants/components.mdx#convertfeaturizer) or pipelines +- Rasa on Apple Silicon does not support the [`ConveRTFeaturizer` component](../nlu-based-assistants/components.mdx#convertfeaturizer) or pipelines containing it. The component relies on `tensorflow-text` which currently isn't available for Apple Silicon. Progress on this can be tracked on the [Tensorflow Text project](https://github.com/tensorflow/text/issues/823). diff --git a/docs/docs/installation/installing-rasa-open-source.mdx b/docs/docs/installation/installing-rasa-open-source.mdx index 8c1270fe4dd9..a4663b654014 100644 --- a/docs/docs/installation/installing-rasa-open-source.mdx +++ b/docs/docs/installation/installing-rasa-open-source.mdx @@ -9,13 +9,11 @@ import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import { Button } from "@theme/Button"; - - -The next generation of Rasa's dialogue management is currently only +The next generation of Rasa's dialogue management is currently only available as a Rasa Labs release, which is restricted to customers of Rasa. If you already have a valid Rasa Pro license, you can install Rasa Labs -releases using the installation instructions from the +releases using the installation instructions from the [Rasa Pro Python Installation](../installation/rasa-pro/installation.mdx#python-package-installation) guide. After you've completed the authentication and configuration steps @@ -26,8 +24,6 @@ using: pip install 'rasa-plus>3.7' --pre ``` - - ## Install Rasa Open Source @@ -81,7 +77,7 @@ poetry install For some machine learning algorithms you need to install additional python packages. They aren't installed by default to keep the footprint small. -The page on [Tuning Your Model](../building-classic-assistants/tuning-your-model.mdx) will help you pick the right +The page on [Tuning Your Model](../nlu-based-assistants/tuning-your-model.mdx) will help you pick the right configuration for your assistant and alert you to additional dependencies. :::tip Just give me everything! diff --git a/docs/docs/migration-guide.mdx b/docs/docs/migration-guide.mdx index ba87129875da..f25b19fa3cfb 100644 --- a/docs/docs/migration-guide.mdx +++ b/docs/docs/migration-guide.mdx @@ -30,7 +30,7 @@ Please check whether your trained model still performs as expected and retrain i ### NLU JSON Format -[NLU training data](./building-classic-assistants/nlu-training-data.mdx) in JSON format is deprecated and will be +[NLU training data](./nlu-based-assistants/nlu-training-data.mdx) in JSON format is deprecated and will be removed in Rasa 4.0. Please use `rasa data convert nlu -f yaml --data ` to convert your NLU JSON data to YAML format before support for NLU JSON data is removed. @@ -52,7 +52,7 @@ to convert your data from Markdown to YAML. Please use the commands described ### Model Configuration It is required to specify the used `recipe` within the -[model configuration](./building-classic-assistants/model-configuration.mdx). As of now Rasa only supports +[model configuration](./nlu-based-assistants/model-configuration.mdx). As of now Rasa only supports the `default.v1` recipe and will continue using it even if you don't specify a recipe in the model configuration. To avoid breaking changes in the future you should to specify `recipe: "default.v1"` at the top of your model configuration: @@ -88,8 +88,8 @@ policies: ### Custom Policies and Custom Components -Rasa 3.0 changed the way [NLU components](./building-classic-assistants/components.mdx) and -[policies](./building-classic-assistants/policies.mdx) are trained and run during inference. As part of these changes +Rasa 3.0 changed the way [NLU components](./nlu-based-assistants/components.mdx) and +[policies](./nlu-based-assistants/policies.mdx) are trained and run during inference. As part of these changes the interfaces for NLU components and policies have been unified and adapted. The next sections outline which adaptions are required to run your custom NLU components @@ -468,8 +468,8 @@ class MyNLUComponent(GraphComponent, IntentClassifier): **Augmenting Training Data in an NLU Component** -NLU Components like [tokenizers](./building-classic-assistants/components.mdx#tokenizers) or -[featurizers](./building-classic-assistants/components.mdx#featurizers) augment the training data with their +NLU Components like [tokenizers](./nlu-based-assistants/components.mdx#tokenizers) or +[featurizers](./nlu-based-assistants/components.mdx#featurizers) augment the training data with their output during the model training. Their output is required by NLU components later in the pipeline. Typically, featurizers require _tokenized_ messages and intent classifiers require _featurized_ training data to train themselves. Rasa @@ -615,8 +615,8 @@ class MyNLUComponent(GraphComponent, IntentClassifier): **Using a Model Provider with your NLU Component** -If your NLU component requires a pretrained model such as a [Spacy](./building-classic-assistants/components.mdx#spacynlp) or -[Mitie](./building-classic-assistants/components.mdx#mitienlp) language model you have to specify the NLU component which +If your NLU component requires a pretrained model such as a [Spacy](./nlu-based-assistants/components.mdx#spacynlp) or +[Mitie](./nlu-based-assistants/components.mdx#mitienlp) language model you have to specify the NLU component which provides this model in your model's pipeline before the NLU component which requires the model. In addition to this you now also need to specify the model loading component in the `model_from` parameter in the `register` decorator. The model will then be passed to your model's @@ -688,7 +688,7 @@ This guide leads you through the migration of a custom policy step by step. Policies are no longer instantiated via their constructor. Instead, all policies have to implement a `create` method. During the policy instantiation the configuration from -the [model configuration](./building-classic-assistants/model-configuration.mdx) is passed in as a dictionary instead +the [model configuration](./nlu-based-assistants/model-configuration.mdx) is passed in as a dictionary instead of as separate parameters. Similarly, the`featurizers` are no longer instantiated outside of policies. Instead, the super class `rasa.core.policies.policy.Policy` instantiates the @@ -960,7 +960,7 @@ class MyPolicy(Policy): **Using End-To-End Features in a Policy** -To use a custom [end-to-end policy](./building-classic-assistants/stories.mdx#end-to-end-training) in Rasa +To use a custom [end-to-end policy](./nlu-based-assistants/stories.mdx#end-to-end-training) in Rasa Open Source 2, you had to use the `interpreter` parameter to featurize the tracker events manually. In Rasa 3.0, you need to [register](./concepts/components/custom-graph-components.mdx#registering-graph-components-with-the-model-configuration) a policy that requires end-to-end features with type `ComponentType.POLICY_WITH_END_TO_END_SUPPORT`. The features @@ -968,7 +968,7 @@ will be precomputed and passed into your policy during training and inference. :::caution End-To-End features will only be computed and provided to your policy if your training -data actually contains [end-to-end training data](./building-classic-assistants/stories.mdx#end-to-end-training). +data actually contains [end-to-end training data](./nlu-based-assistants/stories.mdx#end-to-end-training). ::: @@ -1098,14 +1098,14 @@ class MyPolicy(Policy): **Providing Rule-only Data to a Policy** -Rasa allows excluding [forms](./building-classic-assistants/forms.mdx) or [slots](./concepts/domain.mdx#slots) which +Rasa allows excluding [forms](./nlu-based-assistants/forms.mdx) or [slots](./concepts/domain.mdx#slots) which are completely handled by -[rules](./building-classic-assistants/rules.mdx) from becoming features in other policies. +[rules](./nlu-based-assistants/rules.mdx) from becoming features in other policies. In Rasa 2 this information was passed onto the policies using the `set_shared_policy_states` method which set the policy attribute `_rule_only_data`. Rasa passes the names of rule-only slots and forms via the `predict_action_probabilities` method. The passed `rule_only_data` can be `None` -in case the [`RulePolicy`](./building-classic-assistants/policies.mdx#rule-policy) is not part of your model +in case the [`RulePolicy`](./nlu-based-assistants/policies.mdx#rule-policy) is not part of your model configuration. @@ -1270,8 +1270,8 @@ consult the following migration guides for the individual policies: - `TwoStageFallbackPolicy` [migration guide](#manually-migrating-from-the-two-stage-fallback-policy) - `MappingPolicy` [migration guide](#manually-migrating-from-the-mapping-policy) - `FormPolicy` [migration guide](#forms) -- `SklearnPolicy` should be replaced with [TEDPolicy](./building-classic-assistants/policies.mdx#ted-policy). - It is recommended to use the [default TEDPolicy config](./building-classic-assistants/model-configuration.mdx#suggested-config) as a starting point. +- `SklearnPolicy` should be replaced with [TEDPolicy](./nlu-based-assistants/policies.mdx#ted-policy). + It is recommended to use the [default TEDPolicy config](./nlu-based-assistants/model-configuration.mdx#suggested-config) as a starting point. #### Removed Tokenizers and Featurizers @@ -1347,7 +1347,7 @@ provided instead. To maintain the behavior of forms in the 2.0 format, all migrated slot mappings will include mapping conditions for each form. This can be changed manually according to your use case. -See the docs on [mapping conditions](./building-classic-assistants/domain.mdx#mapping-conditions) for more information. +See the docs on [mapping conditions](./nlu-based-assistants/domain.mdx#mapping-conditions) for more information. #### Manually migrating from 2.0 domain format to the 3.0 format @@ -1419,7 +1419,7 @@ forms: - outdoor_seating ``` -For slots that should be filled only in the context of a form, add [mapping conditions](./building-classic-assistants/domain.mdx#mapping-conditions) +For slots that should be filled only in the context of a form, add [mapping conditions](./nlu-based-assistants/domain.mdx#mapping-conditions) to specify which form(s) should be active, as well as indicate if the `requested_slot` should be the same slot. Adding `conditions` is required to preserve the behavior of slot mappings from 2.0, since without them the mappings will be applied on each user turn regardless of whether a form is active or not. @@ -1453,7 +1453,7 @@ If you have been dynamically filling slots not present in the form's `required_s file, note that this behaviour is no longer supported in 3.x. Any dynamic slots with custom mappings, which are set in the last user turn, will be filled **only if** they are returned by the `required_slots` method of the custom action inheriting from `FormValidationAction`. To maintain the 2.x behaviour, you must now override the `required_slots` method -of this custom action as per the strong recommendation listed in the [dynamic form documentation](./building-classic-assistants/forms.mdx#dynamic-form-behavior). +of this custom action as per the strong recommendation listed in the [dynamic form documentation](./nlu-based-assistants/forms.mdx#dynamic-form-behavior). To extract custom slots that are not defined in any form's `required_slots`, you should now use a global [custom slot mapping](./concepts/domain.mdx#custom-slot-mappings) and extend the [ValidationAction class](./action-server/validation-action.mdx#validationaction-class). @@ -1505,8 +1505,8 @@ models - `DIETClassifier`, `ResponseSelector`. In other words, no extra buffer i advance for additional vocabulary items and space will be dynamically allocated for them inside the model. This means there's no need to specify `additional_vocabulary_size` for -[`CountVectorsFeaturizer`](./building-classic-assistants/components.mdx#countvectorsfeaturizer) or -`number_additional_patterns` for [`RegexFeaturizer`](./building-classic-assistants/components.mdx#regexfeaturizer). +[`CountVectorsFeaturizer`](./nlu-based-assistants/components.mdx#countvectorsfeaturizer) or +`number_additional_patterns` for [`RegexFeaturizer`](./nlu-based-assistants/components.mdx#regexfeaturizer). These parameters are now deprecated. **Before** @@ -1539,7 +1539,7 @@ Based on user feedback, we have identified multiple problems with this option. Therefore, `model_confidence=linear_norm` is now deprecated and will be removed in Rasa `3.0.0`. If you were using `model_confidence=linear_norm` for any of the mentioned components, we recommend to revert it back to `model_confidence=softmax` and re-train the assistant. After re-training, -we also recommend to [re-tune the thresholds for fallback components](./building-classic-assistants/fallback-handoff.mdx#fallbacks). +we also recommend to [re-tune the thresholds for fallback components](./nlu-based-assistants/fallback-handoff.mdx#fallbacks). ## Rasa 2.5 to 2.6 @@ -1549,7 +1549,7 @@ we also recommend to [re-tune the thresholds for fallback components](./building There is a new parameter under Forms called `ignored_intents`. This parameter can be used to prevent any required slots in a form from being filled with the specified -intent or intents. Please see the [Forms documentation](./building-classic-assistants/forms.mdx) for examples and more +intent or intents. Please see the [Forms documentation](./nlu-based-assistants/forms.mdx) for examples and more information on how to use it in your `domain.yml` file. Before, if a user did not want to fill any slots of a form with a specified intent @@ -1648,7 +1648,7 @@ a compatible language is configured for the entire pipeline in `config.yml`, eve specify a `model`. This fallback behavior is temporary and will be deprecated in Rasa 3.0.0. We've updated our docs to reflect these changes. All examples now show a direct link to the -correct spaCy model. We've also added a warning to the [SpaCyNLP](./building-classic-assistants/components.mdx#spacynlp) +correct spaCy model. We've also added a warning to the [SpaCyNLP](./nlu-based-assistants/components.mdx#spacynlp) docs that explains the fallback behavior. ## Rasa 2.3 to Rasa 2.4 @@ -1677,8 +1677,8 @@ NLG Server Response Selector -- The field `template_name` will be deprecated in Rasa 3.0.0. Please use `utter_action` instead. Please see [here](./building-classic-assistants/components.mdx#selectors) for more details. -- The field `response_templates` will be deprecated in Rasa 3.0.0. Please use `responses` instead. Please see [here](./building-classic-assistants/components.mdx#selectors) for more details. +- The field `template_name` will be deprecated in Rasa 3.0.0. Please use `utter_action` instead. Please see [here](./nlu-based-assistants/components.mdx#selectors) for more details. +- The field `response_templates` will be deprecated in Rasa 3.0.0. Please use `responses` instead. Please see [here](./nlu-based-assistants/components.mdx#selectors) for more details. ## Rasa 2.3.3 to Rasa 2.3.4 @@ -1728,7 +1728,7 @@ A new option `model_confidence` has been added to each ML component. It affects 2. `cosine` - Cosine similarity between input and label embeddings. Confidence for each label will be in the range `[-1,1]`. 3. `linear_norm` - Dot product similarities between input and label embeddings are post-processed with a linear normalization function. Confidence for each label will be in the range `[0,1]`. -The default value is `softmax`, but we recommend trying `linear_norm`. This should make it easier to [tune thresholds for triggering fallback](./building-classic-assistants/fallback-handoff.mdx#fallbacks). +The default value is `softmax`, but we recommend trying `linear_norm`. This should make it easier to [tune thresholds for triggering fallback](./nlu-based-assistants/fallback-handoff.mdx#fallbacks). The value of this option does not affect how confidences are computed for entity predictions in `DIETClassifier`. We encourage you to try both the above recommendations. This can be done with the following config: @@ -1740,7 +1740,7 @@ We encourage you to try both the above recommendations. This can be done with th ... ``` -Once the assistant is re-trained with the above configuration, users should also [tune fallback confidence thresholds](./building-classic-assistants/fallback-handoff.mdx#fallbacks). +Once the assistant is re-trained with the above configuration, users should also [tune fallback confidence thresholds](./nlu-based-assistants/fallback-handoff.mdx#fallbacks). **EDIT**: Some post-release experiments revealed that using `model_confidence=cosine` is wrong as it can change the order of predicted labels. That's why this option was removed in Rasa version `2.3.4`. @@ -1807,7 +1807,7 @@ described [here](./migration-guide.mdx#training-data-files). ### Policies -[Policies](./building-classic-assistants/policies.mdx) now require a `**kwargs` argument in their constructor and `load` method. +[Policies](./nlu-based-assistants/policies.mdx) now require a `**kwargs` argument in their constructor and `load` method. Policies without `**kwargs` will be supported until Rasa version `3.0.0`. However when using [incremental training](./command-line-interface.mdx#incremental-training) `**kwargs` **must** be included. @@ -1823,7 +1823,7 @@ However when using [incremental training](./command-line-interface.mdx#increment ### Deprecations -`ConveRTTokenizer` is now deprecated. [ConveRTFeaturizer](./building-classic-assistants/components.mdx#convertfeaturizer) now implements +`ConveRTTokenizer` is now deprecated. [ConveRTFeaturizer](./nlu-based-assistants/components.mdx#convertfeaturizer) now implements its behaviour. To migrate, replace `ConveRTTokenizer` with any other tokenizer, for e.g.: ```yaml @@ -1835,7 +1835,7 @@ pipeline: ``` `HFTransformersNLP` and `LanguageModelTokenizer` components are now deprecated. -[LanguageModelFeaturizer](./building-classic-assistants/components.mdx#languagemodelfeaturizer) now implements their behaviour. +[LanguageModelFeaturizer](./nlu-based-assistants/components.mdx#languagemodelfeaturizer) now implements their behaviour. To migrate, replace both the above components with any tokenizer and specify the model architecture and model weights as part of `LanguageModelFeaturizer`, for e.g.: @@ -1880,7 +1880,7 @@ some additional changes will need to be made as described in their respective se ### Policies -With the introduction of [rules](./building-classic-assistants/rules.mdx) and the [RulePolicy](./building-classic-assistants/policies.mdx#rule-policy), +With the introduction of [rules](./nlu-based-assistants/rules.mdx) and the [RulePolicy](./nlu-based-assistants/policies.mdx#rule-policy), the following policies are deprecated: - [Mapping Policy](https://rasa.com/docs/rasa/2.x/policies#mapping-policy) @@ -1907,7 +1907,7 @@ You can also migrate the individual policies manually, if you don't want to use #### Manually migrating from the Mapping Policy If you previously used the [Mapping Policy](https://rasa.com/docs/rasa/2.x/policies#mapping-policy), you -can follow the documentation on [FAQs](./building-classic-assistants/chitchat-faqs.mdx) to convert your mapped +can follow the documentation on [FAQs](./nlu-based-assistants/chitchat-faqs.mdx) to convert your mapped intents to rules. Suppose you previously mapped an intent `ask_is_bot` as follows: ```yaml-rasa title="domain.yml" @@ -1934,7 +1934,7 @@ intents: ``` Finally, you can replace the Mapping Policy with the -[Rule Policy](./building-classic-assistants/policies.mdx#rule-policy) in your model configuration: +[Rule Policy](./nlu-based-assistants/policies.mdx#rule-policy) in your model configuration: ```yaml-rasa title="config.yml" policies: @@ -1973,7 +1973,7 @@ policies: core_fallback_action_name: "action_default_fallback" ``` -In addition, you need to add a [rule](./building-classic-assistants/rules.mdx) to specify which action to run +In addition, you need to add a [rule](./nlu-based-assistants/rules.mdx) to specify which action to run in case of low NLU confidence: ```yaml-rasa title="rules.yml" @@ -1984,7 +1984,7 @@ rules: - action: utter_please_rephrase ``` -See the documentation on [fallback](./building-classic-assistants/fallback-handoff.mdx#fallbacks) for more +See the documentation on [fallback](./nlu-based-assistants/fallback-handoff.mdx#fallbacks) for more information. #### Manually migrating from the Two-Stage-Fallback Policy @@ -2021,7 +2021,7 @@ policies: core_fallback_action_name: "action_default_fallback" ``` -In addition you need to add a [rule](./building-classic-assistants/rules.mdx) to activate the Two-Stage Fallback for +In addition you need to add a [rule](./nlu-based-assistants/rules.mdx) to activate the Two-Stage Fallback for messages with low NLU confidence. ```yaml-rasa title="rules.yml" @@ -2040,18 +2040,18 @@ Note that the previous parameters `fallback_nlu_action_name` and `deny_suggestion_intent_name` are no longer configurable and have the fixed values `action_default_fallback` and `out_of_scope`. -See the [fallback](./building-classic-assistants/fallback-handoff.mdx#fallbacks) documentation for more +See the [fallback](./nlu-based-assistants/fallback-handoff.mdx#fallbacks) documentation for more information. ### Forms -As of version 2.0 the logic for [forms](./building-classic-assistants/forms.mdx) has been moved from the +As of version 2.0 the logic for [forms](./nlu-based-assistants/forms.mdx) has been moved from the Rasa SDK to Rasa to simplify implementation and make it easier to write action servers in other languages. This means that forms are no longer implemented using a `FormAction`, but instead defined in the domain. Any customizations around requesting slots or -[slot validation](./building-classic-assistants/forms.mdx#validating-form-input) can be handled with a `FormValidationAction`. +[slot validation](./nlu-based-assistants/forms.mdx#validating-form-input) can be handled with a `FormValidationAction`. Consider a custom form action from 1.x like this: @@ -2112,7 +2112,7 @@ class RestaurantForm(FormAction): return [] ``` -Start the migration by removing the FormPolicy and adding the [RulePolicy](./building-classic-assistants/policies.mdx#rule-policy) +Start the migration by removing the FormPolicy and adding the [RulePolicy](./nlu-based-assistants/policies.mdx#rule-policy) (if not there already) to your model configuration: ```yaml-rasa title="config.yml" @@ -2123,7 +2123,7 @@ policies: ``` Then you need to define the form, required slots and their slot mappings -in the domain as described in the documentation on [forms](./building-classic-assistants/forms.mdx#defining-a-form): +in the domain as described in the documentation on [forms](./nlu-based-assistants/forms.mdx#defining-a-form): ```yaml-rasa title="domain.yml" forms: @@ -2234,7 +2234,7 @@ forms: my_form: ``` -See the [forms](./building-classic-assistants/forms.mdx) documentation for more details. +See the [forms](./nlu-based-assistants/forms.mdx) documentation for more details. ### Response Selectors @@ -2298,7 +2298,7 @@ list in your domain. This behavior will work fine when defined as a story, but even better when defined as a rule. You should consider transferring your retrieval stories to rules. More information -on what that looks like in the [chitchat and FAQs documentation](./building-classic-assistants/chitchat-faqs.mdx). +on what that looks like in the [chitchat and FAQs documentation](./nlu-based-assistants/chitchat-faqs.mdx). Response Selectors are now trained on retrieval intent labels by default instead of the actual response text. For most models, this should improve training time @@ -2419,7 +2419,7 @@ session_config: ### Dialogue Featurization -This section is only relevant if you explicitly defined [featurizers](./building-classic-assistants/policies.mdx#featurizers) +This section is only relevant if you explicitly defined [featurizers](./nlu-based-assistants/policies.mdx#featurizers) in your policy configuration. LabelTokenizerSingleStateFeaturizer is deprecated and will be removed in the future. @@ -2477,12 +2477,12 @@ and SQLProducer have been removed. If you used these brokers in your The deprecated EmbeddingIntentClassifier has been removed. If you used this component in your pipeline configuration (`config.yml`) you can replace it -with [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier). +with [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier). It accepts the same configuration parameters. The deprecated KerasPolicy has been removed. If you used this component in your policies configuration (`config.yml`) you can replace it -with [TEDPolicy](./building-classic-assistants/policies.mdx#ted-policy). It accepts the same configuration parameters. +with [TEDPolicy](./nlu-based-assistants/policies.mdx#ted-policy). It accepts the same configuration parameters. ## Rasa 1.7 to Rasa 1.8 @@ -2495,10 +2495,10 @@ model before trying to use it with this improved version. ### General -- The [TED Policy](./building-classic-assistants/policies.mdx#ted-policy) replaced the `keras_policy` as recommended machine +- The [TED Policy](./nlu-based-assistants/policies.mdx#ted-policy) replaced the `keras_policy` as recommended machine learning policy. New projects generated with `rasa init` will automatically use this policy. In case you want to change your existing model configuration to use the - [TED Policy](./building-classic-assistants/policies.mdx#ted-policy) add this to the `policies` section in your `config.yml` + [TED Policy](./nlu-based-assistants/policies.mdx#ted-policy) add this to the `policies` section in your `config.yml` and remove potentially existing `KerasPolicy` entries: ```yaml-rasa @@ -2511,14 +2511,14 @@ model before trying to use it with this improved version. The given snippet specifies default values for the parameters `max_history` and `epochs`. `max_history` is particularly important and strongly depends on your stories. - Please see the docs of the [TED Policy](./building-classic-assistants/policies.mdx#ted-policy) if you want to customize them. + Please see the docs of the [TED Policy](./nlu-based-assistants/policies.mdx#ted-policy) if you want to customize them. - All pre-defined pipeline templates are deprecated. **Any templates you use will be mapped to the new configuration, but the underlying architecture is the same**. - Take a look at [Tuning Your Model](./building-classic-assistants/tuning-your-model.mdx) to decide on what components you should use + Take a look at [Tuning Your Model](./nlu-based-assistants/tuning-your-model.mdx) to decide on what components you should use in your configuration file. -- The Embedding Policy was renamed to [TED Policy](./building-classic-assistants/policies.mdx#ted-policy). The functionality of the policy stayed the same. +- The Embedding Policy was renamed to [TED Policy](./nlu-based-assistants/policies.mdx#ted-policy). The functionality of the policy stayed the same. Please update your configuration files to use `TEDPolicy` instead of `EmbeddingPolicy`. - Most of the model options for `EmbeddingPolicy`, `EmbeddingIntentClassifier`, and `ResponseSelector` got @@ -2549,7 +2549,7 @@ model before trying to use it with this improved version. Old configuration options will be mapped to the new names, and a warning will be thrown. However, these will be deprecated in a future release. -- The Embedding Intent Classifier is now deprecated and will be replaced by [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier) +- The Embedding Intent Classifier is now deprecated and will be replaced by [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier) in the future. `DIETClassfier` performs intent classification as well as entity recognition. If you want to get the same model behavior as the current `EmbeddingIntentClassifier`, you can use @@ -2573,7 +2573,7 @@ model before trying to use it with this improved version. # ... any other parameters ``` - See [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier) for more information about the new component. + See [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier) for more information about the new component. Specifying `EmbeddingIntentClassifier` in the configuration maps to the above component definition, and results in the same behaviour within the same Rasa version. @@ -2610,14 +2610,14 @@ model before trying to use it with this improved version. ``` `CRFEntityExtractor` featurizes user messages on its own, it does not depend on any featurizer. - We extracted the featurization from the component into the new featurizer [LexicalSyntacticFeaturizer](./building-classic-assistants/components.mdx#lexicalsyntacticfeaturizer). Thus, + We extracted the featurization from the component into the new featurizer [LexicalSyntacticFeaturizer](./nlu-based-assistants/components.mdx#lexicalsyntacticfeaturizer). Thus, in order to obtain the same results as before, you need to add this featurizer to your pipeline before the - [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier). + [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier). Specifying `CRFEntityExtractor` in the configuration maps to the above component definition, the behavior is unchanged from previous versions. - If your pipeline contains `CRFEntityExtractor` and `EmbeddingIntentClassifier` you can substitute both - components with [DIETClassifier](./building-classic-assistants/components.mdx#dietclassifier). You can use the following pipeline for that: + components with [DIETClassifier](./nlu-based-assistants/components.mdx#dietclassifier). You can use the following pipeline for that: ```yaml-rasa pipeline: @@ -2668,7 +2668,7 @@ model before trying to use it with this improved version. - Default parameters of `EmbeddingIntentClassifier` are changed. See the Components page for details. Architecture implementation is changed as well, so **old trained models cannot be loaded**. - Default parameters and architecture for `EmbeddingPolicy` are changed. See [Policies](./building-classic-assistants/policies.mdx) for details. + Default parameters and architecture for `EmbeddingPolicy` are changed. See [Policies](./nlu-based-assistants/policies.mdx) for details. It uses transformer instead of lstm. **Old trained models cannot be loaded**. They use `inner` similarity and `softmax` loss by default instead of `cosine` similarity and `margin` loss (can be set in config file). @@ -2681,7 +2681,7 @@ model before trying to use it with this improved version. - Default `max_history` for `EmbeddingPolicy` is `None` which means it'll use the `FullDialogueTrackerFeaturizer`. We recommend to set `max_history` to some finite value in order to use `MaxHistoryTrackerFeaturizer` - for **faster training**. See [Featurizers](./building-classic-assistants/policies.mdx#featurizers) for details. + for **faster training**. See [Featurizers](./nlu-based-assistants/policies.mdx#featurizers) for details. We recommend to increase `batch_size` for `MaxHistoryTrackerFeaturizer` (e.g. `"batch_size": [32, 64]`) @@ -2751,4 +2751,4 @@ model before trying to use it with this improved version. ### HTTP API -- There are numerous HTTP API endpoint changes which can be found [here](./http-api.mdx). +- There are numerous HTTP API endpoint changes which can be found [here](./production/http-api.mdx). diff --git a/docs/docs/building-classic-assistants/business-logic.mdx b/docs/docs/nlu-based-assistants/business-logic.mdx similarity index 98% rename from docs/docs/building-classic-assistants/business-logic.mdx rename to docs/docs/nlu-based-assistants/business-logic.mdx index 89a65342bd9d..174aab6e9363 100644 --- a/docs/docs/building-classic-assistants/business-logic.mdx +++ b/docs/docs/nlu-based-assistants/business-logic.mdx @@ -153,7 +153,7 @@ responses: ### 2. Updating the configuration -A form's [happy path](./glossary.mdx#happy--unhappy-paths) should be defined as a [rule](rules.mdx) which means you'll need to add the [RulePolicy](./policies.mdx#rule-policy) +A form's [happy path](./glossary.mdx#happy-path--conversation-repair) should be defined as a [rule](rules.mdx) which means you'll need to add the [RulePolicy](./policies.mdx#rule-policy) to your policies: ```yaml-rasa title="config.yml" diff --git a/docs/docs/building-classic-assistants/chitchat-faqs.mdx b/docs/docs/nlu-based-assistants/chitchat-faqs.mdx similarity index 100% rename from docs/docs/building-classic-assistants/chitchat-faqs.mdx rename to docs/docs/nlu-based-assistants/chitchat-faqs.mdx diff --git a/docs/docs/building-classic-assistants/components.mdx b/docs/docs/nlu-based-assistants/components.mdx similarity index 99% rename from docs/docs/building-classic-assistants/components.mdx rename to docs/docs/nlu-based-assistants/components.mdx index 550aebb76df0..38eb097086bd 100644 --- a/docs/docs/building-classic-assistants/components.mdx +++ b/docs/docs/nlu-based-assistants/components.mdx @@ -9,6 +9,8 @@ abstract: --- import RasaNLUBasedBanner from "@theme/RasaNLUBasedBanner"; +import RasaLabsLabel from "@theme/RasaLabsLabel"; +import RasaLabsBanner from "@theme/RasaLabsBanner"; @@ -1409,16 +1411,16 @@ See [starspace paper](https://arxiv.org/abs/1709.03856) for details. - + The LLM-based intent classifier is a new intent classifier that uses large language models (LLMs) to classify intents. The LLM-based intent classifier relies on a method called retrieval augmented generation (RAG), which combines -the benefits of retrieval-based and generation-based approaches. +the benefits of retrieval-based and generation-based approaches. During training, the classifier -1. embeds all intent examples and +1. embeds all intent examples and 2. stores their embeddings in a vector store. During prediction the classifier @@ -1428,7 +1430,7 @@ During prediction the classifier 3. The retrieved examples are ranked based on similarity to the current message and 4. the most similar ones are included in an LLM prompt. The prompt guides the LLM to predict the intent of the message. -5. LLM predicts an intent label. +5. LLM predicts an intent label. 6. The generated label is mapped to an intent of the domain. The LLM can also predict a label that is not part of the training data. In this case, the intent from the domain with the most similar embedding is predicted. @@ -1465,8 +1467,6 @@ pipeline: # - ... ``` - - ### FallbackClassifier - **Short** diff --git a/docs/docs/building-classic-assistants/contextual-conversations.mdx b/docs/docs/nlu-based-assistants/contextual-conversations.mdx similarity index 100% rename from docs/docs/building-classic-assistants/contextual-conversations.mdx rename to docs/docs/nlu-based-assistants/contextual-conversations.mdx diff --git a/docs/docs/building-classic-assistants/domain.mdx b/docs/docs/nlu-based-assistants/domain.mdx similarity index 98% rename from docs/docs/building-classic-assistants/domain.mdx rename to docs/docs/nlu-based-assistants/domain.mdx index 95e399b28887..33939bc4798a 100644 --- a/docs/docs/building-classic-assistants/domain.mdx +++ b/docs/docs/nlu-based-assistants/domain.mdx @@ -31,7 +31,7 @@ rasa train --domain path_to_domain_directory ## Intents The `intents` key in your domain file lists all intents -used in your [NLU data](../building-classic-assistants/nlu-training-data.mdx) and [conversation training data](../building-classic-assistants/training-data-format.mdx#conversation-training-data). +used in your [NLU data](../nlu-based-assistants/nlu-training-data.mdx) and [conversation training data](../nlu-based-assistants/training-data-format.mdx#conversation-training-data). ### Ignoring Entities for Certain Intents @@ -102,7 +102,7 @@ and default behaviour remains unchanged. ::: The `entities` section lists all entities that can be -extracted by any [entity extractor](../building-classic-assistants/components.mdx) in your +extracted by any [entity extractor](../nlu-based-assistants/components.mdx) in your NLU pipeline. For example: @@ -118,7 +118,7 @@ entities: When using multiple domain files, entities can be specified in any domain file, and can be used or ignored by any intent in any domain file. -If you are using the feature [Entity Roles and Groups](../building-classic-assistants/nlu-training-data.mdx#entities-roles-and-groups) you also +If you are using the feature [Entity Roles and Groups](../nlu-based-assistants/nlu-training-data.mdx#entities-roles-and-groups) you also need to list the roles and groups of an entity in this section. For example: @@ -786,7 +786,7 @@ see [Responses](../concepts/responses.mdx). Forms are a special type of action meant to help your assistant collect information from a user. Define forms under the `forms` key in your domain file. -For more information on form and how to define them, see [Forms](../building-classic-assistants/forms.mdx). +For more information on form and how to define them, see [Forms](../nlu-based-assistants/forms.mdx). ## Actions @@ -898,5 +898,5 @@ config: :::note looking for config.yml? If you're looking for information on the `config.yml` file, check out the docs on -[Model Configuration](../building-classic-assistants/model-configuration.mdx). +[Model Configuration](../nlu-based-assistants/model-configuration.mdx). ::: diff --git a/docs/docs/building-classic-assistants/fallback-handoff.mdx b/docs/docs/nlu-based-assistants/fallback-handoff.mdx similarity index 100% rename from docs/docs/building-classic-assistants/fallback-handoff.mdx rename to docs/docs/nlu-based-assistants/fallback-handoff.mdx diff --git a/docs/docs/building-classic-assistants/forms.mdx b/docs/docs/nlu-based-assistants/forms.mdx similarity index 98% rename from docs/docs/building-classic-assistants/forms.mdx rename to docs/docs/nlu-based-assistants/forms.mdx index d06c1c62888d..5ea0c8ef2ba6 100644 --- a/docs/docs/building-classic-assistants/forms.mdx +++ b/docs/docs/nlu-based-assistants/forms.mdx @@ -139,7 +139,7 @@ rules: ``` Users might want to break out of a form early. Please see -[Writing Stories / Rules for Unhappy Form Paths](./forms.mdx#writing-stories--rules-for-unhappy-form-paths) on how to +[Writing Stories / Rules for Conversation Repair cases](./forms.mdx#writing-stories--rules-for-conversation-repair-cases) on how to write stories or rules for this case. ### Slot Mappings @@ -153,7 +153,7 @@ Note specifically the role of [Mapping Conditions](./domain.mdx#mapping-conditio [unique entity mapping](./domain.mdx#unique-from_entity-mapping-matching) constraint. ::: -### Writing Stories / Rules for Unhappy Form Paths +### Writing Stories / Rules for Conversation Repair cases Your users will not always respond with the information you ask of them. Typically, users will ask questions, make chitchat, change their mind, or otherwise @@ -180,12 +180,12 @@ you could add a rule to handle this: ```yaml-rasa rules: -- rule: Example of an unhappy path +- rule: Example of conversation repair condition: # Condition that form is active. - active_loop: restaurant_form steps: - # This unhappy path handles the case of an intent `chitchat`. + # This case handles the case of an intent `chitchat`. - intent: chitchat - action: utter_chitchat # Return to form after handling the `chitchat` intent @@ -430,7 +430,7 @@ ignored during conversations. If you want to change this behavior, you need to a the `requested_slot` to your domain file as a categorical slot with `influence_conversation` set to `true`. You might want to do this if you -want to handle your unhappy paths differently, depending on what slot is +want to handle conversation repair differently, depending on what slot is currently being asked from the user. For example, if your users respond to one of the bot's questions with another question, like _why do you need to know that?_ The response to this `explain` intent depends on where we are in the story. diff --git a/docs/docs/building-classic-assistants/glossary.mdx b/docs/docs/nlu-based-assistants/glossary.mdx similarity index 83% rename from docs/docs/building-classic-assistants/glossary.mdx rename to docs/docs/nlu-based-assistants/glossary.mdx index d365c0251585..de753352a80e 100644 --- a/docs/docs/building-classic-assistants/glossary.mdx +++ b/docs/docs/nlu-based-assistants/glossary.mdx @@ -26,15 +26,15 @@ Adding labels to messages and conversations so that they can be used to train a The process of replacing personally identifiable information (PII) with masked, artificial or constant text values. This is done to protect the privacy of users. -## [Business Logic](../building-classic-assistants/business-logic.mdx) +## [Business Logic](../nlu-based-assistants/business-logic.mdx) -Conditions that need to be fulfilled due to business requirements. For example: requiring a first and last name, an address, and a password before an account can be created. In a Rasa assistant, business logic is implemented using rule-based actions like [forms](../building-classic-assistants/forms.mdx). +Conditions that need to be fulfilled due to business requirements. For example: requiring a first and last name, an address, and a password before an account can be created. In a Rasa assistant, business logic is implemented using rule-based actions like [forms](../nlu-based-assistants/forms.mdx). -## [Chitchat](../building-classic-assistants/chitchat-faqs.mdx) +## [Chitchat](../nlu-based-assistants/chitchat-faqs.mdx) A conversation pattern where the user says something that isn't directly related to their goal. This can include things like greetings, asking how you are etc. -Read about handling [Chitchat and FAQs](../building-classic-assistants/chitchat-faqs.mdx) to learn how to implement this with Rasa. +Read about handling [Chitchat and FAQs](../nlu-based-assistants/chitchat-faqs.mdx) to learn how to implement this with Rasa. ## CMS @@ -48,9 +48,9 @@ The process of using user messages and conversation data to influence the design Modified story format that includes the full text of the user message in addition to the intent label. Test conversations are saved to a test set file (conversation_tests.md), which is used to evaluate the model’s predictions across an entire conversation. -## [Component](../building-classic-assistants/components.mdx) +## [Component](../nlu-based-assistants/components.mdx) -An element in the an assistant's [NLU pipeline](../building-classic-assistants/tuning-your-model.mdx#how-to-choose-a-pipeline) in the [Model Configuration](../building-classic-assistants/model-configuration.mdx). +An element in the an assistant's [NLU pipeline](../nlu-based-assistants/tuning-your-model.mdx#how-to-choose-a-pipeline) in the [Model Configuration](../nlu-based-assistants/model-configuration.mdx). Incoming messages are processed by a sequence of components called a pipeline. A component can perform tasks ranging from entity extraction to intent classification to pre-processing. @@ -66,7 +66,7 @@ An action written by a bot developer that can run arbitrary code, mainly to inte A built-in action that comes with predefined functionality. -## [DIET](../building-classic-assistants/components.mdx#dietclassifier) +## [DIET](../nlu-based-assistants/components.mdx#dietclassifier) Dual Intent and Entity Transformer. The default NLU architecture used by Rasa, which performs both intent classification and entity extraction. @@ -76,7 +76,7 @@ Defines the inputs and outputs of an assistant. It includes a list of all the intents, entities, slots, actions, and forms that the assistant knows about. -## [Entity](../building-classic-assistants/training-data-format.mdx#entities) +## [Entity](../nlu-based-assistants/training-data-format.mdx#entities) Keywords that can be extracted from a user message. For example: a telephone number, a person's name, a location, the name of a product @@ -88,19 +88,23 @@ Something that happens in a conversation. For instance, a `UserUttered` event re Frequently asked questions (FAQs) are common questions that your users ask. In the context of building an assistant, this typically means the user sends a message and the assistant send a response without needing to consider the context of the conversation. -Read about handling [Chitchat and FAQs](../building-classic-assistants/chitchat-faqs.mdx) to learn how to implement this with Rasa. +Read about handling [Chitchat and FAQs](../nlu-based-assistants/chitchat-faqs.mdx) to learn how to implement this with Rasa. -## [Form](../building-classic-assistants/forms.mdx) +## [Form](../nlu-based-assistants/forms.mdx) A type of custom action that asks the user for multiple pieces of information. For example, if you need a city, a cuisine, and a price range to recommend a restaurant, you can create a restaurant form to collect the information. You can describe [business logic](#business-logic) inside a form, like offering the customer a different set of menu options if they mention a food allergy. -## Happy / Unhappy Paths +## Happy Path / Conversation Repair -Terms used to describe whether the user’s input is expected or unexpected. If your assistant asks a user for some information and the user provides it, we call that a happy path. Unhappy paths are all possible edge cases. For example, the user refusing to give the requested input, changing the topic of conversation, or correcting something they said earlier. +Terms used to describe whether the user’s input follows an expected flow or diverges +from the expected flow, which happens often in reality. +If your assistant asks a user for some information and the user provides it, we call that a happy path. +Conversation repair intervenes when, for example, the user refuses to give the requested input, +changes the topic of conversation, or corrects something they said earlier. -## [Intent](../building-classic-assistants/nlu-training-data.mdx) +## [Intent](../nlu-based-assistants/nlu-training-data.mdx) In a given user message, the thing that a user is trying to convey or accomplish (e,g., greeting, specifying a location). @@ -130,15 +134,15 @@ Natural Language Generation (NLG) is the process of generating natural language Rasa uses a simple template-based approach for NLG. Data-driven approaches (such as neural NLG) can be implemented by creating a custom NLG component. -## [NLU](../building-classic-assistants/nlu-training-data.mdx) +## [NLU](../nlu-based-assistants/nlu-training-data.mdx) Natural Language Understanding (NLU) deals with parsing and understanding human language into a structured format. -## [Pipeline](../building-classic-assistants/tuning-your-model.mdx) +## [Pipeline](../nlu-based-assistants/tuning-your-model.mdx) The list of NLU components (see [NLU Component](#nlu-component)) that defines a Rasa assistant’s NLU system. A user message is processed by each component one by one, before returning the final structured output. -## [Policy](../building-classic-assistants/policies.mdx) +## [Policy](../nlu-based-assistants/policies.mdx) Rasa components that predict the dialogue system’s next actionPolicies make decisions about how the conversation flow should proceed. A typical configuration includes multiple policies, and the policy with the highest confidence decides the next action to be taken in the conversation. @@ -154,7 +158,7 @@ The dialogue engine that decides what to do next in a conversation based on the Rasa NLU is the part of Rasa that performs Natural Language Understanding ([NLU](#nlu)), including intent classification and entity extraction. -## [NLU Component](../building-classic-assistants/components.mdx) +## [NLU Component](../nlu-based-assistants/components.mdx) An element in the Rasa NLU pipeline (see [Pipeline](#pipeline)) that processes incoming messages. Components perform tasks ranging from entity extraction to intent classification to pre-processing. @@ -162,7 +166,7 @@ An element in the Rasa NLU pipeline (see [Pipeline](#pipeline)) that processes i A tool for conversation-driven development. Rasa X/Enterprise helps teams share and test an assistant built with Rasa, annotate user messages, and view conversations. -## [Retrieval Intent](../building-classic-assistants/chitchat-faqs.mdx) +## [Retrieval Intent](../nlu-based-assistants/chitchat-faqs.mdx) A special type of intent that can be divided into smaller sub-intents. For example, an FAQ retrieval intent has sub-intents that represent each individual question the assistant knows how to answer. @@ -174,21 +178,21 @@ A messaging channel used to build custom connectors. Includes an input channel, A message that an assistant sends to a user. This can include text, buttons, images, and other content. -## [Rules](../building-classic-assistants/rules.mdx) +## [Rules](../nlu-based-assistants/rules.mdx) Special training data to specify rule-like behavior, where a specific condition always predicts a specific next action. Examples include -answering FAQs, filling [Forms](../building-classic-assistants/forms.mdx), or handling -[Fallbacks](../building-classic-assistants/fallback-handoff.mdx#fallbacks). +answering FAQs, filling [Forms](../nlu-based-assistants/forms.mdx), or handling +[Fallbacks](../nlu-based-assistants/fallback-handoff.mdx#fallbacks). ## [Slot](./domain.mdx#slots) A key-value store that Rasa uses to track information over the course of a conversation. -## [Story](../building-classic-assistants/stories.mdx) +## [Story](../nlu-based-assistants/stories.mdx) Training data format for the dialogue model, consisting of a conversation between a user and a bot. The user's messages are represented as annotated intents and entities, and the bot’s responses are represented as a sequence of actions. -## [TED Policy](../building-classic-assistants/policies.mdx#ted-policy) +## [TED Policy](../nlu-based-assistants/policies.mdx#ted-policy) Transformer Embedding Dialogue Policy. TED is the default machine learning-based dialogue policy used by Rasa. TED complements rule-based policies by handling previously unseen situations, where no rule exists to determine the next action. diff --git a/docs/docs/building-classic-assistants/language-support.mdx b/docs/docs/nlu-based-assistants/language-support.mdx similarity index 100% rename from docs/docs/building-classic-assistants/language-support.mdx rename to docs/docs/nlu-based-assistants/language-support.mdx diff --git a/docs/docs/building-classic-assistants/model-configuration.mdx b/docs/docs/nlu-based-assistants/model-configuration.mdx similarity index 100% rename from docs/docs/building-classic-assistants/model-configuration.mdx rename to docs/docs/nlu-based-assistants/model-configuration.mdx diff --git a/docs/docs/nlu-only-server.mdx b/docs/docs/nlu-based-assistants/nlu-only-server.mdx similarity index 88% rename from docs/docs/nlu-only-server.mdx rename to docs/docs/nlu-based-assistants/nlu-only-server.mdx index 3015998e859c..ed37258fa092 100644 --- a/docs/docs/nlu-only-server.mdx +++ b/docs/docs/nlu-based-assistants/nlu-only-server.mdx @@ -8,7 +8,7 @@ abstract: You can run an NLU-only server and use the HTTP API to connect to it. ## Connecting to an NLU server -You can connect a [Rasa NLU-only server](./building-classic-assistants/nlu-only.mdx#running-an-nlu-server) to a separately running Rasa dialogue management only server +You can connect a [Rasa NLU-only server](./nlu-only.mdx#running-an-nlu-server) to a separately running Rasa dialogue management only server by adding the connection details to the dialogue management server's endpoint configuration file: ```yaml title="endpoints.yml" @@ -18,7 +18,7 @@ nlu: token_name: # [optional] (default: token) ``` -The `token` and `token_name` refer to optional [authentication parameters](./http-api.mdx#token-based-auth). +The `token` and `token_name` refer to optional [authentication parameters](../production/http-api.mdx#token-based-auth). The dialogue management server should serve a model that does not include an NLU model. To obtain a dialogue management only model, train a model with `rasa train core` or use diff --git a/docs/docs/building-classic-assistants/nlu-only.mdx b/docs/docs/nlu-based-assistants/nlu-only.mdx similarity index 100% rename from docs/docs/building-classic-assistants/nlu-only.mdx rename to docs/docs/nlu-based-assistants/nlu-only.mdx diff --git a/docs/docs/building-classic-assistants/nlu-training-data.mdx b/docs/docs/nlu-based-assistants/nlu-training-data.mdx similarity index 100% rename from docs/docs/building-classic-assistants/nlu-training-data.mdx rename to docs/docs/nlu-based-assistants/nlu-training-data.mdx diff --git a/docs/docs/building-classic-assistants/policies.mdx b/docs/docs/nlu-based-assistants/policies.mdx similarity index 100% rename from docs/docs/building-classic-assistants/policies.mdx rename to docs/docs/nlu-based-assistants/policies.mdx diff --git a/docs/docs/building-classic-assistants/reaching-out-to-user.mdx b/docs/docs/nlu-based-assistants/reaching-out-to-user.mdx similarity index 100% rename from docs/docs/building-classic-assistants/reaching-out-to-user.mdx rename to docs/docs/nlu-based-assistants/reaching-out-to-user.mdx diff --git a/docs/docs/building-classic-assistants/rules.mdx b/docs/docs/nlu-based-assistants/rules.mdx similarity index 98% rename from docs/docs/building-classic-assistants/rules.mdx rename to docs/docs/nlu-based-assistants/rules.mdx index 3669b60fb71c..7ec5d48f5967 100644 --- a/docs/docs/building-classic-assistants/rules.mdx +++ b/docs/docs/nlu-based-assistants/rules.mdx @@ -192,5 +192,5 @@ how the form is defined, ignoring rules. Rules become applicable again if: - the form fills all required slots - the form rejects its execution (see - [Handling Unhappy Paths](./forms.mdx#writing-stories--rules-for-unhappy-form-paths) for + [Conversation Repair](./forms.mdx#writing-stories--rules-for-conversation-repair-cases) for more details) diff --git a/docs/docs/building-classic-assistants/stories.mdx b/docs/docs/nlu-based-assistants/stories.mdx similarity index 100% rename from docs/docs/building-classic-assistants/stories.mdx rename to docs/docs/nlu-based-assistants/stories.mdx diff --git a/docs/docs/building-classic-assistants/testing-your-assistant.mdx b/docs/docs/nlu-based-assistants/testing-your-assistant.mdx similarity index 97% rename from docs/docs/building-classic-assistants/testing-your-assistant.mdx rename to docs/docs/nlu-based-assistants/testing-your-assistant.mdx index 29a9095617a5..58214ed7ad7a 100644 --- a/docs/docs/building-classic-assistants/testing-your-assistant.mdx +++ b/docs/docs/nlu-based-assistants/testing-your-assistant.mdx @@ -38,7 +38,7 @@ always good to run this check before training a model. By including the `--fail-on-warnings` flag, this step will fail on warnings indicating more minor issues. :::note -Running `rasa data validate` does **not** test if your [rules](../building-classic-assistants/rules.mdx) are consistent with your stories. +Running `rasa data validate` does **not** test if your [rules](../nlu-based-assistants/rules.mdx) are consistent with your stories. However, during training, the `RulePolicy` checks for conflicts between rules and stories. Any such conflict will abort training. ::: @@ -59,7 +59,7 @@ the stories in your training data, but include the user message as well. Here are some examples: - + ```yaml-rasa title="tests/test_stories.yml" {5,9,13} @@ -148,7 +148,7 @@ stories: ``` - + ```yaml-rasa title="tests/test_stories.yml" {5,9,14,21} stories: @@ -386,7 +386,7 @@ confidence histogram (`response_selection_histogram.png`) and errors (`response_ If your pipeline includes multiple response selectors, they are evaluated in a single report. The report logs precision, recall and f1 measure for -each sub-intent of a [retrieval intent](../building-classic-assistants/glossary.mdx#retrieval-intent) and provides an overall average. +each sub-intent of a [retrieval intent](../nlu-based-assistants/glossary.mdx#retrieval-intent) and provides an overall average. You can save these reports as JSON files using the `--report` argument. #### Entity Extraction @@ -423,7 +423,7 @@ multiple tokens. #### Entity Scoring To evaluate entity extraction we apply a simple tag-based approach. We don't consider -[BILOU tags](../building-classic-assistants/nlu-training-data.mdx#bilou-entity-tagging) exactly, but only the +[BILOU tags](../nlu-based-assistants/nlu-training-data.mdx#bilou-entity-tagging) exactly, but only the entity type tags on a per token basis. For location entity like “near Alexanderplatz” we expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`. @@ -475,7 +475,7 @@ policies failed to do so, then the corresponding story will end up in `results/f a failed story. The stories are sorted by the severity of `action_unlikely_intent`'s prediction. -This severity is calculated by [`UnexpecTEDIntentPolicy`](../building-classic-assistants/policies.mdx#unexpected-intent-policy) itself at prediction time. +This severity is calculated by [`UnexpecTEDIntentPolicy`](../nlu-based-assistants/policies.mdx#unexpected-intent-policy) itself at prediction time. The higher the severity, the more unlikely is the intent and hence reviewing that particular conversation path becomes more critical. diff --git a/docs/docs/building-classic-assistants/training-data-format.mdx b/docs/docs/nlu-based-assistants/training-data-format.mdx similarity index 95% rename from docs/docs/building-classic-assistants/training-data-format.mdx rename to docs/docs/nlu-based-assistants/training-data-format.mdx index 0183127aa49c..9ea650412bce 100644 --- a/docs/docs/building-classic-assistants/training-data-format.mdx +++ b/docs/docs/nlu-based-assistants/training-data-format.mdx @@ -21,7 +21,7 @@ You can split the training data over any number of YAML files, and each file can contain any combination of NLU data, stories, and rules. The training data parser determines the training data type using top level keys. -The [domain](../building-classic-assistants/glossary.mdx#domain) uses +The [domain](../nlu-based-assistants/glossary.mdx#domain) uses the same YAML format as the training data and can also be split across multiple files or combined in one file. The domain includes the definitions for [responses](../concepts/responses.mdx) and [forms](./forms.mdx). @@ -117,8 +117,8 @@ training examples. ## NLU Training Data -[NLU](../building-classic-assistants/glossary.mdx#nlu) training data consists of example user utterances categorized by -[intent](../building-classic-assistants/glossary.mdx#intent). Training examples can also include [entities](../building-classic-assistants/glossary.mdx#entity). Entities are structured +[NLU](../nlu-based-assistants/glossary.mdx#nlu) training data consists of example user utterances categorized by +[intent](../nlu-based-assistants/glossary.mdx#intent). Training examples can also include [entities](../nlu-based-assistants/glossary.mdx#entity). Entities are structured pieces of information that can be extracted from a user's message. You can also add extra information such as regular expressions and lookup tables to your training data to help the model identify intents and entities correctly. @@ -168,7 +168,7 @@ nlu: ### Training Examples -Training examples are grouped by [intent](../building-classic-assistants/glossary.mdx#intent) and listed under the +Training examples are grouped by [intent](../nlu-based-assistants/glossary.mdx#intent) and listed under the `examples` key. Usually, you'll list one example per line as follows: ```yaml-rasa @@ -215,7 +215,7 @@ nlu: In this case, the content of the `metadata` key is passed to every intent example. -If you want to specify [retrieval intents](../building-classic-assistants/glossary.mdx#retrieval-intent), then your NLU examples will look as follows: +If you want to specify [retrieval intents](../nlu-based-assistants/glossary.mdx#retrieval-intent), then your NLU examples will look as follows: ```yaml-rasa nlu: @@ -247,7 +247,7 @@ name of your intents. ### Entities -[Entities](../building-classic-assistants/glossary.mdx#entity) are structured pieces of information that can be extracted from a user's message. +[Entities](../nlu-based-assistants/glossary.mdx#entity) are structured pieces of information that can be extracted from a user's message. Entities are annotated in training examples with the entity's name. In addition to the entity name, you can annotate an entity with [synonyms](nlu-training-data.mdx#synonyms), [roles, or groups](nlu-training-data.mdx#entities-roles-and-groups). @@ -461,12 +461,12 @@ stories: #### Forms -A [form](../building-classic-assistants/glossary.mdx#form) is a specific kind of custom action that contains the logic to loop over +A [form](../nlu-based-assistants/glossary.mdx#form) is a specific kind of custom action that contains the logic to loop over a set of required slots and ask the user for this information. You [define a form](forms.mdx#defining-a-form) in the `forms` section in your domain. -Once defined, you should specify the [happy path](../building-classic-assistants/glossary.mdx#happy--unhappy-paths) +Once defined, you should specify the [happy path](../nlu-based-assistants/glossary.mdx#happy-path--conversation-repair) for a form as a [rule](./forms.mdx). You should include interruptions of forms or -other "unhappy paths" in stories so that the model can +other conversation repair in stories so that the model can generalize to unseen conversation sequences. As a step in a story, a form takes the following format: diff --git a/docs/docs/building-classic-assistants/tuning-your-model.mdx b/docs/docs/nlu-based-assistants/tuning-your-model.mdx similarity index 100% rename from docs/docs/building-classic-assistants/tuning-your-model.mdx rename to docs/docs/nlu-based-assistants/tuning-your-model.mdx diff --git a/docs/docs/building-classic-assistants/unexpected-input.mdx b/docs/docs/nlu-based-assistants/unexpected-input.mdx similarity index 99% rename from docs/docs/building-classic-assistants/unexpected-input.mdx rename to docs/docs/nlu-based-assistants/unexpected-input.mdx index 9bd1cb0f1afd..ba19e1b114dc 100644 --- a/docs/docs/building-classic-assistants/unexpected-input.mdx +++ b/docs/docs/nlu-based-assistants/unexpected-input.mdx @@ -11,7 +11,7 @@ import RasaNLUBasedBanner from "@theme/RasaNLUBasedBanner"; -Unexpected input is a deviation from the [happy path](./glossary.mdx#happy--unhappy-paths) +Unexpected input is a deviation from the [happy path](./glossary.mdx#happy-path--conversation-repair) that you have defined. For example: - A user walks away in the middle of a conversation about their subscription, then comes back diff --git a/docs/docs/operating/spaces.mdx b/docs/docs/operating/spaces.mdx index f7cdbc1f3a65..f98e9ce8c78a 100644 --- a/docs/docs/operating/spaces.mdx +++ b/docs/docs/operating/spaces.mdx @@ -23,11 +23,11 @@ If you have feedback (positive or negative) please share it with us via your sup ::: Spaces are a way to modularize your assistant. As assistants grow in scale and scope, the -potential for conflicts between intents, entities, and other [primitives](../building-classic-assistants/glossary.mdx#rasa-primitive) grows as well. +potential for conflicts between intents, entities, and other [primitives](../nlu-based-assistants/glossary.mdx#rasa-primitive) grows as well. That is because in a regular rasa assistant all intents, entities, and actions are available at any time, giving the assistant more choices to distinguish between the more that gets added. Spaces provide a basic separation between parts of the bot through the activation -and deactivation of groups of [rasa primitives](../building-classic-assistants/glossary.mdx#rasa-primitive). Each of these groups is called a Space. +and deactivation of groups of [rasa primitives](../nlu-based-assistants/glossary.mdx#rasa-primitive). Each of these groups is called a Space. Spaces are activated when certain designated intents, called entry intents, are predicted. With the activation of a Space, all the primitives for the Space become available for subsequent interactions with the user. Good ways to think about spaces are @@ -218,7 +218,7 @@ generated during assembly. ### How does filter and rerank work? The filter and rerank component post-processes the intent ranking of the [intent -classification components](../building-classic-assistants/components.mdx#intent-classifiers). It accesses the [tracker](../action-server/sdk-tracker.mdx) +classification components](../nlu-based-assistants/components.mdx#intent-classifiers). It accesses the [tracker](../action-server/sdk-tracker.mdx) and checks which space are active or would be activated, in case an entry intent is at the top. It then removes any intents from the ranking that are not possible. Further it also @@ -254,8 +254,8 @@ retrieval intent belongs to the main space, no prefix is added. That's why ### Lookup tables and Synonyms -[Lookup Tables](../building-classic-assistants/training-data-format.mdx#lookup-tables) and -[Synonyms](../building-classic-assistants/training-data-format.mdx#synonyms) work with spaces. However, they are not +[Lookup Tables](../nlu-based-assistants/training-data-format.mdx#lookup-tables) and +[Synonyms](../nlu-based-assistants/training-data-format.mdx#synonyms) work with spaces. However, they are not truly isolated between spaces. So there can be some unanticipated interactions. For synonyms, specifically: @@ -267,10 +267,8 @@ For synonyms, specifically: For lookup tables no adverse interactions are known. - ## Training Data Importers - Using the [`--data` command line argument](../command-line-interface.mdx) you can specify where Rasa should look for training data on your disk. Rasa then loads any potential training files and uses them to train your assistant. @@ -401,7 +399,7 @@ their training data will be combined. ## Writing a Custom Importer -If you are writing a custom importer, this importer has to implement the interface of +If you are writing a custom importer, this importer has to implement the interface of `TrainingDataImporter`: ```python diff --git a/docs/docs/http-api.mdx b/docs/docs/production/http-api.mdx similarity index 94% rename from docs/docs/http-api.mdx rename to docs/docs/production/http-api.mdx index aa2ec1d61f9a..af5315d5a352 100644 --- a/docs/docs/http-api.mdx +++ b/docs/docs/production/http-api.mdx @@ -31,7 +31,7 @@ dialogue model is needed to process the request. :::caution Make sure to secure your server, either by restricting access to the server (e.g. using firewalls), or -by enabling an authentication method. See [Security Considerations](./http-api.mdx#security-considerations). +by enabling an authentication method. See [Security Considerations](#security-considerations). ::: @@ -41,10 +41,10 @@ recommended that you set the number of workers to the number of available CPU co (check out the [Sanic docs](https://sanicframework.org/en/guide/deployment/running.html#workers) for more details). This will only work in combination with the -`RedisLockStore` (see [Lock Stores](./production/lock-stores.mdx). +`RedisLockStore` (see [Lock Stores](./lock-stores.mdx). :::caution -The [SocketIO channel](./connectors/your-own-website.mdx#websocket-channel) does not support multiple worker processes. +The [SocketIO channel](../connectors/your-own-website.mdx#websocket-channel) does not support multiple worker processes. ::: diff --git a/docs/docs/production/setting-up-ci-cd.mdx b/docs/docs/production/setting-up-ci-cd.mdx index aa36d16651bc..639de970adef 100644 --- a/docs/docs/production/setting-up-ci-cd.mdx +++ b/docs/docs/production/setting-up-ci-cd.mdx @@ -108,7 +108,7 @@ succeeded. ### Deploying Your Rasa Model -If you ran [test stories](../building-classic-assistants/testing-your-assistant.mdx) in your CI pipeline, +If you ran [test stories](../nlu-based-assistants/testing-your-assistant.mdx) in your CI pipeline, you'll already have a trained model. You can set up your CD pipeline to upload the trained model to your Rasa server if the CI results are satisfactory. For example, to upload a model to Rasa X/Enterprise: diff --git a/docs/docs/tutorial.mdx b/docs/docs/tutorial.mdx index a4c390a042c1..17abc598f169 100644 --- a/docs/docs/tutorial.mdx +++ b/docs/docs/tutorial.mdx @@ -6,7 +6,7 @@ className: hide abstract: --- -import useBaseUrl from '@docusaurus/useBaseUrl'; +import useBaseUrl from "@docusaurus/useBaseUrl"; import TutorialActionLabel from "@theme/TutorialActionLabel"; You will build an assistant in this tutorial for helping people transfer money. @@ -14,7 +14,6 @@ This tutorial does not assume and existing knowledge of Rasa or chatbots. The techniques you will learn in this tutorial are fundamental to building any Rasa bot, and understanding it will bring you quite far along to mastering Rasa. - ## What are you building? In this tutorial, you will build a robust assistant that can complete a money transfer, and @@ -22,26 +21,37 @@ doesn't get confused as soon as a user says something unexpected. Here are some of the conversations your assistant will be able to handle: - + - I'd like to transfer some money - Sure - who are we sending money to? - Jen - How much would you like to send? - $50 - Just to confirm: you want to send $50 to Jen? - yes - Ok, it's done! + I'd like to transfer some money + Sure - who are we sending money to? + Jen + How much would you like to send? + $50 + Just to confirm: you want to send $50 to Jen? + yes + Ok, it's done! - I want to send $100 to Joe - Just to confirm, you want to send $100.00 to Joe Smith? - yes. - All done. $100.00 has been sent to Joe Smith. - Is there anything else I can help you with? + I want to send $100 to Joe + + Just to confirm, you want to send $100.00 to Joe Smith? + + yes. + All done. $100.00 has been sent to Joe Smith. + Is there anything else I can help you with? @@ -58,15 +68,15 @@ rasa init --template tutorial ``` By default, this tutorial uses the OpenAI API. -If you'd like to use a different LLM, follow the instructions [here](./llms/llm-configuration.mdx#other-llmsembeddings) +If you'd like to use a different LLM, follow the instructions [here](./concepts/components/llm-configuration.mdx#other-llmsembeddings) To set your OpenAI API key, run the following command: - ```shell - export OPENAI_API_KEY= - ``` +```shell +export OPENAI_API_KEY= +``` @@ -82,15 +92,14 @@ To set your OpenAI API key, run the following command: Replacing `` with the actual API key you obtained from the OpenAI platform. - ## Overview Open up the project folder in your IDE to see the files that make up your new project. In this tutorial you will primarily work with the following files: -* `data/flows.yml` -* `domain.yml` -* `actions.py` +- `data/flows.yml` +- `domain.yml` +- `actions.py` ## Testing your money transfer flow @@ -136,13 +145,11 @@ But it is also helpful for anyone who inspects your code to understand what is g If a user says "I need to transfer some money", the description helps Rasa understand that this is the relevant flow. The `steps` describe the business logic required to do what the user asked for. - There are three steps defined, each with an `id`. The `id` can be anything you like, but it's a good idea to use names that make it clear what is going on. The first step is a `collect_information` step, which is used to fill a `slot`. A `collect_information` step sends a message to the user requesting information, and waits for an answer. - ## Collecting Information in Slots `Slots` are a key-value memory store used during a conversation. @@ -177,7 +184,7 @@ flows: action: utter_transfer_complete ``` -Rasa will look for a `response` called `utter_ask_recipient` in your domain file and use this to +Rasa will look for a `response` called `utter_ask_recipient` in your domain file and use this to phrase the question to the user. ```yaml-rasa title="domain.yml" @@ -265,7 +272,6 @@ Notice that your confirmation question uses curly brackets `{}` to include slot Add a `collect_information` step to your flow and give it the id `confirm_transfer`. Also change the `next` attribute of the `ask_amount` step to match the `id` of your newly created step: `confirm_transfer`. - ```yaml-rasa title="flows.yml" flows: transfer_money: @@ -298,7 +304,6 @@ You can learn more about conditions and advanced logic here (link). To try out the updated version of your assistant, run `rasa train`, and then `rasa shell` to talk to your bot. It should now ask you to confirm before completing the transfer. - ## Integrating an API call An `action` step in a flow can describe two types of actions. @@ -331,9 +336,9 @@ class ActionSufficientFunds(Action): def run(self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[Text, Any]) -> List[Dict[Text, Any]]: - # hard-coded balance for tutorial purposes. in production this + # hard-coded balance for tutorial purposes. in production this # would be retrieved from a database or an API - balance = 1000 + balance = 1000 transfer_amount = tracker.get_slot("amount") has_sufficient_funds = transfer_amount <= balance return [SlotSet("has_sufficient_funds", has_sufficient_funds)] @@ -343,9 +348,10 @@ Slots are the primary way to pass information to and from custom actions. In the `run()` method above, you access the value of the `amount` slot that was set during the conversation, and you pass information back to the conversation by returning a `SlotSet` event to update the `has_sufficient_funds` slot. - -diagram of how slots are used with custom actions - +diagram of how slots are used with custom actions Now you are going to make three additions to your `domain.yml`. You will add a top-level section listing your custom actions. @@ -376,7 +382,7 @@ responses: ``` Now you are going to update your flow logic to handle the cases where the user does or does not have - enough money in their account to make the transfer. +enough money in their account to make the transfer. ```yaml-rasa title="flows.yml" flows: @@ -432,29 +438,36 @@ you don't have enough funds in your account. At this point you have experience using some of the key concepts involved in building with Rasa. Congratulations! Take some time to look over these more advanced topics to learn even more. - ## How-to Guides for Advanced Topics ### Slot Validation + Use custom logic to determine whether to accept or reject the value of a slot provided by the user. ### Handling Corrections + Handle cases where the user changes their mind about something and provides an new answer to an earlier question. ### Handling Digressions + Handle cases where users ask clarifying questions, make small talk, etc. ### Customising Patterns + Customise how Rasa switches between flows and other common conversational patterns. ### Contextual Paraphrasing + Use LLMs to paraphrase templated responses to account for context and increase fluency. ### Linking Flows + Link multiple flows together, build shared sub-flows ### Integrating Search & RAG + Integrate flow-based and knowledge-based dialogue into fluent conversations. ### Handling Disambiguation + Automatically ask clarifying questions when it is not clear which flow applies. diff --git a/docs/sidebars.js b/docs/sidebars.js index 1a3917fa5779..6fb0ad6746a2 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -38,7 +38,7 @@ module.exports = { "concepts/flows", "concepts/dialogue-understanding", "concepts/domain", - "concepts/unhappy-paths", + "concepts/conversation-repair", { type: "category", label: "Actions", @@ -62,8 +62,8 @@ module.exports = { label: "Components", items: [ "concepts/components/overview", - "llms/llm-configuration", - "llms/llm-custom", + "concepts/components/llm-configuration", + "concepts/components/llm-custom", "concepts/components/custom-graph-components", "concepts/components/graph-recipe", ], @@ -71,7 +71,7 @@ module.exports = { "concepts/policies", // TODO: ENG-538 { type: "category", - label: "Building Classic Assistants", + label: "NLU-based Assistants", collapsed: true, items: [ // TODO: ENG-537 @@ -80,33 +80,34 @@ module.exports = { label: "Conversation Patterns", collapsed: true, items: [ - "building-classic-assistants/chitchat-faqs", - "building-classic-assistants/business-logic", - "building-classic-assistants/fallback-handoff", - "building-classic-assistants/unexpected-input", - "building-classic-assistants/contextual-conversations", - "building-classic-assistants/reaching-out-to-user", + "nlu-based-assistants/chitchat-faqs", + "nlu-based-assistants/business-logic", + "nlu-based-assistants/fallback-handoff", + "nlu-based-assistants/unexpected-input", + "nlu-based-assistants/contextual-conversations", + "nlu-based-assistants/reaching-out-to-user", ], }, { type: "category", label: "Config", items: [ - "building-classic-assistants/model-configuration", - "building-classic-assistants/components", // TODO: ENG-538 - "building-classic-assistants/policies", // TODO: ENG-538 - "building-classic-assistants/language-support", + "nlu-based-assistants/model-configuration", + "nlu-based-assistants/components", // TODO: ENG-538 + "nlu-based-assistants/policies", // TODO: ENG-538 + "nlu-based-assistants/language-support", ], }, - "building-classic-assistants/domain", - "building-classic-assistants/rules", - "building-classic-assistants/stories", - "building-classic-assistants/forms", - "building-classic-assistants/training-data-format", - "building-classic-assistants/nlu-training-data", - "building-classic-assistants/tuning-your-model", - "building-classic-assistants/nlu-only", - "building-classic-assistants/testing-your-assistant", + "nlu-based-assistants/domain", + "nlu-based-assistants/rules", + "nlu-based-assistants/stories", + "nlu-based-assistants/forms", + "nlu-based-assistants/training-data-format", + "nlu-based-assistants/nlu-training-data", + "nlu-based-assistants/tuning-your-model", + "nlu-based-assistants/testing-your-assistant", + "nlu-based-assistants/nlu-only", + "nlu-based-assistants/nlu-only-server", ], }, ], @@ -218,6 +219,7 @@ module.exports = { ], }, "production/load-testing-guidelines", + "production/http-api", ], }, { @@ -250,18 +252,12 @@ module.exports = { "operating/spaces", ], }, - { - type: "category", - label: "APIs", - collapsed: true, - items: ["http-api", "nlu-only-server"], - }, { type: "category", label: "Reference", collapsed: true, items: [ - "building-classic-assistants/glossary", + "nlu-based-assistants/glossary", "telemetry/telemetry", "telemetry/reference", require("./docs/reference/sidebar.json"),