+ ![image]({useBaseUrl("/img/intent_confusion_matrix_example.png")})
+
+
+The histogram allows you to visualize the confidence for all predictions,
+with the correct and incorrect predictions being displayed by blue and red bars respectively.
+Improving the quality of your training data will move the blue histogram bars up the plot and the
+red histogram bars down the plot. It should also help in reducing the number of red histogram bars itself.
+
+
+
+ ![image]({useBaseUrl("/img/intent_histogram_example.png")})
+
+
+#### Response Selectors
+
+`rasa test` evaluates response selectors in the same way that it evaluates intent classifiers, producing a
+report (`response_selection_report.json`), confusion matrix (`response_selection_confusion_matrix.png`),
+confidence histogram (`response_selection_histogram.png`) and errors (`response_selection_errors.json`).
+If your pipeline includes multiple response selectors, they are evaluated in a single report.
+
+The report logs precision, recall and f1 measure for
+each sub-intent of a [retrieval intent](../building-classic-assistants/glossary.mdx#retrieval-intent) and provides an overall average.
+You can save these reports as JSON files using the `--report` argument.
+
+#### Entity Extraction
+
+`rasa test` reports recall, precision, and f1-score for each entity type that
+your trainable entity extractors are trained to recognize.
+
+Only trainable entity extractors, such as the `DIETClassifier` and `CRFEntityExtractor` are
+evaluated by `rasa test`. Pretrained extractors like the `DucklingHTTPExtractor` are not evaluated.
+
+If you have multiple entity extractors in your pipeline, or use some custom extractors,
+multiple entities might be associated with the same token. In this case,
+you can use a list notation in the test files, such as
+
+```yaml
+stories:
+ - story: A basic test story with multiple entities for a single token
+ steps:
+ - user: |
+ I like [ice cream][{\"entity\": \"food\"}, {\"entity\": \"desert\"}]
+ intent: inform
+ # ...
+```
+
+:::caution incorrect entity annotations
+If any of your entities are incorrectly annotated, your evaluation may fail. One common problem
+is that an entity cannot stop or start inside a token.
+For example, if you have an example for a `name` entity
+like `[Brian](name)'s house`, this is only valid if your tokenizer splits `Brian's` into
+multiple tokens.
+
+:::
+
+#### Entity Scoring
+
+To evaluate entity extraction we apply a simple tag-based approach. We don't consider
+[BILOU tags](../building-classic-assistants/nlu-training-data.mdx#bilou-entity-tagging) exactly, but only the
+entity type tags on a per token basis. For location entity like “near Alexanderplatz” we
+expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`.
+
+Our approach is more lenient when it comes to evaluation, as it rewards
+partial extraction and does not penalize the splitting of entities.
+For example, given the aforementioned entity “near Alexanderplatz” and a system that extracts
+“Alexanderplatz”, our approach rewards the extraction of “Alexanderplatz” and penalizes the missed out word “near”.
+
+The BILOU-based approach, however, would label this as a complete failure since it expects Alexanderplatz
+to be labeled as a last token in an entity (`L-LOC`) instead of a single token entity (`U-LOC`). Note also that
+a split extraction of “near” and “Alexanderplatz” would get full scores on our approach and zero on the
+BILOU-based one.
+
+Here's a comparison between the two scoring mechanisms for the phrase “near Alexanderplatz tonight”:
+
+| extracted | Simple tags (score) | BILOU tags (score) |
+| --------------------------------------------------- | ------------------- | ---------------------- |
+| `[near Alexanderplatz](loc) [tonight](time)` | loc loc time (3) | B-loc L-loc U-time (3) |
+| `[near](loc) [Alexanderplatz](loc) [tonight](time)` | loc loc time (3) | U-loc U-loc U-time (1) |
+| `near [Alexanderplatz](loc) [tonight](time)` | O loc time (2) | O U-loc U-time (1) |
+| `[near](loc) Alexanderplatz [tonight](time)` | loc O time (2) | U-loc O U-time (1) |
+| `[near Alexanderplatz tonight](loc)` | loc loc loc (2) | B-loc I-loc L-loc (1) |
+
+## Evaluating a Dialogue Model
+
+You can evaluate your trained dialogue model on a set of test stories
+by using the test script:
+
+```bash
+rasa test core --stories test_stories.yml --out results
+```
+
+This will print any failed stories to `results/failed_test_stories.yml`.
+A story fails if at least one of the actions was predicted incorrectly.
+
+The test script will also save a confusion matrix to a file called
+`results/story_confmat.pdf`. For each action in your domain, the confusion
+matrix shows how often the action was correctly predicted and how often an
+incorrect action was predicted instead.
+
+### Interpreting the generated warnings
+
+The test script will also generate a warnings file called `results/stories_with_warnings.yml`.
+This file contains all test stories for which [`action_unlikely_intent`](../concepts/default-actions.mdx#action_unlikely_intent)
+was predicted at any conversation turn but all actions from the original story were predicted correctly.
+However, if a test story originally included an `action_unlikely_intent`, for example to ensure [a rule is designed to
+trigger the conversation path after an `action_unlikely_intent`](../concepts/default-actions.mdx#customization-1) but the ensemble of
+policies failed to do so, then the corresponding story will end up in `results/failed_test_stories.yml` as
+a failed story.
+
+The stories are sorted by the severity of `action_unlikely_intent`'s prediction.
+This severity is calculated by [`UnexpecTEDIntentPolicy`](../building-classic-assistants/policies.mdx#unexpected-intent-policy) itself at prediction time.
+The higher the severity, the more unlikely is the intent and hence reviewing that particular
+conversation path becomes more critical.
+
+Note, that `action_unlikely_intent` is predicted by
+`UnexpecTEDIntentPolicy` which employs a machine learning based model
+under the hood and hence can result in false warnings as well. You can choose to ignore such warnings
+if the conversation paths in these stories are already present in the training stories.
+
+### Comparing Policy Configurations
+
+To choose a configuration for your dialogue model, or to choose hyperparameters for a
+specific policy, you want to measure how well your dialogue model will generalize
+to conversations it hasn't seen before. Especially in the beginning
+of a project, when you don't have a lot of real conversations to train
+your bot on, you may not want to exclude some to use as a test set.
+
+Rasa has some scripts to help you choose and fine-tune your policy configuration.
+Once you are happy with it, you can then train your final configuration on your
+full data set.
+
+To do this, you first have to train models for your different
+configurations. Create two (or more) config files including the policies you want to
+compare, and then provide them to the train script to train your models:
+
+```bash
+rasa train core -c config_1.yml config_2.yml \
+ --out comparison_models --runs 3 --percentages 0 5 25 50 70 95
+```
+
+Similar to how the [NLU model was evaluated](#comparing-nlu-pipelines), the above
+command trains the dialogue model on multiple configurations and different amounts of training data.
+For each config file provided, Rasa will train dialogue models
+with 0, 5, 25, 50, 70 and 95% of your training stories excluded from the training
+data. This is repeated three times to ensure consistent results.
+
+Once this script has finished, you can pass multiple models to the test script
+to compare the models you just trained:
+
+```bash
+rasa test core -m comparison_models --stories stories_folder
+ --out comparison_results --evaluate-model-directory
+```
+
+This will evaluate each model on the stories in `stories_folder`
+(can be either training or test set) and plot some graphs
+to show you which policy performs best. Since the previous train command
+excluded some amount of training data to train each model,
+the above test command can measure how well your model predicts the held-out stories.
+To compare single policies, create config files containing only one policy each.
+
+:::note
+This training process can take a long time, so we'd suggest letting it run
+somewhere in the background where it can't be interrupted.
+
+:::
+
+### Testing Action Code
+
+The approach used to test your action code will depend on how it is
+implemented. For example, if you connect to external APIs, you should write integration tests to ensure
+that those APIs respond as expected to common inputs. However you test your action code, you should
+include these tests in your CI pipeline so that they run each time you make changes.
+
+If you have any questions or problems, please share them with us in the dedicated
+[testing section on our forum](https://forum.rasa.com/tags/testing)!
+
+## End-To-End Testing
+
+
+
-
-
-
-The histogram allows you to visualize the confidence for all predictions,
-with the correct and incorrect predictions being displayed by blue and red bars respectively.
-Improving the quality of your training data will move the blue histogram bars up the plot and the
-red histogram bars down the plot. It should also help in reducing the number of red histogram bars itself.
-
-
-
-
-
-
-#### Response Selectors
-
-`rasa test` evaluates response selectors in the same way that it evaluates intent classifiers, producing a
-report (`response_selection_report.json`), confusion matrix (`response_selection_confusion_matrix.png`),
-confidence histogram (`response_selection_histogram.png`) and errors (`response_selection_errors.json`).
-If your pipeline includes multiple response selectors, they are evaluated in a single report.
-
-The report logs precision, recall and f1 measure for
-each sub-intent of a [retrieval intent](../building-classic-assistants/glossary.mdx#retrieval-intent) and provides an overall average.
-You can save these reports as JSON files using the `--report` argument.
-
-#### Entity Extraction
-
-`rasa test` reports recall, precision, and f1-score for each entity type that
-your trainable entity extractors are trained to recognize.
-
-Only trainable entity extractors, such as the `DIETClassifier` and `CRFEntityExtractor` are
-evaluated by `rasa test`. Pretrained extractors like the `DucklingHTTPExtractor` are not evaluated.
-
-If you have multiple entity extractors in your pipeline, or use some custom extractors,
-multiple entities might be associated with the same token. In this case,
-you can use a list notation in the test files, such as
-
-```yaml
-stories:
- - story: A basic test story with multiple entities for a single token
- steps:
- - user: |
- I like [ice cream][{\"entity\": \"food\"}, {\"entity\": \"desert\"}]
- intent: inform
- # ...
-```
-
-:::caution incorrect entity annotations
-If any of your entities are incorrectly annotated, your evaluation may fail. One common problem
-is that an entity cannot stop or start inside a token.
-For example, if you have an example for a `name` entity
-like `[Brian](name)'s house`, this is only valid if your tokenizer splits `Brian's` into
-multiple tokens.
-
-:::
-
-#### Entity Scoring
-
-To evaluate entity extraction we apply a simple tag-based approach. We don't consider
-[BILOU tags](../building-classic-assistants/nlu-training-data.mdx#bilou-entity-tagging) exactly, but only the
-entity type tags on a per token basis. For location entity like “near Alexanderplatz” we
-expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`.
-
-Our approach is more lenient when it comes to evaluation, as it rewards
-partial extraction and does not penalize the splitting of entities.
-For example, given the aforementioned entity “near Alexanderplatz” and a system that extracts
-“Alexanderplatz”, our approach rewards the extraction of “Alexanderplatz” and penalizes the missed out word “near”.
-
-The BILOU-based approach, however, would label this as a complete failure since it expects Alexanderplatz
-to be labeled as a last token in an entity (`L-LOC`) instead of a single token entity (`U-LOC`). Note also that
-a split extraction of “near” and “Alexanderplatz” would get full scores on our approach and zero on the
-BILOU-based one.
-
-Here's a comparison between the two scoring mechanisms for the phrase “near Alexanderplatz tonight”:
-
-| extracted | Simple tags (score) | BILOU tags (score) |
-| --------------------------------------------------- | ------------------- | ---------------------- |
-| `[near Alexanderplatz](loc) [tonight](time)` | loc loc time (3) | B-loc L-loc U-time (3) |
-| `[near](loc) [Alexanderplatz](loc) [tonight](time)` | loc loc time (3) | U-loc U-loc U-time (1) |
-| `near [Alexanderplatz](loc) [tonight](time)` | O loc time (2) | O U-loc U-time (1) |
-| `[near](loc) Alexanderplatz [tonight](time)` | loc O time (2) | U-loc O U-time (1) |
-| `[near Alexanderplatz tonight](loc)` | loc loc loc (2) | B-loc I-loc L-loc (1) |
-
-## Evaluating a Dialogue Model
-
-You can evaluate your trained dialogue model on a set of test stories
-by using the test script:
-
-```bash
-rasa test core --stories test_stories.yml --out results
-```
-
-This will print any failed stories to `results/failed_test_stories.yml`.
-A story fails if at least one of the actions was predicted incorrectly.
-
-The test script will also save a confusion matrix to a file called
-`results/story_confmat.pdf`. For each action in your domain, the confusion
-matrix shows how often the action was correctly predicted and how often an
-incorrect action was predicted instead.
-
-### Interpreting the generated warnings
-
-The test script will also generate a warnings file called `results/stories_with_warnings.yml`.
-This file contains all test stories for which [`action_unlikely_intent`](../concepts/default-actions.mdx#action_unlikely_intent)
-was predicted at any conversation turn but all actions from the original story were predicted correctly.
-However, if a test story originally included an `action_unlikely_intent`, for example to ensure [a rule is designed to
-trigger the conversation path after an `action_unlikely_intent`](../concepts/default-actions.mdx#customization-1) but the ensemble of
-policies failed to do so, then the corresponding story will end up in `results/failed_test_stories.yml` as
-a failed story.
-
-The stories are sorted by the severity of `action_unlikely_intent`'s prediction.
-This severity is calculated by [`UnexpecTEDIntentPolicy`](../building-classic-assistants/policies.mdx#unexpected-intent-policy) itself at prediction time.
-The higher the severity, the more unlikely is the intent and hence reviewing that particular
-conversation path becomes more critical.
-
-Note, that `action_unlikely_intent` is predicted by
-`UnexpecTEDIntentPolicy` which employs a machine learning based model
-under the hood and hence can result in false warnings as well. You can choose to ignore such warnings
-if the conversation paths in these stories are already present in the training stories.
-
-### Comparing Policy Configurations
-
-To choose a configuration for your dialogue model, or to choose hyperparameters for a
-specific policy, you want to measure how well your dialogue model will generalize
-to conversations it hasn't seen before. Especially in the beginning
-of a project, when you don't have a lot of real conversations to train
-your bot on, you may not want to exclude some to use as a test set.
-
-Rasa has some scripts to help you choose and fine-tune your policy configuration.
-Once you are happy with it, you can then train your final configuration on your
-full data set.
-
-To do this, you first have to train models for your different
-configurations. Create two (or more) config files including the policies you want to
-compare, and then provide them to the train script to train your models:
-
-```bash
-rasa train core -c config_1.yml config_2.yml \
- --out comparison_models --runs 3 --percentages 0 5 25 50 70 95
-```
-
-Similar to how the [NLU model was evaluated](../production/testing-your-assistant.mdx#comparing-nlu-pipelines), the above
-command trains the dialogue model on multiple configurations and different amounts of training data.
-For each config file provided, Rasa will train dialogue models
-with 0, 5, 25, 50, 70 and 95% of your training stories excluded from the training
-data. This is repeated three times to ensure consistent results.
-
-Once this script has finished, you can pass multiple models to the test script
-to compare the models you just trained:
-
-```bash
-rasa test core -m comparison_models --stories stories_folder
- --out comparison_results --evaluate-model-directory
-```
-
-This will evaluate each model on the stories in `stories_folder`
-(can be either training or test set) and plot some graphs
-to show you which policy performs best. Since the previous train command
-excluded some amount of training data to train each model,
-the above test command can measure how well your model predicts the held-out stories.
-To compare single policies, create config files containing only one policy each.
-
-:::note
-This training process can take a long time, so we'd suggest letting it run
-somewhere in the background where it can't be interrupted.
-
-:::
-
-### Testing Action Code
-
-The approach used to test your action code will depend on how it is
-implemented. For example, if you connect to external APIs, you should write integration tests to ensure
-that those APIs respond as expected to common inputs. However you test your action code, you should
-include these tests in your CI pipeline so that they run each time you make changes.
-
-If you have any questions or problems, please share them with us in the dedicated
-[testing section on our forum](https://forum.rasa.com/tags/testing)!
-
## End-To-End Testing
-- Discovery features are highly unstable. We introduce + Discovery features are highly unstable. We introduce them for product exploration. These features will change and should only be used for testing purposes. diff --git a/docs/themes/theme-custom/theme/RasaDiscoveryBanner/styles.module.css b/docs/themes/theme-custom/theme/RasaDiscoveryBanner/styles.module.css deleted file mode 100644 index 4a6e8636517a..000000000000 --- a/docs/themes/theme-custom/theme/RasaDiscoveryBanner/styles.module.css +++ /dev/null @@ -1,15 +0,0 @@ -.label { - background-color: #F6D261; - border: 1px solid transparent; - border-radius: 8px; - padding: 2px 12px; - font-size: 15px !important; - font-weight: 600; - - display: inline-block; -} - -.label[disabled] { - background-color: var(--ifm-color-gray-500); - cursor: default; -} diff --git a/docs/themes/theme-custom/theme/RasaLabsBanner/styles.module.css b/docs/themes/theme-custom/theme/RasaLabsBanner/styles.module.css index eafbc6bd315e..fed55d974a30 100644 --- a/docs/themes/theme-custom/theme/RasaLabsBanner/styles.module.css +++ b/docs/themes/theme-custom/theme/RasaLabsBanner/styles.module.css @@ -1,19 +1,3 @@ -.label { - background-color: #F6D261; - border: 1px solid transparent; - border-radius: 8px; - padding: 2px 12px; - font-size: 15px !important; - font-weight: 600; - - display: inline-block; -} - -.label[disabled] { - background-color: var(--ifm-color-gray-500); - cursor: default; -} - .titleExtension { text-transform: none !important; } diff --git a/docs/themes/theme-custom/theme/RasaNLUBasedBanner/index.jsx b/docs/themes/theme-custom/theme/RasaNLUBasedBanner/index.jsx new file mode 100644 index 000000000000..a12defd5d87a --- /dev/null +++ b/docs/themes/theme-custom/theme/RasaNLUBasedBanner/index.jsx @@ -0,0 +1,30 @@ +import * as React from 'react'; +import useBaseUrl from "@docusaurus/useBaseUrl"; + +function RasaNLUBasedBanner() { + return ( + <> +
+
+
+ )
+}
+
+export default RasaNLUBasedBanner;
diff --git a/docs/themes/theme-custom/theme/RasaProBanner/index.jsx b/docs/themes/theme-custom/theme/RasaProBanner/index.jsx
index ef975c61942f..ae87d08ac5e3 100644
--- a/docs/themes/theme-custom/theme/RasaProBanner/index.jsx
+++ b/docs/themes/theme-custom/theme/RasaProBanner/index.jsx
@@ -1,7 +1,4 @@
import * as React from 'react';
-import clsx from 'clsx';
-
-import styles from './styles.module.css';
function RasaProBanner({isLoading, ...props}) {
return (
diff --git a/docs/themes/theme-custom/theme/RasaProBanner/styles.module.css b/docs/themes/theme-custom/theme/RasaProBanner/styles.module.css
deleted file mode 100644
index 55cfcd71e39b..000000000000
--- a/docs/themes/theme-custom/theme/RasaProBanner/styles.module.css
+++ /dev/null
@@ -1,18 +0,0 @@
-
-
-.label {
- background-color:#F6D261;
- border: 1px solid transparent;
- border-radius: 8px;
- padding: 2px 12px;
- font-size: 15px !important;
- font-weight: 600;
-
- display: inline-block;
-}
-
-.label[disabled] {
- background-color: var(--ifm-color-gray-500);
- cursor: default;
-}
-
+
+ + NLU-based assistants +
+
+
+ + This page refers to building NLU-based assistants. + If you are working with Conversational AI with Language Models (CALM), + this content may not apply to you. +
+