Merge pull request #287 from deeppavlov/dev

Release v0.6.3

# Release notes

- Update README to make it more concise and include reasons to use DFF (#281)
- Add [DFF optimization guide](https://deeppavlov.github.io/dialog_flow_framework/user_guides/optimization_guide.html) (#236)
- Add documentation section description for the [index page](https://deeppavlov.github.io/dialog_flow_framework/index.html) (#281)
- Add async method [dff.messengers.common.interface.CallbackMessengerInterface.on_request_async](https://deeppavlov.github.io/dialog_flow_framework/apiref/dff.messengers.common.interface.html#dff.messengers.common.interface.CallbackMessengerInterface.on_request_async) (#206)
- Move information about messenger interfaces from `pipeline.6_custom_messenger_interface` (removed) to [messengers.web_api_interface.1_fastapi](https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.messengers.web_api_interface.1_fastapi.html) (#206)
- Fix cross-references in [dff.messengers.common.interface](https://deeppavlov.github.io/dialog_flow_framework/apiref/dff.messengers.common.interface.html) (#206)
- Update tutorials to fit into documentation without side-scrolling (#241)

README.md

@@ -1,11 +1,5 @@
-
 # Dialog Flow Framework
 
-The Dialog Flow Framework (DFF) allows you to write conversational services.
-The service is written by defining a special dialog graph that describes the behavior of the dialog service.
-The dialog graph contains the dialog script. DFF offers a specialized language (DSL) for quickly writing dialog graphs.
-You can use it in services such as writing skills for Amazon Alexa, etc., chatbots for social networks, website call centers, etc.
-
 [![Documentation Status](https://github.com/deeppavlov/dialog_flow_framework/workflows/build_and_publish_docs/badge.svg)](https://deeppavlov.github.io/dialog_flow_framework)
 [![Codestyle](https://github.com/deeppavlov/dialog_flow_framework/workflows/codestyle/badge.svg)](https://github.com/deeppavlov/dialog_flow_framework/actions/workflows/codestyle.yml)
 [![Tests](https://github.com/deeppavlov/dialog_flow_framework/workflows/test_coverage/badge.svg)](https://github.com/deeppavlov/dialog_flow_framework/actions/workflows/test_coverage.yml)
@@ -14,6 +8,17 @@ You can use it in services such as writing skills for Amazon Alexa, etc., chatbo
 [![PyPI](https://img.shields.io/pypi/v/dff)](https://pypi.org/project/dff/)
 [![Downloads](https://pepy.tech/badge/dff)](https://pepy.tech/project/dff)
 
+The Dialog Flow Framework (DFF) allows you to develop conversational services.
+DFF offers a specialized domain-specific language (DSL) for quickly writing dialogs in pure Python. The service is created by defining a special dialog graph that determines the behavior of the dialog agent. The latter is then leveraged in the DFF pipeline.
+You can use the framework in various services such as social networks, call centers, websites, personal assistants, etc.
+
+## Why choose DFF
+
+* Written in pure Python, the framework is easily accessible for both beginners and experienced developers.
+* For the same reason, all the abstractions used in DFF can be easily customized and extended using regular language synthax.
+* DFF offers easy and straightforward tools for state management which is as easy as setting values of a Python dictionary.
+* The framework is being actively maintained and thoroughly tested. The team is open to suggestions and quickly reacts to bug reports.
+
 # Quick Start
 ## Installation
 
@@ -26,7 +31,6 @@ pip install dff
 The above command will set the minimum dependencies to start working with DFF. 
 The installation process allows the user to choose from different packages based on their dependencies, which are:
 ```bash
-pip install dff[core]  # minimal dependencies (by default)
 pip install dff[json]  # dependencies for using JSON
 pip install dff[pickle] # dependencies for using Pickle
 pip install dff[redis]  # dependencies for using Redis
@@ -37,13 +41,6 @@ pip install dff[sqlite]  # dependencies for using SQLite
 pip install dff[ydb]  # dependencies for using Yandex Database
 pip install dff[telegram]  # dependencies for using Telegram
 pip install dff[benchmark]  # dependencies for benchmarking
-pip install dff[full]  # full dependencies including all options above
-pip install dff[tests]  # dependencies for running tests
-pip install dff[test_full]  # full dependencies for running all tests (all options above)
-pip install dff[tutorials]  # dependencies for running tutorials (all options above)
-pip install dff[devel]  # dependencies for development
-pip install dff[doc]  # dependencies for documentation
-pip install dff[devel_full]  # full dependencies for development (all options above)
 ```
 
 For example, if you are going to use one of the database backends,
@@ -54,11 +51,15 @@ pip install dff[postgresql, mysql]
 
 ## Basic example
 
+The following code snippet builds a simplistic chat bot that replies with messages
+``Hi!`` and ``OK`` depending on user input, which only takes a few lines of code.
+All the abstractions used in this example are thoroughly explained in the dedicated
+[user guide](https://deeppavlov.github.io/dialog_flow_framework/user_guides/basic_conceptions.html).
+
 ```python
-from dff.script import GLOBAL, TRANSITIONS, RESPONSE, Context, Message
+from dff.script import GLOBAL, TRANSITIONS, RESPONSE, Message
 from dff.pipeline import Pipeline
 import dff.script.conditions.std_conditions as cnd
-from typing import Tuple
 
 # create a dialog script
 script = {
@@ -69,89 +70,56 @@ script = {
         }
     },
     "flow": {
-        "node_hi": {RESPONSE: Message(text="Hi!!!")},
-        "node_ok": {RESPONSE: Message(text="Okey")},
+        "node_hi": {RESPONSE: Message(text="Hi!")},
+        "node_ok": {RESPONSE: Message(text="OK")},
     },
 }
 
 # init pipeline
 pipeline = Pipeline.from_script(script, start_label=("flow", "node_hi"))
 
 
-# handler requests
-def turn_handler(in_request: Message, pipeline: Pipeline) -> Tuple[Message, Context]:
-    # Pass the next request of user into pipeline and it returns updated context with actor response
+def turn_handler(in_request: Message, pipeline: Pipeline) -> Message:
+    # Pass user request into pipeline and get dialog context (message history)
+    # The pipeline will automatically choose the correct response using script
     ctx = pipeline(in_request, 0)
-    # Get last actor response from the context
+    # Get last response from the context
     out_response = ctx.last_response
-    # The next condition branching needs for testing
-    return out_response, ctx
+    return out_response
 
 
 while True:
-    in_request = input("type your answer: ")
-    out_response, ctx = turn_handler(Message(text=in_request), pipeline)
-    print(out_response.text)
+    in_request = input("Your message: ")
+    out_response = turn_handler(Message(text=in_request), pipeline)
+    print("Response: ", out_response.text)
 ```
 
 When you run this code, you get similar output:
 ```
-type your answer: hi
-Okey
-type your answer: Hi
-Hi!!!
-type your answer: ok
-Okey
-type your answer: ok
-Okey
+Your message: hi
+Response:  OK
+Your message: Hi
+Response:  Hi!
+Your message: ok
+Response:  OK
+Your message: ok
+Response:  OK
 ```
 
-To get more advanced examples, take a look at
-[tutorials](https://github.com/deeppavlov/dialog_flow_framework/tree/master/tutorials) on GitHub.
-
-# Context Storages
-## Description
-
-Context Storages allow you to save and retrieve user dialogue states
-(in the form of a `Context` object) using various database backends. 
-
-The following backends are currently supported:
-* [JSON](https://www.json.org/json-en.html)
-* [pickle](https://docs.python.org/3/library/pickle.html)
-* [shelve](https://docs.python.org/3/library/shelve.html)
-* [SQLite](https://www.sqlite.org/index.html)
-* [PostgreSQL](https://www.postgresql.org/)
-* [MySQL](https://www.mysql.com/)
-* [MongoDB](https://www.mongodb.com/)
-* [Redis](https://redis.io/)
-* [Yandex DataBase](https://ydb.tech/)
-
-Aside from this, we offer some interfaces for saving data to your local file system.
-These are not meant to be used in production, but can be helpful for prototyping your application.
-
-## Basic example
-
-```python
-from dff.script import Context
-from dff.pipeline import Pipeline
-from dff.context_storages import SQLContextStorage
-from .script import some_df_script
-
-db = SQLContextStorage("postgresql+asyncpg://user:password@host:port/dbname")
-
-pipeline = Pipeline.from_script(some_df_script, start_label=("root", "start"), fallback_label=("root", "fallback"))
+More advanced examples are available as a part of documentation:
+[tutorials](https://deeppavlov.github.io/dialog_flow_framework/tutorials.html).
 
+## Further steps
 
-def handle_request(request):
-    user_id = request.args["user_id"]
-    new_context = pipeline(request, user_id)
-    return new_context.last_response
-
-```
-
-To get more advanced examples, take a look at
-[tutorials](https://github.com/deeppavlov/dialog_flow_framework/tree/master/tutorials/context_storages) on GitHub.
+To further explore the API of the framework, you can make use of the [detailed documentation](https://deeppavlov.github.io/dialog_flow_framework/index.html). 
+Broken down into several sections to highlight all the aspects of development with DFF,
+the documentation for the library is constantly available online.
 
 # Contributing to the Dialog Flow Framework
 
+We are open to accepting pull requests and bug reports.
 Please refer to [CONTRIBUTING.md](https://github.com/deeppavlov/dialog_flow_framework/blob/master/CONTRIBUTING.md).
+
+# License
+
+DFF is distributed under the terms of the [Apache License 2.0](https://github.com/deeppavlov/dialog_flow_framework/blob/master/LICENSE).

dff/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 
 __author__ = "Denis Kuznetsov"

dff/context_storages/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .database import DBContextStorage, threadsafe_method, context_storage_factory
 from .json import JSONContextStorage, json_available

dff/messengers/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/messengers/common/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .interface import MessengerInterface, PollingMessengerInterface, CallbackMessengerInterface, CLIMessengerInterface
 from .types import PipelineRunnerFunction, PollingInterfaceLoopFunction

dff/messengers/common/interface.py

@@ -30,7 +30,7 @@ async def connect(self, pipeline_runner: PipelineRunnerFunction):
         May be used for sending an introduction message or displaying general bot information.
 
         :param pipeline_runner: A function that should return pipeline response to user request;
-            usually it's a :py:meth:`~Pipeline._run_pipeline(request, ctx_id)` function.
+            usually it's a :py:meth:`~dff.pipeline.pipeline.pipeline.Pipeline._run_pipeline` function.
         :type pipeline_runner: PipelineRunnerFunction
         """
         raise NotImplementedError
@@ -97,7 +97,7 @@ async def connect(
         for most cases the loop itself shouldn't be overridden.
 
         :param pipeline_runner: A function that should return pipeline response to user request;
-            usually it's a :py:meth:`~Pipeline._run_pipeline(request, ctx_id)` function.
+            usually it's a :py:meth:`~dff.pipeline.pipeline.pipeline.Pipeline._run_pipeline` function.
         :type pipeline_runner: PipelineRunnerFunction
         :param loop: a function that determines whether polling should be continued;
             called in each cycle, should return `True` to continue polling or `False` to stop.
@@ -124,18 +124,33 @@ def __init__(self):
     async def connect(self, pipeline_runner: PipelineRunnerFunction):
         self._pipeline_runner = pipeline_runner
 
+    async def on_request_async(self, request: Any, ctx_id: Hashable) -> Context:
+        """
+        Method invoked on user input. This method works just like
+        :py:meth:`~dff.pipeline.pipeline.pipeline.Pipeline._run_pipeline`,
+        however callback message interface may contain additional functionality (e.g. for external API accessing).
+        Return context that represents dialog with the user;
+        `last_response`, `id` and some dialog info can be extracted from there.
+
+        :param request: User input.
+        :param ctx_id: Any unique id that will be associated with dialog between this user and pipeline.
+        :return: Context that represents dialog with the user.
+        """
+        return await self._pipeline_runner(request, ctx_id)
+
     def on_request(self, request: Any, ctx_id: Hashable) -> Context:
         """
-        Method invoked on user input. This method works just like :py:meth:`.__call__(request, ctx_id)`,
+        Method invoked on user input. This method works just like
+        :py:meth:`~dff.pipeline.pipeline.pipeline.Pipeline._run_pipeline`,
         however callback message interface may contain additional functionality (e.g. for external API accessing).
-        Returns context that represents dialog with the user;
+        Return context that represents dialog with the user;
         `last_response`, `id` and some dialog info can be extracted from there.
 
         :param request: User input.
         :param ctx_id: Any unique id that will be associated with dialog between this user and pipeline.
         :return: Context that represents dialog with the user.
         """
-        return asyncio.run(self._pipeline_runner(request, ctx_id))
+        return asyncio.run(self.on_request_async(request, ctx_id))
 
 
 class CLIMessengerInterface(PollingMessengerInterface):
@@ -169,7 +184,7 @@ async def connect(self, pipeline_runner: PipelineRunnerFunction, **kwargs):
         The CLIProvider generates new dialog id used to user identification on each `connect` call.
 
         :param pipeline_runner: A function that should return pipeline response to user request;
-            usually it's a :py:meth:`~Pipeline._run_pipeline(request, ctx_id)` function.
+            usually it's a :py:meth:`~dff.pipeline.pipeline.pipeline.Pipeline._run_pipeline` function.
         :type pipeline_runner: PipelineRunnerFunction
         :param \\**kwargs: argument, added for compatibility with super class, it shouldn't be used normally.
         """

dff/messengers/telegram/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 try:
     import telebot

dff/messengers/telegram/interface.py

@@ -235,7 +235,7 @@ async def endpoint():
 
             json_string = request.get_data().decode("utf-8")
             update = types.Update.de_json(json_string)
-            resp = self.on_request(*extract_telegram_request_and_id(update, self.messenger))
+            resp = await self.on_request_async(*extract_telegram_request_and_id(update, self.messenger))
             self.messenger.send_response(resp.id, resp.last_response)
             return ""
 

dff/pipeline/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 
 from .conditions import (

dff/pipeline/pipeline/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/pipeline/service/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/script/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .core.context import Context
 from .core.keywords import (

dff/script/conditions/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .std_conditions import (
     exact_match,

dff/script/core/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/script/extras/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/script/extras/conditions/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/script/extras/slots/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/script/labels/__init__.py

@@ -1,4 +1,3 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .std_labels import repeat, previous, to_start, to_fallback, forward, backward

dff/script/responses/__init__.py

@@ -1,4 +1,3 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .std_responses import choice

dff/stats/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
 from opentelemetry.sdk.trace.export import ConsoleSpanExporter

dff/utils/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/utils/db_benchmark/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 from dff.utils.db_benchmark.benchmark import (
     time_context_read_write,
     DBFactory,

dff/utils/parser/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

dff/utils/testing/__init__.py

@@ -1,5 +1,4 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 from .common import is_interactive_mode, check_happy_path, run_interactive_mode
 from .toy_script import TOY_SCRIPT, TOY_SCRIPT_ARGS, HAPPY_PATH
 from .response_comparers import default_comparer

dff/utils/turn_caching/__init__.py

@@ -1,4 +1,3 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401
 
 from .singleton_turn_caching import cache_clear, lru_cache, cache

dff/utils/viewer/__init__.py

@@ -1,2 +1 @@
 # -*- coding: utf-8 -*-
-# flake8: noqa: F401

docs/source/get_started.rst

@@ -17,7 +17,6 @@ The installation process allows the user to choose from different packages based
 
 .. code-block:: console
 
-   pip install dff[core]  # minimal dependencies (by default)
    pip install dff[json]  # dependencies for using JSON
    pip install dff[pickle] # dependencies for using Pickle
    pip install dff[redis]  # dependencies for using Redis
@@ -28,13 +27,6 @@ The installation process allows the user to choose from different packages based
    pip install dff[ydb]  # dependencies for using Yandex Database
    pip install dff[telegram]  # dependencies for using Telegram
    pip install dff[benchmark]  # dependencies for benchmarking
-   pip install dff[full]  # full dependencies including all options above
-   pip install dff[tests]  # dependencies for running tests
-   pip install dff[test_full]  # full dependencies for running all tests (all options above)
-   pip install dff[tutorials]  # dependencies for running tutorials (all options above)
-   pip install dff[devel]  # dependencies for development
-   pip install dff[doc]  # dependencies for documentation
-   pip install dff[devel_full]  # full dependencies for development (all options above)
 
 For example, if you are going to use one of the database backends,
 you can specify the corresponding requirements yourself.
@@ -58,7 +50,7 @@ It allows developers to easily write and manage dialog systems by defining a spe
 dialog graph that describes the behavior of the service.
 DFF offers a specialized language (DSL) for quickly writing dialog graphs,
 making it easy for developers to create chatbots for a wide
-range of applications such as social networks, call centers, websites, skills for Amazon Alexa, etc.
+range of applications, such as social networks, call centers, websites, personal assistants, etc.
 
 DFF has several important concepts: