Lesson 2.8. Logging API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 2.8:\n",
    "# Logging API\n",
    "\n",
    "This tutorial introduces the Logging API, which provides a simple extension of the standard logging API to support the GridAPPS-D Log Message format. The API enables applications to subscribe to real-time log messages from a simulation, query previously logged messages from the MySQL database, and publish messages to their either own log or their GridAPPS-D logs.\n",
    "\n",
    "__Learning Objectives:__\n",
    "\n",
    "At the end of the tutorial, the user should be able to use the Logging API to\n",
    "\n",
    "* Subscribe to log messages from GridAPPS-D simulations and apps\n",
    "* Publish log messages to a local app log\n",
    "* Publish log messages to the GridAPPS-D logs\n",
    "* Query the GridAPPS-D logs for prior log messages\n",
    "\n",
    "## Getting Started\n",
    "\n",
    "Before running any of the sample routines in this tutorial, it is first necessary to start the GridAPPS-D Platform and establish a connection to this notebook so that we can start passing calls to the API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Open the Ubuntu terminal and start the GridAPPS-D Platform if it is not running already:_\n",
    "\n",
    "`cd gridappsd-docker`\n",
    "\n",
    "~/gridappsd-docker$ `./run.sh -t develop`\n",
    "\n",
    "_Once containers are running,_\n",
    "\n",
    "gridappsd@[container]:/gridappsd$ `./run-gridappsd.sh`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "* [1. Introduction to the Logging API](#1.-Introduction-to-the-Logging-API)\n",
    "\n",
    "\n",
    "* [2. Using the Logging API](#2.-Using-the-Logging-API)\n",
    "    * [2.1. Specify the Topic](#2.1.-Specify-the-Topic)\n",
    "    * [2.2. Message Structure](#2.2.-Message-Structure)\n",
    "\n",
    "* [3. GridAPPSD-Python Logging API Extensions](#3.-GridAPPSD-Python-Logging-API-Extensions)\n",
    "\n",
    "\n",
    "* [4. Subscribing to Simulation Logs](#4.-Subscribing-to-Simulation-Logs)\n",
    "    * [4.1. Specify the Topic](#4.1.-Specify-the-Topic)\n",
    "    * [4.2. Create Subscription Function](#4.2.-Create-Subscription-Function)\n",
    "    * [4.3. Create Subscription Method](#4.3.-Create-Subscription-Method)\n",
    "\n",
    "\n",
    "* [5. Publish to Simulation Logs](#5.-Publish-to-Simulation-Logs)\n",
    "    * [5.1. Publishing to Local App Logs](#5.1.-Publishing-to-Local-App-Logs)\n",
    "    * [5.2. Publishing to GridAPPS-D Logs](#5.2.-Publishing-to-GridAPPS-D-Logs)\n",
    "\n",
    "\n",
    "* [6. Querying for Save Logs](#6.-Querying-Saved-Logs)\n",
    "    * [6.1. Specify the Topic](#6.1.-Specify-the-Topic)\n",
    "    * [6.2. Structure of the Query Message](#6.2.-Structure-of-the-Query-Message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction to the Logging API\n",
    "\n",
    "The Logging API enables applications to subscribe to real-time log messages from a simulation, query previously logged messages from the MySQL database, and publish messages to their either own log or their GridAPPS-D logs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 2. Using the Logging API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Specify the Topic\n",
    "\n",
    "As with the Simulation API, the logging API uses both static `/queue/` and dynamic `/topic/` communication channel names depending on whether the API is being used for real-time simulation logs or historic logs that have already been saved in the database.\n",
    "\n",
    "For a review of GridAPPS-D topics, see [Lesson 1.4.](Lesson%201.4.%20GridAPPS-D%20Topics.ipynb)\n",
    "\n",
    "The correct topic for each Logging API call will be provided in the corresponding section for each API task below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Message Structure\n",
    "\n",
    "Logging messages in the GridAPPS-D environment follow the format of a python dictionary or equivalent JSON string with the format below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{       KEY               VALUE\n",
    "    \"source\":               filename,\n",
    "    \"processId\":            simulation_id,\n",
    "    \"timestamp\":            epoch time number,\n",
    "    \"processStatus\":        \"STARTED\" or \"STOPPED\" or \"RUNNING\" or \"ERROR\" or \"PASSED\" or \"FAILED\",\n",
    "    \"logMessage\":           string,\n",
    "    \"logLevel\":             \"INFO\" or \"DEBUG\" or \"ERROR\",\n",
    "    \"storeToDb\":            true or false\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the messages from a single instantiation will have the same format, with the only difference being the logMessage. As a result, it is possible to use the shortcuts available from the GridAPPSD-Python library to build out the repetitive portions of the message and pass just the logMessage string. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. GridAPPSD-Python Logging API Extensions\n",
    "\n",
    "The GridAPPSD-Python library uses several extensions to the standard Python logging library that enable applications to easily create log messages using the same syntax. These extensions support the additional log message formatting required by GridAPPS-D, such as simulation_id, log source, and process status.\n",
    "\n",
    "The following code block enables the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "os.environ['GRIDAPPSD_APPLICATION_ID'] = 'gridappsd-sensor-simulator'\n",
    "os.environ['GRIDAPPSD_APPLICATION_STATUS'] = 'STARTED'\n",
    "os.environ['GRIDAPPSD_SIMULATION_ID'] = opts.simulation_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Important!__ Run this import command ___BEFORE___ creating the GridAPPS-D connection object `gapps = GridAPPSD(...)`.\n",
    "\n",
    "__Note:__ If your application is containerized in Docker and registered with the GridAPPS-D platform using the docker-compose file, these extensions will be imported automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 4. Subscribing to Simulation Logs\n",
    "\n",
    "Similar to the two approaches used to subscribe to simulation measurements discussed in [Lesson 2.6](), it is possible to use either a function or a class definition to subscribe to the simulation logs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1. Specify the Topic\n",
    "\n",
    "This is a dynamic `/topic/` communication channel that is best implemented by importing the GriAPPSD-Python library function for generating the correct topic. \n",
    "\n",
    "* `from gridappsd.topics import simulation_log_topic`\n",
    "* `log_topic = simulation_log_topic(simulation_id)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note on Jupyter Notebook environment:__ In the examples below, the Jupyter Notebook environment does not update definitions of the subscription object or function definitions. As a result, it is necessary to restart the notebook kernel. The gapps connection object definition is included again for convenience in executing the notebook code blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Establish connection to GridAPPS-D Platform:\n",
    "viz_simulation_id = \"1160275131\"\n",
    "gapps = GridAPPSD(viz_simulation_id, \"('localhost', 61613)\", username='system', password='manager')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gridappsd.topics import simulation_log_topic\n",
    "log_topic = simulation_log_topic(viz_simulation_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note on Jupyter Notebook environment:__ In the examples below, the Jupyter Notebook environment does not update definitions of the subscription object or function definitions. As a result, it is necessary to restart the notebook kernel. The gapps connection object definition is included again for convenience in executing the notebook code blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Establish connection to GridAPPS-D Platform:\n",
    "viz_simulation_id = \"1160275131\"\n",
    "gapps = GridAPPSD(viz_simulation_id, \"('localhost', 61613)\", username='system', password='manager')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gridappsd.topics import simulation_log_topic\n",
    "log_topic = simulation_log_topic(viz_simulation_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Create Subscription Function\n",
    "\n",
    "The first approach used to subscribe to measurements is to define a function with the correct set of arguments that is then passed to the `.subscribe()` method associated with the `GridAPPPSD()` object.\n",
    "\n",
    "The function does not require a specific name, and is somewhat easier to define and use. However, the arguments of the function need to be named correctly for the GridAPPSD-Python library to process the simulation output correctly.\n",
    "\n",
    "The format for the function definition is \n",
    "\n",
    "```\n",
    "def myLogFunction(header, message):\n",
    "    # do something when receive a log message\n",
    "    # do something else\n",
    "```\n",
    "\n",
    "That function handle is then passed as an argument to the `.subscribe(topic, function_handle)` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def demoLogFunction(header, message):\n",
    "    timestamp = message[\"timestamp\"]\n",
    "    log_message = message[\"logMessage\"]\n",
    "    \n",
    "    print(\"Log message received at timestamp \", timestamp, \"which reads:\")\n",
    "    print(log_message)\n",
    "    print(\"........................\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gapps.subscribe(log_topic, demoLogFunction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3. Create Subscription Method\n",
    "\n",
    "The second approach used to subscribe to simulation logs is to define add a custom method to the same class with `__init__` and `on_message` methods that was created to subscribe to measurements.\n",
    "\n",
    "Unlike the Simulation API, the Logging API does not require a specific name for the method used to subscribe to log messages. \n",
    "\n",
    "It is possible to create additional methods in the subscription class definition to enable the app to subscribe to additional topics, such as the simulation log topic, as shown in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PlatformDemo(object):\n",
    "    # A simple class for interacting with simulation\n",
    "\n",
    "    def __init__(self, simulation_id, gapps_obj):\n",
    "        # Initialize variables and attributes\n",
    "        self._gapps = gapps_obj\n",
    "        self._simulation_id = simulation_id\n",
    "        # self.foo = bar\n",
    "        \n",
    "    def on_message(self, headers, message):\n",
    "        # Do things with measurements\n",
    "        meas_value = message[\"message\"][\"measurements\"]\n",
    "        # Do more stuff with measurements\n",
    "        \n",
    "    def my_custom_method(self, headers, message):\n",
    "        timestamp = message[\"timestamp\"]\n",
    "        log_message = message[\"logMessage\"]\n",
    "        \n",
    "        print(\"Log message received at timestamp \", timestamp, \"which reads:\")\n",
    "        print(log_message)\n",
    "        print(\"........................\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create subscription object\n",
    "demo_obj = PlatformDemo(viz_simulation_id, gapps)\n",
    "\n",
    "# Subscribe to logs using method\n",
    "gapps.subscribe(log_topic, demo_obj.my_custom_method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 5. Publish to Simulation Logs\n",
    "\n",
    "The GridAPPSD-Python library enables use of the standard Python logging syntax to create logs, publish them to the GOSS Message Bus, and store them in the MySQL database. \n",
    "\n",
    "Documentation of the standard Python logging library is available on [Python Docs](https://docs.python.org/3/library/logging.html).\n",
    "\n",
    "\n",
    "It is possible to publish to either local app logs (which are more useful for debugging) or the GridAPPS-D logs (which can be accessed by other applications and should be used for completed applications).\n",
    "\n",
    "## 5.1. Publishing to Local App Logs\n",
    "\n",
    "The first approach is to use the default Python logger to write to local app logs by importing the `logging` library and then use the `.getLogger()` method from the Python library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "_python_log = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Log messages can then be published by invoking the methods \n",
    "\n",
    "* `_python_log.debug(\"log message\")`\n",
    "* `_python_log.info(\"log message\")`\n",
    "* `_python_log.warning(\"log message\")`\n",
    "* `_python_log.error(\"log message\")`\n",
    "* `_python_log.fatal(\"log message\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2. Publishing to GridAPPS-D Logs\n",
    "\n",
    "The second approach is to use the GridAPPS-D logs. Importing the python logging library is not necessary. Instead initialize a logging object using the `.getLogger()` method associated with the GridAPPS-D connection object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_gapps_log = gapps.getLogger()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Log messages can then be published by invoking the methods \n",
    "\n",
    "* `_gapps_log.debug(\"log message\")`\n",
    "* `_gapps_log.info(\"log message\")`\n",
    "* `_gapps_log.warning(\"log message\")`\n",
    "* `_gapps_log.error(\"log message\")`\n",
    "* `_gapps_log.fatal(\"log message\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Querying Saved Logs\n",
    "\n",
    "Log messages published using the Logging API and the GOSS Message Bus are saved to the MySQL database. These log messages can be accessed with a Logging API query. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1. Specify the Topic\n",
    "\n",
    "The query for logs uses a static `/queue/` channel that is imported from the GridAPPS-D Topics library. \n",
    "\n",
    "This topic is used with the `.get_response(topic, message)` method associated with the GridAPPS-D connection object.\n",
    "\n",
    "* `from gridappsd import topics as t`\n",
    "* `gapps.get_response(t.LOGS, message)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2. Structure of the Query Message\n",
    "\n",
    "The first approach to querying with the Logging API is to use a python dictionary or equivalent JSON string that follows formatting similar to the query messages used by all the other GridAPPS-D APIs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gridappsd import topics as t\n",
    "\n",
    "message = {\n",
    "    \"source\": \"ProcessEvent\",\n",
    "    \"processId\": viz_simulation_id,\n",
    "    \"processStatus\": \"INFO\",\n",
    "    \"logLevel\": \"INFO\"\n",
    "}\n",
    "\n",
    "gapps.get_reponse(t.LOGS, message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Return to Top](#Table-of-Contents)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}