model_monitoring.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# @title Copyright & License (click to expand)\n",
    "# Copyright 2021 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fsv4jGuU89rX"
   },
   "source": [
    "# Vertex AI Model Monitoring with Explainable AI Feature Attributions\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmaster%2Fnotebooks%2Fcommunity%2Fmodel_monitoring%2Fmodel_monitoring_feature_attribs.ipynb\">\n",
    "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Cloud Notebook\n",
    "    </a>\n",
    "  </td> \n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/model_monitoring/model_monitoring_feature_attribs.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/model_monitoring/model_monitoring_feature_attribs.ipynb\">\n",
    "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lA32H1oKGgpf"
   },
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t6Cd51FkG09E"
   },
   "source": [
    "### What is Vertex AI Model Monitoring?\n",
    "\n",
    "Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:\n",
    "\n",
    "* software versioning\n",
    "* rigorous deployment processes\n",
    "* event logging\n",
    "* alerting/notication of situations requiring intervention\n",
    "* on-demand and automated diagnostic tracing\n",
    "* automated performance and functional testing\n",
    "\n",
    "You should be able to manage your ML services with the same degree of power and flexibility with which you can manage your applications. That's what MLOps is all about - managing ML services with the best practices Google and the broader computing industry have learned from generations of experience deploying well engineered, reliable, and scalable services.\n",
    "\n",
    "Model monitoring is only one piece of the ML Ops puzzle - it helps answer the following questions:\n",
    "\n",
    "* How well do recent service requests match the training data used to build your model? This is called **training-serving skew**.\n",
    "* How significantly are service requests evolving over time? This is called **drift detection**.\n",
    "\n",
    "[Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview) adds another facet to model monitoring, which we call feature attribution monitoring. Explainable AI enables you to understand the relative contribution of each feature to a resulting prediction. In essence, it assesses the magnitude of each feature's influence.\n",
    "\n",
    "If production traffic differs from  training data, or varies substantially over time, **either in terms of model predictions or feature attributions**, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yG7FcXWKHOhC"
   },
   "source": [
    "### Objective\n",
    "\n",
    "In this notebook, you will learn how to... \n",
    "\n",
    "* deploy a pre-trained model\n",
    "* configure model monitoring\n",
    "* generate some artificial traffic\n",
    "* understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8yVpQt-JHKPF"
   },
   "source": [
    "### Costs \n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertext AI\n",
    "* BigQuery\n",
    "\n",
    "Learn about [Vertext AI\n",
    "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "### The example model\n",
    "\n",
    "The model you'll use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
    "\n",
    "- identity - unique player identitity numbers\n",
    "- demographic features - information about the player, such as the geographic region in which a player is located\n",
    "- behavioral features - counts of the number of times a  player has triggered certain game events, such as reaching a new level\n",
    "- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n",
    "\n",
    "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process it for use in machine learning, and train a model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Google Cloud Storage. In the next section, you will setup your environment and import this model into your own project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ze4-nDLfK4pw"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Setup your dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "020040f91150"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UcwUlnMnfo3i"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pprint as pp\n",
    "import sys\n",
    "\n",
    "import IPython\n",
    "\n",
    "assert sys.version_info.major == 3, \"This notebook requires Python 3.\"\n",
    "\n",
    "# Install Python package dependencies.\n",
    "print(\"Installing TensorFlow and TensorFlow Data Validation (TFDV)\")\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade tensorflow tensorflow_data_validation[visualization]\n",
    "! rm -f /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-api-python-client google-auth-oauthlib google-auth-httplib2 oauth2client requests\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-aiplatform\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade explainable_ai_sdk\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-storage==1.32.0\n",
    "\n",
    "# Automatically restart kernel after installing new packages.\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    print(\"Restarting kernel...\")\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)\n",
    "    print(\"Done.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8rv5wnmA6BTd"
   },
   "outputs": [],
   "source": [
    "# Import required packages.\n",
    "import os\n",
    "import random\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "### Set up your Google Cloud project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
    "\n",
    "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n",
    "\n",
    "1. You'll use the *gcloud* command throughout this notebook. In the following cell, enter your project name and run the cell to authenticate yourself with the Google Cloud and initialize your *gcloud* configuration settings.\n",
    "\n",
    "**For this lab, we're going to use region us-central1 for all our resources (BigQuery training data, Cloud Storage bucket, model and endpoint locations, etc.). Those resources can be deployed in other regions, as long as they're consistently co-located, but we're going to use one fixed region to keep things as simple and error free as possible.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "wxiE6dEWOFm3"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
    "REGION = \"us-central1\"\n",
    "SUFFIX = \"aiplatform.googleapis.com\"\n",
    "API_ENDPOINT = f\"{REGION}-{SUFFIX}\"\n",
    "PREDICT_API_ENDPOINT = f\"{REGION}-prediction-{SUFFIX}\"\n",
    "if os.getenv(\"IS_TESTING\"):\n",
    "    !gcloud --quiet components install beta\n",
    "    !gcloud --quiet components update\n",
    "!gcloud config set project $PROJECT_ID\n",
    "!gcloud config set ai/region $REGION\n",
    "os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sECqTau7Oh6M"
   },
   "source": [
    "### Login to your Google Cloud account and enable AI services"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C6H1vZYjvT6w"
   },
   "outputs": [],
   "source": [
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# If on Google Cloud Notebooks, then don't execute this code\n",
    "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    if \"google.colab\" in sys.modules:\n",
    "        from google.colab import auth as google_auth\n",
    "\n",
    "        google_auth.authenticate_user()\n",
    "\n",
    "    # If you are running this notebook locally, replace the string below with the\n",
    "    # path to your service account key and run this cell to authenticate your GCP\n",
    "    # account.\n",
    "    elif not os.getenv(\"IS_TESTING\"):\n",
    "        %env GOOGLE_APPLICATION_CREDENTIALS ''\n",
    "\n",
    "!gcloud services enable aiplatform.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "btZeLzqQ7pXc"
   },
   "source": [
    "### Define some helper functions and data structures\n",
    "\n",
    "Run the following cell to define some utility functions used throughout this notebook. Although these functions are not critical to understand the main concepts, feel free to expand the  cell if you're curious or want to dive deeper into how some of your API requests are made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "yhDFSB2YDvfT"
   },
   "outputs": [],
   "source": [
    "# @title Utility functions\n",
    "import copy\n",
    "import os\n",
    "\n",
    "from explainable_ai_sdk.metadata.tf.v2 import SavedModelMetadataBuilder\n",
    "from google.cloud.aiplatform_v1.services.endpoint_service import \\\n",
    "    EndpointServiceClient\n",
    "from google.cloud.aiplatform_v1.services.job_service import JobServiceClient\n",
    "from google.cloud.aiplatform_v1.services.prediction_service import \\\n",
    "    PredictionServiceClient\n",
    "from google.cloud.aiplatform_v1.types.io import BigQuerySource\n",
    "from google.cloud.aiplatform_v1.types.model_deployment_monitoring_job import (\n",
    "    ModelDeploymentMonitoringJob, ModelDeploymentMonitoringObjectiveConfig,\n",
    "    ModelDeploymentMonitoringScheduleConfig)\n",
    "from google.cloud.aiplatform_v1.types.model_monitoring import (\n",
    "    ModelMonitoringAlertConfig, ModelMonitoringObjectiveConfig,\n",
    "    SamplingStrategy, ThresholdConfig)\n",
    "from google.cloud.aiplatform_v1.types.prediction_service import (\n",
    "    ExplainRequest, PredictRequest)\n",
    "from google.protobuf import json_format\n",
    "from google.protobuf.duration_pb2 import Duration\n",
    "from google.protobuf.struct_pb2 import Value\n",
    "\n",
    "DEFAULT_THRESHOLD_VALUE = 0.001\n",
    "\n",
    "\n",
    "def create_monitoring_job(objective_configs):\n",
    "    # Create sampling configuration.\n",
    "    random_sampling = SamplingStrategy.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)\n",
    "    sampling_config = SamplingStrategy(random_sample_config=random_sampling)\n",
    "\n",
    "    # Create schedule configuration.\n",
    "    duration = Duration(seconds=MONITOR_INTERVAL)\n",
    "    schedule_config = ModelDeploymentMonitoringScheduleConfig(monitor_interval=duration)\n",
    "\n",
    "    # Create alerting configuration.\n",
    "    emails = [USER_EMAIL]\n",
    "    email_config = ModelMonitoringAlertConfig.EmailAlertConfig(user_emails=emails)\n",
    "    alerting_config = ModelMonitoringAlertConfig(email_alert_config=email_config)\n",
    "\n",
    "    # Create the monitoring job.\n",
    "    endpoint = f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\"\n",
    "    predict_schema = \"\"\n",
    "    analysis_schema = \"\"\n",
    "    job = ModelDeploymentMonitoringJob(\n",
    "        display_name=JOB_NAME,\n",
    "        endpoint=endpoint,\n",
    "        model_deployment_monitoring_objective_configs=objective_configs,\n",
    "        logging_sampling_strategy=sampling_config,\n",
    "        model_deployment_monitoring_schedule_config=schedule_config,\n",
    "        model_monitoring_alert_config=alerting_config,\n",
    "        predict_instance_schema_uri=predict_schema,\n",
    "        analysis_instance_schema_uri=analysis_schema,\n",
    "    )\n",
    "    options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=options)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
    "    response = client.create_model_deployment_monitoring_job(\n",
    "        parent=parent, model_deployment_monitoring_job=job\n",
    "    )\n",
    "    print(\"Created monitoring job:\")\n",
    "    print(response)\n",
    "    return response\n",
    "\n",
    "\n",
    "def get_thresholds(default_thresholds, custom_thresholds):\n",
    "    thresholds = {}\n",
    "    default_threshold = ThresholdConfig(value=DEFAULT_THRESHOLD_VALUE)\n",
    "    for feature in default_thresholds.split(\",\"):\n",
    "        feature = feature.strip()\n",
    "        thresholds[feature] = default_threshold\n",
    "    for custom_threshold in custom_thresholds.split(\",\"):\n",
    "        pair = custom_threshold.split(\":\")\n",
    "        if len(pair) != 2:\n",
    "            print(f\"Invalid custom skew threshold: {custom_threshold}\")\n",
    "            return\n",
    "        feature, value = pair\n",
    "        thresholds[feature] = ThresholdConfig(value=float(value))\n",
    "    return thresholds\n",
    "\n",
    "\n",
    "def get_deployed_model_ids(endpoint_id):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = EndpointServiceClient(client_options=client_options)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
    "    response = client.get_endpoint(name=f\"{parent}/endpoints/{endpoint_id}\")\n",
    "    model_ids = []\n",
    "    for model in response.deployed_models:\n",
    "        model_ids.append(model.id)\n",
    "    return model_ids\n",
    "\n",
    "\n",
    "def set_objectives(model_ids, objective_template):\n",
    "    # Use the same objective config for all models.\n",
    "    objective_configs = []\n",
    "    for model_id in model_ids:\n",
    "        objective_config = copy.deepcopy(objective_template)\n",
    "        objective_config.deployed_model_id = model_id\n",
    "        objective_configs.append(objective_config)\n",
    "    return objective_configs\n",
    "\n",
    "\n",
    "def send_predict_request(endpoint, input, type=\"predict\"):\n",
    "    client_options = {\"api_endpoint\": PREDICT_API_ENDPOINT}\n",
    "    client = PredictionServiceClient(client_options=client_options)\n",
    "    if type == \"predict\":\n",
    "        obj = PredictRequest\n",
    "        method = client.predict\n",
    "    elif type == \"explain\":\n",
    "        obj = ExplainRequest\n",
    "        method = client.explain\n",
    "    else:\n",
    "        raise Exception(\"unsupported request type:\" + type)\n",
    "    params = {}\n",
    "    params = json_format.ParseDict(params, Value())\n",
    "    request = obj(endpoint=endpoint, parameters=params)\n",
    "    inputs = [json_format.ParseDict(input, Value())]\n",
    "    request.instances.extend(inputs)\n",
    "    response = None\n",
    "    try:\n",
    "        response = method(request)\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "    return response\n",
    "\n",
    "\n",
    "def list_monitoring_jobs():\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/us-central1\"\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.list_model_deployment_monitoring_jobs(parent=parent)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "def pause_monitoring_job(job):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.pause_model_deployment_monitoring_job(name=job)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "def delete_monitoring_job(job):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.delete_model_deployment_monitoring_job(name=job)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "# Sampling distributions for categorical features...\n",
    "DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}\n",
    "LANGUAGE = {\n",
    "    \"en-us\": 4807,\n",
    "    \"en-gb\": 678,\n",
    "    \"ja-jp\": 419,\n",
    "    \"en-au\": 310,\n",
    "    \"en-ca\": 299,\n",
    "    \"de-de\": 147,\n",
    "    \"en-in\": 130,\n",
    "    \"en\": 127,\n",
    "    \"fr-fr\": 94,\n",
    "    \"pt-br\": 81,\n",
    "    \"es-us\": 65,\n",
    "    \"zh-tw\": 64,\n",
    "    \"zh-hans-cn\": 55,\n",
    "    \"es-mx\": 53,\n",
    "    \"nl-nl\": 37,\n",
    "    \"fr-ca\": 34,\n",
    "    \"en-za\": 29,\n",
    "    \"vi-vn\": 29,\n",
    "    \"en-nz\": 29,\n",
    "    \"es-es\": 25,\n",
    "}\n",
    "OS = {\"IOS\": 3980, \"ANDROID\": 3798, \"null\": 253}\n",
    "MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}\n",
    "COUNTRY = {\n",
    "    \"United States\": 4395,\n",
    "    \"India\": 486,\n",
    "    \"Japan\": 450,\n",
    "    \"Canada\": 354,\n",
    "    \"Australia\": 327,\n",
    "    \"United Kingdom\": 303,\n",
    "    \"Germany\": 144,\n",
    "    \"Mexico\": 102,\n",
    "    \"France\": 97,\n",
    "    \"Brazil\": 93,\n",
    "    \"Taiwan\": 72,\n",
    "    \"China\": 65,\n",
    "    \"Saudi Arabia\": 49,\n",
    "    \"Pakistan\": 48,\n",
    "    \"Egypt\": 46,\n",
    "    \"Netherlands\": 45,\n",
    "    \"Vietnam\": 42,\n",
    "    \"Philippines\": 39,\n",
    "    \"South Africa\": 38,\n",
    "}\n",
    "\n",
    "# Means and standard deviations for numerical features...\n",
    "MEAN_SD = {\n",
    "    \"julianday\": (204.6, 34.7),\n",
    "    \"cnt_user_engagement\": (30.8, 53.2),\n",
    "    \"cnt_level_start_quickplay\": (7.8, 28.9),\n",
    "    \"cnt_level_end_quickplay\": (5.0, 16.4),\n",
    "    \"cnt_level_complete_quickplay\": (2.1, 9.9),\n",
    "    \"cnt_level_reset_quickplay\": (2.0, 19.6),\n",
    "    \"cnt_post_score\": (4.9, 13.8),\n",
    "    \"cnt_spend_virtual_currency\": (0.4, 1.8),\n",
    "    \"cnt_ad_reward\": (0.1, 0.6),\n",
    "    \"cnt_challenge_a_friend\": (0.0, 0.3),\n",
    "    \"cnt_completed_5_levels\": (0.1, 0.4),\n",
    "    \"cnt_use_extra_steps\": (0.4, 1.7),\n",
    "}\n",
    "\n",
    "DEFAULT_INPUT = {\n",
    "    \"cnt_ad_reward\": 0,\n",
    "    \"cnt_challenge_a_friend\": 0,\n",
    "    \"cnt_completed_5_levels\": 1,\n",
    "    \"cnt_level_complete_quickplay\": 3,\n",
    "    \"cnt_level_end_quickplay\": 5,\n",
    "    \"cnt_level_reset_quickplay\": 2,\n",
    "    \"cnt_level_start_quickplay\": 6,\n",
    "    \"cnt_post_score\": 34,\n",
    "    \"cnt_spend_virtual_currency\": 0,\n",
    "    \"cnt_use_extra_steps\": 0,\n",
    "    \"cnt_user_engagement\": 120,\n",
    "    \"country\": \"Denmark\",\n",
    "    \"dayofweek\": 3,\n",
    "    \"julianday\": 254,\n",
    "    \"language\": \"da-dk\",\n",
    "    \"month\": 9,\n",
    "    \"operating_system\": \"IOS\",\n",
    "    \"user_pseudo_id\": \"104B0770BAE16E8B53DF330C95881893\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1mhT_d_Bi-Kf"
   },
   "source": [
    "### Generate model metadata for explainable AI\n",
    "\n",
    "Run the following cell to extract metadata from the exported model, which is needed for generating the prediction explanations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "d1a984c57356"
   },
   "outputs": [],
   "source": [
    "builder = SavedModelMetadataBuilder(\n",
    "    \"gs://mco-mm/churn\", outputs_to_explain=[\"churned_probs\"]\n",
    ")\n",
    "builder.save_metadata(\".\")\n",
    "md = builder.get_metadata()\n",
    "del md[\"tags\"]\n",
    "del md[\"framework\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lAOk8UqvCL0S"
   },
   "source": [
    "## Import your model\n",
    "\n",
    "The churn propensity model you'll be using in this notebook has been trained in BigQuery ML and exported to a Google Cloud Storage bucket. This illustrates how you can easily export a trained model and move a model from one cloud service to another. \n",
    "\n",
    "Run the next cell to import this model into your project. **If you've already imported your model, you can skip this step.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "50c05f40f1dc"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "MODEL_NAME = \"churn\"\n",
    "IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-5:latest\"\n",
    "ENDPOINT = \"us-central1-aiplatform.googleapis.com\"\n",
    "churn_model_path = \"gs://mco-mm/churn\"\n",
    "\n",
    "request_data = {\n",
    "    \"model\": {\n",
    "        \"displayName\": \"churn\",\n",
    "        \"artifactUri\": churn_model_path,\n",
    "        \"containerSpec\": {\"imageUri\": IMAGE},\n",
    "        \"explanationSpec\": {\n",
    "            \"parameters\": {\"sampledShapleyAttribution\": {\"pathCount\": 5}},\n",
    "            \"metadata\": md,\n",
    "        },\n",
    "    }\n",
    "}\n",
    "\n",
    "with open(\"request_data.json\", \"w\") as outfile:\n",
    "    json.dump(request_data, outfile)\n",
    "output = !curl -X POST \\\n",
    "         -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
    "         -H \"Content-Type: application/json\" \\\n",
    "         https://{ENDPOINT}/v1/projects/{PROJECT_ID}/locations/{REGION}/models:upload \\\n",
    "         -d @request_data.json 2>/dev/null\n",
    "# print(output)\n",
    "MODEL_ID = output[1].split()[1].split(\"/\")[5]\n",
    "print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")\n",
    "# If auto-testing this notebook, wait for model registration\n",
    "if os.getenv(\"IS_TESTING\"):\n",
    "    time.sleep(300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e2030b028cef"
   },
   "source": [
    "This request will return immediately but it spawns an asynchronous task that takes several minutes. Periodically check the Vertex Models page on the Cloud Console and don't continue with this lab until you see your newly created model there. It should like something like this:\n",
    "<br>\n",
    "<br>\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm0.png\" />\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d7cbb0fb73cc"
   },
   "source": [
    "## Deploy your endpoint\n",
    "\n",
    "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet.  Your endpoint is also serverless, in the sense that Google ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
    "\n",
    "Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "M4nHQTVh7PIi"
   },
   "outputs": [],
   "source": [
    "ENDPOINT_NAME = \"churn\"\n",
    "output = !gcloud --quiet beta ai endpoints create --display-name=$ENDPOINT_NAME --format=\"value(name)\"\n",
    "# print(\"endpoint output: \", output)\n",
    "ENDPOINT = output[-1]\n",
    "ENDPOINT_ID = ENDPOINT.split(\"/\")[-1]\n",
    "output = !gcloud --quiet beta ai endpoints deploy-model $ENDPOINT_ID --display-name=$ENDPOINT_NAME --model=$MODEL_ID --traffic-split=\"0=100\"\n",
    "print(f\"Model deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NKsA_lfl9Ryw"
   },
   "source": [
    "## Run a prediction test\n",
    "\n",
    "Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON, along with a pie chart summarizing the results.\n",
    "\n",
    "**Try this now by running the next cell and examine the results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QNEb7fDJ9NXc"
   },
   "outputs": [],
   "source": [
    "# print(ENDPOINT)\n",
    "# pp.pprint(DEFAULT_INPUT)\n",
    "try:\n",
    "    resp = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
    "    for i in resp.predictions:\n",
    "        vals = i[\"churned_values\"]\n",
    "        probs = i[\"churned_probs\"]\n",
    "    for i in range(len(vals)):\n",
    "        print(vals[i], probs[i])\n",
    "    plt.pie(probs, labels=vals)\n",
    "    plt.show()\n",
    "    pp.pprint(resp)\n",
    "except Exception as ex:\n",
    "    print(\"prediction request failed\", ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a1eb4131bb5e"
   },
   "source": [
    "Taking a closer look at the results, we see the following elements:\n",
    "\n",
    "- **churned_values** - a set of possible values (0 and 1) for the target field\n",
    "- **churned_probs** - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)\n",
    "- **predicted_churn** - based on the probabilities, the predicted value of the target field (1)\n",
    "\n",
    "This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "be7830c55cdc"
   },
   "source": [
    "## Run an explanation test\n",
    "\n",
    "We can also run a test of explainable AI on this endpoint. Run the next cell to send a test explanation request. If everything works as expected, you should receive a response encoding the feature importance of this prediction in a text representation called JSON, along with a bar chart summarizing the results.\n",
    "\n",
    "**Try this now by running the next cell and examine the results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dcbd4e755931"
   },
   "outputs": [],
   "source": [
    "# print(ENDPOINT)\n",
    "# pp.pprint(DEFAULT_INPUT)\n",
    "try:\n",
    "    features = []\n",
    "    scores = []\n",
    "    resp = send_predict_request(ENDPOINT, DEFAULT_INPUT, type=\"explain\")\n",
    "    for i in resp.explanations:\n",
    "        for j in i.attributions:\n",
    "            for k in j.feature_attributions:\n",
    "                features.append(k)\n",
    "                scores.append(j.feature_attributions[k])\n",
    "    features = [x for _, x in sorted(zip(scores, features))]\n",
    "    scores = sorted(scores)\n",
    "    fig, ax = plt.subplots()\n",
    "    fig.set_size_inches(9, 9)\n",
    "    ax.barh(features, scores)\n",
    "    fig.show()\n",
    "    # pp.pprint(resp)\n",
    "except Exception as ex:\n",
    "    print(\"explanation request failed\", ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8rF5iLuXCT7i"
   },
   "source": [
    "## Start your monitoring job\n",
    "\n",
    "Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.\n",
    "\n",
    "In this section, you will configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wW2gLBQ3Zkhq"
   },
   "source": [
    "### Configure the following fields:\n",
    "\n",
    "1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n",
    "1. Monitor interval - time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds)\n",
    "1. Target field - prediction target column name in training dataset\n",
    "1. Skew detection threshold - skew threshold for each feature you want to monitor\n",
    "1. Prediction drift threshold - drift threshold for each feature you want to monitor\n",
    "1. Attribution Skew detection threshold - feature importance skew threshold\n",
    "1. Attribution Prediction drift threshold - feature importance drift threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "plpASmM2YIVO"
   },
   "outputs": [],
   "source": [
    "USER_EMAIL = \"\"  # @param {type:\"string\"}\n",
    "JOB_NAME = \"churn\"\n",
    "\n",
    "# Sampling rate (optional, default=.8)\n",
    "LOG_SAMPLE_RATE = 0.8  # @param {type:\"number\"}\n",
    "\n",
    "# Monitoring Interval in seconds (optional, default=3600).\n",
    "MONITOR_INTERVAL = 3600  # @param {type:\"number\"}\n",
    "\n",
    "# URI to training dataset.\n",
    "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\"  # @param {type:\"string\"}\n",
    "# Prediction target column name in training dataset.\n",
    "TARGET = \"churned\"\n",
    "\n",
    "# Skew and drift thresholds.\n",
    "SKEW_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\"  # @param {type:\"string\"}\n",
    "SKEW_CUSTOM_THRESHOLDS = \"cnt_level_start_quickplay:.01\"  # @param {type:\"string\"}\n",
    "DRIFT_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\"  # @param {type:\"string\"}\n",
    "DRIFT_CUSTOM_THRESHOLDS = \"cnt_level_start_quickplay:.01\"  # @param {type:\"string\"}\n",
    "ATTRIB_SKEW_DEFAULT_THRESHOLDS = \"country,cnt_user_engagement\"  # @param {type:\"string\"}\n",
    "ATTRIB_SKEW_CUSTOM_THRESHOLDS = (\n",
    "    \"cnt_level_start_quickplay:.01\"  # @param {type:\"string\"}\n",
    ")\n",
    "ATTRIB_DRIFT_DEFAULT_THRESHOLDS = (\n",
    "    \"country,cnt_user_engagement\"  # @param {type:\"string\"}\n",
    ")\n",
    "ATTRIB_DRIFT_CUSTOM_THRESHOLDS = (\n",
    "    \"cnt_level_start_quickplay:.01\"  # @param {type:\"string\"}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mjVSViZR-dP2"
   },
   "source": [
    "### Create your monitoring job\n",
    "\n",
    "The following code uses the Google Python client library to translate your configuration settings into a programmatic request to start a model monitoring job. Instantiating a monitoring job can take some time. If everything looks good with your request, you'll get a successful API response. Then, you'll need to check your email to receive a notification that the job is running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-62TYm2iYv3K"
   },
   "outputs": [],
   "source": [
    "skew_thresholds = get_thresholds(SKEW_DEFAULT_THRESHOLDS, SKEW_CUSTOM_THRESHOLDS)\n",
    "drift_thresholds = get_thresholds(DRIFT_DEFAULT_THRESHOLDS, DRIFT_CUSTOM_THRESHOLDS)\n",
    "attrib_skew_thresholds = get_thresholds(\n",
    "    ATTRIB_SKEW_DEFAULT_THRESHOLDS, ATTRIB_SKEW_CUSTOM_THRESHOLDS\n",
    ")\n",
    "attrib_drift_thresholds = get_thresholds(\n",
    "    ATTRIB_DRIFT_DEFAULT_THRESHOLDS, ATTRIB_DRIFT_CUSTOM_THRESHOLDS\n",
    ")\n",
    "\n",
    "skew_config = ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(\n",
    "    skew_thresholds=skew_thresholds,\n",
    "    attribution_score_skew_thresholds=attrib_skew_thresholds,\n",
    ")\n",
    "drift_config = ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(\n",
    "    drift_thresholds=drift_thresholds,\n",
    "    attribution_score_drift_thresholds=attrib_drift_thresholds,\n",
    ")\n",
    "explanation_config = ModelMonitoringObjectiveConfig.ExplanationConfig(\n",
    "    enable_feature_attributes=True\n",
    ")\n",
    "\n",
    "training_dataset = ModelMonitoringObjectiveConfig.TrainingDataset(target_field=TARGET)\n",
    "training_dataset.bigquery_source = BigQuerySource(input_uri=DATASET_BQ_URI)\n",
    "objective_config = ModelMonitoringObjectiveConfig(\n",
    "    training_dataset=training_dataset,\n",
    "    training_prediction_skew_detection_config=skew_config,\n",
    "    prediction_drift_detection_config=drift_config,\n",
    "    explanation_config=explanation_config,\n",
    ")\n",
    "model_ids = get_deployed_model_ids(ENDPOINT_ID)\n",
    "objective_template = ModelDeploymentMonitoringObjectiveConfig(\n",
    "    objective_config=objective_config\n",
    ")\n",
    "objective_configs = set_objectives(model_ids, objective_template)\n",
    "\n",
    "monitoring_job = create_monitoring_job(objective_configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OiwOVR4D_xhl"
   },
   "outputs": [],
   "source": [
    "# Run a prediction request to generate schema, if necessary.\n",
    "try:\n",
    "    _ = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
    "    print(\"prediction succeeded\")\n",
    "except Exception:\n",
    "    print(\"prediction failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SaXYVFFslRru"
   },
   "source": [
    "After a minute or two, you should receive email at the address you configured above for USER_EMAIL. This email confirms successful deployment of your monitoring job. Here's a sample of what this email might look like:\n",
    "<br>\n",
    "<br>\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm6.png\" />\n",
    "<br>\n",
    "As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to see an example of the layout of these measurements in Cloud Storage. If you substitute the Cloud Storage URL in your job creation email, you can view the structure and content of the data files for your own monitoring job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XV-vru2Pm1oX"
   },
   "outputs": [],
   "source": [
    "!gsutil ls gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/*/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XgUwU0sDpUUD"
   },
   "source": [
    "You will notice the following components in these Cloud Storage paths:\n",
    "\n",
    "- **cloud-ai-platform-..** - This is a bucket created for you and assigned to capture your service's prediction data. Each monitoring job you create will trigger creation of a new folder in this bucket.\n",
    "- **[model_monitoring|instance_schemas]/job-..** - This is your unique monitoring job number, which you can see above in both the response to your job creation requesst and the email notification. \n",
    "- **instance_schemas/job-../analysis** - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).\n",
    "- **instance_schemas/job-../predict** - This is the first prediction made to your model after the current monitoring job was enabled.\n",
    "- **model_monitoring/job-../serving** - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.\n",
    "- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data.\n",
    "- **model_monitoring/job-../feature_attribution_score** - This folder is used to record data relevant to feature attribution calculations. It contains an ongoing summary of feature attribution scores relative to training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8V2zo7-MMd7G"
   },
   "source": [
    "### You can create monitoring jobs with other user interfaces\n",
    "\n",
    "In the previous cells, you created a monitoring job using the Python client library. You can also use the *gcloud* command line tool to create a model monitoring job and, in the near future, you will be able to use the Cloud Console, as well for this function. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q106INuFCXKX"
   },
   "source": [
    "## Generate test data to trigger alerting\n",
    "\n",
    "Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. This cell runs two five minute tests, one minute apart, so it should take roughly eleven minutes to complete the test.\n",
    "\n",
    "The first test sends 300 fabricated requests (one per second for five minutes) while perturbing two features of interest (cnt_level_start_quickplay and country) by a factor of two. The second test does the same thing but perturbs the selected feature distributions by a factor of three. By perturbing data in two experiments, we're able to trigger both skew and drift alerts.\n",
    "\n",
    "After running this test, it takes at least an hour to assess and report skew and drift alerts so feel free to proceed with the notebook now and you'll see how to examine the resulting alerts later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "obZYLLAuKmG8"
   },
   "outputs": [],
   "source": [
    "def random_uid():\n",
    "    digits = [str(i) for i in range(10)] + [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\n",
    "    return \"\".join(random.choices(digits, k=32))\n",
    "\n",
    "\n",
    "def monitoring_test(count, sleep, perturb_num={}, perturb_cat={}):\n",
    "    # Use random sampling and mean/sd with gaussian distribution to model\n",
    "    # training data. Then modify sampling distros for two categorical features\n",
    "    # and mean/sd for two numerical features.\n",
    "    mean_sd = MEAN_SD.copy()\n",
    "    country = COUNTRY.copy()\n",
    "    for k, (mean_fn, sd_fn) in perturb_num.items():\n",
    "        orig_mean, orig_sd = MEAN_SD[k]\n",
    "        mean_sd[k] = (mean_fn(orig_mean), sd_fn(orig_sd))\n",
    "    for k, v in perturb_cat.items():\n",
    "        country[k] = v\n",
    "    for i in range(0, count):\n",
    "        input = DEFAULT_INPUT.copy()\n",
    "        input[\"user_pseudo_id\"] = str(random_uid())\n",
    "        input[\"country\"] = random.choices([*country], list(country.values()))[0]\n",
    "        input[\"dayofweek\"] = random.choices([*DAYOFWEEK], list(DAYOFWEEK.values()))[0]\n",
    "        input[\"language\"] = str(random.choices([*LANGUAGE], list(LANGUAGE.values()))[0])\n",
    "        input[\"operating_system\"] = str(random.choices([*OS], list(OS.values()))[0])\n",
    "        input[\"month\"] = random.choices([*MONTH], list(MONTH.values()))[0]\n",
    "        for key, (mean, sd) in mean_sd.items():\n",
    "            sample_val = round(float(np.random.normal(mean, sd, 1)))\n",
    "            val = max(sample_val, 0)\n",
    "            input[key] = val\n",
    "        print(f\"Sending prediction {i}\")\n",
    "        try:\n",
    "            send_predict_request(ENDPOINT, input)\n",
    "        except Exception:\n",
    "            print(\"prediction request failed\")\n",
    "        time.sleep(sleep)\n",
    "    print(\"Test Completed.\")\n",
    "\n",
    "\n",
    "start = 2\n",
    "end = 3\n",
    "for multiplier in range(start, end + 1):\n",
    "    test_time = 300\n",
    "    tests_per_sec = 1\n",
    "    sleep_time = 1 / tests_per_sec\n",
    "    iterations = test_time * tests_per_sec\n",
    "    perturb_num = {\n",
    "        \"cnt_level_start_quickplay\": (\n",
    "            lambda x: x * multiplier,\n",
    "            lambda x: x / multiplier,\n",
    "        )\n",
    "    }\n",
    "    perturb_cat = {\"Japan\": max(COUNTRY.values()) * multiplier}\n",
    "    monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)\n",
    "    if multiplier < end:\n",
    "        print(\"sleeping...\")\n",
    "        time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bQohDTJgLQlW"
   },
   "source": [
    "## Interpret your results\n",
    "\n",
    "While waiting for your results, which, as noted, may take up to an hour, you can read ahead to get sense of the alerting experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uGPI92qbOFUR"
   },
   "source": [
    "### Here's what a sample email alert looks like...\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm7.png\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HoaqsxpaRs1m"
   },
   "source": [
    "This email is warning you that the *cnt_level_start_quickplay*, *cnt_user_engagement*, and *country* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* and *country* feature attribution values are skewed relative to your training data, again, as per your threshold specification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w4jVIVq4VzB_"
   },
   "source": [
    "### Monitoring results in the Cloud Console\n",
    "\n",
    "You can examine your model monitoring data from the Cloud Console. Below is a screenshot of those capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2OdIMVBAPZi_"
   },
   "source": [
    "#### Monitoring Status\n",
    "\n",
    "You can verify that a given endpoint has an active model monitoring job via the Endpoint summary page:\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm1.png\" />\n",
    "\n",
    "#### Monitoring Alerts\n",
    "\n",
    "You can examine the alert details by clicking into the endpoint of interest, and selecting the alerts panel:\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm2.png\" />\n",
    "\n",
    "#### Feature Value Distributions\n",
    "\n",
    "You can also examine the recorded training and production feature distributions by drilling down into a given feature, like this:\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm9.png\" />\n",
    "\n",
    "which yields graphical representations of the feature distrubution during both training and production, like this:\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/mco-general/img/mm8.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Clean up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TPP_ImwJDFJf"
   },
   "outputs": [],
   "source": [
    "# Delete endpoint resource\n",
    "!gcloud ai endpoints delete $ENDPOINT_NAME --quiet\n",
    "\n",
    "# Delete model resource\n",
    "!gcloud ai models delete $MODEL_NAME --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j3Dh15h3-NoO"
   },
   "source": [
    "## Learn more about model monitoring\n",
    "\n",
    "**Congratulations!** You've now learned what model monitoring is, how to configure and enable it, and how to find and interpret the results. Check out the following resources to learn more about model monitoring and ML Ops.\n",
    "\n",
    "- [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv)\n",
    "- [Data Understanding, Validation, and Monitoring At Scale](https://blog.tensorflow.org/2018/09/introducing-tensorflow-data-validation.html)\n",
    "- [Vertex Product Documentation](https://cloud.google.com/vertex-ai)\n",
    "- [Vertex AI Model Monitoring Reference Docs](https://cloud.google.com/vertex-ai/docs/reference)\n",
    "- [Vertex AI Model Monitoring blog article](https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai)\n",
    "- [Explainable AI Whitepaper](https://storage.googleapis.com/cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "model_monitoring.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-9.m93",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-9:m93"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}