notebooks/214-vision-paddle-classification/214-vision-paddle-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5206ff2e-8ee6-47b0-abef-eb1b7f9ec5ee",
   "metadata": {},
   "source": [
    "# PaddlePaddle Image Classification with OpenVINO™\n",
    "This demo shows how to run a MobileNetV3 Large PaddePaddle model, using OpenVINO Runtime. Instead of exporting the PaddlePaddle model to ONNX and converting to OpenVINO Intermediate Representation (OpenVINO IR) format using Model Optimizer, you can now read the Paddle model directly without conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29258d2e",
   "metadata": {},
   "source": [
    "## Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0dacf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Downloading model\n",
    "from pathlib import Path\n",
    "import os\n",
    "import urllib.request\n",
    "import tarfile\n",
    "\n",
    "# Inference\n",
    "from openvino.runtime import Core\n",
    "\n",
    "# Preprocessing\n",
    "import cv2\n",
    "import numpy as np\n",
    "from openvino.preprocess import PrePostProcessor, ResizeAlgorithm\n",
    "from openvino.runtime import Layout, Type, AsyncInferQueue, PartialShape\n",
    "\n",
    "# Visualization of the results\n",
    "import time\n",
    "import json\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3dbb77-5cd6-46ad-b6e4-ba75f0ee795f",
   "metadata": {},
   "source": [
    "## Download the MobileNetV3_large_x1_0 Model\n",
    "Download the pre-trained model directly from the server. For more detailed information about the pre-trained model, refer to the [PaddleClas documentation](https://github.com/PaddlePaddle/PaddleClas/blob/release/2.2/deploy/lite/readme_en.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d476b3-0d6e-420b-9e51-9a1629bda494",
   "metadata": {},
   "outputs": [],
   "source": [
    "mobilenet_url = \"https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/MobileNetV3_large_x1_0_infer.tar\"\n",
    "mobilenetv3_model_path = Path(\"model/MobileNetV3_large_x1_0_infer/inference.pdmodel\")\n",
    "if mobilenetv3_model_path.is_file(): \n",
    "    print(\"Model MobileNetV3_large_x1_0 already exists\")\n",
    "else:\n",
    "    # Download the model from the server, and untar it.\n",
    "    print(\"Downloading the MobileNetV3_large_x1_0_infer model (20Mb)... May take a while...\")\n",
    "    # Create a directory.\n",
    "    os.makedirs(\"model\")\n",
    "    urllib.request.urlretrieve(mobilenet_url, \"model/MobileNetV3_large_x1_0_infer.tar\")\n",
    "    print(\"Model Downloaded\")\n",
    "\n",
    "    file = tarfile.open(\"model/MobileNetV3_large_x1_0_infer.tar\")\n",
    "    res = file.extractall(\"model\")\n",
    "    file.close()\n",
    "    if (not res):\n",
    "        print(f\"Model Extracted to {mobilenetv3_model_path}.\")\n",
    "    else:\n",
    "        print(\"Error Extracting the model. Please check the network.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3a72a48-494e-44f3-9ee8-b2c15c845fdc",
   "metadata": {},
   "source": [
    "## Define the callback function for postprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36b52bfa-551c-4d58-a796-7a0cb7e8c357",
   "metadata": {},
   "outputs": [],
   "source": [
    "def callback(infer_request, info) -> None:\n",
    "    \"\"\"\n",
    "    Define the callback function for postprocessing\n",
    "    \n",
    "    :param: infer_request: the infer_request object\n",
    "            info: a tuple includes the submitting time of infer request and iteration of inference\n",
    "    :retuns:\n",
    "            None\n",
    "    \"\"\"\n",
    "    global total_time\n",
    "    submit_time, i = info\n",
    "    imagenet_classes = json.loads(open(\"utils/imagenet_class_index.json\").read())\n",
    "    predictions = next(iter(infer_request.results.values()))\n",
    "    indices = np.argsort(-predictions[0])\n",
    "    if (i == 0):\n",
    "        # Calculate the first inference time\n",
    "        first_latency = (time.time() - submit_time) * 1000\n",
    "        print(\"first inference latency: {:.2f} ms\".format(first_latency))\n",
    "        for n in range(5):\n",
    "            print(\n",
    "                \"class name: {}, probability: {:.5f}\"\n",
    "                .format(imagenet_classes[str(list(indices)[n])][1], predictions[0][list(indices)[n]])\n",
    "            )\n",
    "        total_time = total_time + first_latency\n",
    "    else:\n",
    "        latency = (time.time() - submit_time) * 1000\n",
    "        total_time = total_time + latency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8adfba1-82d2-4526-910f-2d32530d74eb",
   "metadata": {},
   "source": [
    "## Read the model\n",
    "OpenVINO Runtime reads the `PaddlePaddle` model directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "179ae1a7-966d-49fe-9c06-811257c57989",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Intialize OpenVINO Runtime with Core()\n",
    "ie = Core()\n",
    "# MobileNetV3_large_x1_0\n",
    "model = ie.read_model(mobilenetv3_model_path)\n",
    "# Get the information of input and output layers.\n",
    "input_layer = model.input(0)\n",
    "output_layer = model.output(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc6a1ee1-802b-4561-af31-a613bba10fc6",
   "metadata": {},
   "source": [
    "## Integrate preprocessing steps into the execution graph with Preprocessing API\n",
    "If your input data does not fit perfectly in the model input tensor, additional operations/steps are needed to transform the data to a format expected by the model. These operations are known as “preprocessing”.\n",
    "Preprocessing steps are integrated into the execution graph and performed on the selected device(s) (CPU/GPU/VPU/etc.) rather than always executed on CPU. This improves utilization on the selected device(s).\n",
    "\n",
    "For more information, refer to the overview of [Preprocessing API](https://docs.openvino.ai/latest/openvino_docs_OV_Runtime_UG_Preprocessing_Overview.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "998aa8d0-20d7-471a-b4c5-25dd52c51881",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"../001-hello-world/data/coco.jpg\"\n",
    "test_image = cv2.imread(filename) \n",
    "test_image = np.expand_dims(test_image, 0) / 255\n",
    "_, h, w, _ = test_image.shape\n",
    "\n",
    "# Adjust model input shape to improve the performance.\n",
    "model.reshape({input_layer.any_name: PartialShape([1, 3, 224, 224])})\n",
    "ppp = PrePostProcessor(model)\n",
    "# Set input tensor information:\n",
    "# - The `input()` function provides information about a single model input.\n",
    "# - Layout of data is \"NHWC\".\n",
    "# - Set static spatial dimensions to input tensor to resize from.\n",
    "ppp.input().tensor() \\\n",
    "    .set_spatial_static_shape(h, w) \\\n",
    "    .set_layout(Layout(\"NHWC\")) \n",
    "inputs = model.inputs\n",
    "# Here, it is assumed that the model has \"NCHW\" layout for input.\n",
    "ppp.input().model().set_layout(Layout(\"NCHW\"))\n",
    "# Do prepocessing:\n",
    "# - Apply linear resize from tensor spatial dims to model spatial dims.\n",
    "# - Subtract mean from each channel.\n",
    "# - Divide each pixel data to appropriate scale value.\n",
    "ppp.input().preprocess() \\\n",
    "    .resize(ResizeAlgorithm.RESIZE_LINEAR, 224, 224) \\\n",
    "    .mean([0.485, 0.456, 0.406]) \\\n",
    "    .scale([0.229, 0.224, 0.225])\n",
    "# Set output tensor information:\n",
    "# - Precision of a tensor is supposed to be 'f32'.\n",
    "ppp.output().tensor().set_element_type(Type.f32)\n",
    "# Apply preprocessing to modify the original 'model'.\n",
    "model = ppp.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63289a21-3369-4f45-bd36-5f2e4f7aca01",
   "metadata": {},
   "source": [
    "## Run Inference\n",
    "Use Auto Device (or AUTO in short) plugin as the device name to delegate device selection to OpenVINO. AUTO internally recognizes and selects devices from among Intel CPU and GPU depending on the device capabilities and the characteristics of the model(s) (for example, precision). Then, it assigns inference requests to the best device.\n",
    "AUTO starts inference immediately on the CPU and then transparently shifts to the GPU (or VPU) once it is ready, dramatically reducing time to first inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f306c2-7589-4e0a-8de6-5bfb4b94ce0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "total_time = 0\n",
    "# Check the available devices in your system.\n",
    "devices = ie.available_devices\n",
    "for device in devices:\n",
    "    device_name = ie.get_property(device, \"FULL_DEVICE_NAME\")\n",
    "    print(f\"{device}: {device_name}\")\n",
    "\n",
    "# Load the model to a device selected by AUTO from the available devices list.\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\")\n",
    "# Create an infer request queue.\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "infer_queue.set_callback(callback)\n",
    "start = time.time()\n",
    "# Do inference.\n",
    "infer_queue.start_async({input_layer.any_name: test_image}, (time.time(), 0))\n",
    "infer_queue.wait_all()\n",
    "Image(filename=filename) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7a206a3",
   "metadata": {},
   "source": [
    "## Performance Hints: Latency and Throughput\n",
    "Throughput and latency are some of the most widely used metrics that measure the overall performance of an application.\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/160096393-cfd4cfaf-5fb2-4c35-9e6e-bdd216820a82.png\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "- **Latency** measures inference time (ms) required to process a single input or First inference.\n",
    "- To calculate **throughput**, divide number of inputs that were processed by the processing time.\n",
    "\n",
    "High-level Performance Hints in OpenVINO are the new way to configure the performance with the portability in mind. Performance Hints will let the device configure itself, rather than map the application needs to the low-level performance settings, and keep an associated application logic to configure each possible device separately. \n",
    "\n",
    "For more information, see [High-level Performance Hints](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Performance_Hints.html).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be337a5a-1e28-42cc-bb11-0fe9db5986c7",
   "metadata": {},
   "source": [
    "<br/>\n",
    " \n",
    "**Run Inference with \"LATENCY\" Performance Hint**\n",
    "\n",
    "It is possible to define application-specific performance settings with a config key, letting the device adjust to achieve better `LATENCY` oriented performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d46e985-258b-4650-aede-54a83c3aa03d",
   "metadata": {},
   "outputs": [],
   "source": [
    "loop = 100\n",
    "total_time = 0\n",
    "# AUTO sets device config based on hints.\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\", config={\"PERFORMANCE_HINT\": \"LATENCY\"})\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "# Implement AsyncInferQueue Python API to boost the performance in Async mode.\n",
    "infer_queue.set_callback(callback)\n",
    "# Run inference for 100 times to get the average FPS.\n",
    "start_time = time.time()\n",
    "for i in range(loop):\n",
    "    infer_queue.start_async({input_layer.any_name: test_image}, (time.time(), i))\n",
    "infer_queue.wait_all()\n",
    "end_time = time.time()\n",
    "# Calculate the average FPS\\n\",\n",
    "fps = loop / (end_time - start_time)\n",
    "print(\"throughput: {:.2f} fps\".format(fps))\n",
    "print(\"average latency: {:.2f} ms\".format(total_time / loop))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0807fd6b-16ac-4e90-b394-36a88a94abb6",
   "metadata": {},
   "source": [
    " <br/>\n",
    " \n",
    "**Run Inference with \"TRHOUGHPUT\" Performance Hint**\n",
    "\n",
    "It is possible to define application-specific performance settings with a config key, letting the device adjust to achieve better `THROUGHPUT` performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49997f5f-36d6-42b4-bce1-3eb44005fbd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "total_time = 0\n",
    "# AUTO sets device config based on hints.\n",
    "compiled_model = ie.compile_model(model=model, device_name=\"AUTO\", config={\"PERFORMANCE_HINT\": \"THROUGHPUT\"})\n",
    "infer_queue = AsyncInferQueue(compiled_model)\n",
    "infer_queue.set_callback(callback)\n",
    "start_time = time.time()\n",
    "for i in range(loop):\n",
    "    infer_queue.start_async({input_layer.any_name: test_image}, (time.time(), i))\n",
    "infer_queue.wait_all()\n",
    "end_time = time.time()\n",
    "# Calculate the average FPS\\n\",\n",
    "fps = loop / (end_time - start_time)\n",
    "print(\"throughput: {:.2f} fps\".format(fps))\n",
    "print(\"average latency: {:.2f} ms\".format(total_time / loop))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e701661c",
   "metadata": {},
   "source": [
    "## Measure Performance with benchmark_app\n",
    "To generate more accurate performance measurements, use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) in OpenVINO.\n",
    "\n",
    "You can trigger the \"Performance hint\" by using `-hint` parameter, which instructs the OpenVINO device plugin to use the best network-specific settings for `latency` OR `throughput`.\n",
    "\n",
    "> **NOTE**: The performance results from `benchmark_app` exclude \"compilation and load time\" of a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89af7e9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'latency': device performance optimized for LATENCY.\n",
    "! benchmark_app -m $mobilenetv3_model_path -data_shape [1,3,224,224] -hint \"latency\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9671c464",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'throughput' or 'tput': device performance optimized for THROUGHPUT.\n",
    "! benchmark_app -m $mobilenetv3_model_path -data_shape [1,3,224,224] -hint \"throughput\" "
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
  },
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}