3-3-12-QAT.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"Tce3stUlHN0L"},"source":["##### Copyright 2020 The TensorFlow Authors."]},{"cell_type":"code","execution_count":null,"metadata":{"cellView":"form","id":"tuOe1ymfHZPu"},"outputs":[],"source":["#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n","# you may not use this file except in compliance with the License.\n","# You may obtain a copy of the License at\n","#\n","# https://www.apache.org/licenses/LICENSE-2.0\n","#\n","# Unless required by applicable law or agreed to in writing, software\n","# distributed under the License is distributed on an \"AS IS\" BASIS,\n","# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n","# See the License for the specific language governing permissions and\n","# limitations under the License."]},{"cell_type":"markdown","metadata":{"id":"qFdPvlXBOdUN"},"source":["# Quantization aware training in Keras example"]},{"cell_type":"markdown","metadata":{"id":"MfBg1C5NB3X0"},"source":["<table class=\"tfo-notebook-buttons\" align=\"left\">\n","  <td>\n","    <a target=\"_blank\" href=\"https://www.tensorflow.org/model_optimization/guide/quantization/training_example\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/model-optimization/blob/master/tensorflow_model_optimization/g3doc/guide/quantization/training_example.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n","  </td>\n","  <td>\n","    <a target=\"_blank\" href=\"https://github.com/tensorflow/model-optimization/blob/master/tensorflow_model_optimization/g3doc/guide/quantization/training_example.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n","  </td>\n","  <td>\n","    <a href=\"https://storage.googleapis.com/tensorflow_docs/model-optimization/tensorflow_model_optimization/g3doc/guide/quantization/training_example.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n","  </td>\n","</table>"]},{"cell_type":"markdown","metadata":{"id":"Bjmi3qZeu_xk"},"source":["## Overview\n","\n","Welcome to an end-to-end example for *quantization aware training*.\n","\n","### Other pages\n","For an introduction to what quantization aware training is and to determine if you should use it (including what's supported), see the [overview page](https://www.tensorflow.org/model_optimization/guide/quantization/training.md).\n","\n","To quickly find the APIs you need for your use case (beyond fully-quantizing a model with 8-bits), see the\n","[comprehensive guide](https://www.tensorflow.org/model_optimization/guide/quantization/training_comprehensive_guide.md).\n","\n","### Summary\n","\n","In this tutorial, you will:\n","\n","1.   Train a `tf.keras` model for MNIST from scratch.\n","2.   Fine tune the model by applying the quantization aware training API, see the accuracy, and\n","     export a quantization aware model.\n","3.   Use the model to create an actually quantized model for the TFLite\n","     backend.\n","4.   See the persistence of accuracy in\n","     TFLite and a 4x smaller model. To see the latency benefits on mobile, try out the TFLite examples [in the TFLite app repository](https://www.tensorflow.org/lite/models)."]},{"cell_type":"markdown","metadata":{"id":"yEAZYXvZU_XG"},"source":["## Setup"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"zN4yVFK5-0Bf"},"outputs":[],"source":["import subprocess\n","import pkg_resources\n","\n","import tensorflow as tf\n","import tensorflow_hub as hub\n","import tensorflow_datasets as tfds\n","\n","# Desired versions\n","tf_version = \"2.10.0\"\n","hub_version = \"0.12.0\"\n","datasets_version = \"4.6.0\"\n","tfmot_opt_version = \"0.7.3\"\n","\n","# Function to install or upgrade a package to the desired version\n","def install_package(package_name, version):\n","    print(f\"Installing {package_name}=={version}...\")\n","    subprocess.check_call([\"pip\", \"install\", f\"{package_name}=={version}\"])\n","\n","# Check if TensorFlow, TensorFlow Hub, TensorFlow Datasets, and TensorFlow Model Optimization are the correct versions\n","if (tf.__version__ != tf_version or \n","    hub.__version__ != hub_version or \n","    tfds.__version__ != datasets_version):\n","    \n","    print(f\"Current TensorFlow version: {tf.__version__}, switching to {tf_version}\")\n","    print(f\"Current TensorFlow Hub version: {hub.__version__}, switching to {hub_version}\")\n","    print(f\"Current TensorFlow Datasets version: {tfds.__version__}, switching to {datasets_version}\")\n","    \n","    # Uninstall current versions of TensorFlow, TensorFlow Hub, and TensorFlow Datasets\n","    !pip uninstall -y tensorflow tensorflow_hub tensorflow_datasets\n","    \n","    # Install desired versions\n","    install_package(\"tensorflow\", tf_version)\n","    install_package(\"tensorflow_hub\", hub_version)\n","    install_package(\"tensorflow_datasets\", datasets_version)\n","    \n","    print(\"Specified versions of TensorFlow, TensorFlow Hub, and TensorFlow Datasets installed.\")\n","    print(\"Please click on the Runtime > Restart session and run all.\")\n","    \n","else:\n","    print(\"All packages (TensorFlow, TensorFlow Hub, and TensorFlow Datasets) are already at the specified versions.\")\n","\n","# Check for tensorflow-model-optimization\n","try:\n","    package_version = pkg_resources.get_distribution(\"tensorflow_model_optimization\").version\n","    if package_version != tfmot_opt_version:\n","        print(f\"Current tensorflow-model-optimization version: {package_version}, upgrading to {tfmot_opt_version}\")\n","        # Uninstall and install the correct version\n","        subprocess.check_call([\"pip\", \"uninstall\", \"-y\", \"tensorflow-model-optimization\"])\n","        install_package(\"tensorflow-model-optimization\", tfmot_opt_version)\n","    else:\n","        print(f\"tensorflow-model-optimization {tfmot_opt_version} is already installed.\")\n","except pkg_resources.DistributionNotFound:\n","    print(f\"tensorflow-model-optimization not found, installing version {tfmot_opt_version}.\")\n","    install_package(\"tensorflow-model-optimization\", tfmot_opt_version)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"yJwIonXEVJo6"},"outputs":[],"source":["import tempfile\n","import os\n","\n","import tensorflow as tf\n","\n","from tensorflow import keras"]},{"cell_type":"markdown","metadata":{"id":"psViY5PRDurp"},"source":["## Train a model for MNIST without quantization aware training"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"pbY-KGMPvbW9"},"outputs":[],"source":["# Load MNIST dataset\n","mnist = keras.datasets.mnist\n","(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n","\n","# Normalize the input image so that each pixel value is between 0 to 1.\n","train_images = train_images / 255.0\n","test_images = test_images / 255.0\n","\n","# Define the model architecture.\n","model = keras.Sequential([\n","  keras.layers.InputLayer(input_shape=(28, 28)),\n","  keras.layers.Reshape(target_shape=(28, 28, 1)),\n","  keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation='relu'),\n","  keras.layers.MaxPooling2D(pool_size=(2, 2)),\n","  keras.layers.Flatten(),\n","  keras.layers.Dense(10)\n","])\n","\n","# Train the digit classification model\n","model.compile(optimizer='adam',\n","              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.fit(\n","  train_images,\n","  train_labels,\n","  epochs=1,\n","  validation_split=0.1,\n",")"]},{"cell_type":"markdown","metadata":{"id":"K8747K9OE72P"},"source":["## Clone and fine-tune pre-trained model with quantization aware training\n"]},{"cell_type":"markdown","metadata":{"id":"F19k7ExXF_h2"},"source":["### Define the model"]},{"cell_type":"markdown","metadata":{"id":"JsZROpNYMWQ0"},"source":["You will apply quantization aware training to the whole model and see this in the model summary. All layers are now prefixed by \"quant\".\n","\n","Note that the resulting model is quantization aware but not quantized (e.g. the weights are float32 instead of int8). The sections after show how to create a quantized model from the quantization aware one.\n","\n","In the [comprehensive guide](https://www.tensorflow.org/model_optimization/guide/quantization/training_comprehensive_guide.md), you can see how to quantize some layers for model accuracy improvements."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oq6blGjgFDCW"},"outputs":[],"source":["import tensorflow_model_optimization as tfmot\n","\n","quantize_model = tfmot.quantization.keras.quantize_model\n","\n","# q_aware stands for for quantization aware.\n","q_aware_model = quantize_model(model)\n","\n","# `quantize_model` requires a recompile.\n","q_aware_model.compile(optimizer='adam',\n","              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","q_aware_model.summary()"]},{"cell_type":"markdown","metadata":{"id":"uDr2ijwpGCI-"},"source":["### Train and evaluate the model against baseline"]},{"cell_type":"markdown","metadata":{"id":"XUBEn94hXYB1"},"source":["To demonstrate fine tuning after training the model for just an epoch, fine tune with quantization aware training on a subset of the training data."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"_PHDGJryE31X"},"outputs":[],"source":["train_images_subset = train_images[0:1000] # out of 60000\n","train_labels_subset = train_labels[0:1000]\n","\n","q_aware_model.fit(train_images_subset, train_labels_subset,\n","                  batch_size=500, epochs=1, validation_split=0.1)"]},{"cell_type":"markdown","metadata":{"id":"-byC2lYlMkfN"},"source":["For this example, there is minimal to no loss in test accuracy after quantization aware training, compared to the baseline."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6bMFTKSSHyyZ"},"outputs":[],"source":["_, baseline_model_accuracy = model.evaluate(\n","    test_images, test_labels, verbose=0)\n","\n","_, q_aware_model_accuracy = q_aware_model.evaluate(\n","   test_images, test_labels, verbose=0)\n","\n","print('Baseline test accuracy:', baseline_model_accuracy)\n","print('Quant test accuracy:', q_aware_model_accuracy)"]},{"cell_type":"markdown","metadata":{"id":"2IepmUPSITn6"},"source":["## Create quantized model for TFLite backend"]},{"cell_type":"markdown","metadata":{"id":"1FgNP4rbOLH8"},"source":["After this, you have an actually quantized model with int8 weights and uint8 activations."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"w7fztWsAOHTz"},"outputs":[],"source":["converter = tf.lite.TFLiteConverter.from_keras_model(q_aware_model)\n","converter.optimizations = [tf.lite.Optimize.DEFAULT]\n","\n","quantized_tflite_model = converter.convert()"]},{"cell_type":"markdown","metadata":{"id":"BEYsyYVqNgeY"},"source":["## See persistence of accuracy from TF to TFLite"]},{"cell_type":"markdown","metadata":{"id":"saadXD4JQsBK"},"source":["Define a helper function to evaluate the TF Lite model on the test dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"b8yBouuGNqls"},"outputs":[],"source":["import numpy as np\n","\n","def evaluate_model(interpreter):\n","  input_index = interpreter.get_input_details()[0][\"index\"]\n","  output_index = interpreter.get_output_details()[0][\"index\"]\n","\n","  # Run predictions on every image in the \"test\" dataset.\n","  prediction_digits = []\n","  for i, test_image in enumerate(test_images):\n","    if i % 1000 == 0:\n","      print('Evaluated on {n} results so far.'.format(n=i))\n","    # Pre-processing: add batch dimension and convert to float32 to match with\n","    # the model's input data format.\n","    test_image = np.expand_dims(test_image, axis=0).astype(np.float32)\n","    interpreter.set_tensor(input_index, test_image)\n","\n","    # Run inference.\n","    interpreter.invoke()\n","\n","    # Post-processing: remove batch dimension and find the digit with highest\n","    # probability.\n","    output = interpreter.tensor(output_index)\n","    digit = np.argmax(output()[0])\n","    prediction_digits.append(digit)\n","\n","  print('\\n')\n","  # Compare prediction results with ground truth labels to calculate accuracy.\n","  prediction_digits = np.array(prediction_digits)\n","  accuracy = (prediction_digits == test_labels).mean()\n","  return accuracy"]},{"cell_type":"markdown","metadata":{"id":"TuEFS4CIQvUw"},"source":["You evaluate the quantized model and see that the accuracy from TensorFlow persists to the TFLite backend."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VqQTyqz4NsWd"},"outputs":[],"source":["interpreter = tf.lite.Interpreter(model_content=quantized_tflite_model)\n","interpreter.allocate_tensors()\n","\n","test_accuracy = evaluate_model(interpreter)\n","\n","print('Quant TFLite test_accuracy:', test_accuracy)\n","print('Quant TF test accuracy:', q_aware_model_accuracy)"]},{"cell_type":"markdown","metadata":{"id":"z8D7WnFF5DZR"},"source":["## See 4x smaller model from quantization"]},{"cell_type":"markdown","metadata":{"id":"I1c2IecBRCdQ"},"source":["You create a float TFLite model and then see that the quantized TFLite model\n","is 4x smaller."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jy_Lgfh8VkyX"},"outputs":[],"source":["# Create float TFLite model.\n","float_converter = tf.lite.TFLiteConverter.from_keras_model(model)\n","float_tflite_model = float_converter.convert()\n","\n","# Measure sizes of models.\n","_, float_file = tempfile.mkstemp('.tflite')\n","_, quant_file = tempfile.mkstemp('.tflite')\n","\n","with open(quant_file, 'wb') as f:\n","  f.write(quantized_tflite_model)\n","\n","with open(float_file, 'wb') as f:\n","  f.write(float_tflite_model)\n","\n","print(\"Float model in Mb:\", os.path.getsize(float_file) / float(2**20))\n","print(\"Quantized model in Mb:\", os.path.getsize(quant_file) / float(2**20))"]},{"cell_type":"markdown","metadata":{"id":"0O5xuci-SonI"},"source":["## Conclusion"]},{"cell_type":"markdown","metadata":{"id":"O2I7xmyMW5QY"},"source":["In this tutorial, you saw how to create quantization aware models with the TensorFlow Model Optimization Toolkit API and then quantized models for the TFLite backend.\n","\n","You saw a 4x model size compression benefit for a model for MNIST, with minimal accuracy\n","difference. To see the latency benefits on mobile, try out the TFLite examples [in the TFLite app repository](https://www.tensorflow.org/lite/models).\n","\n","We encourage you to try this new capability, which can be particularly important for deployment in resource-constrained environments. \n"]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"3-3-12-QAT.ipynb","provenance":[{"file_id":"https://github.com/tensorflow/model-optimization/blob/master/tensorflow_model_optimization/g3doc/guide/quantization/training_example.ipynb","timestamp":1624567112706}],"toc_visible":true},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}