gemini-2/video_understanding.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lb5yiH5h8x3h"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "906e07f6e562"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMGdicu8PVD9"
      },
      "source": [
        "# Video understanding with Gemini 2.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DR4Ti6Q0QKIl"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/gemini-2/video_understanding.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w14yjWnPVD-"
      },
      "source": [
        "Gemini has from the begining been a multimodal model, capable of analyzing all sorts of medias using its [long context window](https://developers.googleblog.com/en/new-features-for-the-gemini-api-and-google-ai-studio/).\n",
        "\n",
        "[Gemini 2.0](https://ai.google.dev/gemini-api/docs/models/gemini-v2) bring video analysis to a whole new level as illustrated in [this video](https://www.youtube.com/watch?v=Mot-JEU26GQ):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "CumMaR-sts53"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/Mot-JEU26GQ?si=pcb7-_MZTSi_1Zkw\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen></iframe>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#@title Building with Gemini 2.0: Video understanding\n",
        "%%html\n",
        "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/Mot-JEU26GQ?si=pcb7-_MZTSi_1Zkw\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen></iframe>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jexx9acnuDsA"
      },
      "source": [
        "This notebook will show you how to easily use Gemini to perform the same kind of video analysis. Each of them has different prompts that you can select using the dropdown, also feel free to experiment with your own.\n",
        "\n",
        "You can also check the [live demo](https://aistudio.google.com/starter-apps/video) and try it on your own videos on [AI Studio](https://aistudio.google.com/starter-apps/video)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0HWzIEAQYqz"
      },
      "source": [
        "## Setup\n",
        "\n",
        "This section install the SDK, set it up using your [API key](../quickstarts/Authentication.ipynb), imports the relevant libs, downloads the sample videos and upload them to Gemini.\n",
        "\n",
        "Expand the section if you are curious, but you can also just run it (it should take a couple of minutes since there are large files) and go straight to the examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzBKAaL4QYq0"
      },
      "source": [
        "### Install SDK\n",
        "\n",
        "The new **[Google Gen AI SDK](https://ai.google.dev/gemini-api/docs/sdks)** provides programmatic access to Gemini 2.0 (and previous models) using both the [Google AI for Developers](https://ai.google.dev/gemini-api/docs) and [Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/overview) APIs. With a few exceptions, code that runs on one platform will run on both. This means that you can prototype an application using the Developer API and then migrate the application to Vertex AI without rewriting your code.\n",
        "\n",
        "More details about this new SDK on the [documentation](https://ai.google.dev/gemini-api/docs/sdks) or in the [Getting started](../gemini-2/get_started.ipynb) notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IbKkL5ksQYq1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/111.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.9/111.9 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -U -q google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDUGen_kQYq2"
      },
      "source": [
        "### Setup your API key\n",
        "\n",
        "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](../quickstarts/Authentication.ipynb) for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0H_lRdlrQYq3"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "\n",
        "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3Lez1vBQYq3"
      },
      "source": [
        "### Initialize SDK client\n",
        "\n",
        "With the new SDK you now only need to initialize a client with you API key (or OAuth if using [Vertex AI](https://link_to_vertex_AI)). The model is now set in each call."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X3CAp9YrQYq4"
      },
      "outputs": [],
      "source": [
        "from google import genai\n",
        "from google.genai import types\n",
        "\n",
        "client = genai.Client(api_key=GOOGLE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITgsQyaXQYq4"
      },
      "source": [
        "### Select the Gemini 2.0 model\n",
        "\n",
        "Video understanding works best Gemini 2.0 Flash model. You can also select former models to compare their behavior but it is recommended to use the 2.0 one.\n",
        "\n",
        "For more information about all Gemini models, check the [documentation](https://ai.google.dev/gemini-api/docs/models/gemini) for extended information on each of them.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IO7IoqbrQYq5"
      },
      "outputs": [],
      "source": [
        "model_name = \"gemini-2.0-flash-exp\" # @param [\"gemini-1.5-flash-001\",\"gemini-1.5-pro-002\",\"gemini-2.0-flash-exp\"] {\"allow-input\":true}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5sgbw9krwPmo"
      },
      "source": [
        "### System instructions\n",
        "\n",
        " With the new SDK, the `system_instructions` and the `model` parameters must be passed in all `generate_content` calls, so let's save them to not have to type them all the time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6GBdoMUVwP6j"
      },
      "outputs": [],
      "source": [
        "system_instructions = \"\"\"\n",
        "    When given a video and a query, call the relevant function only once with the appropriate timecodes and text for the video\n",
        "  \"\"\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nVfyTD0zwv8p"
      },
      "source": [
        "These system instructions make sure the TODO: add explanation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rv8ULT0lvJ47"
      },
      "source": [
        "### Get sample videos"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fMcwUw48vL1N"
      },
      "outputs": [],
      "source": [
        "# Load sample images\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/Pottery.mp4 -O Pottery.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/Jukin_Trailcam_Videounderstanding.mp4 -O Trailcam.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/post_its.mp4 -O Post_its.mp4 -q\n",
        "!wget https://storage.googleapis.com/generativeai-downloads/videos/user_study.mp4 -O User_study.mp4 -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y4YMNQulz_yY"
      },
      "source": [
        "### Upload the videos\n",
        "\n",
        "Upload all the videos using the File API. You can find modre details about how to use it in the [Get Started](../gemini-2/get_started.ipynb#scrollTo=KdUjkIQP-G_i) notebook.\n",
        "\n",
        "This can take a couple of minutes as the videos will need to be processed and tokenized."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LUUMJ4kE0OZS"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/hc3novqg1cb9\n",
            "Waiting for video to be processed.\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/gtlhr7ubnxnp\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/eqh0uo7mjbt9\n",
            "Waiting for video to be processed.\n",
            "Video processing complete: https://generativelanguage.googleapis.com/v1beta/files/bwfach9o4ay1\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "def upload_video(video_file_name):\n",
        "  video_file = client.files.upload(path=video_file_name)\n",
        "\n",
        "  while video_file.state == \"PROCESSING\":\n",
        "      print('Waiting for video to be processed.')\n",
        "      time.sleep(10)\n",
        "      video_file = client.files.get(name=video_file.name)\n",
        "\n",
        "  if video_file.state == \"FAILED\":\n",
        "    raise ValueError(video_file.state)\n",
        "  print(f'Video processing complete: ' + video_file.uri)\n",
        "\n",
        "  return video_file\n",
        "\n",
        "pottery_video = upload_video('Pottery.mp4')\n",
        "trailcam_video = upload_video('Trailcam.mp4')\n",
        "post_its_video = upload_video('Post_its.mp4')\n",
        "user_study_video = upload_video('User_study.mp4')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IF5tDbb-Q0oc"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B0Z9QzC3Q2wX"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "from PIL import Image\n",
        "from IPython.display import display, Markdown, HTML"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GAa7sCD7tuMW"
      },
      "source": [
        "# Search within videos\n",
        "\n",
        "First, try using the model to search within your videos and describe all the animal sightings in the trailcam video.\n",
        "\n",
        "<video controls width=\"500\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/Jukin_Trailcam_Videounderstanding.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PZw41-lsKKMf"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "```json\n",
              "[\n",
              "  {\n",
              "    \"timecode\": \"00:00:00\",\n",
              "    \"caption\": \"A close up of brown fur with visible plant life in the background.\"\n",
              "  },\n",
              "  {\n",
              "    \"timecode\": \"00:00:00\",\n",
              "    \"caption\": \"A gray fox walks into the scene, followed by another fox. The background is a rocky area with some undergrowth. A high pitched sound effect can be heard.\"\n",
              "  },\n",
              "  {\n",
              "    \"timecode\": \"00:00:16\",\n",
              "     \"caption\": \"A grayscale shot of a mountain lion in the forest, walking on some leaves.\"\n",
              "  },\n",
              "  {\n",
              "     \"timecode\": \"00:00:35\",\n",
              "     \"caption\": \"A nighttime grayscale shot of two foxes in the forest. They appear to be digging.\"\n",
              "  },\n",
              "    {\n",
              "     \"timecode\": \"00:00:50\",\n",
              "      \"caption\": \"An overexposed, blurry shot that is most of one solid color, followed by\"\n",
              "  },\n",
              "  {\n",
              "    \"timecode\": \"00:00:51\",\n",
              "    \"caption\": \"A nighttime shot of some foxes by rocks in a forest. A sound effect can be heard.\"\n",
              "  },\n",
              "  {\n",
              "    \"timecode\": \"00:01:04\",\n",
              "     \"caption\": \"A nighttime shot of a mountain lion in the forest with visible rock formations.\"\n",
              "  },\n",
              "   {\n",
              "    \"timecode\": \"00:01:28\",\n",
              "      \"caption\": \"A night time shot of a bobcat standing by a tree in the forest.\"\n",
              "  },\n",
              "  {\n",
              "      \"timecode\": \"00:01:51\",\n",
              "      \"caption\": \"A medium shot of a black bear in the forest during the day. It turns to its left and moves past the camera.\"\n",
              "  },\n",
              "  {\n",
              "    \"timecode\": \"00:01:56\",\n",
              "    \"caption\": \"A grayscale shot of a mountain lion moving quickly across the screen in a forest.\"\n",
              "  },\n",
              "    {\n",
              "    \"timecode\": \"00:02:05\",\n",
              "     \"caption\":\"A close up of brown fur. The camera moves to reveal a black bear.\"\n",
              "   },\n",
              " {\n",
              "    \"timecode\": \"00:02:22\",\n",
              "    \"caption\": \"A night time grayscale shot of a fox looking at a lit up city in the background.\"\n",
              "  },\n",
              "    {\n",
              "    \"timecode\": \"00:02:35\",\n",
              "    \"caption\": \"A nighttime grayscale shot of a bear walking near the edge of a cliff.\"\n",
              "  },\n",
              " {\n",
              "    \"timecode\": \"00:02:51\",\n",
              "    \"caption\": \"A nighttime shot of a mountain lion in the forest.\"\n",
              "  },\n",
              "{\n",
              "    \"timecode\": \"00:03:04\",\n",
              "    \"caption\":\"A black bear is standing in the forest looking directly at the camera. The bear is moving. Then a lighter brown bear enters the frame from the right.\"\n",
              "    },\n",
              "    {\n",
              "       \"timecode\": \"00:03:22\",\n",
              "       \"caption\": \"A lighter brown bear stands in the forest, looking to the left.\"\n",
              "    },\n",
              "    {\n",
              "     \"timecode\": \"00:04:21\",\n",
              "    \"caption\": \"A nighttime shot of a bobcat with glowing eyes standing near a fallen tree limb.\"\n",
              "  }\n",
              "]\n",
              "```"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"For each scene in this video, generate captions that describe the scene along with any spoken text placed in quotation marks. Place each caption into an object with the timecode of the caption in the video.\"  # @param [\"For each scene in this video, generate captions that describe the scene along with any spoken text placed in quotation marks. Place each caption into an object with the timecode of the caption in the video.\", \"Organize all scenes from this video in a table, along with timecode, a short description, a list of objects visible in the scene (with representative emojis) and an estimation of the level of excitement on a scale of 1 to 10\"] {\"allow-input\":true}\n",
        "\n",
        "video = trailcam_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=model_name,\n",
        "    contents=[\n",
        "        types.Content(\n",
        "            role=\"user\",\n",
        "            parts=[\n",
        "                types.Part.from_uri(\n",
        "                    file_uri=video.uri,\n",
        "                    mime_type=video.mime_type),\n",
        "                ]),\n",
        "        prompt,\n",
        "    ]\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOQzKYGJKAnD"
      },
      "source": [
        "The prompt used is quite a generic one, but you can get even better results if you cutomize it to your needs (like asking specifically for foxes).\n",
        "\n",
        "The [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows how you can postprocess this output to jump directly to the the specific part of the video by clicking on the timecodes. If you are interested, you can check the [code of that demo on Github](https://github.com/google-gemini/starter-applets/tree/main/video)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wog32E7CKnT6"
      },
      "source": [
        "# Extract and organize text\n",
        "\n",
        "Gemini can also read what's in the video and extract it in an organized way. You can even use Gemini reasoning capabilities to generate new ideas for you.\n",
        "\n",
        "<video controls width=\"400\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/post_its.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "baNCeA3GKrfu"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "Certainly! Here's a table of the sticky notes, organized into categories based on their subject matter, along with a few ideas for additional considerations:\n",
              "\n",
              "**Project Name Brainstorm Sticky Notes**\n",
              "\n",
              "| Category            | Sticky Note Names    |\n",
              "|--------------------|--------------------|\n",
              "| **Constellations**  | Canis Major, Leo Minor, Lynx, Delphinus, Serpens, Centaurus, Lyra |\n",
              "| **Celestial Objects**| Lunar Eclipse,  Stellar Nexus,  Comets Tail, Supernova Echo, Orion's Belt,   Celestial Drift|\n",
              "| **Mythological**    |  Prometheus Rising, Chimera Dream, Perseus Shield, Zephyr, Odin, Aether, Athena's Eye, Athena, Medusa, Hera, Cerberus, |\n",
              "| **Scientific Concepts** | Convergence,  Astral Forge,  Draco,  Euler's Path, Equilibrium,  Chaos Theory, Golden Ratio, Infinity Loop, Taylor Series, Stokes Theorem,  Bayes Theorem, Riemann's Hypothesis, Fractal, Symmetry, Vector |\n",
              "| **Other** |   Galactic Core, Titan, Echo, Phoenix, Pandora's Box, Sagitta|\n",
              "\n",
              "\n",
              "**Ideas for What We May Have Missed**\n",
              "\n",
              "1. **Themes & Subcategories**: Is there a desired theme or subcategory (e.g., \"geometric\", \"mythological\", \"technical\", etc.)? If so, some additional terms or more targeted brainstorming in that area might be beneficial.\n",
              "    * **Greek Mythology**: Expand on Greek/Roman myths. Consider names like: *Icarus's Fall*, *Pegasus*, *Hades' Gate*, *Poseidon's Trident*.\n",
              "    * **Astronomy**: More technical names from astronomy like *Solar Flare*, *Accretion Disk*, *Gravitational Lens*, *Quasar Jet*, *Event Horizon*, etc.\n",
              "2. **Style**:\n",
              "    * **Simplicity vs. Complexity**:  Do you want a name that is straightforward or something more complex and evocative?  Consider a combination of scientific and mythological names.\n",
              "    * **Acronyms/Initialisms**: Some project names are short acronyms (e.g., ALICE, ATLAS). Perhaps brainstorm some of those if you need a shorter title.\n",
              "3. **Audience**: Who is the audience for this project? The name could be more or less technical, depending on whether it's an expert audience or a general public one.\n",
              "4. **Uniqueness**: It's often valuable to research if any of these project names are already in use to avoid confusion.\n",
              "5. **Team Name/Internal vs. External Name**: Is this a team name or is this a name for a product that the public will see?\n",
              "\n",
              "Let me know if you’d like me to explore any of these ideas or categories in more detail!"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "prompt = \"Transcribe the sticky notes, organize them and put it in a table. Can you come up with a few more ideas?\" # @param [\"Transcribe the sticky notes, organize them and put it in a table. Can you come up with a few more ideas?\", \"Which of those names who fit an AI product that can resolve complex questions using its thinking abilities?\"] {\"allow-input\":true}\n",
        "\n",
        "video = post_its_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=model_name,\n",
        "    contents=[\n",
        "        types.Content(\n",
        "            role=\"user\",\n",
        "            parts=[\n",
        "                types.Part.from_uri(\n",
        "                    file_uri=video.uri,\n",
        "                    mime_type=video.mime_type),\n",
        "                ]),\n",
        "        prompt,\n",
        "    ]\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjKIsLDMTNk1"
      },
      "source": [
        "# Structure information\n",
        "\n",
        "Gemini 2.0 is not only able to read text but also to reason and structure about real world objects. Like in this video about a display of ceramics with handwritten prices and notes.\n",
        "\n",
        "<video controls width=\"500\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/Pottery.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bqzqedMFT5Wp"
      },
      "outputs": [],
      "source": [
        "prompt = \"Give me a table of my items and notes\" # @param [\"Give me a table of my items and notes\", \"Help me come up with a selling pitch for my potteries\"] {\"allow-input\":true}\n",
        "\n",
        "video = pottery_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=model_name,\n",
        "    contents=[\n",
        "        types.Content(\n",
        "            role=\"user\",\n",
        "            parts=[\n",
        "                types.Part.from_uri(\n",
        "                    file_uri=video.uri,\n",
        "                    mime_type=video.mime_type),\n",
        "                ]),\n",
        "        prompt,\n",
        "    ],\n",
        "    config = types.GenerateContentConfig(\n",
        "        system_instruction=\"Don't forget to escape the dollar signs\",\n",
        "    )\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zsh6i-Z6VHNK"
      },
      "source": [
        "As you can see, Gemini is able to grasp to with item corresponds each note, including the last one."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uIfsFC0pVUTD"
      },
      "source": [
        "# Analyze screen recordings for key moments\n",
        "\n",
        "You can also use the model to analyze screen recordings. Let's say you're doing user studies on how people use your product, so you end up with lots of screen recordings, like this one, that you have to manually comb through.\n",
        "With just one prompt, the model can describe all the actions in your video.\n",
        "\n",
        "<video controls width=\"400\"><source src=\"https://storage.googleapis.com/generativeai-downloads/videos/user_study.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wrMHZ0MxW75y"
      },
      "outputs": [],
      "source": [
        "prompt = \"Generate a paragraph that summarizes this video. Keep it to 3 to 5 sentences with corresponding timecodes.\" # @param [\"Generate a paragraph that summarizes this video. Keep it to 3 to 5 sentences with corresponding timecodes.\", \"Choose 5 key shots from this video and put them in a table with the timecode, text description of 10 words or less, and a list of objects visible in the scene (with representative emojis).\", \"Generate bullet points for the video. Place each bullet point into an object with the timecode of the bullet point in the video.\"] {\"allow-input\":true}\n",
        "\n",
        "video = user_study_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=model_name,\n",
        "    contents=[\n",
        "        types.Content(\n",
        "            role=\"user\",\n",
        "            parts=[\n",
        "                types.Part.from_uri(\n",
        "                    file_uri=video.uri,\n",
        "                    mime_type=video.mime_type),\n",
        "                ]),\n",
        "        prompt,\n",
        "    ],\n",
        ")\n",
        "\n",
        "Markdown(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cizoUEdIYLd0"
      },
      "source": [
        "Once again, you can check the  [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows an example on how to postprocess this output. Check the [code of that demo](https://github.com/google-gemini/starter-applets/tree/main/video) for more details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lND4jB6MrsSk"
      },
      "source": [
        "# Next Steps\n",
        "\n",
        "Try with you own videos using the [AI Studio's live demo](https://aistudio.google.com/starter-apps/video) or play with the examples from this notebook (in case you haven't seen, there are other prompts you can try in the dropdowns).\n",
        "\n",
        "For more examples of the Gemini 2.0 capabilities, check the [Gemini 2.0 folder of the cookbook](https://github.com/google-gemini/cookbook/blob/main/gemini-2/). You'll learn how to use the [Live API](live_api_starter.ipynb), juggle with [multiple tools](./plotting_and_mapping.ipynb) or use Gemini 2.0 [spatial understanding](./spatial_understanding.ipynb) abilities.\n",
        "\n",
        "The [examples](../examples) folder from the cookbook is also full of nice code samples illustrating creative ways to use Gemini multimodal capabilities and long-context."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "video_understanding.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}