Azure OpenAI Insights

The 'Azure OpenAI Insights' workbook offers deep insights into Azure OpenAI usage, helping you manage costs, optimize performance, and make strategic decisions for a robust AI infrastructure.

Introduction

In the ever-evolving world of Artificial Intelligence, organizations and entities across various sectors are on a quest to leverage advanced technologies efficiently. Azure OpenAI opens a realm of possibilities, offering both challenges and excitement, particularly for those at the early stages of AI adoption.

Read more in depth in this Tech Community blog: Azure OpenAI Insights: Monitoring AI with Confidence

This workbook offers deep insights into Azure OpenAI resources and usage (Platform Metrics and Logs) and can be powerful tool in analyzing & monitoring your AI initiatives.

Structure and Views

Structure

This workbook contains 3 main parts:

• Overview - Holistic view of Azure OpenAI resources
• Monitor - Holistic view of Azure OpenAI resources Metrics
• Insights - Holistic view of Azure OpenAI resources Logs
  • Requires by enabling Diagnostic Settings to Log Analytics Workspace.

Views

Types of views this workbook provides:

• Overview
  • Azure OpenAI Resources by
    • SubscriptionId
    • Resource Group
    • Location
    • Kind
    • Public Network Access
    • Private Network Access
  • All Azure OpenAI Resources

The information displayed uses KQL queries to query the Azure Resource Graph.

• Monitor
  • Overview
    • Requests
    • Processed Inference Tokens
    • Processed Prompt Tokens
    • Generated Completions Tokens
    • Processed FineTuned Training Hours
    • Provisioned-managed Utilization
    • Active Tokens
    • Prompt Token Cache Match Rate
    • Time to Response
  • HTTP Requests
    • Requests
      • by Model Name
      • by Model Version
      • by Model Deployment Name
      • by Status Code
      • by StreamType
      • by Operation Name
      • by API Name
      • by Region
    • Time to Response
      • by Model Name
      • by Model Deployment Name
    • Prompt Token Cache Match Rate
      • by Model Name
      • by Model Deployment Name
  • Token-Based Usage
    • Processed Inference Tokens
      • by Model Name
      • by Model Deployment Name
    • Processed Prompt Tokens
      • by Model Name
      • by Model Deployment Name
    • Generate Completion Tokens
      • by Model Name
      • by Model Deployment Name
    • Active Tokens
      • by Model Name
      • by Model Deployment Name
  • PTU Utilization
    • Provisioned-managed Utilization
      • by Model Name
      • Model Version
      • by Model Deployment Name
      • by StreamType
      • by Region
  • Fine-tuning
    • Processed FineTuned Training Hours
      • by Model Name
      • by Model Deployment Name

The information displayed uses Azure OpenAI Platform Metrics and presented by multiple dimensions.

• Insights
  • Overview
    • Requests
      • by Resource
      • by Location
      • by StreamType
      • by Api Version
      • by Model Deployment Name + Operation Name
      • by Model Deployment Name
      • by Model Name + Operation Name
      • by Model Name
      • by Operation Name
      • by Avg Duration (ms)
      • by Avg Request Length (bytes)
      • by Avg Response Length (bytes)
  • By CallerIP
    • Requests
    • Operation Name
    • Model Deployment Name + Operation Name
    • Model Name + Operation Name
    • Avg Duration (ms)
    • Avg Request Length (bytes)
    • Avg Response Length (bytes)
  • All Logs
    • Successful requests
  • Failures
    • Failed requests
      • by Resources
      • by Api Version
      • by Operation name
      • by Stream Type

Filters

This workbook support to filter all the logs by several fields:

• Model Deployment Name
• Model Name
• Model Version
• Api Version
• Operation Name
• Stream Type
• Location

All the filters are related to each other to allow a granular view and simplify the tracking of the logs.

The information displayed uses KQL queries to query the Log Analytics Workspace that store the logs.

Note: The Logs will be available on resources that enabled Diagnostic Settings to Log Analytics Workspace.

Average Duration (ms)

Average Request / Response Length (bytes)

How to use it?

Importing this Workbook to your Azure environment is quite simple.

Follow this steps to use the Workbook:

• Login to Azure Portal
• Go to 'Azure Workbooks'

• Click on '+ Create'

• Click on '+ New'

• Open the Advanced Editor using the '</>' button on the toolbar

• Select the 'Gallery Template' (step 1)
• Replace the JSON code with this JSON code Azure OpenAI Insights JSON (step 2)
  • We use the Gallery Templaty type (step 1), so we need to use the 'Azure OpenAI Insights.workbook' and not the 'Azure OpenAI Insights.json'.
• Click 'Apply' (step 3)

• Click in the ‘Save’ button on the toolbar

• Select a name and where to save the Workbook:
  • Title: 'Azure OpenAI Insights'
  • Subscription: <Subscription Name>
  • Resource group: <Resource Group Name>
  • Location: <Region>
• Click 'Save'

The Workbook is ready to use!

• Click on 'Workbooks'
• Click on 'Azure OpenAI Insights' Workbook.

Start using the Workbook and analyze your Azure OpenAI resources.

(Optional) You can filter by specific subscription/s or resource/s.