This approach is best suited for:
- High throughput low latency scenarios
The below diagram shows a high-level design for implementing online scoring for streaming workloads suitable for classical machine learning scenarios using Azure Machine Learning and Azure Stream Analytics.
The solution consists of the following components:
- Datastores: production-grade data used to develop models.
- Experimentation workstation: workstation where data scientists can access data, explore data and develop machine learning models.
- Orchestration: central solution responsible for triggering pipelines and orchestrating data movement.
- Artefact repository: place to store machine learning models and experiment metrics.
- Training pipeline: machine learning pipeline job used to build a model artifact for deployment.
- Model deployment: managed online endpoint used to host the model artifact for online inferencing.
- Streaming services: data streams will be consumed and processed using the managed online endpoint.
- Monitoring: central monitoring solution for application and model logs and metrics. Inference data and data drift metrics are stored here.
- Data drift pipeline: pipeline job to calculate data drift metrics based on inference data and model training data.
- Data export: solution to export inference data collected from the managed online endpoint. This can be used to retrain the model or determine data drift.
- Source control: solution to track code.
- Automated workflows: workflows to automate the build and deployment of different components used in the solution.
The end-to-end workflow operation consists of:
- Creating a machine learning model as an output from a pipeline job designed to develop a model artifact for the relevant use case.
- The model artifact is registered in the model registry and consumed by the online managed endpoint.
- Data streams will be accepted by Azure Event Hub and sent to Azure Stream Analytics for processing. Azure Stream Analytics will consume the online managed endpoint to generate predictions for incoming data.
- When the online managed endpoint is triggered, the data within the request payload will be logged within Azure Monitor.
- Inference data collected from the request payload can be extracted by an Azure Logic App by querying Azure Monitor and writing the output to blob storage. These can be used as an input to calculating data drift metrics.
- Data drift metrics will be calculated via a pipeline job and sent to Azure Monitor via Application Insights as custom metrics.
- Alerts can be triggered in Azure Monitor to retain and re-deploy models via triggering a pipeline job designed to develop a model artifact.
The below diagram shows the overall CI/CD process as built with GitHub Actions. This approach consists of three environments consisting of an identical set of resources.
Azure Machine Learning artifacts will follow the build and release process shown in the below diagram.
Azure Stream Analytics will follow the build and release process shown in the below diagram. It's important that the associated Azure Machine Learning artifacts have to be created first in each environment.
The environments include:
- Development: used by developers to build and test their solutions.
- Staging: used to test deployments before going to production in a production-like environment. Any integration tests are run in this environment.
- Production: used for the final production environment.
You might also find these references useful: