Testing GitHub SSH Signing

Try Seldon Core

Repository to bring up Seldon core locally with minimal effort. This repo also lists the know issues and work around/fixes needed for installation, run Seldon Core out of the box sample notebooks with ease.

References

Install Locally

Notebooks

Seldon Core

Seldon Core Setup

Prerequisites

1. Docker
2. Kind K8S
3. Kubectl
4. grpcurl for gRPC: Command-line tool for interacting with gRPC servers
5. MINIO - High performance Object Storage - MINIO Setup locally
6. Helm
7. !git clone --branch 1.15.0-dev https://github.com/SeldonIO/seldon-core

Setup Seldon Core locally

Setup Seldon Core with Kind

Setup MINIO

Kubernetes-based Event Driven Autoscaler(KEDA) Setup

Seldon Deploy

istioctl

Useful admin and troubleshooting cli commands

1. kubectl get seldondeployments
2. kubectl describe seldondeployments <seldondeployment name>
3. kubectl describe svc <svc name>
4. kubectl get namespaces
5. kubectl get all -n istio-system
6. kubectl get svc -n istio-system
7. kubectl describe namespace default
8. kubectl describe namespace istio-system
9. kubectl get pods
10. kubectl describe pod <pod name>
11. which istioctl
12. istioctl version
13. istioctl proxy-status
14. istioctl proxy-config cluster <pod name>
15. istioctl install --set profile=default -y
16. istioctl install --set profile=demo -y
17. istioctl analyze
18. kubectl describe service/istio-ingressgateway -n istio-system
19. kubectl get svc istio-ingressgateway -n istio-system -oyaml
20. kubectl get ksvc,configuration,revision,route
21. kubectl get ingress
22. kubectl describe pod <pod name> -n seldon-system
23. kubectl describe svc sklearn-default
24. kubectl get svc istio-ingressgateway -n istio-system
25. kubectl describe svc istio-ingressgateway -n istio-system
26. kubectl get pods -n=istio-system
27. kubectl logs pods/<pod name> -n=istio-system
28. kubectl logs pod/<pod name> --tail=100
29. istioctl profile dump demo
30. kubectl logs <pod name/container id>
31. kubectl get gateway -n istio-system
32. kubectl describe gateway/seldon-gateway -n istio-system
33. kubectl get gateway --all-namespaces
34. kubectl delete all --all
35. Label a K8s namespace for istio injection kubectl label namespace seldon istio-injection=enabled
36. Shows resources available in each node.kubectl describe nodes
37. Load docker image to kind cluster/control plane kind load docker-image nvcr.io/nvidia/tritonserver:21.08-py3 --name seldon
38. docker exec kind-control-plane crictl images
39. Scale Seldon deployment !kubectl scale --replicas=4 sdep/<seldon deployment id>
40. Open interactive shell to kind cluster docker exec -it kind-control-plane bash
41. list nodes/clusters/kubeconfig on kind cluster kind get nodes
42. To get details about K8s nodes kubectl describe clusters/nodes/kubeconfig
43. kubectl cluster-info dump
44. Get pod names with seldon deployment-id=model-server-triton kubectl get pods -l seldon-deployment-id=model-server-triton -o custom-columns=:metadata.name
45. Get all K8S objects in seldon namespace kubectl get all -n seldon
46. Get Kafka with strimzi operator kubectl get Kafka
47. Istio port forwarding kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:8080
48. Minio port forwarding export POD_NAME=$(kubectl get pods --namespace minio-system -l "release=minio" -o jsonpath="{.items[0].metadata.name}") && kubectl port-forward $POD_NAME 9000 -n minio-system
49. K8S component statuses kubectl get componentstatuses
50. delete deployment kubectl delete deploy <deployment name> -n <namespace>
51. scale down kubectl scale --replicas=0 replicaset.apps/<replicaset name>
52. Remove docker images docker rmi $(docker images -f "dangling=true" -q)
53. Delete namespace kubectl delete namespace/<namespace name>
54. Port forward for prometheus analytics. kubectl port-forward -n seldon-system svc/seldon-core-analytics-prometheus-seldon 9007:80. Open http://localhost:9007/.
55. Delete Seldon deployments kubectl delete sdep -n seldon <deployment name>

Example Seldon Core Deployments using Helm with Istio

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/istio_examples.html

1. open istio_example.ipynb

Scaling Seldon Deployments

1. Scale Seldon Deployments Notebook

Auto scaling

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/autoscaling_example.html

1. Auto Scaling Examples

Example Model Servers with Seldon

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/server_examples.html

1. open server_examples.ipynb

Metrics with Prometheus Operator

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/metrics.html https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html

1. Metrics with prometheus Notebook
2. Stateful Model Feedback Metrics Server

Triton Examples

1. open triton_examples.ipynb
2. git clone https://github.com/azure/azureml-examples

Example IRIS Model loaded from MINIO

1. iris_deploy_with_minio.ipynb

Custom model packaging and deployment

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/python/index.html https://docs.primehub.io/docs/model-deployment-language-wrapper-python#pytorch

1. Custom Model deployment with Language Wrapper
2. Pytorch model deployment with Triton Inference Server

AB Tests and Progressive Rollouts

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/istio_canary.html

1. Canary Deployments with istio Notebook
2. Seldon/Iter8 - Progressive AB Test with Single Seldon Deployment
3. Seldon/Iter8 - Progressive AB Test with Multiple Seldon Deployments

Triton Model Server with MINIO Model Repository

1. Triton Model server with MINIO repository

Complex Graph Examples

1. graph-examples

Protocol Examples

1. protocol_examples

Seldon Kafka Integration with KEDA scaling over SSL

1. Notebook

Metrics Server

Reference: https://docs.seldon.io/projects/seldon-core/en/latest/examples/feedback_reward_custom_metrics.htm https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html

1. Metrics Server Notebook

Batch processing with Argo Worfklows

1. Batch processing with Argo Worfklows
2. Batch processing with Kubeflow Pipelines

Example Model Explanations with Seldon

Reference: https://github.com/oegedijk/explainerdashboard/blob/master/notebooks/explainer_examples.ipynb

1. open explainer_examples.ipynb

MLFlow Pre-packaged Model Server AB Test Deployment

1. open mlflow_server_ab_test_ambassador.ipynb

Known issue with K8S >= 1.25 :

1. SeldonIO/seldon-core#4339
2. SeldonIO/seldon-core#2485

Known issues with Istio configuration

1. kserve/kserve#1005
2. https://stackoverflow.com/questions/69826714/fail-to-run-istio-ingressgateway-got-readiness-probe-failed-connection-refused