[Tracker]: Model Serving v2 - KServe Support

[andrewballantyne]

Description

We'd like to support the ability to host KServe Model Serving servers... this differs from Model Mesh (our current structure).

Target Branch

f/model-serving

Requirements

• Model serving phase 2 requirements (I'm gonna update the requirements later today)

1. Model serving runtimes

   • P0: OOTB support Caikit/TGIS
   • P0: (depending on feasibility): NVidia Triton Inference Server
   • P0: Control that some Model Serving Runtimes can only be enabled in kserve

2. Support options for KServe and/or ModelMesh (support KServe - 1 model per pod or modelmesh - multiple models per pod)

   • P0: RHODS admins should be able to configure whether they want to use KServe (single model serving + additional functionality), ModelMesh, or both
   • P0: Users should be able to select kserve or modelmesh on project creation
     • Only one can be enabled per project
   • P0: Users should be able to deploy in the global view to both kserve and modelmesh

Itemized UX Issues

• [UX]: Model serving v2 - enhancements #1809

Itemized Dev Issues

General configuration

• [Story]: Create dashboard configuration to control kserve and modelmesh #1939
• [Story]: Add logic to detect if kserve or modelmesh are installed #1946
• [Story]: Migration of existing clusters to kserve #1975

Custom Serving Runtimes

• [Story]: Adapt Custom Serving Runtimes to kserve #1912
• [Story]: Add TGIS and Caikit OOTB Serving Runtime #1911
• [Story]: Migrate current Custom Serving Runtimes #1940
• [Story]: Display serving platforms available in Custom Serving Runtime view #1973
• [Story]: Add new OVMS OOTB template #1989

Admin Settings

• [Story]: Add admin section to control model serving platforms #1913

Projects

• [Story]: Adapt logic for model serving creation in new projects #1941
• [Story]: Empty State platform selection #1944
• [Story]: Kserve Empty Project View #1942
• [Story]: Filter Serving Runtime by platform #1945
• [Story]: KServe Deploy Modal #1943
• [Story]: KServe Model Table #1947
• [Story]: Handle Cluster Component Installation #1995

Global View

• [Story]: Adapt Global View Table to Kserve #1948

Enhancements

• Revamp "Add server" button #1579
• Disable Model route and Token autorization for Edit user #1581
• Make bucket field mandatory in new data connection for deploy model #1583
• Edit Model server modal enhancements #1626
• [Feature Request]: Validate Serving Runtime when creating Custom Serving Runtime #1353
• [Bug]: failure to create model server with previously used name #1998
• [Story]: Make sure enable-route and enable-route are not enable in kserve and they are not present when disabled in modelmesh. #2050

---

Incubation work

• Unit testing coverage
• Integration testing coverage
• [Story]: Handle Cluster Component Installation #1995
• [Story]: No serving platforms available view #1994
• [Story]: Adapt Global Deploy Model Modal #1956
• [Story]: Improve url handling in kserve #2053
• [Story]: Regression in modelmesh modal for tokens #2049
• [Story]: Support rest calls in caikit #2058
• [Story]: Revert logic of Custom Serving Runtime migration #2059
• [Story]: OVMS OOTB Model does not support KServe #2048
• [Story]: Add annotations for kserve InferenceService #2079

Incubation Enhancements (can be done post-MVP)

• Add popovers to Model Server Modal #1580
• [Story]: Adapt alerting in model serving admin section #2047
• [Story]: Model global view doesn't hot reload when deleting an inference service #2052
• [Story]: Change link button text to Project display name instead of resource name on global model serving page #2132
• [Story]: Show error state on global model serving page when the project is labelled while the platform is not installed #2134

Related artifacts

• Serving runtimes settings mockups
• Cluster settings mockups
• Model Serving Project mockups
• Model Serving v2 Open Discussion
• Model Serving Controller Requirements

[andrewballantyne]

cc @lucferbux please adjust this tracker to fit what we want for v2 -- I imagine some of the issues might come from: #1312

[lucferbux]

Yes, that's right, I would say all the issues for 1312 can go to v2, that old tracker was just a filler until we waited for v2, let me rework this.

[lucferbux]

[UPDATE] We are gonna keep #1312 we have some issues there that can be fixed after kserve is implemented, moving only the tasks that are related to kserve development

[gmarkley-VI]

Is this still on UX? Can you please update the status on this? @lucferbux @vconzola

[vconzola]

@gmarkley-VI This is a Tracker. It looks like it has a Tracker State of "UX investigation", whatever that means. I have first pass designs that were reviewed at last Thursday's UX meeting, but the UX will likely be iterated on for quite a while. This should not prevent dev work from getting started. I'm not sure when we declare Tracker issues to be "off" UX?

[andrewballantyne]

I moved this into progress -- Lucas and Juntao are working on it.

[lucferbux]

Moving the enhancements to #1313