Can the Assistant Accelerator be used with an Azure OpenAI fine-tuned model ?

[PaulineJaquot]

Hi,
We noticed that the model is struggling extracting domain-specific information and answer specific questions on our private documents. We're considering training a model (on the Azure OpenAI Studio) on one of these specific tasks to check the potential. As long as the trained model is deployed and the specific env. variables adjusted (deployment name, resource group, etc.), Is it possible to use it in the Assistant Accelerator ? Thank you, just want to double-check before that we go intro training :)

[WookieeOnTheRun]

Hi Pauline, it is possible - a few things to keep in mind :

1. In the requirements file(s) for the function(s) and backend apps, you'll need to upgrade tiktoken to V0.5.0.
2. In /app/backend/core/modelhelper.py, notice two variables : MODELS_2_TOKEN_LIMITS and AOAI_2_OAI. These Python dictionaries are used in reference to verifying token limits via TikToken. Once you have fine-tuned your base gpt-35-turbo model to create a new base model and then deployed a new model from that base model, ensure the naming conventions you use fit within those parameters ( gpt-35-turbo-, for example ). Tiktoken is specific about naming conventions, since it assumes the user is using a 'default' base model. Tiktoken looks at both the name of the model as well as the model name prefix.
3. Under MODELS_2_TOKEN_LIIMITS : add a dictionary entry for base model name : token count
4. Under AOAI_2_OAI : add a dictionary entry for base model name : deployed model name

I would also point out that, in most cases, fine-tuning can be avoided in lieu of more thorough prompt engineering ( at least in my experience ). The base prompts within Infor. Asst. are very thorough, but are intended to be 'generic' for any type of document. Tuning the prompts for your dataset could make a bigger difference in the quality of returned responses. Fine-Tuning is obviously not a 'one and done' type of effort.

[PaulineJaquot]

Great thank you, we'll keep this option as last resort :)