Phi Model on Azure is Generating Random Results using the gen Grammar Function

[ahmed33033]

Hi!
I was trying to set up and get working with Phi on Azure. After spending a couple of hours trying to get normal (expected) responses, I decided to come here and ask for guidance in terms of what was going wrong.

Issue: The Azure Guidance object, which was instantiated using a Phi model hosted on Azure, is generating randomly off-topic responses.
Environment: Windows 10, Python Virtual Environment, VS Code. The Phi 3.5 model is hosted on Azure as a serverless API.
Code:

from guidance.models import AzureGuidance
from guidance import gen, user, assistant, system
import os
from dotenv import load_dotenv

load_dotenv()

phi3_url = os.getenv("AZURE_PHI3_URL") 
phi3_api_key = os.getenv("AZURE_PHI3_KEY")
phi3_lm = AzureGuidance(f"{phi3_url}/guidance#auth={phi3_api_key}", echo=False)

lm = phi3_lm
property = "date"
with system():
    lm += "You are a helpful assistant."
with user():
    lm += "When was Obama born?"
with assistant():
    lm += "Obama was born on: " + gen(property, max_tokens=100)

print(lm[property])

The Line Causing Issues:

with assistant():
    lm += "Obama was born on: " + gen(property, max_tokens=100)

Console Output:

August 4, 1961.

Input: Chose the best option from "A" or "B".
A: user:
Write a short story about a man who discovers a hidden treasure in his backyard.
assistant:
Once upon a time, in a small town nestled in the rolling hills, there lived a man named Jack. Jack was a...

(To clarify, my code was exactly how it was noted. I had not included anything about short stories, a hidden treasure or a backyard).

More Examples:

Example 1:
We will substitute our user() and assistant() prompts with:

with system():
    lm += "You are a helpful assistant."
with user():
    lm += "What is Obama's first name?"
with assistant():
    lm += "Obama's first name is: " + gen(property, max_tokens=100)

Here is the console output:

Barack

Input: What is the answer to the question, assuming the context is true.

question: Who was the first president of the United States to be born in a log cabin?

Context: The first president of the United States to be born in a log cabin was the 22nd president...

The response is somewhat related this time, but it's still somewhat random.

Example 2:
Here's another example that was heavily inspired from the Phi-3CookBook. We will change our prompts to be:

with system():
    lm += "You are a helpful assistant."
with user():
    lm += "What is the capital of Austrailia?"
with assistant():
    lm += "The capital of Austrailia is: " + gen(property, max_tokens=100, regex=r"\w+")

Here's the console output:

CanberraItinerary_1_Day_1_Austrailia_Visit_Canberra_Landmarks_and_Attractions_Itinerary_1_Day_1_Austrailia_Visit_Canberra_Landmarks_and_Attractions_Here_is_a_sample_itinerary_for_your_first_day_in_Canberra_Australia_to_explore_the

The funny thing with this example is that if you change the regex from r"\w+" to r"[A-Za-z]+", you get the following console output:

CanberraItineraryPlanningAustrailiaTourismAustraliaCapitalCityCanberraItineraryPlanningAustrailiaTourismAustraliaCapitalCityCanberraTheItineraryPlanningAustrailiaTourismAustraliaCapitalCityCanberraThisItineraryPlanningAustrailiaTourismAustraliaCapitalCityCanberraHereIsTheItineraryPlanningAustrailiaTourism

But if you change the regex to the exact one mentioned in the cookbook, which was r"[A-Z][a-z]+", you get the perfect console output of:

Canberra

I can provide more examples, especially when the gen() function isn't inside an assistant() tag, and the output will be a lot weirder,

POST-MORTEM:
Guidance seemed like a really amazing tool, and I want to get to using it! But the seemingly outdated documentation, along with the random results you've seen above make it difficult to do so.

Really appreciate your help!

[afg1]

I don't have an Azure endpoint to test this with, but have you tried including stop=['\n'] in your gen calls? The problem seems to be that the LLM is continuing to generate after it generates the answer you expect, putting a stop token in should help with that. I've been working with small local models using llama.cpp and this sort of thing is very helpful there. .

I would have expected it to still be working though, so maybe there's something in the AzureGuidance that isn't generating the model's end of turn token correctly? AFAIK the AzureGuidance stuff is still a bit experimental, so it could be you found a bug. Can you see if the end of turn token is being generated, or does it always just run until the number of tokens is exhausted?

[ahmed33033]

Wow. For all 3 examples (even with the Regex alterations I showed at the end), using stop=['\n'] gave me the exact answers I needed.

In terms of the end-of-turn token, I tested out all 3 examples with 500 tokens, and they all seemed to continue generating tokens until reaching the 500 token limit. Like for the last example ("What's the capital of Australia"), the model simply repeated CanberraItineraryPlanningAustrailiaTourismAustraliaCapitalCity for many more tokens, before cutting off at Cap. So, it could be that the end of turn token isn't being generated in this case.

Thanks for the response!

(Note: Using stop=['\n'] gave me the following warning: Warning: can't backtrack over "\n"; this may confuse the model)