Python Phi3Vision Sample Error when divided into multiple Functions

[nmoeller]

Describe the bug

Running the phi3vision sample works 20/20 Times, but when i divided the sample Code into multiple function i am experiencing different errors.
When using the og.set_log_options(), i can see the input tensor is different on some runs, indicating something could be wrong with the memory.

The errors only occur when i am using images, when i am only using text everything works as it should.

This is the divided coded of the sample in multiple functions :

import onnxruntime_genai as og
og.set_log_options(enabled=True, model_input_values=True, model_output_values=True, ansi_tags=False)

def get_params(model,processor):
    params = og.GeneratorParams(model)
    image_paths = ["coffee.png"]
    print(image_paths)
    images = og.Images.open(*image_paths)

    prompt = f"<|system|>\n\nYou are a helpful assistant.\nYou know about provided images and the history of the conversation.\n<|end|>\n<|image_1|>\n<|user|>\nDescribe the image.<|end|>\n<|assistant|>\n"
    print("Processing images and prompt...")
    inputs = processor(prompt, images=images)
    params.set_inputs(inputs)
    params.set_search_options(max_length=7680)
    return params

def run():
    print("Loading model...")
    model = og.Model("./cpu-int4-rtn-block-32-acc-level-4")
    processor = model.create_multimodal_processor()
    tokenizer_stream = processor.create_stream()

    print("Generating response...")
    params = get_params(model,processor)

    generator = og.Generator(model, params)
    output = ""

    while not generator.is_done():
        generator.compute_logits()
        generator.generate_next_token()

        new_token = generator.get_next_tokens()[0]
        output += tokenizer_stream.decode(new_token)

    for _ in range(3):
        print()
    print(output)
    # Delete the generator to free the captured graph before creating another one
    del generator


if __name__ == "__main__":
    run()

This code produces out of 20 Runs the following behaviors :

1. Successful runs (12 Times)
2. Empty answers (2 Times)
3. Indices element out of data bounds (6 Times)

Expected behavior
I would expect the script to work 20 out of 20 Times, what you see on the error messages that the model input is different between multiple runs.

Screenshots
Error Message Indicies out of bounds Example 1:

Loading model...
Generating response...
['coffee.png']
Processing images and prompt...
  model_input_values   
Name: input_ids Shape[ 1 2550 ] Type: int64 Location: CPU
Values[ 0 0 0 0 1951801488 ... 29889 32007 13 32001 13 ]
2024-11-15 10:53:16.5129060 [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:516 onnxruntime::ExecuteKernel] Non-zero status code returned while running Gather node. Name:'/model/embed_tokens/Gather' Status Message: indices element out of data bounds, idx=1951801488 must be within the inclusive range [-32064,32063]
Traceback (most recent call last):
  File "c:\GIT\OpenSource\sk-phi3-vision-demo\phi3v.py", line 52, in <module>
    run()
  File "c:\GIT\OpenSource\sk-phi3-vision-demo\phi3v.py", line 38, in run
    generator.compute_logits()
onnxruntime_genai.onnxruntime_genai.OrtException: Non-zero status code returned while running Gather node. Name:'/model/embed_tokens/Gather' Status Message: indices element out of data bounds, idx=1951801488 must be within the inclusive range [-32064,32063]

Error Message Indicies out of bounds Example 2:

Loading model...
Generating response...
['coffee.png']
Processing images and prompt...
  model_input_values   
Name: input_ids Shape[ 1 2550 ] Type: int64 Location: CPU
Values[ 99334336 341 1 0 0 ... 29889 32007 13 32001 13 ]
2024-11-15 10:48:33.8290464 [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:516 onnxruntime::ExecuteKernel] Non-zero status code returned while running Gather node. Name:'/model/embed_tokens/Gather' Status Message: indices element out of data bounds, idx=99334336 must be within the inclusive range [-32064,32063]
Traceback (most recent call last):
  File "c:\GIT\OpenSource\sk-phi3-vision-demo\phi3v.py", line 53, in <module>
    run()
  File "c:\GIT\OpenSource\sk-phi3-vision-demo\phi3v.py", line 39, in run
    generator.compute_logits()
onnxruntime_genai.onnxruntime_genai.OrtException: Non-zero status code returned while running Gather node. Name:'/model/embed_tokens/Gather' Status Message: indices element out of data bounds, idx=99334336 must be within the inclusive range [-32064,32063]

Desktop (please complete the following information):

• OS: [e.g. Windows 11]
• Python 3.11
• onnxgenai 0.4.0 & 0.5.0