Does Google Gemini support Vizro-AI?

[richizo]

Question

I'm trying to follow the example given on Vizro-AI homepage, but instead of using the OpenAI model I switched to the Google Gemini.

But I'm getting the error bellow.

Error:

ChatGoogleGenerativeAIError:** Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[imports].items: missing field.

Stacktrace:

Traceback (most recent call last):
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/graphaito.py", line 42, in <module>
    fig = vizro_ai.plot(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/_vizro_ai.py", line 34, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/_vizro_ai.py", line 87, in plot
    response = _get_pydantic_model(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/vizro_ai/dashboard/_pydantic_output.py", line 84, in _get_pydantic_model
    res = pydantic_llm.invoke(message_content)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2879, in invoke
    input = context.run(step.invoke, input, config)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 5093, in invoke
    return self.bound.invoke(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 277, in invoke
    self.generate_prompt(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 777, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 634, in generate
    raise e
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 624, in generate
    self._generate_with_cache(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 846, in _generate_with_cache
    result = self._generate(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 975, in _generate
    response: GenerateContentResponse = _chat_with_retry(
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 198, in _chat_with_retry
    return _chat_with_retry(**kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/home/ubuntu/Projects/DataAnalysis/graphaito/.venv/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 192, in _chat_with_retry
    raise ChatGoogleGenerativeAIError(
langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[imports].items: missing field.

Python: 3.10

vizro: 0.1.26

vizro_ai: 0.3.2

langchain: 0.2.17

langchain_core: 0.2.43

langchain_google_genai: 1.0.10

langgraph: 0.2.16

langsmith: 0.1.142

Code/Examples

from langchain_google_genai import (
    ChatGoogleGenerativeAI,
    HarmBlockThreshold,
    HarmCategory,
)
from vizro_ai import VizroAI
import vizro.plotly.express as px

safety_settings = {
    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE
}

llm = ChatGoogleGenerativeAI(
    model='gemini-1.5-flash',
    api_key='XXXX',
    safety_settings=safety_settings)

vizro_ai = VizroAI(model=llm)

df = px.data.gapminder()

fig = vizro_ai.plot(
    df,
    """
    Create a line graph for GDP per capita since 1950 for each continent.
    Mark the x axis as Year, y axis as GDP Per Cap and don't include a title.
    Make sure to take average over continent.
    """
)

Which package?

vizro-ai

Code of Conduct

• I agree to follow the Code of Conduct.

[maxschulz-COL]

Dear @richizo ,

thank you so much for your message! This is indeed an interesting error. On the surface I cannot tell immediately whether the bug lies with vizro-ai or actually langchain. When I try out the code below (which is the core of what we are doing), it seems to work. This is only to say that there doesn't seem anything fundamentally wrong with the google model, and the with_structured_output functionality of langchain:

from langchain_google_genai import (
    ChatGoogleGenerativeAI,
)
from pydantic.v1 import BaseModel

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", safety_settings=safety_settings)


class User(BaseModel):
    name: str
    age: int


pydantic_llm = llm.with_structured_output(User)
pydantic_llm.invoke("Jon is 23 years old")

I do need to jump, can only look again tomorrow, but could you seem if you can locate more what argument/situation might be causing this?

[maxschulz-COL]

A quick search is showing that this might be similarly occurring for others:

• langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[action].type: must be specified langchain-ai/langchain-google#535
• Error:  Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].name langchain-ai/langchain-google#471
• https://stackoverflow.com/questions/78402786/chatgooglegenerativeaierror-invalid-argument-provided-to-gemini-400-please-ens (this looks promising, maybe Gemini doesn't work with our system prompts, which would be somewhat annoying from the langchain side of things. Ideally all ChatModels behave in the same ways)

If you could dig a bit deeper here that would be amazing, otherwise I'll try to take a look tomorrow

[maxschulz-COL]

Dear @richizo,

sorry for taking a bit longer to reply, it was a bit hectic on my side. There seems to be an bug on the Google API side in that it cannot properly read a schema produced by pydantic V1 (which we still use, but we are changing that as we speak).

The following code shows that if you uncomment the hobbies, the API request will fail:

from langchain_google_genai import (
    ChatGoogleGenerativeAI,
)
from pydantic.v1 import BaseModel

llm = ChatGoogleGenerativeAI(model="gemini-1.5-pro")


class User(BaseModel):
    name: str
    age: int
    # hobbies: list[str]


pydantic_llm = llm.with_structured_output(User)
pydantic_llm.invoke("Jon is 23 years old and he plays guitar and reads books")

Now further testing suggests that this is not the case for Pydantic V2, so if you change the import to just pydantic, then it works! (although I get strange warnings about the title key). Hence the path forward is either:

• wait for the Gemini API to work with pydantic V1 (not sure that's going to happen)
• us upgrading to pydantic V2 (should happen by the end of the year)
• technically we could upgrade VizroAI before Vizro, but that would be a mess from other angles, so let's stick with option 2

@antonymilne FYI - this is further support :)