Trying to get with_structured_output working with Hugging Face custom endpoints. What am I doing wrong?

[michael-newsrx]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

def display(markdown: str, **kwargs) -> None:
    from IPython.core.display import display_markdown
    display_markdown(markdown, raw=True, **kwargs)

from huggingface_hub import InferenceEndpoint
from langchain.cache import SQLiteCache
from langchain.globals import set_llm_cache
from langchain_openai import ChatOpenAI

import local_utils
from config import use_llm_cache
from local_utils.hf.inference import hf_bearer_token
from notebooks_langchain_clarivate.config import llm_cache_file

bearer_token = hf_bearer_token()

if use_llm_cache:
    set_llm_cache(SQLiteCache(database_path=llm_cache_file))

ep: InferenceEndpoint = local_utils.hf.inference.llama_31_8B_Instruct(wait=True)
model = ChatOpenAI(model="TGI", base_url=ep.url + "/v1", api_key=bearer_token)

from pydantic.v1 import BaseModel, Field


class Joke(BaseModel):
    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")

structured_llm: ChatOpenAI
structured_llm = model.with_structured_output(Joke, method="json_mode")

import pathlib
from openai import UnprocessableEntityError

try:
    joke = structured_llm.invoke("Tell me a joke about cats.")
except UnprocessableEntityError as e:
    pathlib.Path("hf_request_with_error.json").write_text(e.request.content.decode("utf-8"))
    pathlib.Path("hf_response_with_error.json").write_text(e.response.content.decode("utf-8"))
    raise e

joke

Description

I'm trying to use JSON structured output with a custom Hugging Face endpoint using TGI via the OpenAI compatibility option.

I receive an error:

UnprocessableEntityError: Failed to deserialize the JSON body into the target type: response_format: missing field `value` at line 1 column 140

System Info

python -m langchain_core.sys_info

System Information

OS: Linux
OS Version: #44-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 13 13:35:26 UTC 2024
Python Version: 3.12.5 | packaged by conda-forge | (main, Aug 8 2024, 18:36:51) [GCC 12.4.0]

Package Information

langchain_core: 0.2.38
langchain: 0.2.16
langchain_community: 0.2.16
langsmith: 0.1.117
langchain_cli: 0.0.30
langchain_huggingface: 0.0.3
langchain_llm: 0.4.15
langchain_openai: 0.1.23
langchain_text_splitters: 0.2.4
langgraph: 0.2.19
langserve: 0.2.3

Other Dependencies

accelerate: 0.34.2
aiohttp: 3.10.5
async-timeout: Installed. No version info available.
cpm_kernels: 1.0.11
dataclasses-json: 0.6.7
einops: 0.8.0
fastapi: 0.114.0
gitpython: 3.1.43
httpx: 0.27.2
huggingface-hub: 0.24.6
jsonpatch: 1.33
langgraph-checkpoint: 1.0.9
langserve[all]: Installed. No version info available.
libcst: 1.4.0
loguru: 0.7.2
numpy: 1.26.4
openai: 1.44.1
orjson: 3.10.7
packaging: 24.1
peft: 0.12.0
pydantic: 2.9.1
pyproject-toml: 0.0.10
PyYAML: 6.0.2
requests: 2.32.3
sentence-transformers: 3.0.1
sentencepiece: 0.2.0
SQLAlchemy: 2.0.34
sse-starlette: 1.8.2
tenacity: 8.5.0
tiktoken: 0.7.0
tokenizers: 0.19.1
tomlkit: 0.12.5
transformers: 4.44.2
transformers_stream_generator: 0.0.5
typer[all]: Installed. No version info available.
typing-extensions: 4.12.2
uvicorn: 0.23.2

[dosubot[bot]]

To get with_structured_output working with Hugging Face custom endpoints using TGI via the OpenAI compatibility option without encountering the UnprocessableEntityError, you need to ensure that the input schema is correctly defined and matches the expected input format. Here is an example of how to set it up:

1. Define the structured output schema:

   from pydantic import BaseModel, Field
   
   class Joke(BaseModel):
       setup: str = Field(description="The setup of the joke")
       punchline: str = Field(description="The punchline to the joke")

2. Initialize the model with the structured output:

   structured_llm = model.with_structured_output(Joke)

3. Invoke the model and handle errors:

   import pathlib
   from openai import UnprocessableEntityError
   
   try:
       joke = structured_llm.invoke("Tell me a joke about cats.")
   except UnprocessableEntityError as e:
       pathlib.Path("hf_request_with_error.json").write_text(e.request.content.decode("utf-8"))
       pathlib.Path("hf_response_with_error.json").write_text(e.response.content.decode("utf-8"))
       raise e
   
   print(joke)

Ensure that the input_schema and output_schema are correctly defined and match the expected input format to avoid the UnprocessableEntityError. The with_structured_output method should be used as shown in the example above ^[1].

If you continue to encounter issues, you may need to check the specific configurations or settings in the Hugging Face custom endpoint to ensure compatibility with LangChain's with_structured_output method.

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To resolve the UnprocessableEntityError with the message 'Tool error: No function found in generated text' when using model.with_structured_output(Joke) with Hugging Face custom endpoints, you need to ensure that the model is explicitly instructed to call the Joke function in the prompt. Here is an example of how to do this:

from pydantic import BaseModel, Field

class Joke(BaseModel):
    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")

# Ensure the model is instructed to call the Joke function
structured_llm = model.with_structured_output(Joke)
structured_llm.invoke("Tell me a joke about cats. Make sure to call the Joke function.")

Ensure that the prompt explicitly instructs the model to call the Joke function. This should help the model understand that it needs to use the Joke function to generate the structured output. For more detailed guidelines, you can refer to the LangChain documentation on structured output: LangChain Structured Output Documentation ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[michael-newsrx]

@dosu

Why are you repeating the same broken code?

This code does NOT work.

It results in:

UnprocessableEntityError: Error code: 422 - {'error': 'Tool error: No function found in generated text', 'error_type': 'tool_error'}

[dosubot[bot]]

The error UnprocessableEntityError: Error code: 422 - {'error': 'Tool error: No function found in generated text'} occurs because the model did not generate a function call in its response. This can happen if the model does not recognize the need to call the Joke function based on the provided prompt.

To resolve this, ensure that the prompt explicitly instructs the model to call the Joke function. Here is an example of how to modify the prompt:

from pydantic import BaseModel, Field

class Joke(BaseModel):
    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")

# Ensure the model is instructed to call the Joke function
structured_llm = model.with_structured_output(Joke)
response = structured_llm.invoke("Tell me a joke about cats. Make sure to call the Joke function.")

print(response)

Ensure that the prompt "Tell me a joke about cats. Make sure to call the Joke function." is clear and instructs the model to use the Joke function ^[1].

For more detailed guidelines on how to do information extraction with LLMs, you can refer to the LangChain documentation on structured output and extraction use cases:

• Structured Output
• Extraction Use Case

This is a tough nut to crack, could really use your help on this, @eyurtsev.

To continue talking to Dosu, mention @dosu.

[michael-newsrx]

@dosu

hf_request_with_error.json

hf_response_with_error.json

[dosubot[bot]]

The error you're encountering with the Hugging Face custom endpoint and the with_structured_output method in LangChain is likely due to the current limitations and known issues with this functionality. Specifically, the structured output feature is not yet fully implemented for Hugging Face custom endpoints, as indicated by the test for structured output being marked with @pytest.mark.xfail(reason=("Not implemented")) ^[1].

Given this limitation, you might not be able to achieve the desired structured output directly with the current implementation. Here are a few steps you can take to work around this issue:

1. Check the JSON Response: Review the contents of hf_request_with_error.json and hf_response_with_error.json to understand the exact structure and content of the request and response. This can help identify any discrepancies or missing fields that might be causing the error.

2. Manual Parsing: If the structured output feature is not working as expected, you can manually parse the JSON response from the Hugging Face endpoint. Here is an example of how you can do this:

   import json
   from pydantic import BaseModel, Field
   
   class Joke(BaseModel):
       setup: str = Field(description="The setup of the joke")
       punchline: str = Field(description="The punchline to the joke")
   
   # Assuming `response_content` is the JSON response from the Hugging Face endpoint
   response_content = '{"setup": "Why was the cat sitting on the computer?", "punchline": "To keep an eye on the mouse!"}'
   
   # Parse the JSON response manually
   joke_data = json.loads(response_content)
   joke = Joke(**joke_data)
   
   print(joke)

3. Update LangChain: Keep an eye on updates to the LangChain repository. The structured output feature for Hugging Face custom endpoints might be implemented in future releases.

4. Alternative Approaches: Consider using other methods or endpoints that are fully supported by LangChain for structured output.

By manually parsing the JSON response or exploring alternative approaches, you can work around the current limitations and achieve the desired structured output.

To continue talking to Dosu, mention @dosu.

[shawon-majid]

I am also facing the same issue. Maybe hugging face models cannot use tool calling, or the langchain has no support for this