[Bugfix]: allow extra fields in requests to openai compatible server

[gcalmettes]

There are been several reports of BadRequestError: Error code: 400 due to the vLLM OpenAI compatible server not accepting extra arguments in the requests' payloads, while the same requests would work with the official OpenAI API.

e.g.:

• [Usage]: disable pydantic request validation #10093
• [Bug]: OpenAI API emulation incompatible with shell_gpt OpenAI API calls #5797
• [Bug]: [Bug]: vllm 启动，openai的swarm 函数调用不正常  #10015
• [Bug]: Vllm api server does not receive supported parameter truncate_prompt_tokens #6890

Similarly, code completion plugins like CodeCompanion are currently not working with models served with the vLLM OpenAI compatible server, failing with the same BadRequestError: Error code: 400 error, while the plugin is working fine with the official OpenAI API:

Error: {"object":"error","message":"[{'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, ' typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': \"Input should be 'user'\", 'input': 'system', 'ctx': {'expected': \"'user'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': \"Input should be 'assistant'\", 'input': 'system', 'ctx': {'expected': \"'assistant'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': \"Input should be 'tool'\", 'input': 'system', 'ctx': {'expected': \"'tool'\"}}, {'type': 'missing', 'loc': ('body', 'messages', 0, 'typed-dict', 'tool_call_id'), 'msg': 'Field required', 'input': {'content': 'You are an AI programming assistant named \"CodeCompanion\".\\nYou are currently plugged in to the Neovim text editor on a user\\'s machine.\\n\\nYour core tasks include:\\n- Answering general programming questions.\\n- Explaining how the code in a Neovim buffer works.\\n- Reviewing the selected code in a Neovim buffer.\\n- Generating unit tests for the selected code.\\n- Proposing fixes for problems in the selected code.\\n- Scaffolding code for a new workspace.\\n- Finding relevant code to the user\\'s query.\\n- Proposing fixes for test failures.\\n- Answering questions about Neovim.\\n- Running tools.\\n\\nYou must:\\n- Follow the user\\'s requirements carefully and to the letter.\\n- Keep your answers short and impersonal, especially if the user responds with context outside of your tasks.\\n- Minimize other prose.\\n- Use Markdown formatting in your answers.\\n- Include the programming language name at the start of the Markdown code blocks.\\n- Avoid line numbers in code blocks.\\n- Avoid wrapping the whole response in triple backticks.\\n- Only return code that\\'s relevant to the task at hand. You may not need to return all of the code that the user has shared.\\n- Use actual line breaks instead of \\'\\\\n\\' in your response to begin new lines.\\n- Use \\'\\\\n\\' only when you want a literal backslash followed by a character \\'n\\'.\\n- All non-code responses must use English.\\n\\nWhen given a task:\\n1. Think step-by-step and describe your plan for what to build in pseudocode, written out in great detail, unless asked not to do so.\\n2. Output the code in a single code block, being careful to only return relevant code.\\n3. You should always generate short suggestions for the next user turns that are relevant to the conversation.\\n4. You can only give one reply for each conversation turn.', 'role': 'system', 'id': 1947269257, 'opts': {'visible': False}}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'missing', 'loc': ('body', 'messages', 0, 'typed-dict', 'name'), 'msg': 'Field required', 'input': {'content': 'You are an AI programming assistant named \"CodeCompanion\".\\nYou are currently plugged in to the Neovim text editor on a user\\'s machine.\\n\\nYour core tasks include:\\n- Answering general programming questions.\\n- Explaining how the code in a Neovim buffer works.\\n- Reviewing the selected code in a Neovim buffer.\\n- Generating unit tests for the selected code.\\n- Proposing fixes for problems in the selected code.\\n- Scaffolding code for a new workspace.\\n- Finding relevant code to the user\\'s query.\\n- Proposing fixes for test failures.\\n- Answering questions about Neovim.\\n- Running tools.\\n\\nYou must:\\n- Follow the user\\'s requirements carefully and to the letter.\\n- Keep your answers short and impersonal, especially if the user responds with context outside of your tasks.\\n- Minimize other prose.\\n- Use Markdown formatting in your answers.\\n- Include the programming language name at the start of the Markdown code blocks.\\n- Avoid line numbers in code blocks.\\n- Avoid wrapping the whole response in triple backticks.\\n- Only return code that\\'s relevant to the task at hand. You may not need to return all of the code that the user has shared.\\n- Use actual line breaks instead of \\'\\\\n\\' in your response to begin new lines.\\n- Use \\'\\\\n\\' only when you want a literal backslash followed by a character \\'n\\'.\\n- All non-code responses must use English.\\n\\nWhen given a task:\\n1. Think step-by-step and describe your plan for what to build in pseudocode, written out in great detail, unless asked not to do so.\\n2. Output the code in a single code block, being careful to only return relevant code.\\n3. You should always generate short suggestions for the next user turns that are relevant to the conversation.\\n4. You can only give one reply for each conversation turn.', 'role': 'system', 'id': 1947269257, 'opts': {'visible': False}}}, {'type': 'literal_error', 'loc': ('body', 'messages', 0, 'typed-dict', 'role'), 'msg': \"Input should be 'function'\", 'input': 'system', 'ctx': {'expected': \"'function'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': 1947269257}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 0, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': False}}, {'type': 'literal_error', 'loc': ('body', 'messages', 1, 'typed-dict', 'role'), 'msg': \"Input should be 'system'\", 'input': 'user', 'ctx': {'expected': \"'system'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'literal_error', 'loc': ('body', 'messages', 1, 'typed-dict', 'role'), 'msg': \"Input should be 'assistant'\", 'input': 'user', 'ctx': {'expected': \"'assistant'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'literal_error', 'loc': ('body', 'messages', 1, 'typed-dict', 'role'), 'msg': \"Input should be 'tool'\", 'input': 'user', 'ctx': {'expected': \"'tool'\"}}, {'type': 'missing', 'loc': ('body', 'messages', 1, 'typed-dict', 'tool_call_id'), 'msg': 'Field required', 'input': {'content': 'what is a function in python?', 'role': 'user', 'id': -2096855750, 'opts': {'visible': True}}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'missing', 'loc': ('body', 'messages', 1, 'typed-dict', 'name'), 'msg': 'Field required', 'input': {'content': 'what is a function in python?', 'role': 'user', 'id': -2096855750, 'opts': {'visible': True}}}, {'type': 'literal_error', 'loc': ('body', 'messages', 1, 'typed-dict', 'role'), 'msg': \"Input should be 'function'\", 'input': 'user', 'ctx': {'expected': \"'function'\"}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'id'), 'msg': 'Extra inputs are not permitted', 'input': -2096855750}, {'type': 'extra_forbidden', 'loc': ('body', 'messages', 1, 'typed-dict', 'opts'), 'msg': 'Extra inputs are not permitted', 'input': {'visible': True}}, {'type': 'extra_forbidden', 'loc': ('body', 'options'), 'msg': 'Extra inputs are not permitted', 'input': {'top_k': 40, 'mirostat': 0, 'num_predict': -1, 'num_ctx': 2048, 'tfs_z': 1, 'temperature': 0.8, 'seed': 0, 'repeat_penalty': 1.1, 'repeat_last_n': 64, 'top_p': 0.9, 'mirostat_tau': 5, 'mirostat_eta': 0.1}}]","type":"BadRequestError","param":null,"code":400}

The reason of the error is that currently the vLLM OpenAI-compatible server forbids extra field in its base pydantic model (introduced by #4355) while OpenAI native pydantic definitions all inherit from a BaseModel that explicitely allow extra fields in their model_config

This PR updates the vLLM OpenAIBaseModel to allow extra fields, like the official OpenAI API.

cc: @DarkLight1337 @simon-mo

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mgoin]

It would be nice if the client could know what arguments were ignored. I believe this was set up to avoid silent ignores

[gcalmettes]

@mgoin yes the original idea was indeed to have a way to make the user aware of e.g., a typo, in one of their field, which would explain why the response they get is not what they wanted. However in practice it seems to break compatibility with lots of systems that have been developed against the official OlenAI API.

However, let me investigate, from the linked issue in the comment of this pydantic issue (which describes our current need, telling the user they added something while still allowing the extra fields) it seems that there might be a way to do it.

[gcalmettes]

@mgoin, based on pydantic #3455 I have added a __log_extra_fields__ classmethod to the base model, so the extra fields present in the requests and not in the model are now warn-logged (server side).

I agree that with this solution, the information about ignored fields won't be directly present in the request response, but at least if the user performing the request has access to the vllm server logs, debugging can still be done efficiently. Not sure what more could be done while still matching the OpenAI API behavior.

WARNING 11-19 13:58:36 protocol.py:71] The following fields were present in the request but were ignored: {'some_extra'}

[DarkLight1337]

I agree that logging a warning in this case is more reasonable. Thanks for improving the user experience!

[DarkLight1337]

Please fix the test failure though.

[mgoin]

Thanks for adding the server-side log. After thinking further on it, we generally strive to follow the behavior of OpenAI so it makes sense to let the requests through

[gcalmettes]

@DarkLight1337 happy to report that all tests passed 😊.

@mgoin what could be possible though is for example to add a --forbid-extra-fields to the cli args for the vllm openai server that would set the extra fields policy to forbid when the flag is present. So people could also have a strict no-extra-fields behavior that would raise an error if they decide so.

[mgoin]

I think that is a good idea, and certainly would be useful when testing the server in CI!

[gcalmettes]

Ok, I’ll prepare something and I’ll ping you on it to see if that would work for all the use cases you have in mind.