Supporting Cline requests style.

[gahoo]

Feature request / 功能建议

Cline requests are quite different in the content section which might be a list instead of str.

{
	"model": "qwen-2.5-coder-instruct",
	"messages": [{
		"role": "system",
		"content": "You are Cline, a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices......"
	}, {
		"role": "user",
		"content": [{
			"type": "text",
			"text": "<task>......</task>"
		}, {
			"type": "text",
			"text": "<environment_details>.....</environment_details>"
		}, {
			"type": "text",
			"text": "[TASK RESUMPTION] ....."
		}, {
			"type": "text",
			"text": "<environment_details>.......</environment_details>"
		}, {
			"type": "text",
			"text": "[TASK RESUMPTION] ......"
		}, {
			"type": "text",
			"text": "<environment_details>......</environment_details>"
		}]
	}],
	"temperature": 0,
	"stream": true,
	"stream_options": {
		"include_usage": true
	}
}

So the default chat_template might cause the following error:

TypeError: [address=0.0.0.0:33357, pid=544] can only concatenate str (not "list") to str

Motivation / 动机

It would be great that Cline or other similar tools can use Xinference's API.

Similar issues happened in DeepSeek API, and seems fixed. see cline/cline#230 .

Your contribution / 您的贡献

None

[qinxuye]

I have no clue about cline, to be honest, this is my first time hearing of it.

But welcome to contribute on this feature.

[gahoo]

Cline is a VS code extension, previous named Claude Dev.

I think it would be a lot of work since every LLM model has it's own chat_template. Do you think custom chat_template is a solution?

[qinxuye]

Does Cline has its own API router like openai ones /v1, if so, we can tell the message is from Cline, and we can write a function to convert the messages.

[gahoo]

Cline supports API providers like OpenRouter, Anthropic, OpenAI, Google Gemini, AWS Bedrock, Azure, and GCP Vertex. You can also configure any OpenAI compatible API, or use a local model through LM Studio/Ollama. If you're using OpenRouter, the extension fetches their latest model list, allowing you to use the newest models as soon as they're available.

Cline is design to work with Anthropic Claude 3.5 Sonnet, it should be OpenAI API compatible. I think it just a client that construct requests and then send to API. It's hard to tell whether the request is from Cline. The request constructed does not contain any information says it's from Cline.

[qinxuye]

That would a bit more nasty, we have to analyze the content of messages, if it contains the structure like the example, we convert it to flattened normal openai messages.

[gahoo]

Yes, and will bring extra burden to the server on each request. Do you think it's worthy? Are there any other clients that send similar request?

[qinxuye]

Yeah, it costs some, a simple way is just pass the messages to models, if TypeError raised, we then check and covert, IMO, that would be better.