Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add siliconflow text2img tool #7612

Merged
merged 2 commits into from
Aug 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
19 changes: 19 additions & 0 deletions api/core/tools/provider/builtin/siliconflow/siliconflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import requests

from core.tools.errors import ToolProviderCredentialValidationError
from core.tools.provider.builtin_tool_provider import BuiltinToolProviderController


class SiliconflowProvider(BuiltinToolProviderController):
def _validate_credentials(self, credentials: dict) -> None:
url = "https://api.siliconflow.cn/v1/models"
headers = {
"accept": "application/json",
"authorization": f"Bearer {credentials.get('siliconFlow_api_key')}",
}

response = requests.get(url, headers=headers)
if response.status_code != 200:
raise ToolProviderCredentialValidationError(
"SiliconFlow API key is invalid"
)
21 changes: 21 additions & 0 deletions api/core/tools/provider/builtin/siliconflow/siliconflow.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
identity:
author: hjlarry
name: siliconflow
label:
en_US: SiliconFlow
zh_CN: 硅基流动
description:
en_US: The image generation API provided by SiliconFlow includes Flux and Stable Diffusion models.
zh_CN: 硅基流动提供的图片生成 API,包含 Flux 和 Stable Diffusion 模型。
icon: icon.svg
tags:
- image
credentials_for_provider:
siliconFlow_api_key:
type: secret-input
required: true
label:
en_US: SiliconFlow API Key
placeholder:
en_US: Please input your SiliconFlow API key
url: https://cloud.siliconflow.cn/account/ak
44 changes: 44 additions & 0 deletions api/core/tools/provider/builtin/siliconflow/tools/flux.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
from typing import Any, Union

import requests

from core.tools.entities.tool_entities import ToolInvokeMessage
from core.tools.tool.builtin_tool import BuiltinTool

FLUX_URL = (
"https://api.siliconflow.cn/v1/black-forest-labs/FLUX.1-schnell/text-to-image"
)


class FluxTool(BuiltinTool):

def _invoke(
self, user_id: str, tool_parameters: dict[str, Any]
) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]:

headers = {
"accept": "application/json",
"content-type": "application/json",
"authorization": f"Bearer {self.runtime.credentials['siliconFlow_api_key']}",
}

payload = {
"prompt": tool_parameters.get("prompt"),
"image_size": tool_parameters.get("image_size", "1024x1024"),
"seed": tool_parameters.get("seed"),
"num_inference_steps": tool_parameters.get("num_inference_steps", 20),
}

response = requests.post(FLUX_URL, json=payload, headers=headers)
if response.status_code != 200:
return self.create_text_message(f"Got Error Response:{response.text}")

res = response.json()
result = [self.create_json_message(res)]
for image in res.get("images", []):
result.append(
self.create_image_message(
image=image.get("url"), save_as=self.VARIABLE_KEY.IMAGE.value
)
)
return result
73 changes: 73 additions & 0 deletions api/core/tools/provider/builtin/siliconflow/tools/flux.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
identity:
name: flux
author: hjlarry
label:
en_US: Flux
icon: icon.svg
description:
human:
en_US: Generate image via SiliconFlow's flux schnell.
llm: This tool is used to generate image from prompt via SiliconFlow's flux schnell model.
parameters:
- name: prompt
type: string
required: true
label:
en_US: prompt
zh_Hans: 提示词
human_description:
en_US: The text prompt used to generate the image.
zh_Hans: 用于生成图片的文字提示词
llm_description: this prompt text will be used to generate image.
form: llm
- name: image_size
type: select
required: true
options:
- value: 1024x1024
label:
en_US: 1024x1024
- value: 768x1024
label:
en_US: 768x1024
- value: 576x1024
label:
en_US: 576x1024
- value: 512x1024
label:
en_US: 512x1024
- value: 1024x576
label:
en_US: 1024x576
- value: 768x512
label:
en_US: 768x512
default: 1024x1024
label:
en_US: Choose Image Size
zh_Hans: 选择生成的图片大小
form: form
- name: num_inference_steps
type: number
required: true
default: 20
min: 1
max: 100
label:
en_US: Num Inference Steps
zh_Hans: 生成图片的步数
form: form
human_description:
en_US: The number of inference steps to perform. More steps produce higher quality but take longer.
zh_Hans: 执行的推理步骤数量。更多的步骤可以产生更高质量的结果,但需要更长的时间。
- name: seed
type: number
min: 0
max: 9999999999
label:
en_US: Seed
zh_Hans: 种子
human_description:
en_US: The same seed and prompt can produce similar images.
zh_Hans: 相同的种子和提示可以产生相似的图像。
form: form
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
from typing import Any, Union

import requests

from core.tools.entities.tool_entities import ToolInvokeMessage
from core.tools.tool.builtin_tool import BuiltinTool

SDURL = {
"sd_3": "https://api.siliconflow.cn/v1/stabilityai/stable-diffusion-3-medium/text-to-image",
"sd_xl": "https://api.siliconflow.cn/v1/stabilityai/stable-diffusion-xl-base-1.0/text-to-image",
}


class StableDiffusionTool(BuiltinTool):

def _invoke(
self, user_id: str, tool_parameters: dict[str, Any]
) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]:

headers = {
"accept": "application/json",
"content-type": "application/json",
"authorization": f"Bearer {self.runtime.credentials['siliconFlow_api_key']}",
}

model = tool_parameters.get("model", "sd_3")
url = SDURL.get(model)

payload = {
"prompt": tool_parameters.get("prompt"),
"negative_prompt": tool_parameters.get("negative_prompt", ""),
"image_size": tool_parameters.get("image_size", "1024x1024"),
"batch_size": tool_parameters.get("batch_size", 1),
"seed": tool_parameters.get("seed"),
"guidance_scale": tool_parameters.get("guidance_scale", 7.5),
"num_inference_steps": tool_parameters.get("num_inference_steps", 20),
}

response = requests.post(url, json=payload, headers=headers)
if response.status_code != 200:
return self.create_text_message(f"Got Error Response:{response.text}")

res = response.json()
result = [self.create_json_message(res)]
for image in res.get("images", []):
result.append(
self.create_image_message(
image=image.get("url"), save_as=self.VARIABLE_KEY.IMAGE.value
)
)
return result
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
identity:
name: stable_diffusion
author: hjlarry
label:
en_US: Stable Diffusion
icon: icon.svg
description:
human:
en_US: Generate image via SiliconFlow's stable diffusion model.
llm: This tool is used to generate image from prompt via SiliconFlow's stable diffusion model.
parameters:
- name: prompt
type: string
required: true
label:
en_US: prompt
zh_Hans: 提示词
human_description:
en_US: The text prompt used to generate the image.
zh_Hans: 用于生成图片的文字提示词
llm_description: this prompt text will be used to generate image.
form: llm
- name: negative_prompt
type: string
label:
en_US: negative prompt
zh_Hans: 负面提示词
human_description:
en_US: Describe what you don't want included in the image.
zh_Hans: 描述您不希望包含在图片中的内容。
llm_description: Describe what you don't want included in the image.
form: llm
- name: model
type: select
required: true
options:
- value: sd_3
label:
en_US: Stable Diffusion 3
- value: sd_xl
label:
en_US: Stable Diffusion XL
default: sd_3
label:
en_US: Choose Image Model
zh_Hans: 选择生成图片的模型
form: form
- name: image_size
type: select
required: true
options:
- value: 1024x1024
label:
en_US: 1024x1024
- value: 1024x2048
label:
en_US: 1024x2048
- value: 1152x2048
label:
en_US: 1152x2048
- value: 1536x1024
label:
en_US: 1536x1024
- value: 1536x2048
label:
en_US: 1536x2048
- value: 2048x1152
label:
en_US: 2048x1152
default: 1024x1024
label:
en_US: Choose Image Size
zh_Hans: 选择生成图片的大小
form: form
- name: batch_size
type: number
required: true
default: 1
min: 1
max: 4
label:
en_US: Number Images
zh_Hans: 生成图片的数量
form: form
- name: guidance_scale
type: number
required: true
default: 7
min: 0
max: 100
label:
en_US: Guidance Scale
zh_Hans: 与提示词紧密性
human_description:
en_US: Classifier Free Guidance. How close you want the model to stick to your prompt when looking for a related image to show you.
zh_Hans: 无分类器引导。您希望模型在寻找相关图片向您展示时,与您的提示保持多紧密的关联度。
form: form
- name: num_inference_steps
type: number
required: true
default: 20
min: 1
max: 100
label:
en_US: Num Inference Steps
zh_Hans: 生成图片的步数
human_description:
en_US: The number of inference steps to perform. More steps produce higher quality but take longer.
zh_Hans: 执行的推理步骤数量。更多的步骤可以产生更高质量的结果,但需要更长的时间。
form: form
- name: seed
type: number
min: 0
max: 9999999999
label:
en_US: Seed
zh_Hans: 种子
human_description:
en_US: The same seed and prompt can produce similar images.
zh_Hans: 相同的种子和提示可以产生相似的图像。
form: form