Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(model): Support siliconflow models #2157

Merged
merged 1 commit into from
Nov 27, 2024

Conversation

fangyinc
Copy link
Collaborator

Description

Closes #2156

How Has This Been Tested?

  1. Using siliconflow proxyllm in DB-GPT webserver
# .env
LLM_MODEL=silicon_flow_proxyllm
SILICON_FLOW_MODEL_VERSION=Qwen/Qwen2.5-Coder-32B-Instruct
SILICON_FLOW_API_BASE=https://api.siliconflow.cn/v1
SILICON_FLOW_API_KEY={your-siliconflow-api-key}
  1. Using SiliconFlowLLMClient
    Create python file test_proxyllm.py
import asyncio

from dbgpt.core import ModelRequest
from dbgpt.model.proxy import SiliconFlowLLMClient

client = SiliconFlowLLMClient(model_alias="Qwen/Qwen2.5-Coder-32B-Instruct")
print(
    asyncio.run(
        client.generate(
            ModelRequest._build("Qwen/Qwen2.5-Coder-32B-Instruct", "Hi!")
        )
    )
)

Run test_proxyllm.py

SILICON_FLOW_API_KEY={your-siliconflow-api-key} python test_proxyllm.py

The output like this:

ModelOutput(text='Hello! How can I assist you today?', error_code=0, incremental=False, model_context=None, finish_reason=None, usage={'completion_tokens': 9, 'prompt_tokens': 31, 'total_tokens': 40}, metrics=None)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Snapshots:

Include snapshots for easier review.

Checklist:

  • My code follows the style guidelines of this project
  • I have already rebased the commits and make the commit message conform to the project standard.
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Any dependent changes have been merged and published in downstream modules

@github-actions github-actions bot added enhancement New feature or request model Module: model labels Nov 26, 2024
@fangyinc
Copy link
Collaborator Author

Use siliconflow embedding model by DB-GPT SDK:

import os
from dbgpt.rag.embedding import OpenAPIEmbeddings

openai_embeddings = OpenAPIEmbeddings(
    api_url="https://api.siliconflow.cn/v1/embeddings",
    api_key=os.getenv("SILICON_FLOW_API_KEY"),
    model_name="BAAI/bge-large-zh-v1.5",
)

texts = ["Hello, world!", "How are you?"]
openai_embeddings.embed_documents(texts)

Copy link
Collaborator

@csunny csunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

It's cool, and using the Jupyter Notebook makes it easy for code testing.

r+

Copy link
Collaborator

@Aries-ckt Aries-ckt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r+

@Aries-ckt Aries-ckt merged commit e5ec471 into eosphoros-ai:main Nov 27, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request model Module: model
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][model] Support siliconflow models
3 participants