Customize parameters

Warning⚠️

Customizing parameters are not allowed anymore, please ignore this page

Properties

You can see it in hugchat_api.core.config.
please install dataclasses_json using pip which converts a dataclass to json

ModelConfig

Manages model parameters

from dataclasses import dataclass
from dataclasses_json import dataclass_json, DataClassJsonMixin

@dataclass_json
@dataclass
class ModelConfig(DataClassJsonMixin):
    temperature: float | None = None
    top_p: float | None = None
    repetition_penalty: float | None = None
    top_k: int | None = None
    truncate: int | None = None
    watermark: bool | None = None
    max_new_tokens: int | None = None
    return_full_text: bool | None = None
    stop: List[str] | None = None

defaults:

temperature=0.9,
top_p=0.95,
repetition_penalty=1.2,
top_k=50,
truncate=1024,
watermark=False,
max_new_tokens=1024,
return_full_text=False,
stop=["</s>"],

RequestOptions

Specifies conversation

@dataclass_json
@dataclass
class RequestOptions(DataClassJsonMixin):
    id: str | None = None
    response_id: str | None = None
    is_retry: bool | None = None
    use_cache: bool | None = None
    web_search_id: str | None = None

defaults:

is_retry=False,
use_cache=False,

RequestData

post request data

@dataclass_json
@dataclass
class RequestData(DataClassJsonMixin):
    inputs: str | None = None
    parameters: ModelConfig | None = None
    options: RequestOptions | None = None
    stream: bool | None = None

defaults:

stream=True

Not recommended to change stream since the response is different and the parse method may fail to function.

Usage

Just put in the config you want to change, the rest default configs will be filled up automatically.

custom_data: RequestData = RequestData(
    parameters = ModelConfig(
        max_new_tokens = 2048
    )
)
message: Message = bot.chat(custom_data=custom_data)