Add LiteLLM Support - Anthropic, Bedrock, Huggingface, TogetherAI, Replicate, etc.

[krrishdholakia]

Hi @yuhongsun96,

Following up on our conversation here - #283

The only change required is changing ChatOpenAI to ChatLiteLLM.

I wasn't able to get Danswer running locally, so would appreciate some help testing the integration.

[vercel]

@krrishdholakia is attempting to deploy a commit to the Danswer Team on Vercel.

A member of the Team first needs to authorize it.

[Weves]

Hey @krrishdholakia ! Sorry about the delay - finally got around to testing this today.

I'm running into:

openai.error.AuthenticationError: No API key provided. You can set your API key in code using 'openai.api_key = <API-KEY>', or you can set the environment variable OPENAI_API_KEY=<API-KEY>). If your API key is stored in a file, you can point the openai module at it with 'openai.api_key_path = <PATH>'. You can generate API keys in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details.

but if I look at the LLM object, I do see a key present immediately before the call:

Model Class: ChatLiteLLM, Model Config: {'cache': None, 'verbose': False, 'callbacks': None, 'callback_manager': None, 'tags': None, 'metadata': None, 'client': <module 'litellm' from '/Users/chrisweaver/.virtualenvs/danswer/lib/python3.11/site-packages/litellm/__init__.py'>, 'model': 'gpt-3.5-turbo', 'model_name': None, 'openai_api_key': '<OPEN_AI_KEY>', 'azure_api_key': '', 'anthropic_api_key': '', 'replicate_api_key': '', 'cohere_api_key': None, 'openrouter_api_key': '', 'streaming': False, 'api_base': None, 'organization': None, 'custom_llm_provider': None, 'request_timeout': 5.0, 'temperature': 0.0, 'model_kwargs': {'frequency_penalty': 0, 'presence_penalty': 0}, 'top_p': None, 'top_k': None, 'n': 1, 'max_tokens': 1024, 'max_retries': 0}

Any ideas as to what's going wrong here? 🤔

[krrishdholakia]

@Weves is there a code snippet you could share for repro?

[yuhongsun96]

Thanks for this incredible library! Big win for us!