Skip to content

Commit

Permalink
Formatting fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@emooreatx
emooreatx authored Dec 12, 2024
1 parent 183ba39 commit fdca90d
Showing 1 changed file with 64 additions and 58 deletions.
122 changes: 64 additions & 58 deletions website/docs/topics/non-openai-models/cloud-litellm-watsonx.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,108 +25,111 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing WatsonX \n",
"## Installing WatsonX\n",
"\n",
"To set up WatsonX, follow these steps:\n",
"\n",
"1. **Access WatsonX:**\n",
" - Sign up for [WatsonX.ai](https://www.ibm.com/watsonx).\n",
" - Create an API_KEY and PROJECT_ID.\n",
"\n",
" - Sign up for [WatsonX.ai](https://www.ibm.com/watsonx).\n",
" - Create an API_KEY and PROJECT_ID.\n",
"<br>\n",
"<br>\n",
"2. **Validate WatsonX API Access:**\n",
" - Verify access using the following commands:\n"
" - Verify access using the following commands:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tip: Verify access to watsonX APIs before installing LiteLLM\n",
"Get Session Token:\n",
"\n",
"curl -L \"https://iam.cloud.ibm.com/identity/token?=null\" -H \"Content-Type: application/x-www-form-urlencoded\" -d \"grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey\" -d \"apikey=<API_KEY>\"\n",
"\n",
"Get list of LLMs:\n",
" \n",
"curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/foundation_model_specs?version=2024-09-16&project_id=1eeb4112-5f6e-4a81-9b61-8eac7f9653b4&filters=function_text_generation%2C%21lifecycle_withdrawn%3Aand&limit=200\" -H \"Authorization: Bearer <SESSION TOKEN>\"\n",
"\n",
"\n",
"Ask the LLM a question:\n",
" \n",
"curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-02\" -H \"Content-Type: application/json\" -H \"Accept: application/json\" -H \"Authorization: Bearer <SESSION TOKEN>\" \\\n",
"Tip: Verify access to watsonX APIs before installing LiteLLM.\n",
"<br><br>\n",
"Get Session Token: <br>\n",
"```bash\n",
"curl -L \"https://iam.cloud.ibm.com/identity/token?=null\" \n",
"-H \"Content-Type: application/x-www-form-urlencoded\" \n",
"-d \"grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey\" \n",
"-d \"apikey=<API_KEY>\"\n",
"```\n",
"\n",
"Get list of LLMs: <br>\n",
"```bash\n",
"curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/foundation_model_specs?version=2024-09-16&project_id=1eeb4112-5f6e-4a81-9b61-8eac7f9653b4&filters=function_text_generation%2C%21lifecycle_withdrawn%3Aand&limit=200\" \n",
"-H \"Authorization: Bearer <SESSION TOKEN>\"\n",
"```\n",
"\n",
"Ask the LLM a question: <br>\n",
"```bash\n",
"curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-02\" \n",
"-H \"Content-Type: application/json\" \n",
"-H \"Accept: application/json\" \n",
"-H \"Authorization: Bearer <SESSION TOKEN>\" \\\n",
"-d \"{\n",
" \\\"model_id\\\": \\\"google/flan-t5-xxl\\\",\n",
" \\\"input\\\": \\\"What is the capital of Arkansas?:\\\",\n",
" \\\"parameters\\\": {\n",
" \\\"max_new_tokens\\\": 100,\n",
" \\\"time_limit\\\": 1000\n",
" },\n",
" \\\"project_id\\\": \\\"<PROJECT_ID>\"\n",
"}\"\n",
" \\\"project_id\\\": \\\"<PROJECT_ID>\"}\"\n",
"```\n",
"\n",
"\n",
"2.\tWith access to watsonX API’s validated you can install the python library\n",
" \n",
" \n",
"From <https://ibm.github.io/watsonx-ai-python-sdk/install.html> \n"
"With access to watsonX API’s validated you can install the python library from <https://ibm.github.io/watsonx-ai-python-sdk/install.html> \n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing LiteLLM \n",
"\n",
"To install LiteLLM, follow these steps:\n",
"1. **Download LiteLLM Docker Image:**\n",
"\n",
" ```bash\n",
" docker pull ghcr.io/berriai/litellm:main-latest\n",
" ```\n",
"\n",
" **(ALTERNATIVE). Install LiteLLM Python Library:**\n",
" OR\n",
"\n",
"\n",
" **Install LiteLLM Python Library:**\n",
"\n",
" ```bash\n",
" pip install 'litellm[proxy]'\n",
" ```\n",
"\n",
"\n",
"\n",
"---\n",
"\n",
"2. **Create a LiteLLM Configuration File:**\n",
"\n",
" - Save as `litellm_config.yaml` in a local directory.\n",
" - Example content for WatsonX:\n",
"\n",
" ```yaml\n",
" model_list:\n",
" - model_name: llama-3-8b\n",
" litellm_params:\n",
" # all params accepted by litellm.completion()\n",
" model: watsonx/meta-llama/llama-3-8b-instruct\n",
" api_key: \"os.environ/WATSONX_API_KEY\" \n",
" project_id: \"os.environ/WX_PROJECT_ID\"\n",
" ```bash\n",
" model_list:\n",
" - model_name: llama-3-8b\n",
" litellm_params:\n",
" # all params accepted by litellm.completion()\n",
" model: watsonx/meta-llama/llama-3-8b-instruct\n",
" api_key: \"os.environ/WATSONX_API_KEY\" \n",
" project_id: \"os.environ/WX_PROJECT_ID\"\n",
"\n",
" ```\n",
"'''yaml\n",
" - model_name: \"llama_3_2_90\"\n",
" litellm_params:\n",
" model: watsonx/meta-llama/llama-3-2-90b-vision-instruct\n",
" api_key: os.environ[\"WATSONX_APIKEY\"] = \"\" # IBM cloud API key\n",
" max_new_tokens: 4000\n",
"'''\n",
" ```bash\n",
" - model_name: \"llama_3_2_90\"\n",
" litellm_params:\n",
" model: watsonx/meta-llama/llama-3-2-90b-vision-instruct\n",
" api_key: os.environ[\"WATSONX_APIKEY\"] = \"\" # IBM cloud API key\n",
" max_new_tokens: 4000\n",
" ```\n",
"3. **Start LiteLLM Container:**\n",
"\n",
" ```bash\n",
" docker run -v <Directory>\\litellm_config.yaml:/app/config.yaml -e WATSONX_API_KEY=<API_KEY> -e WATSONX_URL=https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-02 -e WX_PROJECT_ID=<PROJECT_ID> -p 4000:4000 ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --detailed_debug\n",
" ```\n",
"\n",
" ---"
"---"
]
},
{
Expand All @@ -147,27 +150,24 @@
"Once installed, AutoGen agents can leverage WatsonX APIs via LiteLLM.\n",
"\n",
"---\n",
"\n",
"```bash\n",
"phi1 = {\n",
" \"config_list\": [\n",
" {\n",
" \"model\": \"llama-3-8b\",\n",
" \"base_url\": \"http://localhost:4000\",\n",
" \"base_url\": \"http://localhost:4000\", #use http://0.0.0.0:4000 for Macs\n",
" \"api_key\":\"watsonx\",\n",
" \"price\" : [0,0]\n",
" },\n",
" ],\n",
" \"cache_seed\": None, # Disable caching.\n",
"}\n",
"\n",
"\n",
"\n",
"\n",
"phi2 = {\n",
" \"config_list\": [\n",
" {\n",
" \"model\": \"llama-3-8b\",\n",
" \"base_url\": \"http://localhost:4000\",\n",
" \"base_url\": \"http://localhost:4000\", #use http://0.0.0.0:4000 for Macs\n",
" \"api_key\":\"watsonx\",\n",
" \"price\" : [0,0]\n",
" },\n",
Expand All @@ -190,8 +190,14 @@
")\n",
"\n",
"#autogen\n",
"chat_result = jack.initiate_chat(emma, message=\"Emma, tell me a joke.\", max_turns=2)\n"
"chat_result = jack.initiate_chat(emma, message=\"Emma, tell me a joke.\", max_turns=2)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
Expand Down

0 comments on commit fdca90d

Please sign in to comment.