Add watsonx to third party models

website/docs/topics/non-openai-models/cloud-litellm-watsonx.ipynb

@@ -0,0 +1,224 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LiteLLM with WatsonX  \n",
+    "\n",
+    "LiteLLM is an open-source, locally run proxy server providing an OpenAI-compatible API. It supports various LLM providers, including IBM's WatsonX, enabling seamless integration with tools like AG2.\n",
+    "\n",
+    "Running LiteLLM with WatsonX requires the following installations:\n",
+    "\n",
+    "1. **AG2** – A framework for building and orchestrating AI agents.\n",
+    "2. **LiteLLM** – An OpenAI-compatible proxy for bridging non-compliant APIs.\n",
+    "3. **IBM WatsonX** – LLM service requiring specific session token authentication.\n",
+    "\n",
+    "### Prerequisites\n",
+    "\n",
+    "Before setting up, ensure **Docker** is installed. Refer to the [Docker installation guide](https://docs.docker.com/get-docker/). Optionally, consider using **Postman** to easily test API requests.\n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installing WatsonX\n",
+    "\n",
+    "To set up WatsonX, follow these steps:\n",
+    "1. **Access WatsonX:**\n",
+    "    - Sign up for [WatsonX.ai](https://www.ibm.com/watsonx).\n",
+    "    - Create an API_KEY and PROJECT_ID.\n",
+    "\n\n",
+    "\n\n",
+    "2. **Validate WatsonX API Access:**\n",
+    "    - Verify access using the following commands:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Tip: Verify access to watsonX APIs before installing LiteLLM.\n",
+    "\n\n\n",
+    "Get Session Token: \n\n",
+    "```bash\n",
+    "curl -L \"https://iam.cloud.ibm.com/identity/token?=null\" \n",
+    "-H \"Content-Type: application/x-www-form-urlencoded\" \n",
+    "-d \"grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey\" \n",
+    "-d \"apikey=<API_KEY>\"\n",
+    "```\n",
+    "\n",
+    "Get list of LLMs: \n\n",
+    "```bash\n",
+    "curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/foundation_model_specs?version=2024-09-16&project_id=1eeb4112-5f6e-4a81-9b61-8eac7f9653b4&filters=function_text_generation%2C%21lifecycle_withdrawn%3Aand&limit=200\" \n",
+    "-H \"Authorization: Bearer <SESSION TOKEN>\"\n",
+    "```\n",
+    "\n",
+    "Ask the LLM a question: \n\n",
+    "```bash\n",
+    "curl -L \"https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-02\" \n",
+    "-H \"Content-Type: application/json\" \n",
+    "-H \"Accept: application/json\" \n",
+    "-H \"Authorization: Bearer <SESSION TOKEN>\" \\\n",
+    "-d \"{\n",
+    "  \\\"model_id\\\": \\\"google/flan-t5-xxl\\\",\n",
+    "  \\\"input\\\": \\\"What is the capital of Arkansas?:\\\",\n",
+    "  \\\"parameters\\\": {\n",
+    "    \\\"max_new_tokens\\\": 100,\n",
+    "    \\\"time_limit\\\": 1000\n",
+    "  },\n",
+    "  \\\"project_id\\\": \\\"<PROJECT_ID>\"}\"\n",
+    "```\n",
+    "\n",
+    "\n",
+    "With access to watsonX API’s validated you can install the python library from <https://ibm.github.io/watsonx-ai-python-sdk/install.html> \n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installing LiteLLM  \n",
+    "To install LiteLLM, follow these steps:\n",
+    "1. **Download LiteLLM Docker Image:**\n",
+    "\n",
+    "   ```bash\n",
+    "   docker pull ghcr.io/berriai/litellm:main-latest\n",
+    "   ```\n",
+    "\n",
+    "   OR\n",
+    "\n",
+    "\n",
+    "   **Install LiteLLM Python Library:**\n",
+    "\n",
+    "   ```bash\n",
+    "   pip install 'litellm[proxy]'\n",
+    "   ```\n",
+    "\n",
+    "\n",
+    "2. **Create a LiteLLM Configuration File:**\n",
+    "\n",
+    "   - Save as `litellm_config.yaml` in a local directory.\n",
+    "   - Example content for WatsonX:\n",
+    "\n",
+    "     ```bash\n",
+    "      model_list:\n",
+    "         - model_name: llama-3-8b\n",
+    "         litellm_params:\n",
+    "            # all params accepted by litellm.completion()\n",
+    "            model: watsonx/meta-llama/llama-3-8b-instruct\n",
+    "            api_key: \"os.environ/WATSONX_API_KEY\" \n",
+    "            project_id: \"os.environ/WX_PROJECT_ID\"\n",
+    "\n",
+    "     ```\n",
+    "      ```bash\n",
+    "      - model_name: \"llama_3_2_90\"\n",
+    "         litellm_params:\n",
+    "            model: watsonx/meta-llama/llama-3-2-90b-vision-instruct\n",
+    "            api_key: os.environ[\"WATSONX_APIKEY\"] = \"\" # IBM cloud API key\n",
+    "            max_new_tokens: 4000\n",
+    "      ```\n",
+    "3. **Start LiteLLM Container:**\n",
+    "\n",
+    "   ```bash\n",
+    "   docker run -v <Directory>\\litellm_config.yaml:/app/config.yaml -e WATSONX_API_KEY=<API_KEY> -e WATSONX_URL=https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-02 -e WX_PROJECT_ID=<PROJECT_ID> -p 4000:4000 ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --detailed_debug\n",
+    "   ```\n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installing AG2  \n",
+    "\n",
+    "AG2 simplifies orchestration and communication between agents. To install:\n",
+    "\n",
+    "1. Open a terminal with administrator rights.\n",
+    "2. Run the following command:\n",
+    "\n",
+    "   ```bash\n",
+    "   pip install ag2\n",
+    "   ```\n",
+    "\n",
+    "Once installed, AG2 agents can leverage WatsonX APIs via LiteLLM.\n",
+    "\n",
+    "---\n",
+    "```bash\n",
+    "phi1 = {\n",
+    "    \"config_list\": [\n",
+    "        {\n",
+    "            \"model\": \"llama-3-8b\",\n",
+    "            \"base_url\": \"http://localhost:4000\", #use http://0.0.0.0:4000 for Macs\n",
+    "            \"api_key\":\"watsonx\",\n",
+    "            \"price\" : [0,0]\n",
+    "        },\n",
+    "    ],\n",
+    "    \"cache_seed\": None,  # Disable caching.\n",
+    "}\n",
+    "\n",
+    "phi2 = {\n",
+    "    \"config_list\": [\n",
+    "        {\n",
+    "            \"model\": \"llama-3-8b\",\n",
+    "            \"base_url\": \"http://localhost:4000\", #use http://0.0.0.0:4000 for Macs\n",
+    "            \"api_key\":\"watsonx\",\n",
+    "            \"price\" : [0,0]\n",
+    "        },\n",
+    "    ],\n",
+    "    \"cache_seed\": None,  # Disable caching.\n",
+    "}\n",
+    "\n",
+    "from AG2 import ConversableAgent, AssistantAgent\n",
+    "\n",
+    "jack = ConversableAgent(\n",
+    "    \"Jack (Phi-2)\",\n",
+    "    llm_config=phi2,\n",
+    "    system_message=\"Your name is Jack and you are a comedian in a two-person comedy show.\",\n",
+    ")\n",
+    "\n",
+    "emma = ConversableAgent(\n",
+    "    \"Emma (Gemma)\",\n",
+    "    llm_config=phi1, \n",
+    "    system_message=\"Your name is Emma and you are a comedian in two-person comedy show.\",\n",
+    ")\n",
+    "\n",
+    "#AG2\n",
+    "chat_result = jack.initiate_chat(emma, message=\"Emma, tell me a joke.\", max_turns=2)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}