From bf03b339086ca84ae86b4f2eba74028573e385b3 Mon Sep 17 00:00:00 2001 From: Ethan Wang Date: Fri, 23 Aug 2024 00:26:04 +0000 Subject: [PATCH 1/2] feat: add header for README --- README.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 72588a51..6737ed49 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,25 @@ -# Nexa SDK +
+ +

Nexa SDK

+ +[![MacOS][MacOS-image]][release-url] [![Linux][Linux-image]][release-url] [![Windows][Windows-image]][release-url] + +[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk) + + + +![Discord](https://img.shields.io/discord/1192186167391682711?logo=discord&color=b19cd9&labelColor=lightgrey) + + +[On-device Model Hub](https://model-hub.nexa4ai.com/) / [Nexa SDK Documentation](https://docs.nexaai.com/) + + +[release-url]: https://github.com/NexaAI/nexa-sdk/releases +[Windows-image]: https://img.shields.io/badge/windows-0078D4?logo=windows +[MacOS-image]: https://img.shields.io/badge/-MacOS-black?logo=apple +[Linux-image]: https://img.shields.io/badge/-Linux-333?logo=ubuntu + +
The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. From a257ea2327fbd2b6336a069799e7f76833d84c8b Mon Sep 17 00:00:00 2001 From: Ethan Wang Date: Fri, 23 Aug 2024 00:46:42 +0000 Subject: [PATCH 2/2] feat: add installation in README --- README.md | 152 +++++++++++++++++++++++++----------------------------- 1 file changed, 71 insertions(+), 81 deletions(-) diff --git a/README.md b/README.md index 6737ed49..0160d481 100644 --- a/README.md +++ b/README.md @@ -2,18 +2,18 @@

Nexa SDK

+ icon + [![MacOS][MacOS-image]][release-url] [![Linux][Linux-image]][release-url] [![Windows][Windows-image]][release-url] -[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk) +[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI&logo=github)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk) -![Discord](https://img.shields.io/discord/1192186167391682711?logo=discord&color=b19cd9&labelColor=lightgrey) - +[![Discord](https://dcbadge.limes.pink/api/server/thRu2HaK4D?style=flat&compact=true)](https://discord.gg/thRu2HaK4D) [On-device Model Hub](https://model-hub.nexa4ai.com/) / [Nexa SDK Documentation](https://docs.nexaai.com/) - [release-url]: https://github.com/NexaAI/nexa-sdk/releases [Windows-image]: https://img.shields.io/badge/windows-0078D4?logo=windows [MacOS-image]: https://img.shields.io/badge/-MacOS-black?logo=apple @@ -21,7 +21,7 @@ -The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. +Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. ## Features @@ -35,7 +35,7 @@ The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** mod - **Vision-Language Models (VLM)** - **Text-to-Speech (TTS)** -Detailed API documentation is available [here](docs/index.html). +Detailed API documentation is available [here](https://docs.nexaai.com/). - **Server:** - OpenAI-compatible API @@ -45,124 +45,114 @@ Detailed API documentation is available [here](docs/index.html). ## Installation -### GPU version(optional) - -check if you have GPU acceleration (torch required) -
- CUDA: - - ``` - import torch - torch.cuda.is_available() - ``` - - if True +### Pre-built Wheels (Recommended) - ``` - CMAKE_ARGS="-DGGML_CUDA=on -DSD_CUBLAS=ON" pip install nexaai - ``` - Or you prefer to install our pre-built wheel: - ```bash - pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple - ``` - Optionally, you can install onnx supported version: - ```bash - pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple - ``` -
-
- Apple M Chip: - Apple icon -> about this mac -> Graphics +We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our [index page](https://nexaai.github.io/nexa-sdk/whl/). - if True: +#### CPU - ``` - CMAKE_ARGS="-DGGML_METAL=on -DSD_METAL=ON" pip install nexaai - ``` - Or you prefer to install our pre-built wheel: - ```bash - pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple - ``` - Optionally, you can install onnx supported version: - ```bash - pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple - ``` -
+```bash +pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple +``` -### CPU version +#### GPU (Metal) -
- Mac with Intel Chips +For the GPU version supporting **Metal (macOS)**: - To install the `nexaai` package on a Mac with Intel chips, use the following command: +```bash +pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple +``` - ```bash - CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai - ``` +#### GPU (CUDA) - **Optional:** To install the version with ONNX support, use: +For the GPU version supporting **CUDA (Linux/Windows)**: - ```bash - CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai[onnx] - ``` +```bash +pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple +``` -
+> [!NOTE] +> The CUDA wheels are built with CUDA 12.4, but should be compatible with all CUDA 12.X -
- Mac with M Chips or Other Operating Systems +### Install from source code distribution - To install the `nexaai` package on a Mac with M chips or other operating systems, use the following command: +If pre-built wheels cannot meet your requirements, you can install Nexa SDK from the source code via cmake. - ```bash - pip install nexaai - ``` +#### CPU - **Optional:** To install the version with ONNX support, use: +```bash +pip install nexaai +``` - ```bash - pip install nexaai[onnx] - ``` +> [!IMPORTANT] +> If you are using a Mac with Intel chips, run the following command: +> +> ```bash +> CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai +> ``` +#### GPU (Metal) -
-If you prefer to install the pre-built wheel for CPU versions: +For the GPU version supporting Metal (macOS): ```bash -pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple +CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai ``` -To include ONNX support: +#### GPU (CUDA) + +For the GPU version supporting CUDA (Linux/Windows), run the following command: ```bash -pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple +CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai ``` +> [!TIP] +> You can accelerate the building process via parallel cmake by appending the following to the commands above: +> +> ```bash +> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) +> ``` +> +> For example: +> +> ```bash +> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL +> ``` + +> [!TIP] +> For Windows users, we recommend running the installation command in Git Bash to avoid unexpected behavior. + ### Docker Usage + Note: Docker doesn't support GPU acceleration + ```bash docker pull nexa4ai/sdk:latest ``` + replace following placeholder with your path and command + ```bash docker run -v :/model -it nexa4ai/sdk:latest [nexa_command] [your_model_relative_path] ``` Example: + ```bash docker run -v /home/ubuntu/.cache/nexa/hub/official:/model -it nexa4ai/sdk:latest nexa gen-text /model/Phi-3-mini-128k-instruct/q4_0.gguf ``` will create an interactive session with text generation - ## Nexa CLI commands ## Model Commands ### NLP Models -| Model | Type | Format | Command | -| ---------------- | ---- | --------- | ------------------------------------ | +| Model | Type | Format | Command | +| ---------------- | ---- | --------- | -------------------------------- | | octopus-v2 | NLP | GGUF | `nexa gen-text octopus-v2` | | octopus-v4 | NLP | GGUF | `nexa gen-text octopus-v4` | | tinyllama | NLP | GGUF | `nexa gen-text tinyllama` | @@ -185,8 +175,8 @@ will create an interactive session with text generation ### Multimodal Models -| Model | Type | Format | Command | -| ---------------- | ---------- | ------ | ------------------------------- | +| Model | Type | Format | Command | +| ---------------- | ---------- | ------ | --------------------------- | | nanollava | Multimodal | GGUF | `nexa vlm nanollava` | | llava-phi3 | Multimodal | GGUF | `nexa vlm llava-phi3` | | llava-llama3 | Multimodal | GGUF | `nexa vlm llava-llama3` | @@ -195,8 +185,8 @@ will create an interactive session with text generation ### Computer Vision Models -| Model | Type | Format | Command | -| --------------------- | --------------- | --------- | ------------------------------------ | +| Model | Type | Format | Command | +| --------------------- | --------------- | --------- | -------------------------------- | | stable-diffusion-v1-4 | Computer Vision | GGUF | `nexa gen-image sd1-4` | | stable-diffusion-v1-5 | Computer Vision | GGUF/ONNX | `nexa gen-image sd1-5` | -| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` | \ No newline at end of file +| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` |