Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update README #39

Merged
merged 2 commits into from
Aug 23, 2024
Merged

update README #39

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf03b33
feat: add header for README
AgainstEntropy Aug 23, 2024
a257ea2
feat: add installation in README
AgainstEntropy Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 89 additions & 78 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,27 @@
# Nexa SDK
<div align="center">

The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.
<h1>Nexa SDK</h1>

<img src="./assets/banner.png" alt="icon"/>

[![MacOS][MacOS-image]][release-url] [![Linux][Linux-image]][release-url] [![Windows][Windows-image]][release-url]

[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI&logo=github)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk)

<!-- ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/nexaai) ![PyPI - Downloads](https://img.shields.io/pypi/dm/nexaai?color=orange) -->

[![Discord](https://dcbadge.limes.pink/api/server/thRu2HaK4D?style=flat&compact=true)](https://discord.gg/thRu2HaK4D)

[On-device Model Hub](https://model-hub.nexa4ai.com/) / [Nexa SDK Documentation](https://docs.nexaai.com/)

[release-url]: https://github.com/NexaAI/nexa-sdk/releases
[Windows-image]: https://img.shields.io/badge/windows-0078D4?logo=windows
[MacOS-image]: https://img.shields.io/badge/-MacOS-black?logo=apple
[Linux-image]: https://img.shields.io/badge/-Linux-333?logo=ubuntu

</div>

Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.

## Features

Expand All @@ -14,7 +35,7 @@ The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** mod
- **Vision-Language Models (VLM)**
- **Text-to-Speech (TTS)**

Detailed API documentation is available [here](docs/index.html).
Detailed API documentation is available [here](https://docs.nexaai.com/).

- **Server:**
- OpenAI-compatible API
Expand All @@ -24,124 +45,114 @@ Detailed API documentation is available [here](docs/index.html).

## Installation

### GPU version(optional)

check if you have GPU acceleration (torch required)
<details>
<summary>CUDA:</summary>

```
import torch
torch.cuda.is_available()
```

if True
### Pre-built Wheels (Recommended)

```
CMAKE_ARGS="-DGGML_CUDA=on -DSD_CUBLAS=ON" pip install nexaai
```
Or you prefer to install our pre-built wheel:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```
Optionally, you can install onnx supported version:
```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```
</details>
<details>
<summary>Apple M Chip:</summary>
Apple icon -> about this mac -> Graphics
We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our [index page](https://nexaai.github.io/nexa-sdk/whl/).

if True:
#### CPU

```
CMAKE_ARGS="-DGGML_METAL=on -DSD_METAL=ON" pip install nexaai
```
Or you prefer to install our pre-built wheel:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```
Optionally, you can install onnx supported version:
```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```
</details>
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
```

### CPU version
#### GPU (Metal)

<details>
<summary>Mac with Intel Chips</summary>
For the GPU version supporting **Metal (macOS)**:

To install the `nexaai` package on a Mac with Intel chips, use the following command:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```

```bash
CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai
```
#### GPU (CUDA)

**Optional:** To install the version with ONNX support, use:
For the GPU version supporting **CUDA (Linux/Windows)**:

```bash
CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai[onnx]
```
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```

</details>
> [!NOTE]
> The CUDA wheels are built with CUDA 12.4, but should be compatible with all CUDA 12.X

<details>
<summary>Mac with M Chips or Other Operating Systems</summary>
### Install from source code distribution

To install the `nexaai` package on a Mac with M chips or other operating systems, use the following command:
If pre-built wheels cannot meet your requirements, you can install Nexa SDK from the source code via cmake.

```bash
pip install nexaai
```
#### CPU

**Optional:** To install the version with ONNX support, use:
```bash
pip install nexaai
```

```bash
pip install nexaai[onnx]
```
> [!IMPORTANT]
> If you are using a Mac with Intel chips, run the following command:
>
> ```bash
> CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai
> ```

#### GPU (Metal)

</details>
If you prefer to install the pre-built wheel for CPU versions:
For the GPU version supporting Metal (macOS):

```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai
```

To include ONNX support:
#### GPU (CUDA)

For the GPU version supporting CUDA (Linux/Windows), run the following command:

```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai
```

> [!TIP]
> You can accelerate the building process via parallel cmake by appending the following to the commands above:
>
> ```bash
> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)
> ```
>
> For example:
>
> ```bash
> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL
> ```

> [!TIP]
> For Windows users, we recommend running the installation command in Git Bash to avoid unexpected behavior.

### Docker Usage

Note: Docker doesn't support GPU acceleration

```bash
docker pull nexa4ai/sdk:latest
```

replace following placeholder with your path and command

```bash
docker run -v <your_model_dir>:/model -it nexa4ai/sdk:latest [nexa_command] [your_model_relative_path]
```

Example:

```bash
docker run -v /home/ubuntu/.cache/nexa/hub/official:/model -it nexa4ai/sdk:latest nexa gen-text /model/Phi-3-mini-128k-instruct/q4_0.gguf
```

will create an interactive session with text generation


## Nexa CLI commands

## Model Commands

### NLP Models

| Model | Type | Format | Command |
| ---------------- | ---- | --------- | ------------------------------------ |
| Model | Type | Format | Command |
| ---------------- | ---- | --------- | -------------------------------- |
| octopus-v2 | NLP | GGUF | `nexa gen-text octopus-v2` |
| octopus-v4 | NLP | GGUF | `nexa gen-text octopus-v4` |
| tinyllama | NLP | GGUF | `nexa gen-text tinyllama` |
Expand All @@ -164,8 +175,8 @@ will create an interactive session with text generation

### Multimodal Models

| Model | Type | Format | Command |
| ---------------- | ---------- | ------ | ------------------------------- |
| Model | Type | Format | Command |
| ---------------- | ---------- | ------ | --------------------------- |
| nanollava | Multimodal | GGUF | `nexa vlm nanollava` |
| llava-phi3 | Multimodal | GGUF | `nexa vlm llava-phi3` |
| llava-llama3 | Multimodal | GGUF | `nexa vlm llava-llama3` |
Expand All @@ -174,8 +185,8 @@ will create an interactive session with text generation

### Computer Vision Models

| Model | Type | Format | Command |
| --------------------- | --------------- | --------- | ------------------------------------ |
| Model | Type | Format | Command |
| --------------------- | --------------- | --------- | -------------------------------- |
| stable-diffusion-v1-4 | Computer Vision | GGUF | `nexa gen-image sd1-4` |
| stable-diffusion-v1-5 | Computer Vision | GGUF/ONNX | `nexa gen-image sd1-5` |
| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` |
| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` |
Loading