Skip to content

Commit

Permalink
feat: add installation in README
Browse files Browse the repository at this point in the history
  • Loading branch information
@AgainstEntropy
AgainstEntropy committed Aug 23, 2024
1 parent bf03b33 commit a257ea2
Showing 1 changed file with 71 additions and 81 deletions.
152 changes: 71 additions & 81 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,26 @@

<h1>Nexa SDK</h1>

<img src="./assets/banner.png" alt="icon"/>

[![MacOS][MacOS-image]][release-url] [![Linux][Linux-image]][release-url] [![Windows][Windows-image]][release-url]

[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk)
[![GitHub Release](https://img.shields.io/github/v/release/NexaAI/nexa-sdk)](https://github.com/NexaAI/nexa-sdk/releases/latest) [![Build workflow](https://img.shields.io/github/actions/workflow/status/NexaAI/nexa-sdk/ci.yaml?label=CI&logo=github)](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain) ![GitHub License](https://img.shields.io/github/license/NexaAI/nexa-sdk)

<!-- ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/nexaai) ![PyPI - Downloads](https://img.shields.io/pypi/dm/nexaai?color=orange) -->

![Discord](https://img.shields.io/discord/1192186167391682711?logo=discord&color=b19cd9&labelColor=lightgrey)

[![Discord](https://dcbadge.limes.pink/api/server/thRu2HaK4D?style=flat&compact=true)](https://discord.gg/thRu2HaK4D)

[On-device Model Hub](https://model-hub.nexa4ai.com/) / [Nexa SDK Documentation](https://docs.nexaai.com/)


[release-url]: https://github.com/NexaAI/nexa-sdk/releases
[Windows-image]: https://img.shields.io/badge/windows-0078D4?logo=windows
[MacOS-image]: https://img.shields.io/badge/-MacOS-black?logo=apple
[Linux-image]: https://img.shields.io/badge/-Linux-333?logo=ubuntu

</div>

The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.
Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI.

## Features

Expand All @@ -35,7 +35,7 @@ The Nexa SDK is a comprehensive toolkit for supporting **ONNX** and **GGML** mod
- **Vision-Language Models (VLM)**
- **Text-to-Speech (TTS)**

Detailed API documentation is available [here](docs/index.html).
Detailed API documentation is available [here](https://docs.nexaai.com/).

- **Server:**
- OpenAI-compatible API
Expand All @@ -45,124 +45,114 @@ Detailed API documentation is available [here](docs/index.html).

## Installation

### GPU version(optional)

check if you have GPU acceleration (torch required)
<details>
<summary>CUDA:</summary>

```
import torch
torch.cuda.is_available()
```

if True
### Pre-built Wheels (Recommended)

```
CMAKE_ARGS="-DGGML_CUDA=on -DSD_CUBLAS=ON" pip install nexaai
```
Or you prefer to install our pre-built wheel:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```
Optionally, you can install onnx supported version:
```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```
</details>
<details>
<summary>Apple M Chip:</summary>
Apple icon -> about this mac -> Graphics
We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our [index page](https://nexaai.github.io/nexa-sdk/whl/).

if True:
#### CPU

```
CMAKE_ARGS="-DGGML_METAL=on -DSD_METAL=ON" pip install nexaai
```
Or you prefer to install our pre-built wheel:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```
Optionally, you can install onnx supported version:
```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```
</details>
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
```

### CPU version
#### GPU (Metal)

<details>
<summary>Mac with Intel Chips</summary>
For the GPU version supporting **Metal (macOS)**:

To install the `nexaai` package on a Mac with Intel chips, use the following command:
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple
```

```bash
CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai
```
#### GPU (CUDA)

**Optional:** To install the version with ONNX support, use:
For the GPU version supporting **CUDA (Linux/Windows)**:

```bash
CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai[onnx]
```
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple
```

</details>
> [!NOTE]
> The CUDA wheels are built with CUDA 12.4, but should be compatible with all CUDA 12.X
<details>
<summary>Mac with M Chips or Other Operating Systems</summary>
### Install from source code distribution

To install the `nexaai` package on a Mac with M chips or other operating systems, use the following command:
If pre-built wheels cannot meet your requirements, you can install Nexa SDK from the source code via cmake.

```bash
pip install nexaai
```
#### CPU

**Optional:** To install the version with ONNX support, use:
```bash
pip install nexaai
```

```bash
pip install nexaai[onnx]
```
> [!IMPORTANT]
> If you are using a Mac with Intel chips, run the following command:
>
> ```bash
> CMAKE_ARGS="-DCMAKE_CXX_FLAGS=-fopenmp" pip install nexaai
> ```
#### GPU (Metal)
</details>
If you prefer to install the pre-built wheel for CPU versions:
For the GPU version supporting Metal (macOS):
```bash
pip install nexaai --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai
```
To include ONNX support:
#### GPU (CUDA)

For the GPU version supporting CUDA (Linux/Windows), run the following command:

```bash
pip install nexaai[onnx] --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple
CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai
```

> [!TIP]
> You can accelerate the building process via parallel cmake by appending the following to the commands above:
>
> ```bash
> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)
> ```
>
> For example:
>
> ```bash
> CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL
> ```
> [!TIP]
> For Windows users, we recommend running the installation command in Git Bash to avoid unexpected behavior.
### Docker Usage
Note: Docker doesn't support GPU acceleration
```bash
docker pull nexa4ai/sdk:latest
```
replace following placeholder with your path and command
```bash
docker run -v <your_model_dir>:/model -it nexa4ai/sdk:latest [nexa_command] [your_model_relative_path]
```
Example:
```bash
docker run -v /home/ubuntu/.cache/nexa/hub/official:/model -it nexa4ai/sdk:latest nexa gen-text /model/Phi-3-mini-128k-instruct/q4_0.gguf
```
will create an interactive session with text generation

## Nexa CLI commands
## Model Commands
### NLP Models
| Model | Type | Format | Command |
| ---------------- | ---- | --------- | ------------------------------------ |
| Model | Type | Format | Command |
| ---------------- | ---- | --------- | -------------------------------- |
| octopus-v2 | NLP | GGUF | `nexa gen-text octopus-v2` |
| octopus-v4 | NLP | GGUF | `nexa gen-text octopus-v4` |
| tinyllama | NLP | GGUF | `nexa gen-text tinyllama` |
Expand All @@ -185,8 +175,8 @@ will create an interactive session with text generation
### Multimodal Models
| Model | Type | Format | Command |
| ---------------- | ---------- | ------ | ------------------------------- |
| Model | Type | Format | Command |
| ---------------- | ---------- | ------ | --------------------------- |
| nanollava | Multimodal | GGUF | `nexa vlm nanollava` |
| llava-phi3 | Multimodal | GGUF | `nexa vlm llava-phi3` |
| llava-llama3 | Multimodal | GGUF | `nexa vlm llava-llama3` |
Expand All @@ -195,8 +185,8 @@ will create an interactive session with text generation
### Computer Vision Models
| Model | Type | Format | Command |
| --------------------- | --------------- | --------- | ------------------------------------ |
| Model | Type | Format | Command |
| --------------------- | --------------- | --------- | -------------------------------- |
| stable-diffusion-v1-4 | Computer Vision | GGUF | `nexa gen-image sd1-4` |
| stable-diffusion-v1-5 | Computer Vision | GGUF/ONNX | `nexa gen-image sd1-5` |
| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` |
| lcm-dreamshaper | Computer Vision | GGUF/ONNX | `nexa gen-image lcm-dreamshaper` |

0 comments on commit a257ea2

Please sign in to comment.