diff --git a/website/blog/2024-01-24-running-tabby-locally-with-rocm/chat-session.png b/website/blog/2024-01-24-running-tabby-locally-with-rocm/chat-session.png new file mode 100644 index 000000000000..49078502efa0 --- /dev/null +++ b/website/blog/2024-01-24-running-tabby-locally-with-rocm/chat-session.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d12bd45beaacab1c8ec3267472d1d195811e6b0ee507f81ae5da3ca79a95719 +size 60514 diff --git a/website/blog/2024-01-24-running-tabby-locally-with-rocm/index.md b/website/blog/2024-01-24-running-tabby-locally-with-rocm/index.md new file mode 100644 index 000000000000..8e1585cc64bc --- /dev/null +++ b/website/blog/2024-01-24-running-tabby-locally-with-rocm/index.md @@ -0,0 +1,34 @@ +--- +slug: running-tabby-locally-with-rocm.md +title: Running Tabby Locally with AMD ROCm +authors: [boxbeam] +tags: [deployment] +--- + +For those using (compatible) **AMD** graphics cards, you can now run Tabby locally with GPU acceleration using AMD's ROCm toolkit! 🎉 + +ROCm is AMD's equivalent of NVidia's CUDA library, making it possible to run highly parallelized computations on the GPU. Cuda is open source and supports using multiple GPUs at the same time to perform the same computation. + +Currently, Tabby with ROCm is only supported on Linux, and can only be run directly from a compiled binary. In the future, Tabby will be able to run with ROCm on Windows, and we will distribute a Docker container capable of running with ROCm on any platform. + +## Install ROCm + +Before starting, please make sure you are on a supported system and have ROCm installed. The AMD website [details how to install it](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/install-overview.html), find the instructions for your given platform. Shown below is a successful installation of ROCm packages on Arch Linux. + +![ROCm installed on Arch Linux](./rocm-packages.png) + +## Install and run Tabby + +Once you have installed ROCm, you can [download the precompiled binary for Tabby](https://github.com/TabbyML/tabby/releases/download/nightly/tabby_x86_64-manylinux2014-rocm57) with ROCm, or you can [compile it yourself](https://github.com/TabbyML/tabby/blob/main/CONTRIBUTING.md#local-setup). If compiling yourself, make sure to use the flag `--features rocm` to enable it. ROCm is currently supported in Tabby's nightly builds only, but will be stable with 0.8.8. + +Once you have a compiled binary, you can run it with this command: + +``` +./tabby serve --model TabbyML/StarCoder-1B --device rocm +``` + +If the command is used correctly and the environment is configured properly, you should see command output similar to the following: +![Tabby running](./tabby-running.png) +And enjoy GPU-accelerated code completions! This should be considerably faster than with CPU (I saw a ~5x speedup with StarCoder-1B using a Ryzen 7 5800X and an RX 6950XT). + +![Completions demo](./using-completions.png) diff --git a/website/blog/2024-01-24-running-tabby-locally-with-rocm/rocm-packages.png b/website/blog/2024-01-24-running-tabby-locally-with-rocm/rocm-packages.png new file mode 100644 index 000000000000..7f68b718b85e --- /dev/null +++ b/website/blog/2024-01-24-running-tabby-locally-with-rocm/rocm-packages.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e98cdf3dc2006562aeca70348b12b701d36d87209249d4378117f75808c153dc +size 20019 diff --git a/website/blog/2024-01-24-running-tabby-locally-with-rocm/tabby-running.png b/website/blog/2024-01-24-running-tabby-locally-with-rocm/tabby-running.png new file mode 100644 index 000000000000..a1e02ed6275e --- /dev/null +++ b/website/blog/2024-01-24-running-tabby-locally-with-rocm/tabby-running.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c91219f58b6b6b154c971535e1e2341b6598c4a524fc59301f136d24f1952166 +size 59278 diff --git a/website/blog/2024-01-24-running-tabby-locally-with-rocm/using-completions.png b/website/blog/2024-01-24-running-tabby-locally-with-rocm/using-completions.png new file mode 100644 index 000000000000..d62450991d5a --- /dev/null +++ b/website/blog/2024-01-24-running-tabby-locally-with-rocm/using-completions.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bc235b3021d14c2124d3a7331118dcff419dc351254168f8b5b87290c6da1b03 +size 15695 diff --git a/website/blog/authors.yml b/website/blog/authors.yml index a03bbdf8c10e..466132e1116d 100644 --- a/website/blog/authors.yml +++ b/website/blog/authors.yml @@ -17,3 +17,8 @@ wwayne: name: Wayne Wang url: https://github.com/wwayne image_url: https://github.com/wwayne.png + +boxbeam: + name: Julien Marcuse + url: https://github.com/boxbeam + image_url: https://github.com/boxbeam.png