Does vllm support the Mac/Metal/MPS?

[Phil-U-U]

I ran into the error when pip install vllm in Mac:
RuntimeError: Cannot find CUDA_HOME. CUDA must be available to build the package.

[WoosukKwon]

Hi @Phil-U-U, vLLM does not support MPS backend at the moment as its main target scenario is to be deployed as a high-throughput server running on powerful accelerators like NVIDIA A100/H100.

[ibehnam]

Hi @Phil-U-U, vLLM does not support MPS backend at the moment as its main target scenario is to be deployed as a high-throughput server running on powerful accelerators like NVIDIA A100/H100.

That's understandable. Although, with Mac Studio, many people and companies are starting to use the Mac as LLM servers. I get very good t/s on Mac Studio M2 Ultra using llama.cpp, but something like vLLM on the Mac would be a game changer. I hope the dev team considers this.

[willtejeda]

I'm with @ibehnam , Mac studio would be great for vllm

[oushu1zhangxiangxuan1]

+1

[joeywen]

+1

[vincent-pli]

+1

[jtoy]

+1

[csmac3144]

+1

[N8python]

+1

[evolu8]

Right, many developers are on Macs while testing out architectures. Yes we'll commonly deploy to linux, but if we can't quickly test and experiment locally on our daily use machines... we'll mostly like look elsewhere for an inference engine. Would be great if it could be supported, even if performance suffers on 'Apple Metal'.

[anisingh1]

+1

[antahiap]

Does this issue still exist?

[ibehnam]

@evolu8 Other solutions such as llama.cpp, LLM-MLC, Apple MLX, etc.?

[sislam-provenir]

Does anyone know about any alternatives to VLLM until this is supported?

[bluenevus]

same here please +1

[shushenghong]

+1
pip install vllm failed on my mac m1 pro

[jtoy]

+1On Feb 20, 2024, at 3:13 AM, Ather Shu ***@***.***> wrote: +1 pip install vllm failed on my mac m1 pro image.png (view on web) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[bluenevus]

+1

[WH2zzZ]

+1

[emitrokhin]

+1

[csawtelle]

+1 I use a macbook and a mac studio. With the M3 architecture's unified memory these are very capable and cost effective systems for development and testing.

[bluenevus]

I'm willing to sponsor that effort because what we pay will be in savings in simply hardware on a mac.

[acodercat]

+1

[chen-bowen]

+10000

[NicolaDonelli]

+1 It would be very useful

[youkaichao]

it is not possible for vllm to support mac when triton https://github.com/openai/triton does not support mac.

[balasaajay]

+1 mac support would be very useful for experimentation

[AllanOricil]

I was about to try it with Snowflake arctic AI on my mac M2 Max hosted in AWS and I couldn't :/

[KE7]

More reason to support Mac. Best in class $/token: https://x.com/awnihannun/status/1786069640948719956

[AllanOricil]

Guys, please add a "like" in the main comment.

[kauabh]

okay

[manbax]

+1

[sd3ntato]

+1

[atbe]

+1

[cderfvc]

Support for Mac M1, M2 ➕

I wish install vllm in m1 mac pro

how to install ?

[mentisXdev]

Support for Mac M1, M2 ➕

I wish install vllm in m1 mac pro

how to install ?

Not supported :(

[stephanj]

If on Apple silicon, then have a look at Exo Platform (OSS), which supports MLX (using python) and allows inference with LLM sharding using multiple Macs @ https://github.com/exoplatform

[clearsitedesigns]

Yeah, this is common, in my experience of running so many models locally, cuda and GPUs is way worse than a simple M1 pro.

[nmadhire]

+1
this is needed to test out basic models locally before deploying

[sokoloveai]

+1

[webboty]

+1

[dzlabs]

+1

[ExtraTon618]

+1

[willtejeda]

+1

…

On Sat, Oct 5, 2024, 9:19 PM ExtraTon618 ***@***.***> wrote: +1 — Reply to this email directly, view it on GitHub <#1441 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALWGZH4M5ZOPLQLP4RIBZITZ2C24DAVCNFSM6AAAAAA6JX4GAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGI4DONBQHA> . You are receiving this because you commented.Message ID: ***@***.***>

[kevin1193]

+1 would love to have vllm support for mac

[achrafmam2]

+1

[x-0D]

+1

[willtejeda]

+1

…

On Wed, Oct 30, 2024, 1:25 PM Nikita Bragin ***@***.***> wrote: +1 — Reply to this email directly, view it on GitHub <#1441 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALWGZH4U2XBFQROZRWIKWJLZ6E6BZAVCNFSM6AAAAAA6JX4GAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGMYDANRZGI> . You are receiving this because you commented.Message ID: ***@***.***>

[Jhonnyr97]

+1

[lemmaa]

+1

[stormstone]

+10086

[zuozuo]

+10087

[white-spring]

+10088

[valenradovich]

+1111111

[youkaichao]

I think #9228 should solve it now. You can build and run using macos.

That's not for performance though, mainly for developing.

[mjtechguy]

+1

[wangxiyuan]

I think #9228 should solve it now. You can build and run using macos.

That's not for performance though, mainly for developing.

Compile failed on mac m3. There are a few problem I found:

1. hard limit to linux system: https://github.com/vllm-project/vllm/blob/main/setup.py#L36
2. hard limit to use linux command "/proc/cpuinfo" https://github.com/vllm-project/vllm/blob/main/cmake/cpu_extension.cmake#L29
3. some other C code is not work on mac, for example, numa.h is missing on macos.