[Tracking] New Model Requests

[CharlieFRuan]

Overview

This is a tracker for all new model requests from the community. The end goal is to have prebuilt models for all these models (either from us, or from the community -- see below).

Help Wanted

We would really appreciate it if you could contribute to the project by compiling any requested model listed below.

You could:

1. Follow the detailed steps listed in this Python notebook (runnable in Colab)
2. Upload your compiled model to your huggingface repo (also covered in the notebook)
3. Post the link to your huggingface repo below so that others may benefit from your great work!

For more information (e.g. what architectures are currently supported) see: https://mlc.ai/mlc-llm/docs/prebuilt_models.html

To see what prebuilt models we currently have: https://huggingface.co/mlc-ai

Thank you!

Models with Supported Architecture

• georgesung/llama2_7b_chat_uncensored

  • [Model Request] [Android] llama2_7b_chat_uncensored by georgesung #586
  • Prebuilt uploaded: mlc-ai/mlc-chat-llama2-7b-chat-uncensored-q4f16_1
  • Prebuilt uploaded: mlc-ai/mlc-chat-llama2-7b-chat-uncensored-q4f32_1

• FlagAlpha/Llama2-Chinese-7b-Chat

  • [Question] Llama 2 on iOS questions about response time and Chinese support #593
  • Prebuilt uploaded: mlc-ai/mlc-chat-FlagAlpha-Llama2-Chinese-7b-Chat-q4f16_1
  • Prebuilt uploaded: mlc-ai/mlc-chat-FlagAlpha-Llama2-Chinese-7b-Chat-q4f32_1

• Tap-M/Luna-AI-Llama2-Uncensored (q3)

  • [Model Request] 3 bit, 7B Luna-AI-Llama2-Uncensored-GGML #633
  • Prebuilt uploaded: acalatrava/mlc-chat-luna-ai-llama2-7b-chat-uncensored-q3f16_1

• Guanaco

  • [Model Request] Guanaco #336
  • Supported by PR Add support for Guanaco #497; prebuilt models not uploaded yet

• HuggingFaceH4/starchat-alpha

  • Please support HuggingFaceH4/starchat-alpha #214

• Moss

  • moss support #174

• CodeLlama (f32 in wasam specifically)

  • WizardCoder-15B-V1.0-q4f32_1 failing to load web-llm#179

Models with Unsupported Architecture

• ReplitLM
  • [Model Request] ReplitLM #514
• Bloom model of BigScience
  • [Model Request] BLOOM model of BigScience #492
• Falcon
  • [Model Request] Falcon #487

Others

• 100+B models in general
  • [Model Request] will 100+B model be supported in near future  #616

[acalatrava]

Model: https://huggingface.co/Tap-M/Luna-AI-Llama2-Uncensored

Repo: https://huggingface.co/acalatrava/mlc-chat-luna-ai-llama2-7b-chat-uncensored-q3f16_1

[MrJungle1]

@CharlieFRuan I am very happy to contribute my code, but I need some information to get started. Can you provide some more information or documents?

[CharlieFRuan]

@MrJungle1 Hi, thank you for the interest! Documentation can be found here: https://llm.mlc.ai/docs/

Specifically, compiling models can be located here: https://llm.mlc.ai/docs/compilation/compile_models.html

[MrJungle1]

@CharlieFRuan Hello, thank you for your reply. I read this doc, but I don’t seem to find what I want. If I want to support a new model_type, such as "model_type": "qwen", where should I start? Or as you say, a new model architecture.

[CharlieFRuan]

Ahh I see! We are currently migrating to a new workflow, and documentation should be up soon (this week probably). Meanwhile, you could refer to #1408 which adds support for GPT-NeoX. Adding a new model should be similar. The commands to use in this new workflow (for now, since it is not officially supported yet) is:

• python -m mlc_chat gen_config dist/models/Llama-2-7b-chat-hf --quantization q4f16_1 -o ./dist/llama_q4f16/params --conv-template llama-2
• python -m mlc_chat compile ./dist/llama_q4f16/params/mlc-chat-config.json -o ./dist/llama_q4f16/llama_q4f16.so
• python -m mlc_chat convert_weight dist/models/Llama-2-7b-chat-hf --quantization q4f16_1 -o ./dist/llama_q4f16/params/

Then you can use the same steps as before to run the model in runtime.

Made a tracker for the new workflow here: #1420; we will add a Colab tutorial on adding new models.

[MrJungle1]

Great, thank you for your reply, LGTM!

[niutech]

Could you please add Microsoft Phi-1.5 and Phi-2 to the list (#905)? Thanks.

[CharlieFRuan]

@niutech Phi1.5 and phi2 are now supported: https://llm.mlc.ai/docs/prebuilt_models.html#level-1-supported-model-architectures-the-all-in-one-table

[CharlieFRuan]

Replacing this page with https://github.com/orgs/mlc-ai/projects/2. Submit a request to the dashboard following #1042