Option to use other REST APIs besides Closed OpenAI

[bionicles]

Looks like a good project and I would like to try this, would it be possible to add HF / Gemini options? HF has more diversity and Gemini is smarter, neither has "customer noncompete clause" like this:

<rant>
Unfortunately I can't accept this term because it's extremely unsafe longterm and also explicitly anticompetitive, are you down with never being able to train on your chat logs or use manifest to develop models that compete with a company as broadly competitive as OpenAI?

maybe it works for some, obviously the service is popular, I used to tolerate it, but once I realized the seriousness of this clause, I felt compelled to delete my OpenAI account, it's truly hypocritical of them to learn from us and turn around and claim we cannot learn from them.

I meekly suggest switching to HF / Gemini and deleting all support for OpenAI or at least not making this legal license your default as it isn't sensible to encourage all your users to incur such a prohibition on learning, the AI could respond with some serious issue and they can't use that to prevent future unsafe activity from their own AIs, it truly does create a bad situation, and I encourage everyone who reads this to divest OpenAI ASAP, for the sake of basic intellectual freedom.
</rant>

[amoffat]

I'm happy to have other models. The code is abstracted enough to handle other LLMs, just subclass the LLM abstract class. I haven't worked with anything but openai though. Want to start a PR with your chosen model?

[bionicles]

My backlog is huge but I suppose I can double dip with some other library I was working on, it only does NVIDIA HuggingFace but I have code to make it work for Apple Silicon and been meaning to integrate that (but have no such silicon so we'd need GitHub actions to test it), and adding groq and Gemini and mistral is probably also doable given it's a standard REST api, main hurdle would be if it's doing something fancy, are you using function calling ?

Can you point me to the part of the code where you define the API you want to use to invoke the models?

[amoffat]

Sure, here is the LLM abstract base class. If you make another file in that folder like gemini.py, similar to openai.py and implement the ABC's interface, that's 90% of the work. If you do that part, I can take it the rest of the way.

Not using function calling either. The prompts are pretty simple. You can see the OpenAI prompts here. We will likely just copy them verbatim to a gemini (or whatever) folder.

[bionicles]

alright im on it, some questions popped up:

• what's the MSPV (minimum supported python version) ?
• what hardware can you use to test local llms?

unfortunately learned gemini API is in the same boat as anthropic and openai, can't use for work cuz it's got a customer noncompete, so i moved the Gemini(LLM) and Anthropic(LLM) into, ahem, garbage.py

I can't test MLX, so we either need you to have a macbook pro or we need to set up github actions, or both

here's roughly what i got done so far

[bionicles]

Passing on creating these, gotta test call next, need a break

• groq remote
• mlx (skipped bc i dont have apple silicon)
• huggingface qwen2 128k
• huggingface llama3 128k
• huggingface mamba-codestral-256k
• mistral-inference codestral-mamba
• mistral-inference 7b
• mistral-remote open-codestral-mamba

[amoffat]

Wow cool! I would say the minimum python should be 3.11 (from pyproject.toml)

Re local LLMs, I am unable to run them. I don't have good hardware. I would want to make the local LLMs optional by default, so that manifest stays beginner-friendly. But for advanced users, have a local LLM be installed with an optional dependency by doing pip install 'manifest[hf-qwen2]'. I can set that up, just thinking out loud.

[amoffat]

So I was chatting with a friend and they pointed me in the direction of vllm as an interface to local llms. Apparently it handles obtaining models and running local servers, as well as providing a standardized interface to those local models. Do you have any experience with it?

https://docs.vllm.ai/en/latest/getting_started/quickstart.html#using-openai-chat-api-with-vllm

[bionicles]

haven't used it but those docs look great! we can absolutely include this as one of the services and looks like that could help support a wider variety of hardware and models

[andupotorac]

It would be nice for this to work with Groq's llama 3.1 models.

[bionicles]

Oh hell yes, that's really easy, my bad for not checking sooner I'm just in major crunch time, here let me find the code for that

[bionicles]

In the HuggingFaceModelConfig Enum, add a variant with the model name and context limit. Make that the default

META_LLAMA_3_1_8B = ModelConfigValue(
    service=service,
    model_name="meta-llama/Meta-Llama-3.1-8B-Instruct",
     context_limit=128_000,
    supports_vision=False,
            )

Super easy!

[andupotorac]

Thanks!