how to get working with Oobabooga?

[Kaszebe]

So I got the docker file up and running and added the server info to Settings.yaml in Oobabooga:

api_base_url: http://localhost:8000/v1
api_backend: local

But I can't figure out where to prepend the model name. Is it done in the drop down menu when I load a model into Oobabooga? The documentation isn't quite clear.

thank you

[codelion]

If in the front-end of Oobabooga there is no easy way to edit the model name, you can run the proxy with a particular model itself. by passing the --model flag along with the --approach flag. This will start the proxy with a particular technique with that particular model. You can also set them using the environment variables

OPTILLM_APPROACH=mcts
OPTILLM_MODEL=gpt-4 

[Kaszebe]

Hi, I think that's where I was unclear.

So I can just change the model name from: Magnum-v2-123b-il-Q6_K.gguf to Agent-Magnum-v2-123b-il-Q6_K.gguf and it will work?

What about for multi-part GGUF files? e.g. I usually load up Mistral-Large-Instruct-2407.Q6_K-00001-of-00005.gguf and llama.cpp somehow loads the other four files in the background. Would I just need to change the first one to Agent-Mistral-Large-Instruct-2407.Q6_K-00001-of-00005.gguf?

For the flags...I wasn't sure if I should add that info to Settings.yaml or CMD_FLAGS.txt (the latter maybe?)

I have this in Settings.yaml:

api_base_url: http://127.0.0.1:8000
api_backend: local

But unsure if that is correct either.

And dumb question--once I get this installed, I will need to start the docker container before I start Oobabooga and then it will run in the background while I use Oobabooga? I normally use the chat-instruct mode in Ooba.

thanks

[codelion]

In llama.cpp you can use the model_alias to create a name for the model. (see #8 (comment) for more details).

I am not very familiar with Oobabooga, but if it works with your model via llama.cpp server it should work similarly when you use optillm as it is a transparent proxy.

Once set, you can start the container and then it should work in background with Oobabooga.