Using models locally via Ollama.ai

[JanMP]

Is there a way to use llms locally installed via Ollama.ai?

[emrekiciman]

Hi @JanMP ,

I'm not familiar with the Ollama.ai class, but if it installs models in a way that is compatible with the hugging face Transformers, it should be pretty easy to load them locally using the Transformers LLM class in Guidance (https://github.com/guidance-ai/guidance/blob/main/guidance/llms/_transformers.py).

There are several subclasses in https://github.com/guidance-ai/guidance/tree/main/guidance/llms/transformers that demonstrate how to add a particular tokenizer or role model syntax to support model-specific needs.

This might be a route to adding support for Ollama's Modelfiles. It would need investigation but would be a great contribution if it's something you are interested in looking into.

[inconnu26]

Oh having such a combo of Ollama and Guidance would be like having a very accessible rocket ship that you can actually steer !

[luciasantamaria]

Is there any updates on this idea, are there plans to work on support of Ollama-run models? Thanks :)

[inconnu26]

Is there a way to use llms locally installed via Ollama.ai?

Did you manage to get it working ? I would be very interested in such implementation. Thank you in advance !!

[xrd]

I downloaded the ollama binary for my linux machine and then ran this:

./ollama-linux-amd64 serve
./ollama-linux-amd64 run llama3

Then, I can use code like this:

from guidance import models, gen
path = "/home/xrdawson/.ollama/models/blobs/sha256:00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29"
llama3 = models.LlamaCpp(path, echo=False)
lm = llama3 + f'Tell me a story about a neanderthal? ' + gen(stop='.', name="test")
print(lm["test"])

Which outputs:

Here's a story about a Neanderthal named Gronk:

Gronk was a sturdy Neanderthal who lived in a small tribe in the rolling hills of Europe

[doomgrave]

I downloaded the ollama binary for my linux machine and then ran this:

./ollama-linux-amd64 serve
./ollama-linux-amd64 run llama3

Then, I can use code like this:

from guidance import models, gen
path = "/home/xrdawson/.ollama/models/blobs/sha256:00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29"
llama3 = models.LlamaCpp(path, echo=False)
lm = llama3 + f'Tell me a story about a neanderthal? ' + gen(stop='.', name="test")
print(lm["test"])

Which outputs:

Here's a story about a Neanderthal named Gronk:

Gronk was a sturdy Neanderthal who lived in a small tribe in the rolling hills of Europe

You are loading the model in memory twice. its not using ollama this way.

[xrd]

Doh, at this point I didn't realize ollama also provided an REST API. Can I retract my answer? 🤦 😄