Using Marvin with Mistral-7B-Instruct via llama.cpp

[lostmygithubaccount]

Rough steps to get here:

• clone the Mistral-7B-Instruct-v0.2 data from huggingface
• use llama.cpp/convert.py to convert to gguf
• run make in llama.cpp to build the server executable
• run the llama.cpp/server pointing at the gguf

This results in an OpenAI-compatible server on http://localhost:8000

Setup Marvin to point there:

MARVIN_LOG_LEVEL=INFO
MARVIN_OPENAI_API_KEY="ayyy"
MARVIN_OPENAI_BASE_URL="http://127.0.0.1:8080/v1"

Unfortunately (and surprisingly), only classification seems to work:

[ins] In [1]: import marvin

[ins] In [2]: result = marvin.classify(
         ...:     "Marvin is so easy to use!",
         ...:     labels=["positive", "negative"],
         ...: )

[ins] In [3]: result
Out[3]: 'positive'

[nav] In [4]: features = marvin.extract(
         ...:     "I love my new phone's camera, but the battery life could be improved.",
         ...:     instructions="product features",
         ...: )
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:95, in run_sync(coroutine)
     94 try:
---> 95     loop = asyncio.get_running_loop()
     96     if loop.is_running():

RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 features = marvin.extract(
      2     "I love my new phone's camera, but the battery life could be improved.",
      3     instructions="product features",
      4 )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:789, in extract(data, target, instructions, model_kwargs, client)
    759 def extract(
    760     data: str,
    761     target: type[T] = None,
   (...)
    764     client: Optional[AsyncMarvinClient] = None,
    765 ) -> list[T]:
    766     """
    767     Extracts entities of a specific type from the provided data.
    768
   (...)
    787         list: A list of extracted entities of the specified type.
    788     """
--> 789     return run_sync(
    790         extract_async(
    791             data=data,
    792             target=target,
    793             instructions=instructions,
    794             model_kwargs=model_kwargs,
    795             client=client,
    796         )
    797     )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:103, in run_sync(coroutine)
    101         return asyncio.run(coroutine)
    102 except RuntimeError:
--> 103     return asyncio.run(coroutine)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:190, in run(main, debug)
    186     raise RuntimeError(
    187         "asyncio.run() cannot be called from a running event loop")
    189 with Runner(debug=debug) as runner:
--> 190     return runner.run(main)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:118, in Runner.run(self, coro, context)
    116 self._interrupt_count = 0
    117 try:
--> 118     return self._loop.run_until_complete(task)
    119 except exceptions.CancelledError:
    120     if self._interrupt_count > 0:

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/base_events.py:653, in BaseEventLoop.run_until_complete(self, future)
    650 if not future.done():
    651     raise RuntimeError('Event loop stopped before Future completed.')
--> 653 return future.result()

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:313, in extract_async(data, target, instructions, model_kwargs, client)
    311     target = str
    312 model_kwargs = model_kwargs or {}
--> 313 return await _generate_typed_llm_response_with_tool(
    314     prompt_template=EXTRACT_PROMPT,
    315     prompt_kwargs=dict(data=data, instructions=instructions),
    316     type_=list[target],
    317     model_kwargs=model_kwargs | dict(temperature=0),
    318     client=client,
    319 )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:161, in _generate_typed_llm_response_with_tool(prompt_template, type_, tool_name, prompt_kwargs, model_kwargs, client)
    152 prompt_kwargs["response_format"] = tool.function.parameters
    154 response = await generate_llm_response(
    155     prompt_template=prompt_template,
    156     prompt_kwargs=prompt_kwargs,
    157     model_kwargs=model_kwargs,
    158     client=client,
    159 )
--> 161 return response.tool_outputs[0]

IndexError: list index out of range

[nav] In [5]: from pydantic import BaseModel, Field
         ...:
         ...:
         ...: class Location(BaseModel):
         ...:     city: str
         ...:     state: str = Field(description='2-letter abbreviation')
         ...:
         ...:
         ...: result = marvin.cast('the big apple', Location)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:95, in run_sync(coroutine)
     94 try:
---> 95     loop = asyncio.get_running_loop()
     96     if loop.is_running():

RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
Cell In[5], line 9
      5     city: str
      6     state: str = Field(description='2-letter abbreviation')
----> 9 result = marvin.cast('the big apple', Location)

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:710, in cast(data, target, instructions, model_kwargs, client)
    686 def cast(
    687     data: str,
    688     target: type[T],
   (...)
    691     client: Optional[AsyncMarvinClient] = None,
    692 ) -> T:
    693     """
    694     Converts the input data into the specified type.
    695
   (...)
    708         T: The converted data of the specified type.
    709     """
--> 710     return run_sync(
    711         cast_async(
    712             data=data,
    713             target=target,
    714             instructions=instructions,
    715             model_kwargs=model_kwargs,
    716             client=client,
    717         )
    718     )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:103, in run_sync(coroutine)
    101         return asyncio.run(coroutine)
    102 except RuntimeError:
--> 103     return asyncio.run(coroutine)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:190, in run(main, debug)
    186     raise RuntimeError(
    187         "asyncio.run() cannot be called from a running event loop")
    189 with Runner(debug=debug) as runner:
--> 190     return runner.run(main)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:118, in Runner.run(self, coro, context)
    116 self._interrupt_count = 0
    117 try:
--> 118     return self._loop.run_until_complete(task)
    119 except exceptions.CancelledError:
    120     if self._interrupt_count > 0:

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/base_events.py:653, in BaseEventLoop.run_until_complete(self, future)
    650 if not future.done():
    651     raise RuntimeError('Event loop stopped before Future completed.')
--> 653 return future.result()

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:269, in cast_async(data, target, instructions, model_kwargs, client)
    255 if (
    256     get_origin(target) == Literal
    257     or (isinstance(target, type) and issubclass(target, Enum))
    258     or isinstance(target, list)
    259     or target is bool
    260 ):
    261     return await classify_async(
    262         data=data,
    263         labels=target,
   (...)
    266         client=client,
    267     )
--> 269 return await _generate_typed_llm_response_with_tool(
    270     prompt_template=CAST_PROMPT,
    271     prompt_kwargs=dict(data=data, instructions=instructions),
    272     type_=target,
    273     model_kwargs=model_kwargs | dict(temperature=0),
    274     client=client,
    275 )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:161, in _generate_typed_llm_response_with_tool(prompt_template, type_, tool_name, prompt_kwargs, model_kwargs, client)
    152 prompt_kwargs["response_format"] = tool.function.parameters
    154 response = await generate_llm_response(
    155     prompt_template=prompt_template,
    156     prompt_kwargs=prompt_kwargs,
    157     model_kwargs=model_kwargs,
    158     client=client,
    159 )
--> 161 return response.tool_outputs[0]

IndexError: list index out of range

[nav] In [6]: from pydantic import BaseModel, Field
         ...:
         ...:
         ...: class Location(BaseModel):
         ...:     city: str
         ...:     state: str = Field(description='2-letter abbreviation')
         ...:
         ...:
         ...: result = marvin.generate(
         ...:     n=4,
         ...:     target=Location,
         ...:     instructions='US cities named after presidents',
         ...: )
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:95, in run_sync(coroutine)
     94 try:
---> 95     loop = asyncio.get_running_loop()
     96     if loop.is_running():

RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
Cell In[6], line 9
      5     city: str
      6     state: str = Field(description='2-letter abbreviation')
----> 9 result = marvin.generate(
     10     n=4,
     11     target=Location,
     12     instructions='US cities named after presidents',
     13 )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:831, in generate(target, instructions, n, use_cache, temperature, model_kwargs, client)
    800 def generate(
    801     target: Optional[type[T]] = None,
    802     instructions: Optional[str] = None,
   (...)
    807     client: Optional[AsyncMarvinClient] = None,
    808 ) -> list[T]:
    809     """
    810     Generates a list of 'n' items of the provided type or based on instructions.
    811
   (...)
    829         list: A list of generated items.
    830     """
--> 831     return run_sync(
    832         generate_async(
    833             target=target,
    834             instructions=instructions,
    835             n=n,
    836             use_cache=use_cache,
    837             temperature=temperature,
    838             model_kwargs=model_kwargs,
    839             client=client,
    840         )
    841     )

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:103, in run_sync(coroutine)
    101         return asyncio.run(coroutine)
    102 except RuntimeError:
--> 103     return asyncio.run(coroutine)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:190, in run(main, debug)
    186     raise RuntimeError(
    187         "asyncio.run() cannot be called from a running event loop")
    189 with Runner(debug=debug) as runner:
--> 190     return runner.run(main)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:118, in Runner.run(self, coro, context)
    116 self._interrupt_count = 0
    117 try:
--> 118     return self._loop.run_until_complete(task)
    119 except exceptions.CancelledError:
    120     if self._interrupt_count > 0:

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/base_events.py:653, in BaseEventLoop.run_until_complete(self, future)
    650 if not future.done():
    651     raise RuntimeError('Event loop stopped before Future completed.')
--> 653 return future.result()

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:413, in generate_async(target, instructions, n, use_cache, temperature, model_kwargs, client)
    411 result = [0] * (n + 1)
    412 while len(result) != n:
--> 413     result = await _generate_typed_llm_response_with_tool(
    414         prompt_template=GENERATE_PROMPT,
    415         prompt_kwargs=dict(
    416             type_=target,
    417             n=n,
    418             instructions=instructions,
    419             previous_responses=previous_responses,
    420         ),
    421         type_=list[target],
    422         model_kwargs=(model_kwargs or {}) | dict(temperature=temperature),
    423         client=client,
    424     )
    426     if len(result) > n:
    427         result = result[:n]

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:161, in _generate_typed_llm_response_with_tool(prompt_template, type_, tool_name, prompt_kwargs, model_kwargs, client)
    152 prompt_kwargs["response_format"] = tool.function.parameters
    154 response = await generate_llm_response(
    155     prompt_template=prompt_template,
    156     prompt_kwargs=prompt_kwargs,
    157     model_kwargs=model_kwargs,
    158     client=client,
    159 )
--> 161 return response.tool_outputs[0]

IndexError: list index out of range

[nav] In [7]: @marvin.fn
         ...: def list_fruits(n: int, color: str) -> list[str]:
         ...:     """Generates a list of `n` fruits that are `color`"""
         ...:
         ...:
         ...: fruits = list_fruits(3, color='red')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:95, in run_sync(coroutine)
     94 try:
---> 95     loop = asyncio.get_running_loop()
     96     if loop.is_running():

RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
Cell In[7], line 6
      1 @marvin.fn
      2 def list_fruits(n: int, color: str) -> list[str]:
      3     """Generates a list of `n` fruits that are `color`"""
----> 6 fruits = list_fruits(3, color='red')

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:516, in fn.<locals>.sync_wrapper(*args, **kwargs)
    514 @wraps(func)
    515 def sync_wrapper(*args, **kwargs):
--> 516     return run_sync(async_wrapper(*args, **kwargs))

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/utilities/asyncio.py:103, in run_sync(coroutine)
    101         return asyncio.run(coroutine)
    102 except RuntimeError:
--> 103     return asyncio.run(coroutine)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:190, in run(main, debug)
    186     raise RuntimeError(
    187         "asyncio.run() cannot be called from a running event loop")
    189 with Runner(debug=debug) as runner:
--> 190     return runner.run(main)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/runners.py:118, in Runner.run(self, coro, context)
    116 self._interrupt_count = 0
    117 try:
--> 118     return self._loop.run_until_complete(task)
    119 except exceptions.CancelledError:
    120     if self._interrupt_count > 0:

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/asyncio/base_events.py:653, in BaseEventLoop.run_until_complete(self, future)
    650 if not future.done():
    651     raise RuntimeError('Event loop stopped before Future completed.')
--> 653 return future.result()

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:494, in fn.<locals>.async_wrapper(*args, **kwargs)
    491 else:
    492     type_ = model.return_annotation
--> 494 result = await _generate_typed_llm_response_with_tool(
    495     prompt_template=FUNCTION_PROMPT,
    496     prompt_kwargs=dict(
    497         fn_definition=model.definition,
    498         bound_parameters=model.bound_parameters,
    499         return_value=model.return_value,
    500     ),
    501     type_=type_,
    502     model_kwargs=model_kwargs,
    503     client=client,
    504 )
    506 if post_processor is not None:
    507     result = post_processor(result)

File ~/repos/ibis/venv/lib/python3.11/site-packages/marvin/ai/text.py:161, in _generate_typed_llm_response_with_tool(prompt_template, type_, tool_name, prompt_kwargs, model_kwargs, client)
    152 prompt_kwargs["response_format"] = tool.function.parameters
    154 response = await generate_llm_response(
    155     prompt_template=prompt_template,
    156     prompt_kwargs=prompt_kwargs,
    157     model_kwargs=model_kwargs,
    158     client=client,
    159 )
--> 161 return response.tool_outputs[0]

IndexError: list index out of range

This would be awesome to get working with all of Marvin. I'm not sure where to start

[zzstoatzz]

hey @lostmygithubaccount - i think there's 2 big things here (from my exploration with mistral, which has been admittedly limited compared to what I would prefer)

• the model's ability to do function calling
  • for all text utils (cast, extract etc) we currently need to do function calling to go from str -> SomeType. I know that some mistral models can do function calling, but the trace you've shown makes it seem like there's not function calling happening here
• our default (you can pass your own) system prompts are tested primarily against OpenAI models

can you say more about what you've done here:

use llama.cpp/convert.py to convert to gguf

so I can spend some time with this myself sometime?

[lostmygithubaccount]

@zzstoatzz hello!

I've tried some stuff since opening this -- long story short, the OSS LLM framework/ecosystem is a bit of a shitshow. I'm skeptical any of these smaller (7B-ish) models can even fit into the Marvin paradigm how I'm using it...but to explore a bit:

• llama.cpp is the most full-featured implementation that works across platforms
• mlx is a nice macOS way to work with LLMs
• candle is something
• ollama is a wrapper around llama.cpp that provides docker-like usage

there are a few Marvin-esque frameworks for getting structured output:

• https://github.com/jxnl/instructor -- have not gotten this to work well
• https://github.com/outlines-dev/outlines -- no luck w/ this, pointed to by mlx: JSON output? ml-explore/mlx#402

I somewhat recently tried the relatively new Ollama OpenAI compatible API thingy: https://ollama.com/blog/openai-compatibility and https://github.com/ollama/ollama/blob/main/docs/openai.md to little success

Ideally, I could get GPT-4 quality Marvin functionality on my laptop...but that feels like a pipedream...

so basic steps:

• download weights from huggingface with git clone (after installing/making lfs default) for an OSS LLM like Mistral7B
• convert the weights to GGUF so llamap.cpp/ollama can work with them (or convert to MLX format)
• this involves cloning llama.cpp, creating a virtual environment, installing the dependencies, and running that convert.py script. some hacking may be needed depending on the model
• setup an OpenAPI compatible local server
• set Marvin to point to that server
• try running Marvin things

people convert and upload GGUF models for most of the OSS LLMs that exist, but I strongly prefer to convert them myself because 1) I trust the original sources a bit more and 2) I'm on satellite internet so re-downloading the same weights multiple times is annoying. but if download speed isn't a problem for you, directly downloading the weights in the format you intend to run in (GGUF for llama.cpp/Ollama, whatever it's called for MLX) might be better

I will retry the Ollama route this week and can post back on where I get using it with Marvin, but the last time I tried it wasn't very far. I'm very skeptical these smaller OSS models will give me the accuracy/speed I need for my applications

[zzstoatzz]

thanks for the thorough reply! I'll eventually get some time to delve into this world a bit more 🙂

the OSS LLM framework/ecosystem

your context is super appreciated - excited to hear any update on your exploration!

[lostmygithubaccount]

I'm going to write up the journey on the Ibis blog, here's the issue if you want to follow along: ibis-project/ibis#8731

probably not getting to it this week but hopefully in the next 2 weeks or so. also hope it's clear but love using Marvin! it's a great library

[lostmygithubaccount]

most recent issue with ollama and Mistral7B:

KeyError: 'Could not automatically map mistral-small to a tokeniser. Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.'

the tokenizer model is right there with the weights, but I'm unclear on using it: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/tree/main
tiktoken has no docs which is super cool, so not sure if there's some way to easily load it in there. I'm not seeing anything obvious

perhaps just load it with pytorch...?

I'm also struggling to figure out how to set the encoder w/ an environment variable or something

and trying to set it from the settings gets this:

[ins] In [3]: marvin.settings.openai.chat.completions.encoder = "a"
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 marvin.settings.openai.chat.completions.encoder = "a"

File ~/repos/lm/venv/lib/python3.11/site-packages/marvin/settings.py:32, in MarvinSettings.__setattr__(self, name, value)
     30     if SecretStr in base_types and not isinstance(value, SecretStr):  # type: ignore # noqa: E501
     31         value = SecretStr(value)
---> 32 super().__setattr__(name, value)

File ~/repos/lm/venv/lib/python3.11/site-packages/pydantic/main.py:790, in BaseModel.__setattr__(self, name, value)
    788 attr = getattr(self.__class__, name, None)
    789 if isinstance(attr, property):
--> 790     attr.__set__(self, value)
    791 elif self.model_config.get('validate_assignment', None):
    792     self.__pydantic_validator__.validate_assignment(self, name, value)

AttributeError: property 'encoder' of 'ChatCompletionSettings' object has no setter