Any docs or examples for using Guidance NOT in a notebook?

[AnaCoda]

I can't find any tutorial for using Guidance in a regular .py file. Is there non-hacky support for this?

[kar-alex]

The documentation isn't great. I discovered you can use await statements to make the program execute and return results normally.

import guidance
import transformers
model = guidance.llms.Transformers("huggingface model")
program = guidance("""
your program here
""")
out = await program(inputs, llm=model)

[dnhkng]

Thanks, but now I am more confused than before.

Is Guidance designed around Jupyter notebooks?! Seems a very weird choice!

[marcotcr]

The reply above is for the previous version of guidance. Nothing in the new version depends on jupyter notebook other than the streaming visualization.

[dnhkng]

@marcotcr

Is there documentation on how this works?

With the following code:

import guidance
from guidance import gen, models

print(f"{guidance.__version__=}")
llama2 = models.LlamaCpp("./models/mistral-7b-instruct-v0.1.Q8_0.gguf")
llama2 + f"Do you want a joke or a poem? " + gen(stop=".")
print("done")

In a Jupyter Notebook, I get:

Do you want a joke or a poem? I can do both
done

In this example, the text guidance.__version__='0.1.5' is briefly shown before being overwritten by Do you want a joke or a poem? I can do both.

As a script, I get the output:

$ python test_lmql.py
guidance.__version__='0.1.5'
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
done
Exception ignored in: <function _LlamaContext.__del__ at 0x7f8d152b2710>
Traceback (most recent call last):
  File "/home/dnhkng/miniforge3/envs/piper2/lib/python3.10/site-packages/llama_cpp/llama.py", line 422, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _LlamaModel.__del__ at 0x7f8d152b17e0>
Traceback (most recent call last):
  File "/home/dnhkng/miniforge3/envs/piper2/lib/python3.10/site-packages/llama_cpp/llama.py", line 240, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _LlamaBatch.__del__ at 0x7f8d152b3b50>
Traceback (most recent call last):
  File "/home/dnhkng/miniforge3/envs/piper2/lib/python3.10/site-packages/llama_cpp/llama.py", line 670, in __del__
TypeError: 'NoneType' object is not callable

I would really like to use this package, but outside of Juptyer, I am having trouble understanding how it should be used correctly.

[mcchung52g]

Same issue. I have the latest version as of now, 0.1.8, which still outputs as above.