Where to find the llama models?

[gael-vanderlee]

Hello,
I'm pretty new to all this, apologies if the answer is obvious.
I'm trying to make this (and similar) libraries work locally but they all as the user to load the model weights. I found a model on huggingFace but when loading I get the following error:

>>> LlamaCpp(model_path="models\\ggml-model-q4_0.bin")
  Could not load Llama model from path: [...]models\ggml-model-q4_0.bin. Received error  (type=value_error)

I assume this is is because I downloaded the wrong model. Where do I find them and how do I know which are compatible with this library ?

Thanks for the help !

[gjmulder]

As of now, the llama models and their derivatives are licensed for restricted distribution by Facebook, so they will never be distributed from or linked to in this repo.

Please refer to the llama.cpp README.md for more information. Or you could try Googling site:huggingface.co+llama+ggml...

Note that there are currently three versions of models compatible with different versions of python-llama-cpp. You would want to install the latest python-llama-cpp version and use the latest (ggml v3) llama.cpp models to ensure both the best compatibility and performance.

HuggingFace is now providing a leaderboard of the best quality models. Note again, however that the models linked off the leaderboard are not directly compatible with llama.cpp (and therefore python-llama-cpp). You can, again with a bit of searching, find the converted ggml v3 llama.cpp equivalent models.

[gael-vanderlee]

Got it, thanks for the response!

[gael-vanderlee]

It seems that I'm still getting the following error:

llama.cpp: loading model from \models\llama-13b-ggml-v3-q4_0.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2022.2.3\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
__root__
  Could not load Llama model from path: \models\llama-13b-ggml-v3-q4_0.bin. Received error  (type=value_error)

Is this an error with the model file or does the issue lie somewhere else ?

[gjmulder]

pip list | grep llama ?

[gael-vanderlee]

Package                 Version
----------------------- -----------
aiohttp                 3.8.4
charset-normalizer      3.1.0
chromadb                0.3.23
click                   8.1.3
clickhouse-connect      0.5.24
colorama                0.4.6
colorclass              2.2.2
commonmark              0.9.1
compressed-rtf          1.0.6
cryptography            40.0.2
dataclasses-json        0.5.7
Deprecated              1.2.13
duckdb                  0.8.0
easygui                 0.98.3
ebcdic                  1.1.1
et-xmlfile              1.1.0
extract-msg             0.41.1
fastapi                 0.95.2
filelock                3.12.0
frozenlist              1.3.3
fsspec                  2023.5.0
greenlet                2.0.2
h11                     0.14.0
hnswlib                 0.7.0
httpcore                0.16.3
httptools               0.5.0 
httpx                   0.23.3
huggingface-hub         0.14.1
idna                    3.4
IMAPClient              2.3.1
Jinja2                  3.1.2
joblib                  1.2.0
langchain               0.0.171
lark-parser             0.12.0
llama-cpp-python        0.1.50
lxml                    4.9.2
lz4                     4.3.2
Markdown                3.4.3
MarkupSafe              2.1.2
marshmallow             3.19.0
marshmallow-enum        1.5.1
monotonic               1.6
mpmath                  1.3.0
msg-parser              1.2.0
msoffcrypto-tool        5.0.1
multidict               6.0.4
mypy-extensions         1.0.0
networkx                3.1
nltk                    3.8.1
numexpr                 2.8.4
numpy                   1.23.5
olefile                 0.46
oletools                0.60.1
openapi-schema-pydantic 1.2.4
openpyxl                3.1.2
packaging               23.1
pandas                  1.5.3
pandoc                  2.3
pcodedmp                1.2.6
pdfminer.six            20221105
Pillow                  9.5.0
pip                     23.1.2
plumbum                 1.8.1
ply                     3.11
posthog                 3.0.1
pycparser               2.21
pydantic                1.10.7
Pygments                2.15.1
pygpt4all               1.1.0
pygptj                  2.0.3
pyllamacpp              2.3.0
pypandoc                1.11
pyparsing               2.4.7
python-dateutil         2.8.2
python-docx             0.8.11
python-dotenv           1.0.0
python-magic            0.4.27
python-pptx             0.6.21
pytz                    2023.3
pytz-deprecation-shim   0.1.0.post0
pywin32                 306
PyYAML                  6.0
red-black-tree-mod      1.20
regex                   2023.5.5
requests                2.30.0
rfc3986                 1.5.0
rich                    13.0.1
RTFDE                   0.0.2
scikit-learn            1.2.2
scipy                   1.10.1
sentence-transformers   2.2.2
sentencepiece           0.1.99
setuptools              65.5.1
six                     1.16.0
sniffio                 1.3.0
soupsieve               2.4.1
SQLAlchemy              2.0.15
starlette               0.27.0
sympy                   1.12
tabulate                0.9.0
tenacity                8.2.2
threadpoolctl           3.1.0
tokenizers              0.13.3
torch                   2.0.1
torchvision             0.15.2
tqdm                    4.65.0
transformers            4.29.2
typer                   0.9.0
typing_extensions       4.5.0
typing-inspect          0.8.0
tzdata                  2023.3
tzlocal                 4.2
unstructured            0.6.6
urllib3                 2.0.2
uvicorn                 0.22.0
watchfiles              0.19.0
websockets              11.0.3
wheel                   0.38.4
win-unicode-console     0.5
wrapt                   1.14.1
XlsxWriter              3.1.1
yarl                    1.9.2
zstandard               0.21.0

llama-13b-ggml-v3-q4_0.bin

[gjmulder]

You forgot the grep:

llama-cpp-python 0.1.50

This version (from memory) supports v2.

Try to upgrade to the latest 0.1.53, which supports v3:

pip install --force-reinstall --ignore-installed --no-cache-dir llama-cpp-python

[gael-vanderlee]

Sorry about the grep, I'm on windows on this machine.
Your fix worked, tysm!

[frankandrobot]

i'm getting the same error as the OP but am on llama-cpp-python 0.1.63. Trying to load https://huggingface.co/TheBloke/LLaMa-13B-GGML. Every file there gives this or a simlar error

It looks like it was a git lfs issue. However, when I tried llama-13b.ggmlv3.q6_K.bin which did download properly, getting this error:

ggml_metal_add_buffer: buffer 'data' size 10678861824 is larger than buffer maximum of 8589934592
llama_init_from_file: failed to add buffer

Does this only support 4-bit models?

[Kan-wal]

from where i can found the v3 ggml model?

[frankandrobot]

https://huggingface.co/TheBloke/LLaMa-13B-GGML

but see above... getting an error for random GGML v3 files