llamafile for balena

Run your own chatGPT like chatbot locally on a balena device, including vision (explain this image) on CPU. Tested to work on x86 on balena.

Requirements

Hardware

• i7 computer

Software

• A balenaCloud account (sign up here)
• balenaEtcher

Deploy

You have two options here:

One-click deploy via Balena Deploy

You can deploy this project to a new balenaCloud application in one click using the button below:

Or, you can create an application in your balenaCloud dashboard and balena push this code to it the traditional way.

In-control deploy via balena CLI

If you are a balena CLI expert, feel free to use balena CLI. This option lets you configure in detail some options, like adding new services to your deploy or configure de DNS Server to use.

• Sign up on balena.io
• Create a new application on balenaCloud.
• Add a new device and download the image of the balenaOS it creates.
• Burn an USB stick, connect it to the device and boot it up.

While the device boots (it will eventually show up in the Balena dashboard) we will prepare de services:

cd ~/workspace
git clone https://github.com/mpous/llamafile-balena
cd llamafile-balena

• Using Balena CLI, push the code with balena push <fleet-name>
• See the magic happening, your device is getting updated 🌟Over-The-Air🌟!

Use llamafile-balena

• Wait until the llamafile file is downloaded successfully on the device during the first startup.

• go to your local IP address or balena Public Device URL with the port 8080.
• say 'hello'!
• you can upload an image and ask it to explain what's on it

troubleshooting:

• if the error is

run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4`

then try;

   sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
    sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
    sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"