App Submission: localai

[highghlow]

noGPUpasthu4u

App Submission

App name

LocalAI

256x256 SVG icon

(Submit an icon with no rounded corners as it will be dynamically rounded with CSS. GitHub doesn't allow uploading SVGs directly, so please upload your icon to an alternate service, like https://svgur.com, and paste the link below.)
We will help finalize this icon before the app goes live in the Umbrel App Store.

For some reason I was only able to find this as a jpeg

https://github.com/go-skynet/LocalAI/assets/2420543/0966aa2a-166e-4f99-a3e5-6c915fc997dd

Gallery images

(Upload 3 to 5 high-quality gallery images (1440x900px) of your app in PNG format, or just upload 3 to 5 screenshots of your app and we'll help you design the gallery images.)
We will help finalize these images before the app goes live in the Umbrel App Store.
...

I can't get more screenshots because running this without gpu passthrough is really, really slow.

I have tested my app on:

• umbrelOS on a Raspberry Pi
• umbrelOS on an Umbrel Home
• umbrelOS on Linux

[nmfretz]

Thanks for getting this together @highghlow. I've been playing around with it this morning to package it so that it installs on a users machine with no/minimal models downloaded initially. The image and the models are quite large, so install time is very long.

But I've just realized embarrassingly late in my testing that the localai images are not multi-architecture. They are only available for x86 right now. This means that the install would fail for arm users (e.g., RPi 5) and leave users in a broken state. Let's close this for now until an official arm image is available. I will open an Issue in their repo asking about official arm support.

[nmfretz]

Ohhhhh, hold up I see that arm support is on the horizon so let's keep this PR open!
mudler/LocalAI#1249 (comment)

Once they are available I'm hoping we can use their "standard container images" which don't include pre-installed models:
https://localai.io/basics/container/#standard-container-images

[nmfretz]

Looks like some images are now multi-arch. e.g.,:

I will trigger our new github actions linter to start the review process.

[al-lac]

@nmfretz can this be looked at again? Would be great to have this.

[nmfretz]

Ah ya, thanks for the ping @al-lac. Haven't heard from highglow in a long time, so I think it's okay to move this forward if you want.

[highghlow]

It seems like the localai docker image is still missing arm64 support, which is the main issue here

[nmfretz]

It seems like the localai docker image is still missing arm64 support, which is the main issue here

Ah that's too bad. Thanks for checking @highghlow. This multiarch image I noticed here might have just been a trial they were doing that didn't pan out longterm for them.

[al-lac]

Not sure where you are checking, but using the latest version (2.24.1) i was able to install it on my raspberry pi 5:

See the container here: https://registry.hub.docker.com/layers/localai/localai/v2.24.1-aio-cpu/images/sha256-592397759ab3fd3fc58643fd1c9437ad556ee32b31e944ec6cc622bdb547e0d7?context=explore

Did not get a response yet for my query though, so really not performant, at least on a pi 5.

[nmfretz]

@al-lac, ya that's where I was checking previously as well. So looks like at least the all-in-one cpu image is still multiarch.

I recall that the AIO image downloads a bunch of models on install, correct? How long did it take for the models to download, and was the UI available during that time?

[al-lac]

Yeah exactly. So it downloads around 2.25GB for the container, which results in about 6GB uncompressed. The models it downloads automatically come up to around 14GB. So really depends on the network speed. Not sure if the UI was available during the model download as I was away for a bit.

But after downloading everything and even downloading a really small model i was never able to get a response or get anything to properly work. Also had to stop nearly every other application running and during its peak it consumed around 4,8GB of Memory which is really a lot to handle for the 8GB Raspberry Pi.

All of this could of course work better on the Umbrel Home or on other machines.

[github-actions]

⚠️   Linting finished with 1 warning   ⚠️

Thank you for your submission! This is an automated linter that checks for common issues in pull requests to the Umbrel App Store.

Please review the linting results below and make any necessary changes to your submission.

Linting Results

Severity	File	Description
ℹ️	localai/docker-compose.yml	Mounted file/directory "/localai/data/models" doesn't exist: The volume "${APP_DATA_DIR}/data/models:/build/models:cached" tries to mount the file/directory "/localai/data/models", but it is not present. This can lead to permission errors!
ℹ️	localai/docker-compose.yml	Potentially using unsafe user in service "api": The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
⚠️	localai/umbrel-app.yml	"icon" and "gallery" needs to be empty for new app submissions: The "icon" and "gallery" fields must be empty for new app submissions as it is being created by the Umbrel team.

Legend

Symbol	Description
❌	Error: This must be resolved before this PR can be merged.
⚠️	Warning: This is highly encouraged to be resolved, but is not strictly mandatory.
ℹ️	Info: This is just for your information.

[highghlow]

Boom, done. @nmfretz