App submission mosquitto #1789

dirstel · 2024-11-11T16:35:20Z

App Submission

App name

Eclipse Mosquitto

256x256 SVG icon

Gallery images

(not really much to screenshot)

I have tested my app on:

umbrelOS on a Raspberry Pi
umbrelOS on an Umbrel Home
umbrelOS on Linux VM

added version

added v2.0.20 incl. hash

sync

nmfretz · 2024-11-13T11:55:57Z

Thanks for another submission @dirstel! We'll try to get to this early next week. Thanks for your patience.

fixed typo in tag <strong>

fixed permissions

nmfretz

Nice work repurposing the Samba/Snowflake index.html to give the app a web UI with instructions @dirstel 👌

Can mosquitto even serve the html page via it's websocket port though? I think that port is just so that mqtt clients can connect using websockets. I tried quickly installing this on my umbrel and nothing shows up.

Regardless, I've got some suggestions below on how to serve the UI and make your life much easier when updating this app in the future.

Package the UI as a separate Docker image

The way umbrelOS currently works is that on app install, everything from the app's repo is copied over to a users app-data directory. But on app updates only a whitelist of files is copied over so that user data is not overwritten:
https://github.com/getumbrel/umbrel/blob/49d37581c8233f808fc5a79c2cd38810c6cd1b0b/packages/umbreld/source/modules/apps/legacy-compat/app-script#L594-L595

What that means for you here is that if you need to make changes to your UI (e.g., changing the instructions in the html), only new installs would receive the changes. Existing installs that update won't get the changes. You would need to resort to some really hacky things like using a hook that runs on app update and makes modifications to the html/image files. It would end up adding a lot of complexity and overhead for you.

Instead though, you can just package the UI as a super simple Docker image that serves your html and then point the app_proxy service to it. That way if you ever need to update the UI, you just build and push a new Docker image to your repo and then change it in the docker-compose.yml when submitting an app update.

I have not tested this at all, but something as simple as this could work for a Dockerfile:

# Use the joseluisq/static-web-server as the base image
FROM joseluisq/static-web-server:2.33.1

# Copy the directory with your ui web assets to the /var/public directory which is set by the SERVER_ROOT env var.
# e.g., if your directory is called `public`:
COPY public /var/public

You'll then need to make sure to build a multi-architecture image for amd64 and arm64. You'll run something like this:

docker buildx build --platform linux/amd64,linux/arm64 -t your-docker-hub-username/static-web-server-mosquitto:version-tag .

nmfretz · 2024-11-28T04:52:14Z

mosquitto/config/mosquitto.conf.default

+# websockets listener (including webserver)
+listener 9001
+protocol websockets
+http_dir /mosquitto/www


If the webserver ends up as its own Docker service, this needs to be modified.

But I'm not sure it works like this anyways. Really sorry if I'm totally wrong here!

nmfretz · 2024-11-28T04:57:26Z

mosquitto/config/mosquitto.conf.default

+http_dir /mosquitto/www
+
+# security
+allow_anonymous true


For this use-case, are we good to allow this? Or should this require authentication?

nmfretz · 2024-11-28T05:02:58Z

mosquitto/docker-compose.yaml

@@ -0,0 +1,18 @@
+services:


This docker-compose file and the umbrel-app manifest file need to have the .yml suffix instead of .yaml ,otherwise they won't render in the app store and you won't be able to install them via the UI or CLI. Were you still able to install it somehow?

nmfretz · 2024-11-28T05:04:04Z

mosquitto/docker-compose.yaml

+    environment:
+      APP_HOST: mosquitto_broker_1
+      APP_PORT: 9001
+    container_name: mosquitto_app_proxy_1


The container_name for the app_proxy and broker services can be removed.

nmfretz · 2024-11-28T05:06:09Z

mosquitto/docker-compose.yaml

+    ports:
+      - '1883:1883'
+      - '9001:9001'


Port 1883 is free, but 9001 is used by Penpot. You could change to 9991:9001 and update the instructions accordingly.

nmfretz · 2024-11-28T05:11:32Z

mosquitto/docker-compose.yaml

+    volumes:
+      - ${APP_DATA_DIR}/config:/mosquitto/config:rw
+      - ${APP_DATA_DIR}/data:/mosquitto/data:rw
+      - ${APP_DATA_DIR}/log:/mosquitto/log:rw
+      - ${APP_DATA_DIR}/www:/mosquitto/www:r


To keep everything under one data directory like other apps I'd suggest going with something like this:

data ├── mosquitto-data │ └── .gitkeep ├── config │ └── mosquitto.conf.default └── log └── .gitkeep

And then if serving the UI from a Docker container as described above, just get rid of - ${APP_DATA_DIR}/www:/mosquitto/www:r

nmfretz · 2024-11-28T05:21:05Z

mosquitto/hooks/pre-start

+  # set correct owner
+  chown umbrel:umbrel "${CONFIG_FILE}"


Nice hook 🫡

Looking at the mosquitto Dockerfile, we're actually stuck running the container as user:group 1883:1883 since they have hardcoded this into the image unfortunately. I think this should still work as-is though.

You can see that all data directories for mosquitto end up owned by 1883

:~$ ls -l ~/umbrel/app-data/mosquitto/ total 20 drwxr-xr-x 1 1883 1883 72 Nov 28 05:14 config drwxr-xr-x 1 1883 1883 0 Nov 28 02:04 data -rw-r--r-- 1 umbrel umbrel 574 Nov 28 04:59 docker-compose.yaml -rw-r--r-- 1 umbrel umbrel 583 Nov 28 05:14 docker-compose.yml drwxr-xr-x 1 umbrel umbrel 18 Nov 28 05:14 hooks drwxr-xr-x 1 1883 1883 0 Nov 28 02:04 log -rw-r--r-- 1 root root 17 Nov 28 05:14 settings.yml -rw-r--r-- 1 umbrel umbrel 1008 Nov 28 04:59 umbrel-app.yaml -rw-r--r-- 1 umbrel umbrel 1008 Nov 28 04:59 umbrel-app.yml drwxr-xr-x 1 1883 1883 58 Nov 28 02:04 www

nmfretz · 2024-11-28T05:21:08Z

mosquitto/umbrel-app.yaml

+id: mosquitto
+name: Mosquitto
+tagline: An open source MQTT broker
+icon: https://raw.githubusercontent.com/eclipse-mosquitto/mosquitto/refs/heads/master/logo/mosquitto-logo-only.svg


This entire icon key-value pair can be deleted. We'll host the icon at https://github.com/getumbrel/umbrel-apps-gallery

nmfretz · 2024-11-28T05:21:38Z

mosquitto/www/index.html

+<!DOCTYPE html>
+<!-- saved from url=(0024)http://server.home:9445/ -->
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
+    <title>Samba</title>


Can change to Mosquitto

…default renamed: data/.gitkeep -> data/mosquitto/.gitkeep renamed: log/.gitkeep -> data/log/.gitkeep renamed: www/favicon.png -> data/www/favicon.png renamed: www/index.html -> data/www/index.html renamed: www/logo.png -> data/www/logo.png renamed: docker-compose.yaml -> docker-compose.yml renamed: umbrel-app.yaml -> umbrel-app.yml

modified: data/www/index.html modified: docker-compose.yml modified: hooks/pre-start modified: umbrel-app.yml

dirstel · 2024-11-28T15:06:41Z

Addressed the following concerns:

ports: 9991 is used for MQTT via WebService / Webserver
app-port changed to 9021 (9011 is also used - is there a list of all used ports so far?)
fixed directories
fixed file-suffixes
fixed user by explizit setting in docker-compose.yml

Allow anonymous MQTT-access
The MQTT-Service is usuable without permissions. When exposing the service to the internet this should definetly be changed. as there are lots of possibilities (one user, user per client/sensor, acl, aditonal plugins, ...) I think this should not be set in default installation. Maybe a hint or two in the webpage can be added.

Serving webpage by dedicated container
First, Mosquitto can serve a webpage on its own.
Second the availability of the webpage is a decent indicator of Moswuitto really running.
I agree with you in terms of updates. Maybe there should be a detailed description of what is done by an update? As far as I can see, the hook used is not needed too. In general, headless services should be given the opportunity to display a webpage by umbrel or the app-proxy. In my opinion it is somewhat cumbersome to build a dedicated container for each of this app.

added zigbee2mqtt/data/configuration.yaml.default added zigbee2mqtt/docker-compose.yml added zigbee2mqtt/hooks/pre-start added zigbee2mqtt/umbrel-app.yml

nmfretz · 2024-12-10T03:55:10Z

Thanks for making those changes @dirstel.

First, Mosquitto can serve a webpage on its own.
Second the availability of the webpage is a decent indicator of Moswuitto really running.

Thanks for the clarification. With your changes the webpage is now being served correctly for me by mosquitto on app install.

The MQTT-Service is usuable without permissions. When exposing the service to the internet this should definetly be changed. as there are lots of possibilities (one user, user per client/sensor, acl, aditonal plugins, ...) I think this should not be set in default installation. Maybe a hint or two in the webpage can be added.

Adding a hint to the UI is a great idea. While you're doing that you may want to also change the following:

accessinformation --> access information
Settings > Advanced Settings > Terminal --> Settings > Advanced settings > Terminal > umbrelOS
nano umbrel/app-data/... --> sudo nano ~/umbrel/app-data/...

I agree with you in terms of updates. Maybe there should be a detailed description of what is done by an update?

I agree with you. We are redoing the apps framework and will make sure to have detailed descriptions for this new framework.

Under the current framework, an app update copies over the following dirs/files:
https://github.com/getumbrel/umbrel/blob/49d37581c8233f808fc5a79c2cd38810c6cd1b0b/packages/umbreld/source/modules/apps/legacy-compat/app-script#L594-L595
docker-compose.yml *.template exports.sh torrc hooks/ umbrel-app.yml

Any other files/dirs are not copied over on update. As an example, the data directory in this mosquitto submission won't be copied over meaning that a users data/settings won't get clobbered when they update.

As far as I can see, the hook used is not needed too.

That's correct. If all you are doing in the hook is creating the config, then you can just delete the entire hooks/ directory and rename the default config file to mosquitto.conf.default.

In general, headless services should be given the opportunity to display a webpage by umbrel or the app-proxy. In my opinion it is somewhat cumbersome to build a dedicated container for each of this app.

We're redoing the apps framework, so will take your suggestion under advisement 🤝.

The idea with the current framework is that an app should not be tied to umbrelOS. The manifest (umbrel-app.yml) is umbrelOS-specific, but the actual app containers themselves are not. The app_proxy is providing transparent proxying and auth (password, 2fa). If the app_proxy were to provide a way for headless services to include a webUI then we would be dictating how an app needs to be developed (umbrelOS-specific) instead of just having services be containerized and thus doing anything they want to create a UI.

App functionality

Can you please do some additional testing to make sure this is working correctly for you? Unfortunately my time is limited for the next while, so I can't spend too much time troubleshooting.

I just quickly tested mosquitto as of your commit ac6824c. The webUI works for me, but I cannot connect via MQTTX Web using port 9991. This is because your compose file is exposing 9991 on the host and binding to internal port 9001 in the container. So when you try to connect to something like http://umbrel.local:9991 you'll hit the internal container port 9001 which isn't associated with a listener in your mosquitto.conf.

What you can do instead is simplify the exposed ports since what the app_proxy container does is proxy requests directly to a port INSIDE the broker container. There is no need to expose 9991. The app proxy will upgrade the request to ws automatically. See my screen-recording below for the changes to make:

mosquitto.mp4

We may need to whitelist certain routes in the app proxy like /mqtt if external (to umbrelOS) applications need to access.

Before considering implementing this though, can you please first test using your commit ac6824c to make sure that I have not made a mistake while briefly playing around.

renamed: data/config/mosquitto.conf.default -> data/config/mosquitto.conf modified: data/config/mosquitto.conf modified: data/www/index.html modified: docker-compose.yml modified: umbrel-app.yml

modified: data/www/index.html modified: docker-compose.yml modified: umbrel-app.yml

removed unnecessary port-mapping

dirstel · 2024-12-10T15:50:42Z

Sorry for the confusion about the port(mapping)s - I'm glad, the app_proxy is able to serve the web page AND the mqtt-via-webservice as well at the same time. This takes away some confusing entries.

removed hook as it is not needed + downgraded manifestVersion to 1 as no hooks needed
removed port mapping in docker container: webpage and mqtt via webservice is handled and exposed by app_proxy
added information about configuring authentication + corrected typos in web page
successfully tested access to web page
successfully tested access via Port 1883 and mqtt-protocol
successfully tested access via Port 9021 and mqtt-via-webservice

github-actions · 2024-12-18T11:32:21Z

⚠️ Linting finished with 1 warning ⚠️

Thank you for your submission! This is an automated linter that checks for common issues in pull requests to the Umbrel App Store.

Please review the linting results below and make any necessary changes to your submission.

Linting Results

Severity	File	Description
ℹ️	`mosquitto/docker-compose.yml`	External port mapping "1883:1883": Port mappings may be unnecessary for the app to function correctly. Docker's internal DNS resolves container names to IP addresses within the same network. External access to the web interface is handled by the app_proxy container. Port mappings are only needed if external access is required to a port not proxied by the app_proxy, or if an app needs to expose multiple ports for its functionality (e.g., DHCP, DNS, P2P, etc.).
⚠️	`mosquitto/umbrel-app.yml`	"icon" and "gallery" needs to be empty for new app submissions: The "icon" and "gallery" fields must be empty for new app submissions as it is being created by the Umbrel team.

Legend

Symbol	Description
❌	Error: This must be resolved before this PR can be merged.
⚠️	Warning: This is highly encouraged to be resolved, but is not strictly mandatory.
ℹ️	Info: This is just for your information.

nmfretz · 2024-12-18T11:35:21Z

Excellent, thanks for making those changes and thoroughly testing @dirstel 🤝. I have tested as well and everything is working nicely.

The gallery assets are complete and we're going live. Thanks for all your efforts bringing this second app to the store!

dirstel added 14 commits November 11, 2024 17:12

Create .gitkeep

cc90fb4

Create .gitkeep

a4f4d30

Create pre-start

48d7b2d

Create mosquitto.conf.default

d35afb6

Delete mosquitto/config/.gitkeep

01ebda9

Create docker-compose.yaml

55110d8

Create umbrel-app.yaml

f356642

Create .gitkeep

85567c2

added www. dir

7520983

Delete mosquitto/www/.gitkeep

520827f

Create .gitkeep

dc7298e

Update umbrel-app.yaml

d8b0288

added version

Update docker-compose.yaml

ba47b04

added v2.0.20 incl. hash

Merge pull request #1 from getumbrel/master

9e702ed

sync

dirstel added 3 commits November 14, 2024 09:39

chmod +x for pre-start hook

5284e42

Update index.html

907af39

fixed typo in tag <strong>

Update pre-start

caba267

fixed permissions

nmfretz reviewed Nov 28, 2024

View reviewed changes

dirstel added 3 commits November 28, 2024 14:28

modified: data/config/mosquitto.conf.default

782b0e0

modified: data/www/index.html modified: docker-compose.yml modified: hooks/pre-start modified: umbrel-app.yml

modified: docker-compose.yml

746f4ef

dirstel added 5 commits November 29, 2024 07:46

inital commit

410202f

added zigbee2mqtt/data/configuration.yaml.default added zigbee2mqtt/docker-compose.yml added zigbee2mqtt/hooks/pre-start added zigbee2mqtt/umbrel-app.yml

Delete zigbee2mqtt/umbrel-app.yml

6255a01

Delete zigbee2mqtt/hooks/pre-start

9c60781

Delete zigbee2mqtt/docker-compose.yml

1d27444

Delete zigbee2mqtt/data/configuration.yaml.default

ac6824c

deleted: hooks/pre-start

fa35a00

renamed: data/config/mosquitto.conf.default -> data/config/mosquitto.conf modified: data/config/mosquitto.conf modified: data/www/index.html modified: docker-compose.yml modified: umbrel-app.yml

dirstel added 4 commits December 10, 2024 15:30

modified: data/config/mosquitto.conf

541f32b

modified: data/www/index.html modified: docker-compose.yml modified: umbrel-app.yml

Update docker-compose.yml

55f2431

removed unnecessary port-mapping

modified: data/www/index.html

7233fbf

modified: data/www/index.html

516dc79

nmfretz added 2 commits December 18, 2024 21:28

Add gallery assets

9b529c3

Remove icon field for official app store launch

e30d202

nmfretz merged commit 7547975 into getumbrel:master Dec 18, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

App submission mosquitto #1789

App submission mosquitto #1789

dirstel commented Nov 11, 2024 •

edited

Loading

nmfretz commented Nov 13, 2024

nmfretz left a comment

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

nmfretz Nov 28, 2024

dirstel commented Nov 28, 2024

nmfretz commented Dec 10, 2024 •

edited

Loading

dirstel commented Dec 10, 2024

github-actions bot commented Dec 18, 2024

nmfretz commented Dec 18, 2024

App submission mosquitto #1789

App submission mosquitto #1789

Conversation

dirstel commented Nov 11, 2024 • edited Loading

App Submission

App name

256x256 SVG icon

Gallery images

I have tested my app on:

nmfretz commented Nov 13, 2024

nmfretz left a comment

Choose a reason for hiding this comment

Package the UI as a separate Docker image

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dirstel commented Nov 28, 2024

nmfretz commented Dec 10, 2024 • edited Loading

App functionality

dirstel commented Dec 10, 2024

github-actions bot commented Dec 18, 2024

⚠️ Linting finished with 1 warning ⚠️

Linting Results

Legend

nmfretz commented Dec 18, 2024

dirstel commented Nov 11, 2024 •

edited

Loading

nmfretz commented Dec 10, 2024 •

edited

Loading