Automatic base URLs for each web service

[mtekman]

Motivation

I currently have the following 4 web services:

• Gitea (running on port 5583)
• Home Assistant (running on port 5961)
• Pi-hole (running on port 8080)
• Jellyfin (running on port 8097)

To access each of these, I have to type in http://<my.local.ip>:<port> (e.g. http://local.home:5961) for me to access these services.

It would be easier if I could simply type in http://<my.local.ip>/<service> (e.g. http://local.home/homeassistant).

Further, if I expose my Dietpi device via a reverse proxy, I would only need to forward a single port, with the baseURL → port mappings happening on the Dietpi device instead of the reverse proxy device.

Implementation

This can be done in lighttpd with a custom rule:

# /etc/lighttpd/conf-enabled/99-my-custom.conf
server.modules += ( "mod_proxy" )

$HTTP["url"] =~ "^/gitea" {
    proxy.server = ( "" => ( ( "host" =>  "127.0.0.1", "port" => 5583 ) ) )
}
$HTTP["url"] =~ "^/pihole" {
    proxy.server = ( "" => ( ( "host" =>  "127.0.0.1", "port" => 8080 ) ) )
}
$HTTP["url"] =~ "^/homeassistant" {
    proxy.server = ( "" => ( ( "host" =>  "127.0.0.1", "port" => 5961 ) )
}
$HTTP["url"] =~ "^/jellyfin" {
    proxy.server = ( "" => ( ( "host" =>  "127.0.0.1", "port" => 8097 ) ) )
}

but... it doesn't quite work, and I see other complicated rules in the same directory.

I'm sadly not versed enough to debug this problem properly, but I think it could be a major win for this platform if if such services were structured like this already.

Caveats

I do realise that each application would need configuring to be compatible with this setup, and that it would be a large undertaking for anyone submitting a new web service software to the software list.

[MichaIng]

Discussions around this topic came up a lot and you already mentioned the major problem:

I do realise that each application would need configuring to be compatible with this setup, and that it would be a large undertaking for anyone submitting a new web service software to the software list.

Each application would need an individual webserver config for each of the 3 webservers we offer. Lighttpd is probably the most difficult one, as it, in our experience, requires significantly more tinkering and directives compared to Nginx and Apache. It seems to not forward/set headers a way that is commonly needed. And yes: Every new web application would again need to additional proxy config for each webserver and any new webserver we want to add would require an additional proxy config for each web application. I am already aiming to implement Caddy for some years, and what makes it such a hard undertaking is only the essential webserver configurations for those web applications which do use the webservers (i.e. no the ones which listen on their own ports). If now there comes a proxy config for each standalone web application on top, the tast becomes impossible without a significant team systematically working on it, i.e. a paid project task would be the only thing I could see this done.

The idea we had so far to approach this topic:

• Find or write and test proxy configs for those web applications and web servers you use.
• Add them to the respective software documentation in our docs, e.g. here for Gitea: https://dietpi.com/docs/software/cloud/#gitea
• Starting from the other end, a generic proxy guide for the 3 webservers could be written, as a starting point: Add reverse proxy guide DietPi-Docs#404
  For Lighttpd it could look like your example, with a note that individual backend apps require individual additions, on their own config as well as on the proxy config (headers to forward or reset). If there is a pattern, e.g. a directive commonly required in the Lighttpd proxy block, it can be added there as well.
• From the web application docs (like Gitea) we can add a link to the generic webserver-based proxy docs, until we have individual/Gitea proxy configs for each webserver.

When the docs have been extended by this and it has been tested/proved for a while by some users, we can start adding it as option (definitely not mandatory!) to individual web application installs in dietpi-software. So it would not be consistent for every web app, but more for those where proven proxy configs for all 3 webservers have been found, not implying the need to have this done/extended for every new web app or webserver.

[mtekman]

Lighttpd is probably the most difficult one, as it, in our experience, requires significantly more tinkering and directives compared to Nginx and Apache. It seems to not forward/set headers a way that is commonly needed.

At your comment, I've switched lighttpd to nginx and can definitely vouch that the configuration is significantly easier and "just works", so thanks!

Every new web application would again need to additional proxy config for each webserver and any new webserver we want to add would require an additional proxy config for each web application. I am already aiming to implement Caddy for some years,

Ah yep, this was a thought at the back of my mind I didn't quite voice. Maybe it would be good if DietPi could become super opinionated about the web server it uses to simplify the task?

Or barring that, allow for complete webstack freedom of choice (as it is now), but use a dedicated Caddy process purely for binding baseURLs to subdomains.

The setup with Caddy looks really nice:

Once Caddy is running as a service, you can change the config file to be a reverse proxy. There is no 1 config that works for everyone, so please, customize this as necessary:

example.com

reverse_proxy /service-a/* 192.168.0.2:8001
reverse_proxy /service-b/* 192.168.0.2:8002

We could then simply add a line for each service in Dietpi! Caddy seems like a really elegant solution to this.

I am already aiming to implement Caddy for some years, and what makes it such a hard undertaking is only the essential webserver configurations for those web applications which do use the webservers (i.e. no the ones which listen on their own ports). If now there comes a proxy config for each standalone web application on top, the tast becomes impossible

I'm a bit lost here, surely we just need to add a single line to the Caddy config for each webserver?

Starting from the other end, a generic proxy guide for the 3 webservers could be written, as a starting point: MichaIng/DietPi-Docs#404
For Lighttpd it could look like your example, with a note that individual backend apps require individual additions, on their own config as well as on the proxy config (headers to forward or reset). If there is a pattern, e.g. a directive commonly required in the Lighttpd proxy block, it can be added there as well.

Yeah, a proxying guide in the official docs would be great! The forums are a great resource, but I had to dig quite a bit to find a solution that worked for me

[mtekman]

Related and super helpful guide:

https://caddy.community/t/the-subfolder-problem-or-why-cant-i-reverse-proxy-my-app-into-a-subfolder/8575

[MichaIng]

Ah yep, this was a thought at the back of my mind I didn't quite voice. Maybe it would be good if DietPi could become super opinionated about the web server it uses to simplify the task?

Not sure what you mean with "super opinionated"? You mean offering a single webserver/proxy software only for this feature? Indeed that would make it easier, or at least is a good start and can be extended if someone finds the time to implement more. Generally I think this is something to better break down into smaller doable tasks; one backend after another, in case one webserver/proxy software after another.

Or barring that, allow for complete webstack freedom of choice (as it is now), but use a dedicated Caddy process purely for binding baseURLs to subdomains.

Before thinking about using Caddy as proxy, we need to implement Caddy in general 🙂. Caddy is not significantly easier or harder in average to setup as proxy, compared to Apache or Nginx. All of them allow to setup a reverse proxy with a single config line, as long as the backend software natively supports it. A more pure proxy choice would be btw HAProxy or frp, which we have both implemented already. All of them allow to proxy via subdomain and sub path, the problem really is the individual needs of the backend and whether it supports base paths or not, as explained in the guide you linked. Sub domains are indeed a good and common option to eliminate most difficulties, but it requires related DNS entries and HTTPS certificates, so is not feasible in all cases.

[mtekman]

"super opinionated"

Yeah a single proxy-to-backend service, and an optional guide for how to get server-to-proxy working. But after getting deeper into the topic and reading your comments, I can see a bit more clearly that some services might need their headers rewritten, others let you set the baseurls, and I'm sure many other complicated parameters particular to each service.

Caddy

N=1: I'm finding Caddy generally much easier to configure than either nginx or lighttpd/apache, as it sets a lot of the needed parameters automatically.

HAProxy or frp

Ah okay. These I need to explore more deeply if they're good viable options. My current plan was to use caddy as server-to-proxy and caddy as proxy-to-backend, and I've got a nice simpl(ish) solution running for some apps, but not all.