FCGIwrap Overhaul: Unified Socket Activation

[iliajie]

Hey Jamie!

As we discussed earlier in the chat, it would be better and simpler to use socket activation for FCGIwrap.

I’ve spent some time looking into this and searching for the best approach, and here’s what I’ve found.

---

Please check these suggested changes:

1. During Virtualmin installation (package) time we create:

   1.1. The file named [email protected] in /usr/lib/systemd/system with the following content:

   [Unit]
   Description=Virtualmin FCGIwrap server for %i user
   After=nss-user-lookup.target
   
   [Service]
   EnvironmentFile=/etc/webmin/virtual-server/fcgiwrap
   ExecStart=/usr/sbin/fcgiwrap ${DAEMON_OPTS} -c ${DAEMON_PROCS}
   User=%i
   Group=%i
   StandardInput=socket
   StandardOutput=journal
   StandardError=journal
   RuntimeMaxSec=300
   
   [Install]
   Also=fcgiwrap@%i.socket
   

   1.2. File named [email protected] in /usr/lib/systemd/system with the following content:

   [Unit]
   Description=Virtualmin FCGIwrap socket for fcgiwrap-virtualmin@%i.service
   
   [Socket]
   ListenStream=/run/fcgiwrap/%i.sock
   RuntimeDirectory=fcgiwrap
   SocketUser=%i
   SocketGroup=%i
   SocketMode=0660
   
   [Install]
   WantedBy=sockets.target
   

   1.3. File named fcgiwrap in /etc/webmin/virtual-server/ with the following content:

   # fcgiwrap configuration parameters
   
   # Specify the number of fcgiwrap processes to prefork
   DAEMON_PROCS=1
   
   # Specify additional daemon options. See man fcgiwrap.
   DAEMON_OPTS=-f
   

2. We stop creating individual fcgiwrap-domain.com.service files as we do now;

3. We slightly change the socket name in Apache/Nginx to include domain owner username only:

SetHandler proxy:unix:/var/fcgiwrap/username.sock/socket|fcgi://localhost

4. We activate (enable and start) the socket (only!) in the following way:

systemctl enable --now [email protected]

---

Disabling FCGIWrap for a specific domain can be done as simply as:

systemctl disable --now [email protected] [email protected]

---

With templated systemd units for FCGIwrap, we’re making things way easier by cutting out the need for separate service files for each domain. Instead, socket activation kicks in, starting FCGIwrap only when it’s needed, which saves resources; plus, the RuntimeMaxSec=300 setting ensures idle services don’t hang around forever, wasting system resources. The clear, consistent socket names tied to usernames also make management simpler.

[swelljoe]

I'll test this tonight on software.virtualmin.com, so we can see if maybe it resolves the "hanging" issue we've seen with the old long-running service. It looks great.

[iliajie]

Great! Thanks, Joe! That will be really helpful!

[iliajie]

1. The file named [email protected] in /usr/lib/systemd/system with the following content:
2. 1.2. File named [email protected] in /usr/lib/systemd/system with the following content:

As we just discovered with Joe, these file names will conflict with the packaged version of fcgiwrap on Rocky and Alma, so it’s better to use a different name, like fcgiwrap-virtualmin.

Also, Jamie, why didn’t we use /run/fcgiwrap in the first place and opted for /var/fcgiwrap instead?

[jcameron]

Oh this is very interesting, I didn't realize that templeatable systemd services could exist! But does this still mean that we need to create a socket for each domain, and also a per-domain fcgiwrap@$user.socket service for each domain?

Also, is the advantage of this over the current approach that the fcgiwrap server only gets launched when it is needed?

[swelljoe]

You don't create anything. You just enable/start it. Only one file for the socket and one for the service ever exists, no matter how many users get an instance of it.

The advantage of template units is we can ship one socket and one service unit file in our packages, and never touch them again, and Virtualmin doesn't need to create/manage those files. Just start/stop/enable/disable the service as appropriate. The templated units can still be managed like actual individual units, they just don't need their own unit files. This is the right way to do user owned services and sockets in systemd; the way we're currently doing it is pretty messy (lots of files, generated and managed by Virtualmin, when we should be outsourcing as much process management as possible to systemd).

The advantage of socket activation is that it doesn't start fcgiwrap until needed, the process is completely managed by systemd, and it may fix the problem with fcgiwrap freezing up periodically. The fcgiwrap documentation uses socket activation, and the RPM package includes a socket-activated system configuration. I believe we can say with confidence the author of fcgiwrap would like it to be socket-activated. It's been running socket-activated on software.virtualmin.com for about 24 hours without failure (without a cronjob restarting it), which may or may not mean it fixes the problem.

Even if it doesn't fix the periodic freezing problem (which I'm hopeful it will), this is still a cleaner implementation that takes advantage of features of systemd to make it easier to use and more in line with how everybody else is doing things (many services work this way, so it is familiar and documented and will become more familiar as people get more experience with systemd).

We should know in another couple days if the freezing problem is fixed. I think with current traffic we were freezing at least once a day before, so we should have seen it already, but I won't commit to saying it's definitely fixed for at least another couple days.

[jcameron]

OK I think it should be pretty simple to implement this, as the Apache-side config is still mostly the same. Let me know how it goes on virtualmin.com, and I'll look into adding support for it in Virtualmin.

[iliajie]

I've been running additional cross-platform tests and made a few small fixes, specifically adding the EnvironmentFile option in an additional step 1.3 to make it configurable. The template file names should be named as [email protected] and [email protected].

I think it should be pretty simple to implement this, as the Apache-side config is still mostly the same. Let me know how it goes on virtualmin.com, and I'll look into adding support for it in Virtualmin.

The socket in webserver should be called as domain owner user for both top-level and sub-server domains. The sub-domains will use the logs path for the main user.

However, sadly it seems impossible not to hardcode /home for logs, because %h doesn't resolve to user's home set in User=. Also using environmental variables in StandardOutput or StandardError doesn't work. $HOME is set and even if I set it in /etc/webmin/virtual-server/fcgiwrap it still doesn't work if used like:

StandardOutput=file:${HOME}/logs/fcgiwrap.log
StandardError=file:${HOME}/logs/fcgiwrap-error.log

The reason for this, I believe, is that logs are written as the root user by default, and %h expands to /root. It isn't clear how to write logs as a specific user with systemd.

Most probably, the best and cleanest solution would be to write logs to the journal, like this:

StandardOutput=journal
StandardError=journal

I haven’t changed it yet, but let’s discuss it.

[swelljoe]

I would rather we use the journal by default. We should try to be like the systems we support, and they've all moved most stuff to the journal. And, lots of good stuff falls out of the decision for free...if you're ingesting logs into something like Grafana Loki or Splunk or whatever, you don't have to micromanage it, you just grab the journal.

[iliajie]

Agreed. I modified my initial suggestion.

[iliajie]

Where are we at with this? What’s the plan moving forward?

[jcameron]

Are we confident enough in this CGI mode that I should add support for it in Virtualmin?

[iliajie]

I think we do. It’s much simpler and more reliable this way.

I don’t think we need to overcomplicate it either. We can pre-create those systemd-related files somewhere in the virtual-server directory, and the post-install script can deploy them if it’s a systemd system. By deploy, I mean simply copying the systemd FCGIwrap config files to the necessary directories mentioned earlier or even creating a symlink. It’s just two files…

[jcameron]

Ok I'll take a look at this for a future release

[iliajie]

↑

[jcameron]

Now that 7.30.0 is out, I will take a look at this.

[iliajie]

Great, thanks! Please let me know if you have any questions about FCGIWrap templates.

[iliajie]

I have updated the original instructions to use RuntimeDirectory.

[iliajie]

I'm bumping this up so it won't be buried.