Apache failure

[Changaco]

When I got up today the Liberapay website was down. As usual, manually restarting Apache fixed the problem, proving once again that Apache isn't resilient.

Error log (times are UTC):

[Tue Apr 07 00:46:34.353046 2020] [cgi:error] [pid 19317] [client 172.31.5.110:32374] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: http://liberapay.com/cgi-bin/libagent.cgi?type=J
[Tue Apr 07 00:46:36.228922 2020] [cgi:error] [pid 25148] [client 172.31.5.110:32392] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: https://liberapay.com/
[Tue Apr 07 00:46:36.934885 2020] [cgi:error] [pid 25149] [client 172.31.23.19:33580] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi
[Tue Apr 07 00:46:37.591394 2020] [cgi:error] [pid 24528] [client 172.31.5.110:32376] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: http://liberapay.com/cgi-bin/libagent.cgi?type=J
[Tue Apr 07 00:46:37.652239 2020] [cgi:error] [pid 21905] [client 172.31.5.110:32378] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: https://liberapay.com/
[Tue Apr 07 00:46:38.396692 2020] [cgi:error] [pid 25227] [client 172.31.37.154:19138] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: https://liberapay.com/
[Tue Apr 07 00:46:42.349172 2020] [cgi:error] [pid 25706] [client 172.31.37.154:19216] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: https://liberapay.com/
[Tue Apr 07 00:46:42.557455 2020] [cgi:error] [pid 25227] [client 172.31.37.154:19138] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi
[Tue Apr 07 00:46:44.510820 2020] [cgi:error] [pid 25508] [client 172.31.5.110:32464] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi, referer: https://liberapay.com/
[Tue Apr 07 00:46:44.932843 2020] [cgi:error] [pid 25176] [client 172.31.23.19:33722] AH02811: script not found or unable to stat: /var/www/cgi-bin/libagent.cgi
[Tue Apr 07 00:46:46.491278 2020] [mpm_prefork:error] [pid 12441] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Tue Apr 07 00:46:51.014615 2020] [core:error] [pid 19317] [client 172.31.5.110:32374] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.014670 2020] [core:error] [pid 21763] [client 172.31.5.110:32500] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018545 2020] [core:error] [pid 25509] [client 172.31.5.110:32460] End of script output before headers: main.py
[Tue Apr 07 00:46:51.018614 2020] [core:error] [pid 25467] [client 172.31.5.110:32448] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018662 2020] [core:error] [pid 26208] [client 172.31.23.19:33672] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018708 2020] [core:error] [pid 25308] [client 172.31.37.154:19152] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018759 2020] [core:error] [pid 25901] [client 172.31.23.19:33648] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018829 2020] [core:error] [pid 21906] [client 172.31.5.110:32380] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018875 2020] [core:error] [pid 25898] [client 172.31.5.110:32596] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.018921 2020] [core:error] [pid 25173] [client 172.31.5.110:32572] End of script output before headers: main.py
[Tue Apr 07 00:46:51.019060 2020] [core:error] [pid 26070] [client 172.31.5.110:32504] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.019174 2020] [core:error] [pid 26179] [client 172.31.5.110:32598] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.019291 2020] [core:error] [pid 23955] [client 172.31.5.110:32386] End of script output before headers: main.py, referer: https://liberapay.com/
[Tue Apr 07 00:46:51.019506 2020] [core:error] [pid 25230] [client 172.31.23.19:33626] End of script output before headers: main.py, referer: https://liberapay.com/

[Changaco]

This issue is very similar to #1405.

[Urahara]

Its not worth migrate to nginx?

[Changaco]

@Uraha Since the origin server is already behind Cloudflare and an AWS load balancer, and doesn't accept traffic from the Internet, nginx is probably not useful. Instead we would use gunicorn directly, like we've always done when running Liberapay locally for development.

[Changaco]

Happened again today, and some people noticed: https://diaspodon.fr/@dada/104035181281283115

[Changaco]

Closing as obsolete. (Apache was dropped in #2029.)