[v2.x] Add logging/notification/alerting facility to BMC

[srcshelton]

Is your feature request related to a problem? Please describe.
As an end-user, I would like to know what events the BMC has observed. Such events might include:

• Node powered on or off via WebUI;
• Node powered on or off via API;
• Node which was powered on turns itself off (if we can detect this?);
• BMC SSL certificate near to expiry;
• BMC SSL certificate expired;
• BMC rebooted/restarted cleanly;
• BMC power lost;
• BMC firmware updated;
• Node firmware updated;
• Possibly, log changes in the number of bad/spare NAND blocks?
• Allow the BMC to act as a remote log destination for nodes (or other network services?) to provide a common log collection point;
• Allow the BMC to forward all log messages to a remote syslog server.

Due to concerns over frequent writes to the BMC filesystem, logs could be held in memory and only committed to storage at specified intervals, or the feature could only be allowed when an SD card is mounted to provide external storage.

Describe the solution you'd like
Logging and alerting capabilities added to the BMC and exposed in the WebUI.

[svenrademakers]

I like this, and the possibilities to polish such a feature are endless. But I think there are two separate components here: Logging and Tracing.

As a starter, I considered focusing on a trivial logging component that logs to syslog instead of stdout. Syslog comes with log rotation. Then, in the UI, we can dump the syslog buffer in a text field. The upside to this is we can also display system logs. The downside is that logs are cluttered with noise.

What you are describing sounds more like tracing to me. I know there is quite a popular tracing crate, but I have yet to use it.

[srcshelton]

Additionally, it would be great to expose/log changes in any environment-monitoring data that exists on the BMC, but also provide a simple per-node service which pushes the node environment data (temperature, voltages, perhaps even clock-speed) to the BMC as the aggregator/single-source-of-truth.

As a more ambitious extension, it would be great if the BMC WebUI were then able to graph all of this data!

[soxrok2212]

Many other embedded systems also log to a tmpfs by default for live debugging. I’d love to see this even without a persistent storage option.

[j0ju]

I would scope this issue later to logging only: notification and alerting are two seperate topics.

I personally modify the syslogd/klogd usage in the BMC to log to memory only ring-buffer (which needs to be enabled in buildroot during compile time.)
I also add support for remote logging in busybox-syslogd. This allows me to send the logs to a central server, e.g. loki, rsyslog or similar.
With the in memory ring-buffer, busybox brings logread for reading the logs.
If the bmcd is logging to syslog, too, everything would be in one central place.

#!/bin/sh

BUSYBOX=/bin/busybox

SYSLOG_OPTS="-C256"
#SYSLOG_OPTS="-C256 -R[2001:db8::514] -L"
KLOG_OPTS=""

[ ! -r /etc/default/syslogd ] ||
  . /etc/default/syslogd

start() {
  stop
  $BUSYBOX syslogd -n $SYSLOG_OPTS &
  echo $! > /var/run/syslogd.pid
  $BUSYBOX klogd -n $KLOG_OPTS &
  echo $! >> /var/run/klogd.pid
}

stop() (
  kill $(cat /var/run/syslogd.pid /var/run/klogd.pid 2> /dev/null) 2> /dev/null
)

restart() { stop; start; }
reload() { restart; }

case "$1" in
  start | restart | reload | stop ) "$@" ;;
esac

# ts=2 sw=2 et ft=sh

[svenrademakers]

@j0ju i would like to integrate a syslog daemon like you mentioned. I think its too soon for the upcoming release that comes around the same time as the v2.5 boards. But i will start on this soon anyway

[svenrademakers]

if there is more input on this from anyone, feel free to dump it here