HotSync aims to reduce downtime in case of failure, syncing your NethServer with another one, that will be manually activated in case of master server failure.
Normally, when a hardware damage occurs, the time needed to restore service is:
- fix or buy another server: from 4h to 2 days
- install the OS: 30 minutes
- restore backup: from 10 minutes to 8 hours
In summary, users are able to start working again with data from the night before failure after a few hours/days. Using HotSync, time 1 and 3 are 0, 2 is 5 minutes (time to activate spare server). Users are able to start working again in few minutes, using data from a few minutes before the crash.
By default all data included in backup are synchronized every 15 minutes. MariaDB databases are synchronized too, unless databases synchronization isn’t disabled. Applications that use PostgreSQL are synchronized (Mattermost, Webtop5) unless databases synchronization isn’t disabled.
Two server machines are needed:
- MASTER: active server on production
- SLAVE: warm backup server
For a correct restore, it’s suggested to configure HotSync on two identical servers or two servers with same network cards number, name and position. If the master and slave servers differ, the restore procedure may behave unexpectedly (see Troubleshooting section on NethServer docs).
MASTER machine is the currently production server and it makes a backup in the SLAVE machine every "n" minutes. When a crash occurs on the master, slave becomes active taking the same IP address of the master. Doing so the downtime is minimal, only the time to launch some commands on the SLAVE.
On both MASTER and SLAVE, install nethserver-hotsync:
# yum install nethserver-hotsync --enablerepo=nethforge
MASTER configuration:
[root@master]# config setprop rsyncd password <PASSWORD> [root@master]# config setprop hotsync role master [root@master]# config setprop hotsync SlaveHost <SLAVE_IP> [root@master]# signal-event nethserver-hotsync-save
SLAVE configuration:
[root@slave]# config setprop rsyncd password <PASSWORD> [root@slave]# config setprop hotsync role slave [root@slave]# config setprop hotsync MasterHost <MASTER_IP> [root@slave]# signal-event nethserver-hotsync-save
The <PASSWORD> must be the same on both master and slave.
If MySQL or PostgreSQL are installed, they will be synchronized by default. To disable databases synchronization:
[root@master]# config setprop hotsync databases disabled [root@master]# signal-event nethserver-hotsync-save
You can update these settings also from Cockpit interface.
The following procedure puts the SLAVE in production when the master has crashed.
switch off MASTER
if the SLAVE machine must run as network gateway, connect it to the router/modem with a network cable
following command changes the IP address and cuts off the network connection. If you are connected through an ssh console, launch the
screencommand[root@slave]# screen
on SLAVE launch command, and read carefully its output
[root@slave]# hotsync-promote
If no Internet connection is detected (e.g. you are restoring a firewall on a machine that was passing through crashed master for Internet connection), the scripts will purpose you some options:
- Restore master network configuration (IMPORTANT: use this option only if two servers are identical - NIC number, names and positions must be identical)
- Fix network configuration from Cockpit GUI (when restoring on different hardware)
- Continue without internet: assign correct roles before proceed with this option. Some events could fails (not recommended)
If necessary go to Server Manager or Cockpit GUI, in page Network and reassign roles to network interfaces as master one. Remember also to recreate bridge if you have configured DC. In case of DC errors consult troubleshooting section before proceed with network restore.
launch command
[root@slave]# /sbin/e-smith/signal-event post-restore-data
if an USB backup is configured on MASTER, connect the backup HD to SLAVE
To put again in production original crashed server:
configure it as SLAVE
synchronize it
switch off current MASTER
from bash execute:
# hotsync-slave
restore configuration backup following above instructions
It is possible to customize HotSync adding all kind of data through the use of plugins. The directory of plugins is /etc/hotsync.d. Executable files in this directory, are executed before the synchronization.
INCLUDE_FILE and EXCLUDE_FILE are the files that contain the list of paths to include and to exclude to/from HotSync. Those two variables are passed as arguments to the scripts of this directory when they are executed.
If you want to add files to the synchronization, append them to the INCLUDE_FILE. Append them to EXCLUDE_FILE to remove from the synchronization.
From MASTER bash launch the command:
# hotsync
"hotsync-slave" script extracts from MASTER configuration backup the list of packages to install and install them. You can force the operation executing bash command:
# hotsync-slave
is a shell script launched by cron every 15 minutes
uses a lockfile to ensure that only one instance at a time is executed
uses secure communication with rsync over stunnel
creates a list of files to be included and another one to be excluded from rsync (using backup-data configuration and hotsync own logic)
launch backup on MASTER
launch an rsync that copy listed files from MASTER to SLAVE in a secure manner using stunnel
if something fails, root is notified with an email
you can check files that will be copied on the next synchronization using the command:
hotsync --dry-run
Automatically executed on SLAVE every 60 minutes, extracts from MASTER configuration backup the list of packages to install and install them.
- nethserver-antivirus
- nethserver-backup-config
- nethserver-backup-data
- nethserver-base
- nethserver-c-icap
- nethserver-cockpit
- nethserver-collectd
- nethserver-cups
- nethserver-dante
- nethserver-dc
- nethserver-dedalo
- nethserver-directory
- nethserver-dnsmasq
- nethserver-duc
- nethserver-ejabberd
- nethserver-evebox
- nethserver-fail2ban
- nethserver-firewall-base
- nethserver-freepbx > 14.0.3
- nethserver-httpd
- nethserver-hylafax
- nethserver-iaxmodem
- nethserver-ipsec-tunnels
- nethserver-janus
- nethserver-letsencrypt
- nethserver-lightsquid
- nethserver-mail
- nethserver-mattermost
- nethserver-mysql
- nethserver-ndpi
- nethserver-netdata
- nethserver-nextcloud
- nethserver-ntopng
- nethserver-nut
- nethserver-openssh
- nethserver-openvpn
- nethserver-pulledpork
- nethserver-restore-data
- nethserver-roundcubemail
- nethserver-samba
- nethserver-samba-audit
- nethserver-squid
- nethserver-squidclamav
- nethserver-squidguard
- nethserver-sssd
- nethserver-subscription
- nethserver-suricata
- nethserver-vpn-ui
- nethserver-vsftpd
- nethserver-webtop5 (z-push state is not synchronized)
Packages nethserver-ntopng and nethserver-evebox are reinstalled without migrating history.
Please read the docs before proceed with restore.