High availability with IP fail-over
For critical services, high availability can be provided through node duplication with an IP fail-over mechanism. This can be implemented with a virtual IP address (VIP) and the Keepalive daemon as described below. For example, the bastion host is a function that is recommended to be implemented with high availability.
VRRP (Virtual Router Redundancy Protocol) allows a number of hosts sharing a (virtual) IP address. One of the hosts is set to be master node and, during normal conditions, is the only host that will respond on the VIP. The other nodes are backup nodes and monitor the master node periodically to ensure that it is still running. If the master node goes down, one of the backup nodes take over the role as master and starts replying on the VIP. This facilitates configurations without any single point of failure at node level.
Initially, we have two servers with private IP addresses
<server2-ip> and a public (floating) IP address
Create VIP port
The first step is to create a port (here called
vip-port) on the desired network and without any security group, and assign
<floating-ip> to it. The port is created with
openstack port create --network <internal-network> --no-security-group vip-port
The private IP of this port is the virtual IP address (VIP)
<virtual-ip>. This can be read in the output from
openstack port list
Assign the available floating IP address to the port with
openstack floating ip set --port vip-port <floating-ip>
The chosen network can be created and dedicated for VRRP, or an existing internal network.
Update server ports
Update the server’s network ports using the allowed-address attribute set to the VIP
<virtual-ip>. To find the port resource ID of a server, list all ports with
openstack port list
and note the IDs of the ports having the IP addresses
<server2-ip>, respectively - here denoted
<port2-id>. Update the ports with
openstack port set --allowed-address ip-address=<virtual-ip> <port1-id>
openstack port set --allowed-address ip-address=<virtual-ip> <port2-id>
By allowing this address pairing with the VIP, enables
keepalived to listen to VRRP.
Configure Keepalive daemon
keepalived is used for high availability architectures. It performs process monitoring, and contains a VRRP stack and low-level supporting functions. The daemon is installed on the remote servers with
sudo apt install keepalived
It needs a configuration file with details such as IP addresses and interface name. The interface names on the servers can be found by listing them with
In the present Ubuntu configuration, the interface is named
ens3. Letting server 1 be the master node, login to server 1 and create the file
sudo nano /etc/keepalived/keepalived.conf
and copy in the content
The daemon is started and enabled with
sudo systemctl enable --now keepalived
keepalived on server 2 and configure it as backup node with
changing the VRRP instance name, priority, the unicast source and the unicast peer IP addresses accordingly. To verify that it is running, do
systemctl status keepalived
and to see that it has been enabled, enter
systemctl list-unit-files | grep enabled
which will list all enabled processes (Figure 1).
ip addr show <interface-name>
now shows both the private and the virtual IP addresses (in Figure 2 the addresses
After making changes in the configuration file, the service needs to be restarted with
sudo systemctl reload keepalived
syslog also contains information related to the daemon activities.
Testing of the IP fail-over amounts to ensuring operation in a simulated server failure scenario. The simplest test is pinging the public IP associated with the virtual IP address, and disable the master server, say server 1, with
openstack server stop <server1-name>
Pinging again should still yield a response, now generated by the backup server. Restore operation with
openstack server start <server1-name>.
To identify the responding back-end server, test web servers can be installed on server 1 and 2 as described in https://pannet.atlassian.net/l/c/HQSvC5Hh, and test with
before and after disabling the master server.