Configure security

Security can be implemented on different layers in a cloud deployment, and having a strict security policy enforced by multiple systems is strongly recommended. This guide contains an overview of security systems readily available on the DT Cloud Services cloud.

Contents

A security group consists of a set of filter rules defining which ports to allow traffic on, where all incoming traffic initially is blocked. The rules in security groups defines allowed inbound (ingress) or outbound (egress) traffic based on network addresses and ports. No traffic can be received by an instance unless a security group rule explicitly allows it.

By default, egress traffic is allowed on all ports, all ingress traffic is blocked and rules are defined to allow certain ingress traffic only. This strategy decreases the lookup workload on the Compute node.

The a security group called "default" is created at instance initiation, and it cannot be deleted. It is good practice not to edit the default security group, but define separate new security groups (that is, lists of IP filter rules) for different roles of the instances, such as front-end servers, back-end servers, database servers, etc.

For each new functionality or interface added to an instance, it will likely be necessary to adjust its security group rules. The rules should also be updated when an interface or role is removed or changed.

Security groups

An OpenStack security group acts as a virtual firewall used to protect servers on a network. It is manifested in a collection of rules allowing specific types of IP traffic from (or to) a set range of IP addresses and ports.

Enable remote access

To create a security group for SSH only, we first create a new security group and then add a rule for allowing TCP over port 22. A new security group is created by

openstack security group create <security-group-name>

To create a SSH security group called “ssh-only”, execute

openstack security group create ssh-only

Create a new security group rule with the OpenStack command

openstack security group rule create --dst-port <port-range> --protocol <protocol> --ingress <security-group-name>

The most commonly used command arguments are:

• protocol (default TCP) - referring to the IP protocol, such as: ah, dccp, egp, esp, gre, icmp, igmp, ipv6-encap, ipv6-frag, ipv6-icmp, ipv6-nonxt, ipv6-opts, ipv6-route, ospf, pgm, rsvp, sctp, tcp, udp, udplite, vrrp

• dst-port - destination port number as a single port or a port range, for example 137:139. The argument is required for TCP and UDP.

• ingress / egress - rules applying to inbound (default) or outbound traffic

• project - user project, that is the tenant name or id

The SSH rule is created with

openstack security group rule create --dst-port 22 --protocol tcp --ingress ssh-only

In Horizon, security groups and rules are defined under Network/Security Groups. Click the button Create Security Group to open a form where the security group name is required and a description can be added in the free text field. Clicking on Create Security Group creates it (Figure 1).

Figure 1. Form for creation of security group in Horizon.

In the Network/Security Groups window, each security group is listed with its name and (optional) description. To set rules, click on Manage Rules corresponding to the security group to be edited. Click on Add Rule to open a form to define a new rule, and Delete Rule to remove an existing rule from the security group (Figure 2). Note the default egress traffic rules created.

Figure 2. Modify a security group by adding or deleting rules.

Under Add Rule, there are many pre-defined traffic types, including SSH. Select the traffic to allow from the drop-down menu, and click Add (Figure 3).

Figure 3. Template for allowed traffic rule.

The newly created security group now looks like in Figure 4. Rules for other traffic types are defined in the same way.

Figure 4. Security group for SSH only.

A list of rules is obtained from the command-line by

openstack security group rule list

A large number of instances or rules to process in a security group can cause the creation of an instance to fail. It is therefore recommended to open egress communication on all ports unless there are good reasons not to do so. The number of maximum rules per security group are controlled by security_group_rules.

Enable ICMP ping

Filter rules need to have a specified Classless Inter-Domain Routing (CIDR) field. Setting CIDR to 0.0.0.0/0 allows traffic over the specified port for all IP addresses. The settings (for an example IP address) to restrict access to a specific IP address range are listed in Table 1.

_{Table 1. CIDR settings and corresponding rules.}

To enable ping (ICMP), we can create the rule and add it to ssh-only by

openstack security group rule create --protocol icmp --remote-ip 0.0.0.0/0 ssh-only

Enable HTTP and HTTPS

For many practical purposes, it is useful to have a security group for HTTP traffic (on port 80) as well. It is created, calling the security group “http-global”, by

openstack security group create http-global

and the corresponding rule with

openstack security group rule create --protocol tcp --dst-port 80:80 --remote-ip 0.0.0.0/0 http-global

Similarly, HTTPS traffic is allowed by the rule

openstack security group rule create --protocol tcp --dst-port 443:443 --remote-ip 0.0.0.0/0 http-global

Wide TCP and UDP rules

The servers should not be assigned security groups allowing protocols and traffic types that it does not need to use. However, for testing purposes and certain applications, wide TCP or UDP security groups can be useful. These rules are created by

openstack security group rule create --protocol tcp --remote-ip 0.0.0.0/0 --dst-port 1:65525 <security-group-name>
openstack security group rule create --protocol udp --remote-ip 0.0.0.0/0 --dst-port 1:65525 <security-group-name>

Assign security group

A security group can be assigned to an instance when created, or added to it later to a running instance from OpenStack client. The command

openstack server add security group <server> <group>

takes the arguments (ID or name) of the server and the security group. Valid values for <server> and <group> can be found by running

openstack server list

and

openstack security group list

Note that all relevant security groups need to be added to an instance.

Manage SSH users

SSH security policies differ from that of a physical remote server in that all configuration is done over SSH, and there is no local terminal available.

Users are authenticated through SSH keys rather than passwords. The default user has superuser status and should be protected, so it is recommended to create user accounts which can be given different levels of access. Creation of key pairs for general SSH access to the instance is described in https://pannet.atlassian.net/wiki/spaces/DocEng/pages/524321364/Configure+client#Create-key-pair-in-tenant

Create user

An SSH user is a normal Linux user with an SSH key. In the following procedure, a user account called guest will be created. User administration on a Compute instance needs to be performed by the superuser.

The guest user will be authenticated by a dedicated SSH key, generated on the client (the machine from which guest is supposed to log in) with

ssh-keygen -t rsa -b 4096 -f guest.key
ssh-add guest.key

which also generates the public key guest.key.pub that needs to be copied to the server. After logging in to the server as ubuntu (or any other superuser) over SSH, create a new user with

sudo adduser --disabled-password guest

This opens a dialog where details can be entered (Figure 5), or left blank.

Figure 5. Dialog shown by the adduser command.

Create and open the file /home/guest/.ssh/authenticated_keys with

sudo mkdir /home/guest/.ssh
sudo nano /home/guest/.ssh/authenticated_keys

The public key, that is, the entire content of the file guest.pub, is now copied into the file authenticated_keys. Save and close the file.

At this point it is essential that the permissions of the file is set correctly. Here, the file was created with root file ownership, and it is necessary to give read access to the system with

sudo chmod 0755 /home/guest/.ssh
sudo chmod 0644 /home/guest/.ssh/authenticated_keys

After exiting the SSH session as ubuntu, it should now be possible to login as guest with

ssh -i guest.key guest@<ip-address>

In case the server is behind a bastion host, create a new server-user entry in the file ~/.ssh/config on the client. If the guest user account was created on the server server1 and the bastion host is called bastion, the new entry would look something like

Host server1-guest
  IdentityFile guest
  User guest
  HostName 10.0.0.130
  ProxyJump bastion

To log in as guest on server1, execute

ssh server1-guest

in the client terminal.

The user guest account created does not have any password associated with it and it is not in the sudoers list, so from it you cannot perform any superuser tasks. The file authorized_keys can contain multiple public keys for the same user account. This removes the need to move private keys between client devices and provides a flexible method to provision user access.

A user can be given sudo rights by a superuser by opening the file /etc/sudoers with the command visudo and adding it in the format

<username> ALL=(ALL) NOPASSWD: ALL

Alternatively, it can be added directly from command-line with

echo “<username> ALL=(ALL) NOPASSWD: ALL” | sudo tee /etc/sudoers.d/<username>

which will create a separate file for <username>, which can be convenient when this user’s privileges needs to be modified or removed.

Restricted access levels

When there are multiple user accounts with separate authorized_keys files, access levels can be restricted for non-essential accounts by the forced execution of a script. To enable this, some action on the server is typically coded in a script file, say user_script.sh, and the authorized_keys file of the restricted user is modified by including

command=”./user_script.sh”

before the public key section.

For example, a minimal script that changes the directory to a user-specific location at login and then enables the bash terminal could be

command=”cd /var/logs ; bash”

Without the bash instruction, the connection would be closed after the change directory command has been executed.

The given command is a single forced command, so multiple choices have to be embedded in the script itself, for example in the script file server_monitor.sh below (adapted from Barrett et al.).

#!/bin/sh

/bin/echo "Server monitor
1       Print current date
2       Get status of SSH and HTTP connections
3       List running processes
q       Quit"

/bin/echo "Enter an item (1-3), or q to quit:"

read ans

while [ "$ans" != "q" ]
do
   case "$ans" in
      1)
         /bin/date
         ;;
      2)
         nmap `hostname` -PN -p ssh,http
         ;;
      3)
         /usr/bin/top
         ;;
      q)
         /bin/echo "Goodbye"
         exit 0
         ;;
      *)
         /bin/echo "Invalid choice '$ans': please try again"
         ;;
   esac
   /bin/echo "Enter an item (1-3), or q to quit:"

   read ans
done
exit 0

The file permissions need to so that the restricted user guest has right to execute the script (chmod 755), and if located in its home directory, the command is added to authorized_keys as

command=”./server_monitor.sh” ssh-rsa <key>

Do not do this for user ubuntu, since it will be locked out from any other operations.

The nmap utility shows the selected port status on the server (Figure 6).

Figure 6. Output from the nmap command.

If the provided argument is the entire network address range (for example 10.0.0.0/24), a list of all host ports in the network and their status is printed.

Access monitoring

In some situations, it is convenient to have notifications delivered by e-mail, for example when someone logs in to the bastion host. Shell scripts of this type can be added to the file /.bashrc, which will be executed each time a terminal is launched (that is, after a SSH session has been set up).

E-mail service

An e-mail dispatching service can implemented based on mailutils and ssmtp. To set this up, install the utilities with

sudo apt install ssmtp
sudo apt install mailutils

The ssmtp utility uses the SMTP service on a mail server, so the e-mail login credentials and the SMTP server address need to be entered into /etc/ssmtp/ssmtp.conf. The SMTP server address usually is in the format smtp.<domain> (in the example, the gmail domain has been used). The last argument specifies the login procedure.

AuthUser=<username>@gmail.com
AuthPass=<password>
mailhub=smtp.gmail.com:587
UseSTARTTLS=YES

Note that Gmail security settings may need to be adjusted to permit login from the ssmtp. An e-mail can be sent from the command-line or a script with

echo “hello” | mail -s test <username>@gmail.com

Activity logging

Login activity can also be written to syslog with the script

DEBUG="logger"
if [[ -n $SSH_CONNECTION ]] ; then
    $DEBUG "${USER} logged in to ${HOSTNAME}"
fi

The snippet can be added directly to the local ~/.bashrc in a user’s home directory, so that all logins are printed to syslog (Figure 7), read by

cat /var/log/syslog

Figure 7. User login information in syslog.

For sudoers, a separate log can be set up by adding the following line to /etc/sudoers

Default logfile=”/var/log/sudo.log”

Figure 8 shows the information contained in the log file.

Figure 8. Details in sudo log.

Create accounts with Ansible

User accounts can be created consistently across servers with Ansible. In the following example, two back-end servers have been added with access details in ~/.ssh/config and the server names in the inventory file inventory.ini:

[webservers]
server0
server1

The playbook, called create_user.yaml, contains definitions for the user account guest, which is member of the admin group with sudo rights. A key pair with the same name as the account (guest.key and guest.key.pub) needs to be created and stored in the directory keyfiles before running the playbook with

ansible-playbook -i inventory.ini create_user.yaml

---
- hosts: webservers
  remote_user: ubuntu
  become: true
  vars:
    users:
    - username: "guest"
      groups: "admin"
    remove_users:
    - "test"
  handlers:
  - name: "Restart sshd"
    service:
      name: "sshd"
      state: "restarted"
  tasks:
  - name: "Create user accounts"
    user:
      name: "{{ item.username }}"
      groups: "{{ item.groups }}"
      state: "present"
    with_items: "{{ users }}"
  - name: "Remove old user accounts in remove_users"
    user:
      name: "{{ item }}"
      state: "absent"
    with_items: "{{ remove_users }}"
  - name: "Add authorized keys"
    authorized_key:
      user: "{{ item.username }}"
      key: "{{ lookup('file', 'keyfiles/'+ item.username + '.key.pub') }}"
    with_items: "{{ users }}"
  - name: "Allow admin users to sudo without a password"
    lineinfile:
      dest: "/etc/sudoers" # path: in version 2.3
      state: "present"
      regexp: "^%admin"
      line: "%admin ALL=(ALL) NOPASSWD: ALL"
  - name: "Disable root login via SSH"
    lineinfile:
      dest: "/etc/ssh/sshd_config"
      regexp: "^PermitRootLogin"
      line: "PermitRootLogin no"
    notify: "Restart sshd"

The playbook also removes obsolete accounts, in this example the account test.

Access control files

Server access is defined in two access control files, /etc/hosts.allow and /etc/hosts.deny. These are text files with rules expressed in patterns, wildcards, operators and shell scripts.

Each row defines a daemon-client pair. The hosts.allow file is read first, and if a match is found the connection is allowed and the search is stopped. If no allowed match if found, the hosts.deny file is read. If a match is found the connection is refused - otherwise it is allowed.

The most open allow rule is

ALL: ALL

The second part, representing clients by IP address (or domain name), can be set to allow local traffic from some IP address range,
ALL: 192.168.1.0/24

The deny rules are formulated similarly. The most restrictive rule in hosts.deny is

ALL: ALL

In this case, only explicitly authorized hosts are permitted access, either as an allow rule or as an exception to the deny rule, for example

ALL: ALL EXCEPT 192.168.1.110

The rule can therefore be set to a CIDR range which the clients use. To allow SSH access from local traffic on a network with CIDR 10.0.0.0/24, the file /etc/hosts.allow would contain

sshd: 10.0.0.0/24

and /etc/hosts.deny
sshd: ALL

This is a suitable configuration for back-end servers managed through a bastion host.

Uncomplicated Firewall (UFW)

Even with restrictive security group settings, additional protection by a separate firewall on servers is recommended - in particular on gateways such as bastion hosts and reverse proxy servers. This section shows how to configure the Linux Netfilter firewall through the iptables with the front-end Uncomplicated Firewall (UFW) command-line interface. This utility is part of the Ubuntu distribution.

UFW is by default set to deny all incoming traffic and allow all outgoing traffic. It has the general syntax

sudo ufw [--dry-run] [options] [rule syntax]

Note that since the bastion host is configured over SSH, enabling the firewall without having set a rule to allow SSH will block this connection. With the --dry-run option when the UFW is switched on, UFW will not apply the specified rule, but it will show status results if it had been.

The current set of rules (in optional verbose mode) is displayed with

sudo ufw status verbose

Allow SSH

When UFW is disabled, it will only show Status: inactive. Note that switching on the firewall may break the SSH connection and block further attempts. It is therefore important that the exception of SSH to the default rule of blocking incoming traffic is in place before starting it. The rule is given by

sudo ufw allow ssh

This produces the output (Rules updated) shown in Figure 9.

Figure 9. Updating SSH rules in UFW.

It is also important to consider rules for IPv6. Here, it is assumed that IPv4 is used for communication and apply some simple complementary rules to restrict IPv6. To enable this, open the firewall configuration file with a standard text editor, for example

sudo nano /etc/default/ufw

and to apply all rules to IPv6 as well as IPv4 (which is default) set IPV6=yes (Figure 10).

Figure 10. UFW configuration file.

Enable and disable the firewall

The firewall is switched off and on respectively with

sudo ufw disable

and

sudo ufw enable

When enabling the firewall, it will show a warning prompt (Figure 11).

Figure 11. Enabling UFW.

Type y and press ENTER to acknowledge. Now sudo ufw status verbose should show a result like in Figure 12.

Figure 12. UFW status showing default rules and exception for SSH.

General rules

The general (default) rules, allowing all outgoing and blocking all incoming traffic are set with

sudo ufw default allow outgoing

and

sudo ufw default deny incoming

If the firewall had been switched on with only these rules, your SSH connection would be blocked so you would be locked out from the server.

To prevent this from happening, allow SSH with

sudo ufw allow ssh

or the equivalent command with port number and protocol

sudo ufw allow 22/tcp

Now further rules can be added with similar commands. These rules would typically match the security group rules.

TCP rules

To allow FTP, HTTP and HTTPS, respectively, enter

sudo ufw allow ftp or sudo ufw allow 21/tcp

sudo ufw allow http or sudo ufw allow 80/tcp

sudo ufw allow https or sudo ufw allow 443/tcp

In these rule definitions, the protocol need not be specified, only the port. However, when combining these rule definitions now including multiple ports as

sudo ufw allow proto tcp from any to any port 80, 443

the protocol needs to be specified. With similar syntax, rules restricted to specific interfaces and IP address ranges can also be created.

Intrusion prevention with fail2ban

Intrusion prevention systems (IPS) are utilities designed to detect and stop intrusions, typically by monitoring access logs and applying a set of rules to incoming requests and login attempts. Such a system, Fail2ban, can easily be deployed to provide protection on the bastion host or proxy server.

To begin, do an update and install with

sudo apt update
sudo apt install fail2ban

Note that after a kernel upgrade, the server will has to be rebooted.

Start and enable fail2ban with

sudo systemctl start fail2ban
sudo systemctl enable fail2ban

Configure jails

Monitoring SSH

Blocked IP address are stored in jails configured in the /etc/fail2ban directory. The configuration file is called jail.conf. Do not edit this file directly, but create a new file jail.local (or make a copy of jail.conf with this name) and make changes in this local file, which overrides settings in jail.conf. For SSH, fail2ban will monitor the log file /var/log/auth.log using the fail2ban sshd filter. To edit the configuration, do

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

Under the SSH section, beginning with [sshd], add (allowing three failed attempts):

[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

Save and close the file, and restart the utility with

sudo systemctl restart fail2ban

Any attempt to login to the server failing three times (within a configurable time span) will be blocked from further attempts by iptables blocking the originating IP address (for a configurable amount of time).

To see the enabled traffic type jails, type

sudo fail2ban-client status

The output looks something like in Figure 13.

Figure 13. List of enabled jails.

Please be aware of the risk of being locked out testing the system. It is useful to set

ignoreself = true
ignoreip = <Your-IP-address>

The time limits governing the blocking behavior are set in findtime, the time frame in which maxretry applies to trigger blocking, and bantime, the time an IP address is blocked. The parameters are set in jail.local under the section [DEFAULT], for example (with default values)

findtime = 10m
bantime = 10m

Monitoring HTTP

Fail2ban contains filters which are software specific. For HTTP, there are filters for Apache and Nginx, for example. Using Nginx as example, a jail rule protecting HTTP authentication can be defined as

[nginx-http-auth]
enabled  = true
filter   = nginx-http-auth
port     = http,https
logpath  = /var/log/nginx/error.log

Rules can also be defined to block activities such as trying to run scripts, using a server as proxy and blocking bad bots.

Unblock IP address

A blocked IP address is released (unbanned) with the command

sudo fail2ban-client set sshd unbanip <IP-address>

Additional resources

Barrett et al. "SSH, The Secure Shell: The Definitive Guide (2nd ed.), O'Reilly Media (2005)