Configure client
The client in this section is the physical computer and software environment used to interface with DT Cloud Services. In this documentation, we assume that the client is configured on a PC running on Ubuntu 20.04 and with Python 3 installed. The procedures described should be easily adapted to any Unix-like system.
Contents
Python virtual environments
It is good practice is to create a virtual environment for the project to prevent Python conflict with other applications. The commands in the text are invoking Python 3 using python and pip, but these may have to be replaced by python3 and pip3, respectively, depending on the Python installation.
Virtual environments in Python are managed by the builtin module venv which is used to create dedicated work spaces for projects, making it possible to use different site package versions. A new virtual environment is easily created by
python -m venv .venv
where the argument .venv is the name of the virtual environment folder, and which can be changed to anything convenient (as long as it is a proper folder name). To use the virtual environment, it needs to be activated with
source .venv/bin/activate
When a virtual environment is active, its name shows before the terminal prompt, for example (.venv) $. This name should therefore be rather short. Any site packages can now be installed in the usual way, but accessible only locally in the virtual environment. To exit the environment, type deactivate after the prompt:
(.venv) $ deactivate
From this point onward, we assume that all Python commands are executed from within a virtual environment, unless otherwise stated.
The OpenStack CLI
Install OpenStack client
OpenStack commands are run from the client, so it needs to have the OpenStack client software installed. It is easily installed through the Python package manager pip by the command
pip install python-openstackclient
In some Python configurations, the pip command may fail. In that case replacing pip by pip3 in the command above should work. See also the relevant section in Troubleshooting. Installing the CLI via Python’s pip is recommended over using the distribution package, which may be outdated. For further details, please see the OpenStack installation guide.
Download OpenStack RC file
On the DT Cloud Services portal, an OpenStack RC resource script file with environment variables is generated at the same time as the tenancy space and it can be downloaded from there following Project > Compute > API Access > Download OpenStack RC File v2.0 (Figure 1) and stored in a project folder.
Alternatively, it can be found in the user profile in the right-hand upper corner (Figure 2).
The name of the OpenStack RC file is <project-name>-openrc.sh, which is a bash script file that can be opened in any text editor. The <project-name> is the name given to your tenant at the time of its creation. Your credentials should be visible in the following variables in the downloaded file
export OS_TENANT_ID=<project-id> export OS_TENANT_NAME="<project-name>" export OS_USERNAME="<openstack-username>"
Differences between Beryllium and Boron
The authentication in the OpenStack client is handled by the Keystone module. Beryllium uses version 2 whereas in Boron version 3 is used. The version used can be seen in the resource file, where the URL has either v2 or v3 as part of it.
Recent versions of OpenStack client support both Keystone versions, but may be using version 2 by default. By passing the optional parameter os-identity-api-version with value 3 to the openstack command, it is forced to use version, for example
openstack flavor list --os-identity-api-version 3
To set this version to be used permanently, modify the resource file to include the environment variable
export OS_IDENTITY_API_VERSION=3
Connect to cloud via OpenStack CLI
Next, move into the location of the downloaded bash script file and run it by
source <project-name>-openrc.sh
When the file contains spaces, it is necessary to type brackets around the file name, for example
source “<project-name> - openrc.sh”
to execute the command. The script will ask for a password. Type the same password as used to login to Horizon and press <Enter>. Now the cloud connection information is set with the correct environment variables.
To test the connection, run a simple Openstack CLI command, like
openstack image list
If the connection is established with the proper credentials, the command will print out a list of accessible images which can be used to build a VM.
Note that when the client is restarted, the exported environment variables can be lost. This raises an error like in Figure 3. It is the necessary to re-execute the source command.
Secure communication
The protocols used for secure communication with the cloud and encryption of data are based on Public Key Infrastructure (KPI), including TLS (HTTPS), SSH and GPG. The methods build on two keys - a public key used for encryption which is publicly available, and a private key used for decryption which kept secret.
Transport layer security (TLS)
OpenStack uses TLS (HTTPS) for encrypted API. This requires a certificate signed by a trusted party being accessible on the client (known as a root certificate). In most cases, such a certificate is already present in the OS distribution. On Ubuntu, such certificates reside in the folder /etc/ssl/certs.
In some cases, a special certificate is issued to distribute the public key and other information. It is typically digitally signed by a Certification Authority (CA), a trusted third-party organization that has validated that the information in the certificate is accurate. All users of DT Cloud Services should install the DT Cloud Services root certificate.
A new certificate is installed on the client with the ca-certificates utility in a few simple steps. Install the utility with
sudo apt install ca-certificates -y
Next, under /usr/share/ca-certificates create a sub-directory, with some more or less meaningful name, say, extras:
sudo mkdir /usr/share/ca-certificates/extras
Download the DT Cloud Services root certificate, copy it to the new directory and update the CA certificate store with the commands
wget http://info.pki.sec.in.pan-net.eu/crts/rootcax1.crt sudo cp rootcax1.crt /usr/local/share/ca-certificates/extras sudo update-ca-certificates
Secure shell (SSH)
The Secure Shell protocol uses certificates for host and user authentication. An open source implementation of SSH (client and server) is contained in the OpenSSH package. On Ubuntu, SSH is installed by default. To verify this, run
ssh -V
which prints out the version details. To use SSH it also requires that an SSH server is running on the remote host (which it is after successful instance launch).
Connect to the remote host with
ssh -v <username>@<ip_address>
where the flag -v gives verbose output. See also Bastion host on how to configure a bastion host.
Generate key pair
SSH uses key pairs, where a public (shared) key is exchanged between the client and the tenant, and further associated with an instance for secure communication. The key pair can be generated either in the tenant or in the client. Generating the keys in the tenant has the advantage that the same key can be used to access an instance from different clients, as long as the same public key is used.
Create key pair in tenant
A key pair can be generated and shared by a single OpenStack command. The shared key is locally stored in PEM (Privacy Enhanced Mail) format, so the following command creates a key with name <key-pair-name> and stores the file as <key-pair-name>.pem.
openstack keypair create <key-pair-name> > <key-pair-name>.pem
Note that the variable <key-pair-name> is contained in angle brackets, followed by the scripting output operator >, which writes the result to the specified file, <key-pair-name>.pem.
A suitable location to store the file is a project directory created for the tenant on the client machine. Set the correct permissions by
chmod 0600 <key-pair-name>.pem
Finally, import the file to the SSH client by
ssh-add <key-pair-name>.pem
To establish a connection with an instance with the key <key-pair-name> associated to it, no password is needed, only the command
ssh <user>@<IP address>
Create key pair in client
Linux operating systems have the SSH client pre-installed, and a key pair is generated by the command ssh-keygen. The keyword takes a number of arguments. We will use the following arguments to ensure correct format of the keys:
-t key_typespecifies the algorithm used to generate the keys. The value replacingkey_typeshould bersa(which is also the default)-b bitsspecifies the key size. The default value ofbitsis 2048. A stronger key (larger key size) is generated by setting it to 4096. A larger key may slow down the initial connection but has no effect on the speed of the encryption after the connection has been established.-f <filename>To avoid the keyword to prompt for a file name, it can be passed directly to it before execution. It is advisable to name the key so that it easily can be associated with the intended project, for example<project-name>.key.
First, change the directory to the standard directory for security keys by
cd ~/.ssh
Generate the key pair by executing
ssh-keygen -t rsa -b 4096 -f <project-name>.key
If you don’t specify the filename as an argument to ssh-keygen, it will ask for it or use the default nameid_rsa. You will also be prompted to enter a passphrase to protect your key. Either enter the same memorable passphrase twice, or press <ENTER> twice to continue without passphrase protection. SSH will prompt for the passphrase before establishing a connection to your instance, so please select an adequate level of protection for the key. The output will look something like shown in Figure 4.
Your key pair has now been created and is ready to be used. You can inspect the content of the public key by open it in a text editor of just by typing in your console
cat <project-name>.key.pub
The final step to enable SSH is to import the public key to the portal. In Horizon, open the window Compute/Key Pairs. Click on the button Import Key Pair to open the window form shown in Figure 5.
In the form, enter a key pair name - it is advisable to use a name that can be used to identify the name of the key previously generated. Next, copy and paste the full contents of the public key in the designated free-text field and press the button Import Key Pair.
After importing the key, it is visible in the Key Pairs window (Figure 6). If there are more than one key pair in the list, and the new key pair name is not selected as active, click on the arrow buttons to deselect the old key pair and select the new one.
File management
Secure file transfer is typically utilizing secure file transfer protocol (SFTP) or secure copy protocol (SCP), which are both related to SSH. File transfer can be performed from a terminal or a file client with a GUI, including the Ubuntu native Nautilus.
In particular for Windows, an alternative to SSH authentication from the terminal is access through an SSH client, such as PuTTY. The client includes a command-line SFTP component, called PSFTP.
There is a number of graphical SFTP clients available that can be used for secure file management, such as FileZilla, which can be a convenient solution for both Linux and Windows.
Command-line file transfer
The SCP utility (using the same acronym as the protocol it uses) is installed by default on Ubuntu and provides an easy-to-use method to copy files between two hosts securely.
SCP relies on ssh for the data transfer, so an ssh key or password is needed to authenticate access on remote. The operation also requires read permission on the source file and write permission on the destination host.
The syntax is
scp [OPTIONS] [[<username-src>@]<host-src>:]file1 [[<username-dst>@]<host-dst>:]file2
The colon (:) is used by SCP to distinguish between local and remote locations. If the source or destination host specification is omitted, the end-point is the local client itself. Remote hosts can be reached with the usual ssh argument, ubuntu@<ip-address>. The arguments file1 and file2 are the relative paths and file names to the file being copied and the file name it is saved as at the destination. Note that the folder in the path at the destination must exist for the transfer to succeed. To transfer a file.txt from the client to a remote host with IP address 188.125.16.110, the syntax is
scp file.txt ubuntu@188.125.16.110:file.txt
To transfer a file in the other direction, simply reverse the source and destination arguments:
scp ubuntu@188.125.16.110:file.txt file.txt
Command options include:
-Cuse data compression during the transfer-iinclude key file (the same as forssh)-Pspecify the SSH port used by the remote host-ppreserve file modification and access timestamps-qsuppress the progress bar and warning messages-rcopy recursively, that is, all files in a folder
SCP also works nicely with back-end servers accessed through a bastion host. Simply replace the <username>@<ip-address> part with the server nickname defined in ~/.ssh/config. If a server can be reached with, say,
ssh my_backend
then file transfers to and from the server are done with
scp file.txt my_backend:file.txt
scp my_backend:file.txt file.txt
SCP shows a progress bar of the transfer (Figure 7).
How to set up and configure a bastion host is described in https://pannet.atlassian.net/l/c/e4T10CtR
Persistent sessions
The stability of an SSH session can be a problem when handling large files. A session can be made persistent by launching it inside a terminal multiplexer, such as GNU screen or tmux. Below, GNU screen is described briefly.
The utility is installed through
sudo apt install screen
The installation can be tested with screen --version, which prints out the version of screen currently installed. Screen is controlled by a special control sequence (similarly to text editors). The control mode is invoked by ^A, meaning pressing the Ctrl and A keys (lowercase) simultaneously, followed by a control character.
To run a SSH or SCP session in screen, launch the utility from a regular terminal with
screen
Available commands are listed by ^A followed by the ? character (which may require simultaneously pressing the Shift key). The sequence ^A and c opens a terminal from which a session is started as usual, but from within the screen environment. Exit screen with ^A and \. A notification is shown as in Figure 8. Press y to close the utility and return to the regular terminal.
Using a file manager
There are many file managers providing a graphical user interface for file transfer between the client and the cloud. Unfortunately, many of these clients can only connect directly to instances having a floating IP assigned, but not to servers and volumes behind an SSH proxy (bastion host).
Based on SFTP, the file browser Nautilus can be used for file operations. Highlight the item Other Locations and in the field Server type
sftp://ubuntu@<ip-address>
and press ENTER. This opens the home directory on the remote server (Figure 9).
An virtual machine accessible by an IP address can also be mapped to a file manager such as FileZilla which supports both FTPS (SSL) and SFTP (SSH).
The mapping can be set up over SFTP using the IP address, login details and the key or certificate. A file manager typically has a site or connection manager, where the user can enter the details needed to connect to the VM. In FileZilla the site manager looks as shown in Figure 10.
In the Site Manager form, the user only needs to add host (IP address), user name and link to the key file. After connecting, the file manager displays the folder structure in the virtual machine and allows file upload, download and remote file operations (Figure 11).
Encrypted data storage
To use data encryption, the data needs to be encrypted on the client side before being uploaded to the cloud. This means that the encrypted files have to be visible to the local file system.
The Linux-native Gnu Privacy Guard (GPG) invoked by the gpg command relies on key pairs and can be used to encrypt and decrypt files for secure storage and file sharing. Encrypted files can then be stored or shared on the cloud.
Backup and restore with rsync
It is crucial for any system to have a well-defined backup and restore strategy. For a cloud deployment there are a few important points to consider. OpenStack provides methods that can be used for backup purposes, but these backup copies are mainly remaining in the cloud. It is often desirable to implement an incremental backup procedure. This can be done with rsync - even for servers behind a bastion host.
A backup plan should detail all systems that need to be backed up regularly, with which frequency, and where to store the copies or database replicas.
OpenStack can be used to create snapshots or images of entire volumes (see Manage volumes ). This is usually not a suitable method in many situations where some files or folders are updated much more often than others. A more granular level of backup on folder level can be performed with rsync.
Remote sync (rsync) is a command-line incremental backup utility included in the Ubuntu 18.04 distribution. It is used to pull and back up files from the cloud to the user client for secure storage. It uses SSH (among other possibilities) and can be used to fetch files and folders from back-end servers through a bastion host
If the remote host has a public IP assigned, the files can be downloaded directly to the client with SSH or a file manager. When the host is located behind a bastion host, the rsync command takes the argument -e ssh specifying that SSH is to be used as remote shell - the SSH addressing and key pair handling is conveniently implemented in the bastion host.
To back up files from the cloud, located in the remote folder <folder-on-remote> to the folder <folder-on-client>, simply issue a command like
rsync -arvz -e ssh <remote-nickname>:<folder-on-remote> <folder-on-client>
which will pull the content of the remote folder <folder-on-remote> and store the entire remote folder under <folder-on-client>. The folder <folder-on-client> must exist, but any subfolders will be created if necessary.
The options suggested here are
-a- enable archive mode, which preserves file permissions, ownership, and preserves symbolic links-r- pull files recursively, that is, all files in a folder and its subfolders-v- enable verbose output mode-z- use file compression during transfer--progress- show progress bar during transfer
In the example shown in Figure 12, the server <remote-nickname> is griselda and the contents of the remote folder www is stored on the client under test/www. The folder test on the client has to exist, but the subfolder www is created by rsync.
The rsync utility is very flexible and can be tailored to specific needs using a selection of its many available options. Please refer to the rsync man page for details. To push files, only swap the order of the source and destination folders.
It simplifies the backup process to have server data, log files etc. on a volume separate from the running instances themselves. Also note that databases and other applications should be backed up before the file transfer is initiated.
Note that for maximum security, a dedicated key pair should be used for the rsync operations. It is also possible to restrict SSH to a set of predefined commands.
Useful techniques
View tenant resources
The OpenStack command quota is used to list the initial resources in a tenant (project). For a complete list, execute
openstack quota show
Sometimes it is more convenient to print only a specific resource, which is done by giving the option --column (or, equivalently, the flag -c) before a resource type, for example
openstack quota show -c gigabytes
Web console
There are several different interfaces available for instance configuration:
OpenStack via Horizon or CLI
SSH remote access
Web console of an instance
To be able to login directly on the virtual machine, create a username and set a password with
sudo useradd <username>
sudo passwd <username>
The web console can be opened following a link generated by Horizon or by issuing the command
openstack console url show <server-id>
The output contains an URL that can be copied into a browser window (Figure 13). On the login screen that is shown, enter the credentials for <username> to login. The console is for example useful for monitoring boot processes and troubleshooting networking issues.
Please note that the web console requires an encrypted connection, that is, connection over a virtual private network.
Copy object data to variable
Whenever the identity of an OpenStack object is needed repeatedly, it is convenient to store it in an environment variable. The value can be extracted from the command
openstack <object> show <object-name>
The displayed table contains key-value-pairs for all important parameters. For example, the identity string has key id, and can be filtered out from the show request - in this case the identity string of an image with name <image-name>:
IMG_ID=`openstack image show -f value -c id <image-name>`
The variable can called anything as long as it is not used as environment variable in any other context. Note the slanted apostrophes surrounding the OpenStack command. The variable can then be used, returning its value when prefixed by $, like this
glance image-download --file <img-copy-name> --progress $IMG_ID
Retrieve metadata
Nova provides a Compute instance metadata service, which is reachable through a REST API. Metadata can be provided by the user, by Nova itself or by DT Cloud Services Cloud at creation time.
The OpenStack metadata API has a number of supported versions concurrently. To retrieve a list of supported versions for the metadata API, log in to the Compute instance and make a GET request to http://169.254.169.254/openstack, which will return a list of directories
curl http://189.254.169.254/openstack
The content of a directory (version) is retrieved with
curl http://189.254.169.254/openstack/<version>
The output is shown in Figure 14.
Other platforms
For the client, a computer running on Linux is strongly recommended but not strictly necessary. However, some utility software currently run only under Linux. On Linux, superuser (or root) access is required to install software, modify system files and change file permissions. On a Windows PC or Mac, administrator rights may be needed to install some software. For Windows, there are several possibilities to enable the necessary CLI tools:
Activate Windows Subsystem for Linux (WSL2)
Install a virtual machine (VM), such as VirtualBox or VMWare
Using a public cloud service with a browser-based terminal as entry point (for example Google cloud). To enable the cloud-based client
Register with Google cloud and set up a simple VM
Share key pairs with this account
Use Google browser-based terminal options to SSH into DT Cloud Services VM from the Google VM
Additional resources
Installation of the OpenStack CLI
Creation of key pairs and using SSH on Linux, following link or link
Installation and usage of the PuTTY SSH client for Windows
Creation of key pairs on Windows using PuTTY
Creation of key pairs on MacOS using PuTTY