Troubleshooting

Please review this chapter for common technical errors or misconfigurations. Contact details to our service operations team are listed in Technical Support

Contents

General connectivity issues

How to access Horizon

Horizon is accessed through a common web browser by opening a link and entering your tenant credentials. After creating a tenant through the DT Cloud Services portal, the link to the data center where it is hosted can be found on the page My Products. For more information, see https://pannet.atlassian.net/l/c/qJ7fbXA0

How to access end-points

The common access to end-points is over SSH. For direct SSH connectivity, the instance need to have a public (floating) IP assigned. With several servers, access can be managed using SSH port forwarding through a bastion host. This relaxes the need for floating IP addresses, and improves security. How to configure a bastion host is described in https://pannet.atlassian.net/l/c/4sXb1xPE

How to troubleshoot the tenant API

When the API fails, the root cause can be many things. First, make sure that the OpenStack client is up to date. The client may throw errors, but when the fault lies in the API proxy, no messages will be shown on the client. It is updated with

pip install -U python-openstackclient

A common cause of failed connectivity is improperly configured security groups or firewall settings. Please check that the instance has a security group allowing the protocol, IP version, IP address range and port used to connect to it. If a firewall is used, it needs to allow the same type of ingress traffic.

If the compute instance has several network interfaces, it is advisable to disable all interface and bring up only the one associated with the subnet being tested.

Sometimes external network issues can cause problems. In such cases, the problem may disappear after waiting a while before trying to connect again (say 30 minutes).

Choice of DNS

For general internet access, any publicly available DNS can be used. In the How-to guide on networking, the Google DNS with IP addresses 8.8.8.8 and 8.8.4.4 are used as examples.

There is no need to use DT Cloud Services DNS for plain internet access, but is needed for access to special services (DTH).

Password recovery

Lost password to data center

Should you forget your password to the data center you can recover your password on the DT Cloud Services portal under the user settings section, see https://pannet.atlassian.net/wiki/spaces/DocEng/pages/638058567/Settings#Reset-Product-Password

for a data center password reset guide.

Lost password to DT Cloud Services portal

Should you forget your password to the DT Cloud Services portal, please use the “Forgot Password?” link on portal login page to have a new password sent by e-mail (Figure 1).

Figure 1. Password reset link on the DT Cloud Services portal.

OpenStack client not working

Installation using pip3 instead of pip

On some Linux PC configurations there can be problems with the OpenStack client installed with pip. If you receive any of the following error messages after trying to use OpenStack client, this can indicate such a problem:

ImportError: No module named queue

Segmentation fault (core dumped)

Currently there is an issue that may affect at least users with Ubuntu 18.04 trying to use OpenStack client installed with pip through

pip install python-openstackclient

This problem can be resolved by using pip3.

1. Uninstall the OpenStack client previously installed with pip or apt-get by pip uninstall python-openstackclien or apt-get remove python-openstackclient

2. Install pip3 through apt install python3-pip

3. Install the OpenStack client with pip3 using pip3 install python-openstackclient

4. Check if the client is working with openstack --version

Cannot connect to tenant through OpenStack CLI

Lost environment variables

Failure to connect to the cloud from the OpenStack client can generate the error message (see also Figure 2)

Missing value auth-url required for auth plugin password

In this case the OpenStack exported environment variables are missing or have been lost.

The source file can be downloaded through the Horizon GUI, which is a file in the format <project-name>-openrc.sh. The resource file contains the environment variables needed to connect to the tenant from the OpenStack client.

The error may also occur when when the host is restarted and the exported environment variables are lost. The variables are loaded by the source command

source <project-name>-openrc.sh

You will be prompted to enter your tenant password, which is the same password used when logging in to the Horizon portal.

Wrong password

When entering the wrong password to the source command, it prints a message similar to

The request you have made requires authentication. (HTTP 401) (Request-ID: req-964fce7e-c39e-713d-c0c6-69e433bface0)

When running the downloaded RC script to set the environment variables please make sure that you are using the correct password where prompted. You must use tenant password, which is the same password used when logging in to the Horizon portal.

Load CA certificate

Problems with OpenStack CLI or SSL may be caused by a missing or incorrectly loaded CA certificate.

In most cases standard certificates are used, and for this no action is needed. To load a specific CA certificate, please see instructions Configure client

Cannot connect to instance

When an instance cannot be reached, it is usually caused by too restrictive security groups on the virtual machine or some problem related to SSH. External network problems, such as overload, can also cause connection problems. An improperly configured bastion host can also cause routing errors within the tenant.

Security groups not properly set

When trying to connect to an instance over SSH, the error message

ssh: connect to host <ip-address> port 22: Connection refused

is probably caused by improperly set security groups. If SSH on port 22 is not set as allowed in the security groups, you will not be able to access the instance. Please note that SSH is not allowed by default and you have to add SSH traffic to the security group associated with the instance. For more details please see Configure security

SSH Permission denied

There are a number of causes which make SSH fail to connect and return Permission denied (publickey). Should SSH fail, it is useful to use the flag -v which produces a verbose output. The number of v’s determine the level of verbosity, the maximum being -vvv.

Wrong Key or key not assigned to the instance

The most common problem is that the wrong key is used by the SSH client, or that it has not been assigned to the instance.

You can check the name of the SSH key assigned to an instance through OpenStack client or Horizon. Please note that if you have more SSH keys in OpenStack you must select an SSH key when launching an instance, otherwise no key will be assigned. Also, if trying to assign an SSH key to an already running instance, the SSH key will not be assigned to the instance.

If you are using more SSH keys you must select a specific key when setting up a connection. Use the argument -i with the key name, for example

ssh -i <key-name>.key ubuntu@<IP address>

File permissions

The key files need to have the proper file permissions, which are changed with

chmod <permission> <file>

The key files are stored in the hidden folder .ssh, and the permissions need to be set according to Table 1.

_{Table 1. File permissions for key files.}

SSH configuration

The SSH configuration file is located in /etc/ssh/sshd_config and contains settings for which type of authentication that should be allowed. The default configuration should normally work with public key authentication. It might be needed to change the configuration settings. To do so, open the file in a text editor, like nano, with

sudo nano /etc/ssh/sshd_config

Verify that the entry PubkeyAuthentication yes is uncommented, save the file and restart sshd with

systemctl restart sshd.service

SSH Too many authentication failures

The error typically occurs when there are many keys on the client, and SSH tries them one after the other, leading to a number of failed login attempts.

This can be resolved by adding the argument IdentitiesOnly=yes to the SSH login command,

ssh -o IdentitiesOnly=yes -i <key-name> ubuntu@<IP address>

An example is shown in Figure 2.

The variable can be set permanently in the SSH config file ~/.ssh/config for a host <server-nickname> as

Host <server-nickname>
  IdentityFile <key-name>

It is also necessary to set the file permissions properly by as shown in Table 1. For details, see https://pannet.atlassian.net/l/c/d60euKpm

Rebuild server

As a last resort the server can be rebuilt with the same key pair. By omitting all optional arguments in the command, the settings of the previous build will be used:

openstack server rebuild <server-name>

This creates a clean install, including the SSH configuration, which may make it accessible from the client. At the first connection, the client will print a notification that the DSA host key is not matching the record in ~/.ssh/known_hosts together with the command to clear that record. After executing the command, the server should be accessible like any newly deployed server.

Flavors and tenant resources

Spawning errors and quota limits

The quotas of tenant resources can be listed with the command

openstack quota list

When trying to create a server exceeding available resources, an error message is displayed. In some cases, resources can be freed by deleting some nodes, or a smaller flavor used for some services. Please see also Configure client

Custom flavors

For special applications, custom virtual machine flavors (server configurations) can be provided by DT Cloud Services. Please not that this is possible mainly for configurations with larger capacity than that of any standard flavor.

To order a custom flavor, please contact Technical Support

How to use metadata

User-defined instance metadata can be set at creation time through command-line arguments (or in Horizon), or by using configdrive. Access to the metadata from the Nova REST API is briefly described in https://pannet.atlassian.net/l/c/h9mX401i

Optional connectivity services

How to deploy a jump host

A jump host (or bastion host) is an SSH gateway for tenant management. It can easily be configured on a standard compute instance. Instructions how to do this can be found in https://pannet.atlassian.net/l/c/mnnD3EnY

How to configure DTH connectivity

Internet connectivity is provided by the DT Cloud Services Cloud tenant.

To be able to reach the internet for package download from the DTH network, the HTTP proxy needs to be configured for DTH access. This needs to be enabled by Technical Support

How to configure NTP

To deploy an NTP server and clients in a tenant is straightforward. On Ubuntu, the package is installed with

sudo apt install ntp

and suitable servers are added to /etc/ntp.conf. A list of NTP pool servers can be found on ntp.org. Several NTP configuration tutorials are available on-line.

Failing IPsec routing

Failing IPsec gateway routing or port security issues can be related to the Interface Route configuration, which needs intervention by Technical Support

LBaaS in operating status ERROR

A listing of an Octavia LBaaS load balancer with

openstack loadbalancer list

may show operating status ERROR in place of the expected ONLINE, without having raised any errors during its creation.

The error state is showing when there is in fact no operational fault. This is is a bug, and the operational status can be safely ignored when configuring the load balancer.