Resource scaling

One the most important uses of orchestration is to allow consistent resource scaling. Such scaling can be done by manually updating a stack, through web hooks, or automatically when some alarm is triggered.


The term scaling in this part refers to cluster scaling, that is, changing the number of resources in a cloud configuration such as servers and volumes. The example templates take as parameters the names of desired flavor and image, as well as existing key pair, local network and security group.

OpenStack Heat resource groups implement container types for resources that provide a convenient way to create multiple identically configured resources by command or a triggered by a signal. The two types of resource groups are OS::Heat::ResourceGroup which can be used to change the group size by modifying the count property in the parent stack, and OS::Heat::AutoScalingGroup with one or more OS::Heat::ScalingPolicy which allows scale operations triggered by signals (webhooks or workload alarms).

Update stack

Scaling can be enabled by creating a OS::Heat::ResourceGroup object, which typically encapsulates a resource template. This allows the number of resource instances to be controlled with a node count parameter. The example template below has a resource group containing the declaration of a single server. Due to its simplicity, the resource group contains the server declaration directly in the main template.

The stack is created with

openstack stack create --template <template-name> <stack-name>

where the default value for count (on line 29 and 35 in the template) is used.

When Heat processes a template, it creates a parent stack for the main template and a number of child stacks for resource templates which are encapsulated in the main template. This creates a hierarchical structure of stacks, which can be inspected with

openstack stack list --nested

In the template, the output section contains reference to the resource group only. This section is printed with

openstack stack output show <stack-name> --all

The stack content can also be listed with

openstack stack resource list <stack-name> -n 1

This output after stack creation is shown in Figure 1.

Figure 1. Stack output and resource list after stack creation.

Scaling the stack created from this template is done by changing the count parameter value through the declared parameter cluster_size. The stack update command is

openstack stack update <stack-name> --template <template-name> --parameter <parameter-name:value>

where the parameter argument is a key-value pair in the format "cluster_size=2". An increase of the node count to two is reflected in the stack output (Figure 2) and the resources list (Figure 3).

Figure 2. Stack output with IP addresses after scaling up to two servers.
Figure 3. Resource list after scaling up to two servers.

Scaling down is done in the same way, by decreasing the count parameter.

Template for scaling by stack update

heat_template_version: 2018-08-31

description: Scaling by stack update

    type: string
    description: Image used for instance
    default: <image>
    type: string
    description: Flavor used for instance
    default: <flavor>
    type: string
    description: Key pair used on instance
    default: <key-pair>
    type: string
    description : Security group used on instance
    default: <security-group>
    type: string
    description: Local network
    default: <local-network>
    type: number
    description: Number of instances in the cluster
    default: 1

    type: OS::Heat::ResourceGroup
      count: { get_param: cluster_size }
        type: OS::Nova::Server
          image: { get_param: server_image }
          flavor: { get_param: server_flavor }
          key_name: { get_param: server_key }
            - network: { get_param: local_network }
          security_groups: [{ get_param: server_secgroup }]

    description: "Private IP addresses"
    value: { get_attr: [rg, "attributes", first_address] }
    description: "Resource ID"
    value: { get_attr: [rg, refs] }


The Heat type OS::Heat::AutoScalingGroup, takes one or more rules of type OS::Heat::ScalingPolicy for scaling operations which may be triggered by an event such as a webhook or an alarm.

A parent template with an AutoScalingGroup is shown below. It has an embedded template called instance.yaml with the actual resource declarations, such as the templates described in

The minimum and maximum size limits are set by min_size and max_size.

By defining scaling policies to AutoScalingGroup, the scaling operations can be initiated by sending a POST request to the URL created, which are shown in the template output section. The two scaling policies contains the parameters

  • adjustment_type - specifies how the cluster is scaled: by a fixed increment, change_in_capacity, a percentage of the current size, percent_change_in_capacity, or an exact target size, exact_capacity.

  • auto_scaling_group_id - reference to the AutoScalingGroup the policy applies to.

  • cooldown - time interval in seconds where the cluster will be on stand-by, allowing monitoring of the workload. A large value of this parameter leads to fewer scaling operations and a stable cluster, but also a slower response to urgent load situations. This is not used for manually triggered scaling operations.

  • scaling_adjustment - a number or percentage, depending on adjustment_type. Note the negative sign of the capacity change in the scale-down policy.

When the stack has been created, its resources are listed with

openstack stack resource list <stack-name> -n 2

The switch -n - short for nested-depth - must in this case be set to 2 to display the actual servers. To scale up the cluster by the amount scaling_adjustment, fetch the URL with

openstack stack output show scale_up_url

and make a POST request with

curl -X POST '<webhook>'

where the URL <webhook> is copied from the stack output and must be surrounded by quotation marks (single or double) to conform with required format. The curl utility may fail due to a missing certificate. The path to the certificate is added to the command with the parameter --cacert <file-path>.

Figure 4 shows the number of servers before and after a scaling up the stack triggered by the webhook for this operation.

Figure 4. Using curl to trigger scaling by webhook.

Scaling by capacity step

When the adjustment type is change_in_capacity, the scaling adjustment take as argument an integer specifying a fixed capacity step. To achieve down-scaling the number has to be negative.

heat_template_version: 2018-08-31

description: Scaling through webhooks

    type: OS::Heat::AutoScalingGroup
        type: instance.yaml
      min_size: 1
      desired_capacity: 2
      max_size: 5

    type: OS::Heat::ScalingPolicy
      adjustment_type: change_in_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '1'

    type: OS::Heat::ScalingPolicy
      adjustment_type: change_in_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '-1'

    description: >
      The URL is the webhook to scale up the group. The scale-up
      operation is invoked by an HTTP POST request to this URL. No
      body nor extra headers are needed.
    value: { get_attr: [scale_up_policy, alarm_url] }
    description: >
      This URL is the webhook to scale down the group. The scale-down
      operation is invoked by an HTTP POST request to this URL. No
      body nor extra headers are needed.
    value: { get_attr: [scale_dn_policy, alarm_url] }

Scaling by percentage

When the adjustment type is percent_change_in_capacity, the scaling adjustment take as argument an integer denoting percentage of current capacity. The realized adjustment will be the nearest attainable capacity configuration. To achieve down-scaling the number has to be negative. The policy is used for scaling based on used capacity rather than fixed capacity increments. In the following example, the scaling are to the double and half of the current capacity, respectively.

    type: OS::Heat::ScalingPolicy
      adjustment_type: percent_change_in_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '100'

    type: OS::Heat::ScalingPolicy
      adjustment_type: percent_change_in_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '-50'

Exact capacity

When the adjustment type is exact_capacity, the scaling adjustment take as argument an integer the fixed target capacity, which always is greater than zero. The policy is used to switch between a small number of pre-determined capacity steps.

    type: OS::Heat::ScalingPolicy
      adjustment_type: exact_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '3'

    type: OS::Heat::ScalingPolicy
      adjustment_type: exact_capacity
      auto_scaling_group_id: { get_resource: asg }
      scaling_adjustment: '1'


The environment is a separate file used with a template to create a stack. There is also a default OpenStack and possibly operator-defined global environments used in the cloud. An entry in the user-defined environment takes precedence over the global environment.


An environment file is used with a template and is included in the openstack stack create command by supplying the -e option. The two prominent features are parameter handling and resource mapping with examples given in the subsequent two subsections.

The environment file can contain some of the following sections:

  • parameters - parameter values to be used with the template resources

  • parameter_defaults - default parameters passed to all template resources

  • parameter_merge_strategies - merge strategies for parameters from input sources

  • resource registry - resource mapping and registry for custom resource types

Other optional sections not described in this tutorial are:

  • encrypted_parameters - list of encrypted parameters

  • event_sinks - list of endpoints that would receive stack events

Parameter values

The environment extends the parameter handling possibilities, and there are certain rules that need to be adhered to. It basically separates the parameter values from the resource declarations.

In the template, a resource property can be assigned a value directly or through a declared parameter whose value can be overwritten either from CLI using the -p option, or by the environment. When used with an environment file, it therefore needs to include a parameter declaration, but no value needs to be specified if it is to be passed from the environment. An example of a minimal parameter declaration in the template looks like

    type: string
    type: string
    type: comma_delimited_list

The type has to be included in the parameter declaration. The values of the parameter names declared in the template are set in the environment without any other properties, like

  server_flavor: g1.standard-1-1
  server_image: ubuntu-20.04-x86_64
  server_secgroup: ssh_only

Values defined under parameters override default values under parameter_defaults. In case parameter values are entered from different sources, it is necessary to follow some rule for how to handle the presence of multiple declarations. Such rules, that can be specified down to parameter level, are collected under the section parameter_merge_strategies.

The options are overwrite (default strategy, unless otherwise specified), merge (for Python list-type arguments) and deep_merge (for JSON arguments). If no merge strategy is provided in an environment file, overwrite becomes the default strategy for all parameters in the environment file. This can be changed by default: merge.

The strategy must conform with the parameter type declared in the template. If declared as string, the merge strategy will concatenate two strings. To merge values into a list argument, the type must be comma_delimited_list.

To illustrate handling of parameters from different sources, consider a situation where we wish to deploy a server with a given image and where the key pair is user specific, and other parameters determined by the project.

The image parameter is considered part of the resource type and specified directly in the template, the flavor and security group have default values, and the server name, network, and key pair and a second security group are given values in the parameters section of environment file.

Say we create the stack with

openstack stack create --environment environment.yaml --template template.yaml --parameter "server_secgroup=http_only" test-stack

This would launch an instance with the property values

name: web_server
image: ubuntu-20.04-x86_64
flavor: g1.standard-1-1
key_name: common
network: internal_network 
security_groups: [ssh_only, http_only]

Template (template.yaml):

heat_template_version: 2018-08-31

description: Test instance

    type: string
    type: string
    type: string
    type: comma_delimited_list
    type: string

    type: OS::Nova::Server
      name: { get_param: server_name }
      image: ubuntu-20.04-x86_64
      flavor: { get_param: server_flavor }
      key_name: { get_param: server_key }
        - network: { get_param: local_network }
      security_groups: { get_param: server_secgroup }

Environment file (environment.yaml):

  server_name: web_server
  server_key: common
  server_secgroup: ssh_only
  local_network: internal_network

  server_flavor: g1.standard-1-1
  server_secgroup: default

    server_secgroup: merge


The OpenStack component Ceilometer is a telemetry service that gathers data from the cloud, including events (such as creation of an instance) and samples (like load figures for VCPUs). Sample data are stored in a database, managed by the Gnocchi service that allows to fetch data for project health monitoring and alarms.

The Aodh service collects measures from Gnocchi, performs logical operations on the data and trigger actions according to specified rules. This is used to trigger autoscaling in Heat as shown in Figure 5.

Figure 5. Autoscaling with Heat triggered by alarms.

To design an alarm, a number of factors need to be considered, such as

  • Source object - is data from the correct resource (for example server or server group) used for the alarm?

  • Does the relevant Gnocchi metric and measure exist, and presented in useful units and scale?

  • Does the alarm trigger the desired action?

  • How should the alarm be tuned with threshold setting and repeated action?

Note that the Gnocchi client is a plugin and installed as part of the OpenStack client suite. The individual Gnocchi client can be installed with pip install python-gnocchiclient.

Available metrics

When the Gnocchi service is active, the command

openstack metric resource list

prints a list of Gnocchi resources, such as a measurement time series. Each OpenStack resource can have several metric resources associated to it. A specific Gnocchi time series associated to a OpenStack resource with ID <resource-id> can be found in two steps:

openstack metric resource show <resource-id> -c metrics
openstack metric measures show <gnocchi-id>

which displays measurements and timestamps for the given resource (for example cpu) as shown in Figure 6.

Figure 6. Metric resources and aggregated values in Gnocchi.

The two operations can be combined in

openstack metric measures show --resource-id <resource-id> cpu

The commands of the Gnocchi client are invoked with the prefix gnocchi or, equivalently, with openstack metric. The definitions of the Gnocchi resources listed in Figure 6 are:

  • compute.instance.booting.time - booting time of instance (s)

  • cpu - accumulated CPU time (ns)

  • disk.ephemeral.size - size of ephemeral disk, by default 0 (GB)

  • disk.root.size - size of root disk in server configuration (GB)

  • memory.usage - instantaneous RAM usage (MB)

  • memory - size of RAM in server configuration (MB)

  • vcpus - number of VCPUs in server configuration

Archives and aggregation

Gnocchi archive policies are used to define the aggregation and storage policies for data. An archive policy definition uses has a fixed number of data points in a given time interval.

The default archive policy contains two definitions and one rule - metrics for seven days with granularity of one minute and for 365 days with granularity of one hour. 

Archive policy names are listed with

openstack metric archive-policy list

Details of an archive policy, including aggregation methods are displayed with

openstack metric archive-policy show <policy-name>

as shown in Figure 7.

Figure 7. Details of policy medium.

The cpu metric cannot be directly compared with a static threshold since it has an accumulated value. To compute a representative load figure for each sample, the data has to be aggregated with one of the built-in policies. The details of available policies (Figure 8) is produced with

openstack metric archive-policy list

Figure 8. List of archive policies.

The average CPU load is given by the policy rate:mean, the value of the aggregation method argument. The aggregated values of the CPU data are shown with

openstack metric measures show --aggregation rate:mean --resource-id <server-id> cpu

The periodical load figures are shown in Figure 9.

Figure 9. Aggregated CPU load after stress test.

Note that the aggregated values are given in nanoseconds, so with the granularity 300 seconds, 100% load corresponds to 3.0*1011 ns per core. This implies that the alarm threshold has to be multiplied by the number of cores and expressed in nanoseconds.

Stress testing

In stress testing a script or process is run on a server to induce high resource utilization. A simple stress test on a server can performed by running the stress utility through SSH, which generates load that can be controlled both in intensity and duration. It is installed on a tested server with

sudo apt install stress

To generate a certain CPU load for a specified time duration, the command is passed values to the parameters --cpu and --timeout.

The command uptime returns values of the current time, total system time in running state, number of users currently logged in, and the average load for the past 1, 5 and 15 minutes respectively. This can be useful to inspect the load figures on the server before and after the stress test,

sudo stress --cpu 100 --timeout 300

Figure 10 shows a stress test on a single core server of duration 300 seconds.

Figure 10. Stress test on Compute instance.

Create alarm

Alarms provide configurable monitoring services for resources running on OpenStack. It is used for resource scaling through Heat orchestration, but can in principle also be used for health monitoring of cloud resources. This section describes alarms based on performance data from the Gnocchi data base; alarms can also be based on events, such as resource creation or power state.

Alarms are managed with the relevant command following openstack alarm, for example

openstack alarm list

Gnocchi alarms follow a tri-state model:

  • ok - the rule governing the alarm has been evaluated as False

  • alarm - the rule governing the alarm has been evaluated as True

  • insufficient data - there are not enough data points available in the evaluation periods to determine the alarm state meaningfully

Roughly speaking, an alarm is defined to perform an action when its state changes to its target state, based on some input. Unfortunately, not all diagnostic tools for alarms, such as notifications, are available to non-administrator OpenStack users, so an alarm should be designed carefully to avoid lengthy troubleshooting. The alarm parameters should be set properly so that

  • Desired change in Gnocchi data changes the state of the alarm

  • Alarm actions result in some observable event

Alarms are created with openstack alarm create, which requires a number of parameters for its full specification.

The alarm may be triggered by comparing a single metric on a resource with a threshold value, aggregate data across metrics or resources. The case determines the alarm type, where the single metric case is gnocchi_resources_threshold.

The alarm type determines the arguments needed to create it. For a simple CPU load alarm, the metric is cpu, aggregation method is rate:mean, comparison operator is gt (greater than) and the threshold is expressed in nanoseconds. The alarm granularity must match the granularity of the metric configured in Gnocchi (in this case 300).

Alarms are evaluated on a periodic basis, defaulting to once every minute. The evaluation period parameter specifies the number of periods to evaluate over before an action is invoked. If repeat-actions is set to True, the actions are repeatedly notified while the alarm remains in its target state.

Finally, the resource type is set to instance and the parameter resource_id specifies the instance the alarm is collecting data from.

An alarm typically invokes an action through a webhook. For this purpose, openstack alarm create accepts the arguments

  • --alarm-action <webhook-url>

  • --ok-action <webhook-url>

  • --insufficient-data-action <webhook-url>

The parameters can be combined with repeat-actions and time-constraint <time-constraint> to control the number of alarm actions.

Figure 11 shows the command for creating an alarm with threshold 50% (for testing purposes) and the alarm action set to the webhook scale-up policy from

Figure 11. Create alarm with specified parameter values.

The details of an alarm can be printed with

openstack alarm show <alarm-id>

To trigger the alarm, high CPU load can induced on the target server running the stress utility through SSH as described in It then enters the alarm state and activates the webhook, which scales up the stack by adding a new server.

Testing an alarm action can also be done directly by manually setting it in its target state, through

openstack alarm state set --state alarm <alarm-id>

Its current state is read with

openstack alarm state get <alarm-id>

Create alarm with Heat

In autoscaling, the alarm should typically aggregate data from a cluster of servers, or server group. The command below computes the periodic CPU load on each instance, and aggregates these values by taking their mean.

openstack metric aggregates --resource-type=instance \
--granularity 300 \
"(aggregate mean (metric cpu rate:mean))" "project_id!='dummy'"

The type of alarm used in such situations is OS::Aodh::GnocchiAggregationByResourcesAlarm. To define it, the following four arguments are required:

  • resource-type

  • metric

  • threshold

  • query

The query is the last line of the command and defines the server group and resource aggregation method. The expression "project_id!='dummy'" filters instances with respect to project_id, where any ID will do (apart from 'dummy'), which returns data for all instances in the project.

In a Heat template, a better choice of server group might be stack_id which is created at the same time as the autoscaling group and filters out to the instances belonging to it. This is can be implemented with metadata. In the properties of the server, add the line

metadata: {"metering.server_group": {get_param: "OS::stack_id"}}

No other identifiers are required; Gnocchi has the keyword server_group defined for instances by default. In a Heat template, a query to filter data on server group would be

    - ''
    - - {'=': {server_group: {get_param: "OS::stack_id"}}}

The query can also be used to format the CPU load to a percentage figure, for example for a server group with two single core instances

openstack metric aggregates --resource-type instance  \
  "(* ( / (aggregate rate:mean (metric cpu mean)) 60000000000.0) 100)"  \

The result of the data transformation is shown in Figure 12.

Figure 12. Transformation of server group data to CPU load percentages.

The details of the query can be seen in the printout from openstack alarm show <alarm-id>.

Additional resources