Horizontally autoscale Kubernetes deployments on custom metrics

If you followed our blog series on Autoscaling on Kubernetes, you should already be familiar with Kubernetes’ Cluster autoscaler and the Vertical Pod Autoscaler used with Java 10 applications. This post will show you how to use the Horizontal Pod Autoscaler to autoscale your deployments based on custom metrics obtained from Prometheus. As a deployment example we’ve chosen our JEE Petstore example application on Wildfly to show that, beside metrics like cpu and memory, which are provided by default on Kubernetes, using our Wildfly Operator, all Java and Java Enterprise Edition / Wildfly specific metrics are automatically placed at your fingertips, available in Prometheus, allowing you to easily autoscale deployments.

Learn more about the different types of autoscaling features supported and automated by the Banzai Cloud Pipeline platform platform:

• Autoscaling Kubernetes clusters
• Vertical pod autoscaler
• Horizontally autoscale Kubernetes deployments on custom metrics
• Horizontal Pod Autoscaler Kubernetes Operator
• The Horizontal Pod Autoscaler operator reloaded
• Horizontal Pod Autoscaling based on custom Istio metrics

tl;dr: 🔗︎

The Pipeline platform supports and configures all these autoscaling options for you - with out of the box monitoring and metrics.

• Cluster autoscaler
• Vertical Pod Autoscaler
• Horizontal Pod Autoscaler

For custom metrics we open sourced Helm charts to deploy Metrics Adapter for Prometheus and Metrics Server in our Banzai Cloud Chart repository.

How Horizontal Pod Autoscaler works 🔗︎

Before we start, let’s recap the basic idea of autoscaling in Kubernetes. Autoscaling has two dimensions:

• Cluster Autoscaling, which deals with node scaling operations
• And the Horizontal Pod Autoscaler, which automatically scales the number of pods in a deployment or replica set

Cluster Autoscaling, together with the Horizontal Pod Autoscaler, can be used to dynamically adjust computing power, as well as the level of parallelism that your system needs to meet SLAs. While Cluster Autoscaling is highly dependent on the underlying capabilities of the cloud provider that’s hosting your cluster, the HPA operates independently of your IaaS/PaaS provider.

The HPA is included in Kubernetes by default, and is implemented as a control loop, with a period controlled by the controller manager’s --horizontal-pod-autoscaler-sync-period flag, whose default value is 30 seconds. The controller manager queries resource utilization against the metrics specified in each HorizontalPodAutoscaler definition.

Note that the Vertical Pod Autoscaler is a special case - for further information about VPA read this post

There are three kinds of metrics:

• per-pod resource metrics - like cpu, memory whose metrics are fetched from the resource metrics API for each pod targeted by the HorizontalPodAutoscaler, then compared against a targetAverageUtilization value or a raw targetAverageValue value.
• per-pod custom metrics - like per-pod resource metrics, but which are fetched from the custom metrics API and for who you can’t specify targetAverageUtilization values, only raw targetAverageValue values.
• object metrics - a single metric that is fetched (which describes the object in question), and compares it to the target value

Metrics are fetched either from the resource metrics API (for per-pod resource metrics), or custom ones from Custom Metrics API. The latter was introduced in Kubernetes 1.6, and enables HPA to access arbitrary metrics. Kubernetes 1.7 introduced the aggregation layer that allows 3rd party applications to extend the Kubernetes API by registering themselves as API add-ons. The Custom Metrics API, along with the aggregation layer, make it possible for monitoring systems like Prometheus to expose application-specific metrics to the HPA controller.

A Pipeline deployment will automatically install and dynamically configure Prometheus to collect metrics in a centralized (or federated Prometheus) cluster

To start autoscaling your deployments, all you have to do is to create a HorizontalPodAutoscaler resource, like the one below, which is an example of how to scale replica count for a wildfly-example deployment based on average cpu and memory usage.

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: wildfly-example
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: wildfly-example
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 1000Mi

As we’ve said, HPA is included by default in Kubernetes, while in order to use custom metrics you have to deploy Metrics Adapter for Prometheus so that the Metrics Server is able to query Prometheus.

You can find Helm charts to deploy Metrics Adapter for Prometheus and Metrics Server in our Chart repository.

It’s important to check your relabel rules in Prometheus for kubernetes-pods, because Metrics Server searches by pod and namespace, so they should be targeted as follows:

- source_labels: [__meta_kubernetes_namespace]
  action: replace
  target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
  action: replace
  target_label: pod

Example of scaling an example JEE application using custom metrics from Prometheus 🔗︎

As usual, our starting point requires that we have a cluster up and ready. To create a Kubernetes cluster in any of the supported cloud providers with Pipeline, follow the steps described in our previous post about Cluster Autoscaler.

Since Cluster Autoscaler and Horizontal Pod Autoscaler are complimentary, we advise that you enable autoscaling for your node pools, so that they are automatically expanded.

Aggregation API is enabled by default on AKS from version 1.9.0 onwards, and the same is true for GKE clusters from version 1.7.0 onwards, so make sure you use at least version 1.9. When utilizing Amazon clusters, Pipeline takes care of enabling Aggregation API for you.

The below HTTP request examples are from our Postman collection described here. Use Postman collection Create Deployment requests to deploy Helm charts.

1. Deploy our Prometheus chart for Monitoring and Prometheus Adapter

        POST {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/deployments
        {
       	"name": "banzaicloud-stable/pipeline-cluster-monitor"
        }
        

   Get release name from response and set in the url for Prometheus Adapter deployment:

        POST {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/deployments
        {
       	"name": "banzaicloud-stable/prometheus-adapter",
       	"values": {
       		"prometheus": {
       			"url": "http://{PROMETHUES_RELEASE_NAME}-prometheus-server.default.svc/prometheus"
       		}
       	}
       }
       

   TLS certificates are disabled by default. If you want to enable them, you have to set tls.enable=true and generate your certificates, and set tls.key & tls.certificate.

2. Install Metrics Server if you’re using Amazon or Azure (on GKE you have it by default)

       POST {{url}}/api/v1/orgs/{{orgId}}/clusters/{{cluster_id}}/deployments
       {
        "name": "banzaicloud-stable/metrics-monitor"
       }
       

3. Make sure you’re able to query custom metrics from Metrics Server so the connection to Prometheus Adapter works fine:

       kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
       

4. Deploy Wildfly operator and JEE 7 Petstore Application

       wget https://raw.githubusercontent.com/banzaicloud/wildfly-operator/master/deploy/rbac.yaml
       wget https://raw.githubusercontent.com/banzaicloud/wildfly-operator/master/deploy/operator.yaml
       wget https://raw.githubusercontent.com/banzaicloud/wildfly-operator/master/deploy/cr.yaml
       kubectl apply -f .
       

   Wildfly Operator starts the Prometheus JMX exporter as a Java agent and attaches annotations necessary for Prometheus to scrape your pods. If you’re using other Java applications, you can use the Prometheus JMX Operator to get all these out of the box.

5. Generate load for Application

   To this end we will use a simple Go load generator app, which makes number GET requests concurrently to a certain URL. First obtain the Application endpoint:

       kubectl describe WildflyAppServer
   
       Status:
         External Addresses:
           Application:  35.225.18.78:8080
           Management:   35.225.18.78:9990
         Nodes:
           wildfly-example-65c687d55f-fr4g7
       

   Install and run load test:

       go get -u github.com/rakyll/hey
       hey -n 10000 -q 15  http://35.225.18.78:8080/applicationPetstore/shopping/searchresult.xhtml
       

6. List available POD related metrics from Metrics API

       kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep "pods/"
       

   Alternatively, you can open up the Prometheus UI to check all the available metrics and choose the right ones.

   For demonstrative purposes, and to simulate a realistic use case, we inserted a random sleep time in our Petstore Application code. As a result of the load test we see that org_xnio_Xnio_BusyWorkerThreadCount has increased sharply, and so we choose this metric.

7. Create HorizontalPodAutoscaler resource

       apiVersion: autoscaling/v2beta1
       kind: HorizontalPodAutoscaler
       metadata:
         name: wildfly-example
       spec:
         scaleTargetRef:
           apiVersion: extensions/v1beta1
           kind: Deployment
           name: wildfly-example
         minReplicas: 1
         maxReplicas: 5
         metrics:
         - type: Pods
           pods:
             metricName: **org_xnio_Xnio_BusyWorkerThreadCount**
             targetAverageValue: 10
       

   After some time, if you describe your HPA you should see something like the below:

       kubectl describe hpa wildfly-example
       Name:                                             wildfly-example
       Namespace:                                        default
       Labels:                                           <none>
       Annotations:                                      <none>
       CreationTimestamp:                                Wed, 27 Jun 2018 01:19:35 +0200
       Reference:                                        Deployment/wildfly-example
       Metrics:                                          ( current / target )
         "org_xnio_Xnio_BusyWorkerThreadCount" on pods:  17 / 10
       Min replicas:                                     1
       Max replicas:                                     10
       Conditions:
         Type            Status  Reason              Message
         ----            ------  ------              -------
         AbleToScale     False   BackoffBoth         the time since the previous scale is still within both the downscale and upscale forbidden windows
         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from pods metric org_xnio_Xnio_BusyWorkerThreadCount
         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
       Events:
         Type     Reason                        Age   From                       Message
         ----     ------                        ----  ----                       -------
         Normal   SuccessfulRescale             2m    horizontal-pod-autoscaler  New size: 4; reason: pods metric org_xnio_Xnio_BusyWorkerThreadCount above target
       

What’s next 🔗︎

At Banzai Cloud Prometheus is one of our favorite tools. It’s no surprise that we’re very enthusiastic about autoscaling based on metrics obtained from Prometheus, and we’re continuing to explore possibilities that will allow us to autoscale based on, not just simple metrics, but more complex Prometheus queries, so stay tuned.

Learn more about the different types of autoscaling features supported and automated by the Banzai Cloud Pipeline platform platform:

• Autoscaling Kubernetes clusters
• Vertical pod autoscaler
• Horizontally autoscale Kubernetes deployments on custom metrics
• Horizontal Pod Autoscaler Kubernetes Operator
• The Horizontal Pod Autoscaler operator reloaded
• Horizontal Pod Autoscaling based on custom Istio metrics