HPA with Prometheus

This topic describes how to configure HPA to read pod scaling recommendations from Prometheus.

HPA can be run with external custom application metrics using the Prometheus adapter and Avesha's external metric, smartscaler_hpa_num_pods.

note

The Smart Scaler agent chart, version 2.0.2 and later, supports Prometheus to extract metrics from the Inference Agent. The HPA can then read the extracted metrics from Prometheus.

Step 1: Install the Prometheus Adapter

1. Create the Prometheus adapter using this helm values file.

2. In the values.yaml file, update the Prometheus URL and port.

   prometheus:
     # Value is templated
     url: http://prometheus.default.svc
     port: 9090

3. Use the following commands to install the Prometheus Adapter:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install my-release prometheus-community/prometheus-adapter -f values.yaml

Step 2: Update the Prometheus Adapter ConfigMap

Update the ConfigMap to add the external app custom metric query. This will make the smartscaler_hpa_num_pods metrics available to the API Service through the Prometheus Adapter.

1. Use the following command to edit the ConfigMap:

   kubectl -n monitoring edit configmap prometheus-adapter

2. Add the following externalRules in the ConfigMap. Note that, by default, an HPA adds its own namespace. Hence, the metric should be from that namespace.

   externalRules:
   - seriesQuery: 'smartscaler_hpa_num_pods{ss_deployment_name!="",ss_namespace!=""}'
     resources:
       namespace: false #Setting false lets the metric to be queried from any namespace and not limit to the namespace where HPA is deployed.
       #name:
       #matches: ^(.*)
       #as: "smartscaler_hpa_num_pods"
     metricsQuery: 'sum by (ss_deployment_name, ss_namespace) (smartscaler_hpa_num_pods{<<.LabelMatchers>>})'

Step 3: Create the Prometheus Adapter Service

Create the Prometheus Adapter Service using the following template.

apiVersion: v1
kind: Service
metadata:
  name: external-metrics-apiserver
  namespace: monitoring
spec:
  ports:
  - port: 443
    targetPort: 6443
  selector:
    app: external-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: external-metrics-apiserver
    namespace: monitoring
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

Step 4: Verify the Metrics on APIService

Verify the metrics on APIService using the following command:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq 

Expected Output

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "smartscaler_hpa_num_pods",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

To verify more details, use the following command:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/smartscaler_hpa_num_pods" | jq '.'

Expected Output

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "smartscaler_hpa_num_pods",
      "metricLabels": {
        "ss_deployment_name": "cart",
        "ss_namespace": "<namespace>" #Metric can be any queried with any namespace 
      },
      "timestamp": "2024-05-08T17:04:55Z",
      "value": "2"
    },
    {
      "metricName": "smartscaler_hpa_num_pods",
      "metricLabels": {
        "ss_deployment_name": "catalog",
        "ss_namespace": "<namespace>"
      },
      "timestamp": "2024-05-08T17:04:55Z",
      "value": "2"
    },
    {
      "metricName": "smartscaler_hpa_num_pods",
      "metricLabels": {
        "ss_deployment_name": "frontend",
        "ss_namespace": "<namespace>"
      },
      "timestamp": "2024-05-08T17:04:55Z",
      "value": "11"
    },
    {
      "metricName": "smartscaler_hpa_num_pods",
      "metricLabels": {
        "ss_deployment_name": "order",
        "ss_namespace": "<namespace>"
      },
      "timestamp": "2024-05-08T17:04:55Z",
      "value": "4"
    },
    {
      "metricName": "smartscaler_hpa_num_pods",
      "metricLabels": {
        "ss_deployment_name": "payment",
        "ss_namespace": "<namespace>"
      },
      "timestamp": "2024-05-08T17:04:55Z",
      "value": "2"
    }
  ]
}

Step 5: Create an HPA Object to Read Recommendations

info

You must create an HPA object that points to your Prometheus data source to collect pod scaling recommendations.

1. Create an HPA object file called smart-scaler-hpa.yaml that points to the smartscaler_hpa_num_pods metric to scale the deployments using the following template.

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: <hpa name>
     namespace: <namespace>
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nginx
     behavior:
       scaleDown:
         policies:
           - type: Pods
             value: 4
             periodSeconds: 60
           - type: Percent
             value: 10
             periodSeconds: 60
         stabilizationWindowSeconds: 10
     minReplicas: 2
     maxReplicas: 30
     metrics:
       - type: External
         external:
           metric:
             name: smartscaler_hpa_num_pods
             selector:
               matchLabels:
                 ss_deployment_name: "<deployment name>"
                 ss_namespace: "<namespace>"
                 ss_cluster_name: "<cluster name>"
           target:
             type: AverageValue
             averageValue: "1"

2. Apply the smart-scaler-hpa.yaml object file using the following command:

   kubectl apply -f smart-scaler-hpa.yaml -n <application-namespace>

The HPA will now collect pod scaling recommendations from your Prometheus data source.