Integrate Keda for HPA

Sysdig supports Keda to deploy Kubernetes Horizontal Pod Autoscaler (HPA) using custom metrics exposed by Sysdig Monitor. You can do this by configuring Prometheus queries and endpoints in Keda. Keda uses that information to query your Prometheus server and create HPA. The HPA will takee care of scaling pods based on your usage of resources, such as CPU and memory.

This option replaces Sysdig’s existing custom metric server for HPA.

Install Keda

Requirements:

• Helm
• Keda v2.3 or above (Endpoint authentication)

Install Keda with helm by running the following command:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace \
  --set image.metricsApiServer.tag=2.4.0 --set image.keda.tag=2.4.0 \
  --set prometheus.metricServer.enabled=true

Create Authentication for Sysdig Prometheus Endpoint

Do the following in each namespace where you want to use Keda. This example uses the namespace, keda.

1. Create the secret with the API key as the bearer token:

   kubectl create secret generic keda-prom-secret --from-literal=bearerToken=<API_KEY> -n keda
   

2. Create the triggerAuthentication.yaml file:

   apiVersion: keda.sh/v1alpha1
   kind: TriggerAuthentication
   metadata:
     name: keda-prom-creds
   spec:
     secretTargetRef:
     - parameter: bearerToken
       name: keda-prom-secret
       key: bearerToken
   

3. Apply the configurations in the triggerAuthentication.yaml file :

   kubectl apply -f -n keda triggerAuthentication.yaml
   

Configure HPA

You can configure HPA for a Deployment, StatefulSet, or CRD. Keda uses a CRD to configure the HPA. You create a ScaledObject and it automatically sets up the metrics server and the HPA object under the hood.

1. To create a ScaledObject, specify the following:

   • spec.scaleTargetRef.name: The unique name of the Deployment.
   • spec.scaleTargetRef.kind: The kind of object to be scaled: Deployment, SStatefulSet, CustomResource.
   • spec.minReplicaCount: The minimum number of replicas that the Deployment should have.
   • spec.maxReplicaCount: The maximum number of replicas that the Deployment should have.

2. In the ScaledObject, use a trigger of type prometheus to get the metrics from your Sysdig Monitor account. To do so, specify the following:

   • triggers.metadata.serverAddress: The address of the Prometheus endpoint. It is the Sysdig Monitor URL with prefix /prometheus. For example: https://app.sysdigcloud.com/prometheus.
   • triggers.metadata.query: The PromQL query that will return a value. Ensure that the query returns a vector/scalar single element response.
   • triggers.metadata.metricName: The name of the metric that will be created in the kubernetes API endpoint, /apis/external.metrics.k8s.io/v1beta1.
   • triggers.metadata.threshold: The threshold that will be used to scale the Deployment.

3. Ensure that you add the authModes and authenticationRef to the trigger.

4. Check the ScaledObject. Here is an example of a ScaledObject:

   apiVersion: keda.sh/v1alpha1
   kind: ScaledObject
   metadata:
     name: keda-web
   spec:
     scaleTargetRef:
       kind: Deployment
       name: web
     minReplicaCount: 1
     maxReplicaCount: 4
     triggers:
     - type: prometheus
       metadata:
         serverAddress: https://app.sysdigcloud.com/prometheus
         metricName: sysdig_container_cpu_cores_used
         query: sum(sysdig_container_cpu_cores_used{kube_cluster_name="my-cluster-name", kube_namespace_name="keda", kube_workload_name = "web"} * 10
         threshold: "5"
         authModes: "bearer"
       authenticationRef:
         name: keda-prom-creds
   

Advanced Configurations

The ScaledObject permits additional options:

spec.pollingInterval:

Specify the interval to check each trigger on. By default KEDA will check each trigger source on every ScaledObject every 30 seconds.

Warning: setting this to a low value will cause Keda to make frequent API calls to the Prometheus endpoint. The minimum value for pollingInterval is 10 seconds. The scraping frequency of the Sysdig Agent is 10 seconds.

spec.cooldownPeriod:

The wait period between the last active trigger reported and scaling the resource back to 0. By default the value is 5 minutes (300 seconds).

spec.idleReplicaCount:

Enabling this property allows KEDA to scale the resource down to the specified number of replicas. If some activity exists on the target triggers, KEDA will scale the target resource immediately to the value of minReplicaCount and scaling is handed over to HPA. When there is no activity, the target resource is again scaled down to the value specified by idleReplicaCount. This setting must be less than minReplicaCount.

spec.fallback:

This property allows you to define a number of replicas if consecutive connection errors happens with the Prometheus endpoint of your Sysdig account.

• spec.fallback.failureThreshold: The number of consecutive errors to apply the fallback.
• spec.fallback.replicas: The number of replicas to apply in case of connection error.

spec.advanced.horizontalPodAutoscalerConfig.behavior:

This property allows you to define the behavior of the Kubernetes HPA Object. See the Kubernetes documentation for more information.

Learn More

• Trigger a Kubernetes HPA with Prometheus Metrics
• Trigger a Kubernetes HPA with Sysdig Metrics