This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

    Integrate Keda for HPA

    Sysdig supports Keda to deploy Kubernetes Horizontal Pod Autoscaler (HPA) using custom metrics exposed by Sysdig Monitor. You can do this by configuring Prometheus queries and endpoints in Keda. Keda uses that information to query your Prometheus server and create HPA. The HPA will takee care of scaling pods based on your usage of resources, such as CPU and memory.

    This option replaces Sysdig’s existing custom metric server for HPA.

    Install Keda

    Requirements:

    • Helm
    • Keda v2.3 or above (Endpoint authentication)

    Install Keda with helm by running the following command:

    helm repo add kedacore https://kedacore.github.io/charts
    helm repo update
    helm install keda kedacore/keda --namespace keda --create-namespace \
      --set image.metricsApiServer.tag=2.4.0 --set image.keda.tag=2.4.0 \
      --set prometheus.metricServer.enabled=true
    

    Create Authentication for Sysdig Prometheus Endpoint

    Do the following in each namespace where you want to use Keda. This example uses the namespace, keda.

    1. Create the secret with the API key as the bearer token:

      kubectl create secret generic keda-prom-secret --from-literal=bearerToken=<API_KEY> -n keda
      
    2. Create the triggerAuthentication.yaml file:

      apiVersion: keda.sh/v1alpha1
      kind: TriggerAuthentication
      metadata:
        name: keda-prom-creds
      spec:
        secretTargetRef:
        - parameter: bearerToken
          name: keda-prom-secret
          key: bearerToken
      
    3. Apply the configurations in the triggerAuthentication.yaml file :

      kubectl apply -f -n keda triggerAuthentication.yaml
      

    Configure HPA

    You can configure HPA for a Deployment, StatefulSet, or CRD. Keda uses a CRD to configure the HPA. You create a ScaledObject and it automatically sets up the metrics server and the HPA object under the hood.

    1. To create a ScaledObject, specify the following:

      • spec.scaleTargetRef.name: The unique name of the Deployment.
      • spec.scaleTargetRef.kind: The kind of object to be scaled: Deployment, SStatefulSet, CustomResource.
      • spec.minReplicaCount: The minimum number of replicas that the Deployment should have.
      • spec.maxReplicaCount: The maximum number of replicas that the Deployment should have.
    2. In the ScaledObject, use a trigger of type prometheus to get the metrics from your Sysdig Monitor account. To do so, specify the following:

      • triggers.metadata.serverAddress: The address of the Prometheus endpoint. It is the Sysdig Monitor URL with prefix /prometheus. For example: https://app.sysdigcloud.com/prometheus.
      • triggers.metadata.query: The PromQL query that will return a value. Ensure that the query returns a vector/scalar single element response.
      • triggers.metadata.metricName: The name of the metric that will be created in the kubernetes API endpoint, /apis/external.metrics.k8s.io/v1beta1.
      • triggers.metadata.threshold: The threshold that will be used to scale the Deployment.
    3. Ensure that you add the authModes and authenticationRef to the trigger.

    4. Check the ScaledObject. Here is an example of a ScaledObject:

      apiVersion: keda.sh/v1alpha1
      kind: ScaledObject
      metadata:
        name: keda-web
      spec:
        scaleTargetRef:
          kind: Deployment
          name: web
        minReplicaCount: 1
        maxReplicaCount: 4
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://app.sysdigcloud.com/prometheus
            metricName: sysdig_container_cpu_cores_used
            query: sum(sysdig_container_cpu_cores_used{kube_cluster_name="my-cluster-name", kube_namespace_name="keda", kube_workload_name = "web"} * 10
            threshold: "5"
            authModes: "bearer"
          authenticationRef:
            name: keda-prom-creds
             
      
      

    Note: The HPA will divide the value of the metric by the number of current replicas, therefore, try to avoid using the AVERAGE aggregation. Use SUM instead to aggregate the metrics by workload. For example, if the sum of all the values of all the pods is 100 and there are 5 replicas, the HPA will calculate that the value of the metric is 20.

    Advanced Configurations

    The ScaledObject permits additional options:

    spec.pollingInterval:

    Specify the interval to check each trigger on. By default KEDA will check each trigger source on every ScaledObject every 30 seconds.

    Warning: setting this to a low value will cause Keda to make frequent API calls to the Prometheus endpoint. The minimum value for pollingInterval is 10 seconds. The scraping frequency of the Sysdig Agent is 10 seconds.

    spec.cooldownPeriod:

    The wait period between the last active trigger reported and scaling the resource back to 0. By default the value is 5 minutes (300 seconds).

    spec.idleReplicaCount:

    Enabling this property allows KEDA to scale the resource down to the specified number of replicas. If some activity exists on the target triggers, KEDA will scale the target resource immediately to the value of minReplicaCount and scaling is handed over to HPA. When there is no activity, the target resource is again scaled down to the value specified by idleReplicaCount. This setting must be less than minReplicaCount.

    spec.fallback:

    This property allows you to define a number of replicas if consecutive connection errors happens with the Prometheus endpoint of your Sysdig account.

    • spec.fallback.failureThreshold: The number of consecutive errors to apply the fallback.
    • spec.fallback.replicas: The number of replicas to apply in case of connection error.

    spec.advanced.horizontalPodAutoscalerConfig.behavior:

    This property allows you to define the behavior of the Kubernetes HPA Object. See the Kubernetes documentation for more information.

    Learn More