Enable Kube State Metrics

The latest versions of Sysdig agent collects the following Kube State Metrics (KSM) families out of the box. These can not be disabled as they are required for certain parts of the Sysdig platform to function:

• cronjobs
• daemonsets
• deployments
• jobs
• namespaces
• nodes
• pods
• replicasets
• replicationcontrollers
• statefulsets

The following Kube State Metrics familes are collected by default but can be disabled if desired:

• services
• resourcequotas
• persistentvolumes
• persistentvolumeclaims
• storageclasses
• configmaps

However you can further customize KSM collection for specific metrics as follows:

Customize KSM Collection

Agent Versions 12.13.0 and Above

Several metrics are available to be scraped but are not enabled by default. Set the following configuration in dragent.yaml to enable them:

Pod Metrics

To enable metrics such as kube_pod_status_ready_time and kube_pod_start_time metrics, add the following configuration:

k8s_send_pod_times: true

To collect the poddisruptionbudgets metric family, use the following configuration.

k8s_extra_resources:
    include:
      - poddisruptionbudgets

For example, use the following configuration to collect the default resources as well as poddisruptionbudgets metrics:

k8s_extra_resources:
    include:
      - poddisruptionbudgets
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses

ConfigMap Metrics

To collect kube_configmap_info, use the following configuration:

k8s_extra_resources:
    include:
      - configmaps

Use sysdig-deploy chart v1.45.0 or above to collect thekube_configmap_info metrics.

Enable Node Annotations

By default, Sysdig Agent monitors Kubernetes nodes, so configuring k8s_extra_resources is unnecessary. However, to collect annotation metrics like kube_node_annotations, you must configure k8s_annotations_allowlist. Each annotation should be specified individually in the kubernetes.<resource-type>.annotation.<annotation-key> format.

For example, the following configuration collectes the kubernetes.io/foo and kubernetes.io/bar annotations on nodes:

k8s_annotations_allowlist:
  - "kubernetes.node.annotation.kubernetes.io/foo"
  - "kubernetes.node.annotation.kubernetes.io/bar"

Example KSM Configuration

k8s_extra_resources:
    include:
      - poddisruptionbudgets
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses
k8s_annotations_allowlist:
    - "kubernetes.node.annotation.kubernetes.io/foo"
    - "kubernetes.node.annotation.kubernetes.io/bar"

Agent Versions 12.9.0 and Above

The following metric familes are available to be scraped by the Sysdig Agent but are not enabled by default.

• certificatesigningrequests
• horizontalpodautoscalers
• ingresses

To enable the agent to collect the above metric families, you must edit the agent configuration file, dragent.yaml, and include them along with the other resources you would like to collect.

For example, to collect all configurable resources including ingresses and certificatesigningrequests, add the following to dragent.yaml:

k8s_extra_resources:
    include:
      - ingresses
      - certificatesigningrequests
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses

NOTE: When configuring k8s_extra_resources you must include all configurable Kube State Metrics families in order to collect metrics from those families. If you add ingresses but remove services, for example, the Sysdig agent will no longer collect services metrics. Adding only the ingresses or certificatesigningrequests configuration as follows will instruct the Sysdig agent to not collect the other configurable KSM services.

k8s_extra_resources:
    include:
      - ingresses
      - certificatesigningrequests

Therefore, ensure that you include the entire block of configuration corresponding to all the Kubernetes resources you wish to collect.

Agent Versions 12.5.0 and Above

horizontalpodautoscalers (HPA) kube state metrics are not collected by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml, and include it along with the other resources you would like to collect.

For example, to collect all supported resources including HPAs, add the following to dragent.yaml:

k8s_extra_resources:
    include:
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

NOTE: When configuring k8s_extra_resources you must include all configurable Kube State Metrics families in order to collect metrics from those families. If you add horizontalpodautoscalers but remove services, for example, the Sysdig agent will no longer collect services metrics. Adding only the horizontalpodautoscalers configuration as follows will instruct the Sysdig agent to not collect the other configurable KSM services.

k8s_extra_resources:
    include:
      - horizontalpodautoscalers

Therefore, ensure that you include the entire block of configuration corresponding to all the Kubernetes resources you wish to collect.

Agent Versions 12.3.x and 12.4.x

The Sysdig agent collects HPA, PVS, PV, Resourcequota, and Services kube state metrics by default.

To disable some of them, you must edit the agent config file, dragent.yaml, as follows:

k8s_extra_resources:
    include:
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

The above list includes all the supported resources so you must remove the resources you are not interested in. For example, if you want to disable services, use the following:

k8s_extra_resources:
    include:
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

For more information, see Understanding the Agent Configuration Files.