Enable Kube State Metrics

Sysdig can collect a number of Kube State Metrics (KSMs) when Cluster Shield is correctly configured. All KSMs are default metrics, and do not count towards time series usage.

Supported Kubernetes Resources

Cluster Shield can collect the following Kubernetes resources:

  • certificatesigningrequest
  • configmap
  • cronjob
  • daemonset
  • deployment
  • horizontalpodautoscaler
  • ingress
  • job
  • namespace
  • node
  • persistentvolume
  • persistentvolumeclaim
  • pod
  • poddisruptionbudget
  • replicaset
  • replicationcontroller
  • resourcequota
  • service
  • statefulset
  • storageclass

Collect KSMs

To collect KSMs, configure Cluster Shield.

This replaces the legacy classic agent method. See Cluster Shield 1.11.0.

Cluster Shield 1.11.0 and Above

To collect Kubernetes State Metrics (KSM) with Cluster Shield, add this configuration to values.yaml:

features:
  monitor:
    kube_state_metrics:
      enabled: true

You can also apply it with a Helm command:

  --set clusterShield.cluster_shield.features.monitor.kube_state_metrics.enabled=true

To enable KSM, Kubernetes metadata, Kubernetes events, and metadata messages carrying CostAdvisor data, use this command:

  --set clusterShield.cluster_shield.features.kubernetes_metadata.enabled: true \
  --set clusterShield.cluster_shield.features.monitor.kube_state_metrics.enabled=true \
  --set clusterShield.cluster_shield.features.monitor.kubernetes_events.enabled=true \

Collect KSMs (Legacy)

To collect KSM with the classic agent, enable the kubernetes_metadata feature. This configuration also enables the collection of Kubernetes events.

Required KSM Families

Some metric families are enabled by default, while others must be manually enabled.

The following KSM families cannot be disabled as they are required for certain parts of the Sysdig Platform to function:

  • cronjobs
  • daemonsets
  • deployments
  • jobs
  • namespaces
  • nodes
  • pods
  • replicasets
  • replicationcontrollers
  • statefulsets

The following Kube State Metrics families are collected by default, but can be disabled if desired:

  • services
  • resourcequotas
  • persistentvolumes
  • persistentvolumeclaims
  • storageclasses
  • configmaps

Agent Versions 12.13.0 to 13.8.1

Several metrics are available to be scraped but are not enabled by default. Set the following configuration in dragent.yaml to enable them. To apply configurations, see Configure the Agent.

Pod Metrics

To enable metrics such as kube_pod_status_ready_time and kube_pod_start_time metrics, add the following configuration:

k8s_send_pod_times: true

To collect the poddisruptionbudgets metric family, use the following configuration.

k8s_extra_resources:
    include:
      - poddisruptionbudgets

For example, use the following configuration to collect the default resources as well as poddisruptionbudgets metrics:

k8s_extra_resources:
    include:
      - poddisruptionbudgets
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses

ConfigMap Metrics

To collect kube_configmap_info, use the following configuration:

k8s_extra_resources:
    include:
      - configmaps

Use sysdig-deploy chart v1.45.0 or above to collect thekube_configmap_info metrics.

Enable Node Annotations

By default, Sysdig monitors Kubernetes nodes, so configuring k8s_extra_resources is unnecessary. However, to collect annotation metrics like kube_node_annotations, you must configure k8s_annotations_allowlist. Each annotation should be specified individually in the kubernetes.<resource-type>.annotation.<annotation-key> format.

For example, the following configuration collects the kubernetes.io/foo and kubernetes.io/bar annotations on nodes:

k8s_annotations_allowlist:
  - "kubernetes.node.annotation.kubernetes.io/foo"
  - "kubernetes.node.annotation.kubernetes.io/bar"

Example KSM Configuration

k8s_extra_resources:
    include:
      - poddisruptionbudgets
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses
k8s_annotations_allowlist:
    - "kubernetes.node.annotation.kubernetes.io/foo"
    - "kubernetes.node.annotation.kubernetes.io/bar"

Agent Versions 12.9.0 and Above

The following metric families are available to be scraped by the Sysdig Agent but are not enabled by default.

  • certificatesigningrequests
  • horizontalpodautoscalers
  • ingresses

To enable the agent to collect the above metric families, you must edit the agent configuration file, dragent.yaml, and include them along with the other resources you would like to collect. To apply configurations, see Configure the Agent.

For example, to collect all configurable resources including ingresses and certificatesigningrequests, add the following to dragent.yaml:

k8s_extra_resources:
    include:
      - ingresses
      - certificatesigningrequests
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - storageclasses

NOTE: When configuring k8s_extra_resources you must include all configurable Kube State Metrics families in order to collect metrics from those families. If you add ingresses but remove services, for example, the Sysdig agent will no longer collect services metrics. Adding only the ingresses or certificatesigningrequests configuration as follows will instruct the Sysdig agent to not collect the other configurable KSM services.

k8s_extra_resources:
    include:
      - ingresses
      - certificatesigningrequests

Therefore, ensure that you include the entire block of configuration corresponding to all the Kubernetes resources you wish to collect.

Agent Versions 12.5.0 and Above

horizontalpodautoscalers (HPA) kube state metrics are not collected by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml, and include it along with the other resources you would like to collect. To apply configurations, see Configure the Agent.

For example, to collect all supported resources including HPAs, add the following to dragent.yaml:

k8s_extra_resources:
    include:
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

NOTE: When configuring k8s_extra_resources you must include all configurable Kube State Metrics families in order to collect metrics from those families. If you add horizontalpodautoscalers but remove services, for example, the Sysdig agent will no longer collect services metrics. Adding only the horizontalpodautoscalers configuration as follows will instruct the Sysdig agent to not collect the other configurable KSM services.

k8s_extra_resources:
    include:
      - horizontalpodautoscalers

Therefore, ensure that you include the entire block of configuration corresponding to all the Kubernetes resources you wish to collect.

Agent Versions 12.3.x and 12.4.x

The Sysdig agent collects HPA, PVS, PV, Resourcequota, and Services kube state metrics by default.

To disable some of them, you must edit the agent config file, dragent.yaml, as follows:

k8s_extra_resources:
    include:
      - services
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

The above list includes all the supported resources so you must remove the resources you are not interested in. For example, if you want to disable services, use the following:

k8s_extra_resources:
    include:
      - resourcequotas
      - persistentvolumes
      - persistentvolumeclaims
      - horizontalpodautoscalers

Learn More