This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

  • 1:
    • 2:
      • 3:
        • 4:

          Metrics and Label Mapping

          This topic outlines the mapping between the metrics and label naming conventions in the Sysdig legacy datastore and the new Sysdig datastore.

          1 -

          Mapping Classic Metrics with Context-Specific PromQL Metrics

          Sysdig classic metrics such as cpu.used.percent previously returned values from a process, container, or host depending on the query segmentation or scope. You can now use context-explicit metrics which aligns with the flat model and resource specific semantics of Prometheus naming schema. Your existing dashboards and alerts will be automatically migrated to the new naming convention.

          Sysdig Classic MetricsContext-Specific Metrics in Prometheus Notation
          cpu.cores.usedsysdig_container_cpu_cores_used
          sysdig_host_cpu_cores_used
          sysdig_program_cpu_cores_used
          cpu.cores.used.percentsysdig_container_cpu_cores_used_percent
          sysdig_host_cpu_cores_used_percent
          sysdig_program_cpu_cores_used_percent
          cpu.used.percentsysdig_container_cpu_used_percent
          sysdig_host_cpu_used_percent
          sysdig_program_cpu_used_percent
          fd.used.percentsysdig_container_fd_used_percent
          sysdig_host_fd_used_percent
          sysdig_program_fd_used_percent
          file.bytes.insysdig_container_file_in_bytes
          sysdig_host_file_in_bytes
          sysdig_program_file_in_bytes
          file.bytes.outsysdig_container_file_out_bytes
          sysdig_host_file_out_bytes
          sysdig_program_file_out_bytes
          file.bytes.totalsysdig_container_file_total_bytes
          sysdig_host_file_total_bytes
          sysdig_program_file_total_bytes
          file.error.open.countsysdig_container_file_error_open_count
          sysdig_host_file_error_open_count
          sysdig_program_file_error_open_count
          file.error.total.countsysdig_container_file_error_total_count
          sysdig_host_file_error_total_count
          sysdig_program_file_error_total_count
          file.iops.insysdig_container_file_in_iops
          sysdig_host_file_in_iops
          sysdig_program_file_in_iops
          file.iops.outsysdig_container_file_out_iops
          sysdig_host_file_out_iops
          sysdig_program_file_out_iops
          file.iops.totalsysdig_container_file_total_iops
          sysdig_host_file_total_iops
          sysdig_program_file_total_iops
          file.open.countsysdig_container_file_open_count
          sysdig_host_file_open_count
          sysdig_program_file_open_count
          file.time.insysdig_container_file_in_time
          sysdig_host_file_in_time
          sysdig_program_file_in_time
          file.time.outsysdig_container_file_out_time
          sysdig_host_file_out_time
          sysdig_program_file_out_time
          file.time.totalsysdig_container_file_total_time
          sysdig_host_file_total_time
          sysdig_program_file_total_time
          fs.bytes.freesysdig_container_fs_free_bytes
          sysdig_fs_free_bytes
          sysdig_host_fs_free_bytes
          fs.bytes.totalsysdig_container_fs_total_bytes
          sysdig_fs_total_bytes
          sysdig_host_fs_total_bytes
          fs.bytes.usedsysdig_container_fs_used_bytes
          sysdig_fs_used_bytes
          sysdig_host_fs_used_bytes
          fs.free.percentsysdig_container_fs_free_percent
          sysdig_fs_free_percent
          sysdig_host_fs_free_percent
          fs.inodes.total.countsysdig_container_fs_inodes_total_count
          sysdig_fs_inodes_total_count
          sysdig_host_fs_inodes_total_count
          fs.inodes.used.countsysdig_container_fs_inodes_used_count
          sysdig_fs_inodes_used_count
          sysdig_host_fs_inodes_used_count
          fs.inodes.used.percentsysdig_container_fs_inodes_used_percent
          sysdig_fs_inodes_used_percent
          sysdig_host_fs_inodes_used_percent
          fs.largest.used.percentsysdig_container_fs_largest_used_percent
          sysdig_host_fs_largest_used_percent
          fs.root.used.percentsysdig_container_fs_root_used_percent
          sysdig_host_fs_root_used_percent
          fs.used.percentsysdig_container_fs_used_percent
          sysdig_fs_used_percent
          sysdig_host_fs_used_percent
          host.error.countsysdig_container_syscall_error_count
          sysdig_host_syscall_error_count
          infosysdig_agent_info
          sysdig_container_info
          sysdig_host_info
          memory.bytes.totalsysdig_host_memory_total_bytes
          sysdig_container_memory_used_bytes
          sysdig_host_memory_used_bytes
          sysdig_program_memory_used_bytes
          memory.bytes.virtualsysdig_container_memory_virtual_bytes
          sysdig_host_memory_virtual_bytes
          memory.swap.bytes.usedsysdig_container_memory_swap_used_bytes
          sysdig_host_memory_swap_used_bytes
          memory.used.percentsysdig_container_memory_used_percent
          sysdig_host_memory_used_percent
          net.bytes.insysdig_connection_net_in_bytes
          sysdig_container_net_in_bytes
          sysdig_host_net_in_bytes
          sysdig_program_net_in_bytes
          net.bytes.outsysdig_connection_net_out_bytes
          sysdig_container_net_out_bytes
          sysdig_host_net_out_bytes
          sysdig_program_net_out_bytes
          net.bytes.totalsysdig_connection_net_total_bytes
          sysdig_container_net_total_bytes
          sysdig_host_net_total_bytes
          sysdig_program_net_total_bytes
          net.connection.count.insysdig_connection_net_connection_in_count
          sysdig_container_net_connection_in_count
          sysdig_host_net_connection_in_count
          sysdig_program_net_connection_in_count
          net.connection.count.outsysdig_connection_net_connection_out_count
          sysdig_container_net_connection_out_count
          sysdig_host_net_connection_out_count
          sysdig_program_net_connection_out_count
          net.connection.count.totalsysdig_connection_net_connection_total_count
          sysdig_container_net_connection_total_count
          sysdig_host_net_connection_total_count
          sysdig_program_net_connection_total_count
          net.request.countsysdig_connection_net_request_count
          sysdig_container_net_request_count
          sysdig_host_net_request_count
          sysdig_program_net_request_count
          net.error.countsysdig_container_net_error_count
          sysdig_host_net_error_count
          sysdig_program_net_error_count
          net.request.count.insysdig_connection_net_request_in_count
          sysdig_container_net_request_in_count
          sysdig_host_net_request_in_count
          sysdig_program_net_request_in_count
          net.request.count.outsysdig_connection_net_request_out_count
          sysdig_container_net_request_out_count
          sysdig_host_net_request_out_count
          sysdig_program_net_request_out_count
          net.request.timesysdig_connection_net_request_time
          sysdig_container_net_request_time
          sysdig_host_net_request_time
          sysdig_program_net_request_time
          net.request.time.insysdig_connection_net_request_in_time
          sysdig_container_net_request_in_time
          sysdig_host_net_request_in_time
          sysdig_program_net_request_in_time
          net.request.time.outsysdig_connection_net_request_out_time
          sysdig_container_net_request_out_time
          sysdig_host_net_request_out_time
          sysdig_program_net_request_out_time
          net.server.bytes.insysdig_container_net_server_in_bytes
          sysdig_host_net_server_in_bytes
          net.server.bytes.outsysdig_container_net_server_out_bytes
          sysdig_host_net_server_out_bytes
          net.server.bytes.totalsysdig_container_net_server_total_bytes
          sysdig_host_net_server_total_bytes
          net.sql.error.countsysdig_container_net_sql_error_count
          sysdig_host_net_sql_error_count
          net.sql.request.countsysdig_container_net_sql_request_count
          sysdig_host_net_sql_request_count
          net.tcp.queue.lensysdig_container_net_tcp_queue_len
          sysdig_host_net_tcp_queue_len
          sysdig_program_net_tcp_queue_len
          proc.countsysdig_container_proc_count
          sysdig_host_proc_count
          sysdig_program_proc_count
          thread.countsysdig_container_thread_count
          sysdig_host_thread_count
          sysdig_program_thread_count
          uptimesysdig_container_up
          sysdig_host_up
          sysdig_program_up

          2 -

          Mapping Between Classic Metrics and PromQL Metrics

          Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.

          Entity

          Type

          PromQL Metric Name

          Classic Metric Name

          Label

          Classic Label

          host

          info

          sysdig_host_info

          Not exposed

          • host_mac

          • host

          • instance_id

          • agent_tag_{*}

          • host.mac

          • host.hostName

          • host.instanceId

          • agent.tag.{*}

          sysdig_cloud_provider_info

          • host_mac

          • provider_id

          • account_id

          • region

          • availability_zone

          • instance_type

          • tag_{*}

          • security_groups

          • host_ip_public

          • host_ip_private

          • host_name

          • name

          • host.mac

          • cloudProvider.id

          • cloudProvider.account.id

          • cloudProvider.region

          • cloudProvider.availabilityZone

          • cloudProvider.instance.type

          • cloudProvider.tag.{*}

          • cloudProvider.securityGroups

          • cloudProvider.host.ip.public

          • cloudProvider.host.ip.private

          • cloudProvider.host.name

          • cloudProvider.name

          data

          sysdig_host_cpu_used_percent

          cpu.used.percent

          • host_mac

          • host

          • host.mac

          • host.hostname

          sysdig_host_cpu_cores_used

          cpu.cores.used

          sysdig_host_cpu_user_percent

          cpu.user.percent

          sysdig_host_cpu_idle_percent

          cpu.idle.percent

          sysdig_host_cpu_iowait_percent

          cpu.iowait.percent

          sysdig_host_cpu_nice_percent

          cpu.nice.percent

          sysdig_host_cpu_stolen_percent

          cpu.stolen.percent

          sysdig_host_cpu_system_percent

          cpu.system.percent

          sysdig_host_fd_used_percent

          fd.used.percent

          sysdig_host_file_error_open_count

          file.error.open.count

          sysdig_host_file_error_total_count

          file.error.total.count

          sysdig_host_file_in_bytes

          file.bytes.in

          sysdig_host_file_in_iops

          file.iops.in

          sysdig_host_file_in_time

          file.time.in

          sysdig_host_file_open_count

          file.open.count

          sysdig_host_file_out_bytes

          file.bytes.out

          sysdig_host_file_out_iops

          file.iops.out

          sysdig_host_file_out_time

          file.time.out

          sysdig_host_load_average_15m

          load.average.15m

          sysdig_host_load_average_1m

          load.average.1m

          sysdig_host_load_average_5m

          load.average.5m

          sysdig_host_memory_available_bytes

          memory.bytes.available

          sysdig_host_memory_total_bytes

          memory.bytes.total

          sysdig_host_memory_used_bytes

          memory.bytes.used

          sysdig_host_memory_swap_available_bytes

          memory.swap.bytes.available

          sysdig_host_memory_swap_total_bytes

          memory.swap.bytes.total

          sysdig_host_memory_swap_used_bytes

          memory.swap.bytes.used

          sysdig_host_memory_virtual_bytes

          memory.bytes.virtual

          sysdig_host_net_connection_in_count

          net.connection.count.in

          sysdig_host_net_connection_out_count

          net.connection.count.out

          sysdig_host_net_error_count

          net.error.count

          sysdig_host_net_in_bytes

          net.bytes.in

          sysdig_host_net_out_bytes

          net.bytes.out

          sysdig_host_net_tcp_queue_len

          net.tcp.queue.len

          sysdig_host_proc_count

          proc.count

          sysdig_host_system_uptime

          system.uptime

          sysdig_host_thread_count

          thread.count

          container

          info

          sysdig_container_info

          Not exposed

          container_id

          container_id

          container_full_id

          none

          host_mac

          host.mac

          container

          container.name

          container_type

          container.type

          image

          container.image

          image_id

          container.image.id

          mesos_task_id

          container.mesosTaskId

          Only available in Mesos orchestrator.

          cluster

          kubernetes.cluster.name

          Present only if the container is part of Kubernetes.

          pod

          kubernetes.pod.name

          Present only if the container is part of Kubernetes

          namespace

          kubernetes.namespace.name

          Present only if the container is part of Kubernetes.

          data

          sysdig_container_cpu_used_percent

          cpu.used.percent

          • host_mac

          • container_id

          • container_type

          • container

          • host.mac

          • container.id

          • container.type

          • container.name

          sysdig_container_cpu_cores_used

          cpu.cores.used

          sysdig_container_cpu_cores_used_percent

          cpu.cores.used.percent

          sysdig_container_cpu_quota_used_percent

          cpu.quota.used.percent

          sysdig_container_cpu_shares

          cpu.shares.count

          sysdig_container_cpu_shares_used_percent

          cpu.shares.used.percent

          sysdig_container_fd_used_percent

          fd.used.percent

          sysdig_container_file_error_open_count

          file.error.open.count

          sysdig_container_file_error_total_count

          file.error.total.count

          sysdig_container_file_in_bytes

          file.bytes.in

          sysdig_container_file_in_iops

          file.iops.in

          sysdig_container_file_in_time

          file.time.in

          sysdig_container_file_open_count

          file.open.count

          sysdig_container_file_out_bytes

          file.bytes.out

          sysdig_container_file_out_iops

          file.iops.out

          sysdig_container_file_out_time

          file.time.out

          sysdig_container_memory_limit_bytes

          memory.limit.bytes

          sysdig_container_memory_limit_used_percent

          memory.limit.used.percent

          sysdig_container_memory_swap_available_bytes

          memory.swap.bytes.available

          sysdig_container_memory_swap_total_bytes

          memory.swap.bytes.total

          sysdig_container_memory_swap_used_bytes

          memory.swap.bytes.used

          sysdig_container_memory_used_bytes

          memory.bytes.used

          sysdig_container_memory_virtual_bytes

          memory.bytes.virtual

          sysdig_container_net_connection_in_count

          net.connection.count.in

          sysdig_container_net_connection_out_count

          net.connection.count.out

          sysdig_container_net_error_count

          net.error.count

          sysdig_container_net_in_bytes

          net.bytes.in

          sysdig_container_net_out_bytes

          net.bytes.out

          sysdig_container_net_tcp_queue_len

          net.tcp.queue.len

          sysdig_container_proc_count

          proc.count

          sysdig_container_swap_limit_bytes

          swap.limit.bytes

          sysdig_container_thread_count

          thread.count

          Process/ Program

          Info

          sysdig_program_info

          not exposed

          program

          proc.name

          cmd_line

          proc.commandLine

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          data

          sysdig_program_cpu_used_percent

          cpu.used.percent

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          sysdig_program_memory_used_bytes

          memory.bytes.used

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          sysdig_program_net_in_bytes

          net.bytes.in

          container_id

          container.id

          host_mac

          host.mac

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          sysdig_program_net_out_bytes

          net.bytes.out

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          sysdig_program_proc_count

          proc.count

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          sysdig_program_thread_count

          thread.count

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          program

          proc.name

          cmd_line

          proc.commandLine

          fs

          info

          sysdig_fs_info

          not exposed

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          device

          fs.device

          mount_dir

          fs.mountDir

          type

          fs.type

          data

          sysdig_fs_free_bytes

          fs.bytes.free

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          device

          fs.device

          sysdig_fs_inodes_total_count

          fs.inodes.total.count

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          device

          fs.device

          sysdig_fs_inodes_used_count

          fs.inodes.used.count

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          device

          fs.device

          sysdig_fs_total_bytes

          fs.bytes.total

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          device

          fs.device

          fs.bytes.used

          host_mac

          host.mac

          container_id

          container.id

          container_type

          container.type

          devide

          fs.device

          3 -

          Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

          Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.

          For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.

          Resource

          Sysdig Metrics

          Kubernetes State Metrics

          Label

          Example / More Information

          Pod

          kubernetes.pod.containers.waiting

          kube_pod_container_status_waiting

          • container=<container-name>

          • pod=<pod-name>

          • namespace=<pod-namespace>

          kubernetes.pod.resourceLimits.cpuCores

          kubernetes.pod.resourceLimits.memBytes

          kube_pod_container_resource_limits

          kube_pod_sysdig_resource_limits_memory_bytes

          kube_pod_sysdig_resource_limits_cpu_cores

          • resource=<resource-name>

          • unit=<resource-unit>

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • node=< node-name>

          {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"}

          {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}

          kubernetes.pod.resourceRequests.cpuCores

          kubernetes.pod.resourceRequests.memBytes

          kube_pod_container_resource_requests

          kube_pod_sysdig_resource_requests_cpu_cores

          kube_pod_sysdig_resource_requests_memory_bytes

          • resource=<resource-name>

          • unit=<resource-unit>

          • container=<container-name>

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • node=< node-name>

          {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"}

          {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}

          kubernetes.pod.status.ready

          kube_pod_status_ready

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • condition=<true|false|unknown>

          kube_pod_info

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • host_ip=<host-ip>

          • pod_ip=<pod-ip>

          • node=<node-name>

          • uid=<pod-uid>

          {namespace="default",pod="pod0",host_ip="1.1.1.1",pod_ip="1.2.3.4",uid="abc-0",node="node1",created_by_kind="<none>",created_by_name="<none>",priority_class=""}

          kube_pod_owner

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • owner_kind=<owner kind>

          • owner_name=<owner name>

          {namespace="default",pod="pod0",owner_kind="<none>",owner_name="<none>;",owner_is_controller="<none>"}

          kube_pod_labels

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • label_POD_LABEL=<POD_LABEL>

          {namespace="default",pod="pod0", label_app="myApp"}

          kube_pod_container_info

          • pod=<pod-name>

          • namespace=<pod-namespace>

          • container_id=<containerid>

          {namespace="default",pod="pod0",container="container2",image="k8s.gcr.io/hyperkube2",image_id="docker://sha256:bbb",container_id="docker://cd456"}

          node

          kubernetes.node.allocatable.cpuCores

          kube_node_status_allocatable_cpu_cores

          • node=<node-address>

          • resource=<resource-name>

          • unit=<resource-unit>

          • node=<node-address>

          resource/unit have one of the values: (cpu, core); (memory, byte); (pods, integer). Sysdig currently supports only CPU, pods, and memory resources for kube_node_status_capacity metrics.

          "# HELP kube_node_status_capacity The capacity for different resources of a node.
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""memory"",unit=""byte""} 4.16342016e+09
kube_node_status_capacity{node=""k8s-master"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""cpu"",unit=""core""} 2
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node2"",resource=""cpu"",unit=""core""} 2

          kubernetes.node.allocatable.memBytes

          kube_node_status_allocatable_memory_bytes

          kubernetes.node.allocatable.pods

          kube_node_status_allocatable_pods

          kubernetes.node.capacity.cpuCores

          kube_node_status_capacity_cpu_cores

          • node=<node-address>

          • resource=<resource-name>

          • unit=<resource-unit>

          • node=<node-address>

          kubernetes.node.capacity.memBytes

          kube_node_status_capacity_memory_bytes

          kubernetes.node.capacity.pod

          kube_node_status_capacity_pods

          kubernetes.node.diskPressure

          kube_node_status_condition

          • node=<node-address

          • condition=<node-condition>

          • status=<true|false|unknown>

          kubernetes.node.memoryPressure

          kubernetes.node.networkUnavailable

          kubernetes.node.outOfDisk

          kubernetes.node.ready

          kubernetes.node.unschedulable

          kube_node_spec_unschedulable

          • node=<node-address>

          kube_node_info

          • node=<node-address>

          kube_node_labels

          • node=<node-address>

          • label_NODE_LABEL=<NODE_LABEL>

          Deployment

          kubernetes.deployment.replicas.available

          kube_deployment_status_replicas_available

          • deployment=<deployment-name>

          • namespace=<deployment-namespace>

          kubernetes.deployment.replicas.desired

          kube_deployment_spec_replicas

          kubernetes.deployment.replicas.paused

          kube_deployment_spec_paused

          kubernetes.deployment.replicas.running

          kube_deployment_status_replicas

          kubernetes.deployment.replicas.unavailable

          kube_deployment_status_replicas_unavailable

          kubernetes.deployment.replicas.updated

          kube_deployment_status_replicas_updated

          kube_deployment_labels

          job

          kubernetes.job.completions

          kube_job_spec_completions

          • job_name=<job-name>

          • namespace=<job-namespace>

          kubernetes.job.numFailed

          kube_job_failed

          kubernetes.job.numSucceeded

          kube_job_complete

          kubernetes.job.parallelism

          kube_job_spec_parallelism

          kube_job_status_active

          kube_job_info

          kube_job_owner

          • job_name=<job-name>

          • namespace=<job-namespace>

          • owner_kind=<owner kind>

          • owner_name=<owner name>

          kube_job_labels

          • job_name=<job-name>

          • namespace=<job-namespace>

          • label_job_label=<job_label>

          daemonSet

          kubernetes.daemonSet.pods.desired

          kube_daemonset_status_desired_number_scheduled

          • daemonset=<daemonset-name>

          • namespace=<daemonset-namespace>

          kubernetes.daemonSet.pods.misscheduled

          kube_daemonset_status_number_misscheduled

          kubernetes.daemonSet.pods.ready

          kube_daemonset_status_number_ready

          kubernetes.daemonSet.pods.scheduled

          kube_daemonset_status_current_number_scheduled

          kube_daemonset_labels

          • daemonset=<daemonset-name>

          • namespace=<daemonset-namespace>

          • label_daemonset_label=<daemonset_label>

          replicaSet

          kubernetes.replicaSet.replicas.fullyLabeled

          kube_replicaset_status_fully_labeled_replicas

          • replicaset=<replicaset-name>

          • namespace=<replicaset-namespace>

          kubernetes.replicaSet.replicas.ready

          kube_replicaset_status_ready_replicas

          kubernetes.replicaSet.replicas.running

          kube_replicaset_status_replicas

          kubernetes.replicaSet.replicas.desired

          kube_replicaset_spec_replicas

          kube_replicaset_owner

          • replicaset=<replicaset-name>

          • namespace=<replicaset-namespace>

          • owner_kind=<owner kind>

          • owner_name=<owner name>

          kube_replicaset_labels

          • label_replicaset_label=<replicaset_label>

          • replicaset=<replicaset-name>

          • namespace=<replicaset-namespace>

          statefulset

          kubernetes.statefulset.replicas

          kube_statefulset_replicas

          • statefulset=<statefulset-name>

          • namespace=<statefulset-namespace>

          kubernetes.statefulset.status.replicas

          kube_statefulset_status_replicas

          kubernetes.statefulset.status.replicas.current

          kube_statefulset_status_replicas_current

          kubernetes.statefulset.status.replicas.ready

          kube_statefulset_status_replicas_ready

          kubernetes.statefulset.status.replicas.updated

          kube_statefulset_status_replicas_updated

          kube_statefulset_labels

          hpa

          kubernetes.hpa.replicas.min

          kube_horizontalpodautoscaler_spec_min_replicas

          • hpa=<hpa-name>

          • namespace=<hpa-namespace>

          kubernetes.hpa.replicas.max

          kube_horizontalpodautoscaler_spec_max_replicas

          kubernetes.hpa.replicas.current

          kube_horizontalpodautoscaler_status_current_replicas

          kubernetes.hpa.replicas.desired

          kube_horizontalpodautoscaler_status_desired_replicas

          kube_horizontalpodautoscaler_labels

          resourcequota

          kubernetes.resourcequota.configmaps.hard

          kubernetes.resourcequota.configmaps.used

          kubernetes.resourcequota.limits.cpu.hard

          kubernetes.resourcequota.limits.cpu.used

          kubernetes.resourcequota.limits.memory.hard

          kubernetes.resourcequota.limits.memory.used

          kubernetes.resourcequota.persistentvolumeclaims.hard

          kubernetes.resourcequota.persistentvolumeclaims.used

          kubernetes.resourcequota.cpu.hard

          kubernetes.resourcequota.memory.hard

          kubernetes.resourcequota.pods.hard

          kubernetes.resourcequota.pods.used

          kubernetes.resourcequota.replicationcontrollers.hard

          kubernetes.resourcequota.replicationcontrollers.used

          kubernetes.resourcequota.requests.cpu.hard

          kubernetes.resourcequota.requests.cpu.used

          kubernetes.resourcequota.requests.memory.hard

          kubernetes.resourcequota.requests.memory.used

          kubernetes.resourcequota.requests.storage.hard

          kubernetes.resourcequota.requests.storage.used

          kubernetes.resourcequota.resourcequotas.hard

          kubernetes.resourcequota.resourcequotas.used

          kubernetes.resourcequota.secrets.hard

          kubernetes.resourcequota.secrets.used

          kubernetes.resourcequota.services.hard

          kubernetes.resourcequota.services.used

          kubernetes.resourcequota.services.loadbalancers.hard

          kubernetes.resourcequota.services.loadbalancers.used

          kubernetes.resourcequota.services.nodeports.hard

          kubernetes.resourcequota.services.nodeports.used

          kube_resourcequota

          • resourcequota=<quota-name>

          • namespace=<namespace>

          • resource=<ResourceName>

          • type=<quota-type>

          namespace

          kube_namespace_labels

          • namespace=<namespace-name>

          • label_ns_label=<ns_label>

          replicationcontroller

          kubernetes.replicationcontroller.replicas.desired

          kube_replicationcontroller_spec_replicase

          • replicationcontroller=<replicationcontroller-name>

          • namespace=<replicationcontroller-namespace>

          kubernetes.replicationcontroller.replicas.running

          kube_replicationcontroller_status_replicas

          kube_replicationcontroller_status_fully_labeled_replicas

          kube_replicationcontroller_status_ready_replicas

          kube_replicationcontroller_status_available_replicas

          kube_replicationcontroller_status_observed_generation

          kube_replicationcontroller_metadata_generation

          kube_replicationcontroller_created

          kube_replicationcontroller_owner

          • replicationcontroller=<replicationcontroller-name>

          • namespace=<replicationcontroller-namespace>

          • owner_kind=<owner kind>

          • owner_name=<owner name>

          service

          kube_service_info

          • service=<service-name>

          • namespace=<service-namespace>

          • cluster_ip=<service cluster ip>

          • external_name=<service external name>

          • load_balancer_ip=<service load balancer ip>

          kube_service_labels

          • service=<service-name>

          • namespace=<service-namespace>

          • label_service_label=<service_label>

          persistentvolume

          kubernetes.persistentvolume.storage

          kube_persistentvolume_capacity_bytes

          • persistentvolume=<pv-name>

          kube_persistentvolume_info

          • persistentvolume=<pv-name>

          kube_persistentvolume_labels

          • persistentvolume=<pv-name>

          • namespace=<label_persistentvolume_label=<persistentvolume_label>

          persistentvolumeclaim

          kubernetes.persistentvolumeclaim.requests.storage

          kube_persistentvolumeclaim_resource_requests_storage_bytes

          • namespace=<persistentvolumeclaim-namespace>

          • persistentvolumeclaim=<persistentvolumeclaim-name>

          kube_persistentvolumeclaim_info

          kube_persistentvolumeclaim_labels

          • persistentvolumeclaim=<persistentvolumeclaim-name>

          • namespace=<persistentvolumeclaim-namespace>

          • label_persistentvolumeclaim_label=<persistentvolumeclaim_label>

          4 -

          Run PromQL Queries Faster with Extended Label Set

          Sysdig allows you to run PromQL queries smoother and faster with the extended label set. The extended label set is created by augmenting the incoming data with the rich metadata associated with your infrastructure and making it available in PromQL.

          With this, you can troubleshoot a problem or building Dashboards and Alerts without the need to write complex queries. Sysdig automatically enriches your metrics with Kubernetes and application context without the need to instrument additional labels in your environment. This reduces operational complexity and cost—the enrichment takes place in Sysdig metric ingestion pipeline after time series have been sent to the backend.

          Calculate Memory Usage by Deployment in a Cluster

          Using the vector matching operation, you could run the following query and calculate the memory usage by deployment in a cluster:

          sum by(cluster,namespace,owner_name) ((sysdig_container_memory_used_bytes * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~".+",namespace=~".+"})

          To get the result, you need to write a query to perform a join (vector match) of various metrics, usually in the following order:

          • Grab a metric you need that is defined on a container level. For example, a Prometheus metric or some of the Sysdig provided metrics, such as sysdig_container_memory_used_byte.

          • Perform a vector match on container ID with the metric kube_pod_container_info to get the pod metadata.

          • Perform a vector match on the pod, namespace, and cluster with the kube_pod_owner metric.

          In the case of Sysdig’s extended label set for PromQL, all the metrics inherit the metadata, so that necessary container, host, and Kubernetes metadata are set on all the metrics. This simplifies the query so you can build and run it quickly.

          Likewise, the above query can be simplified as follows:

          sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name) (sysdig_container_memory_used_bytes{kube_cluster_name!="",kube_namespace_name!="",kube_deployment_name!=""})

          The advantages of using a simplified query are:

          • Complex vector matching operations (the group_left and group_right operators) are no longer required. All the labels are already available on each of the metrics, and therefore, any filtering can be performed directly on the metric itself.

          • The metrics now will have a huge amount of labels. You can use PromQL Explorer to deal with this rich metadata.

          • The metadata is distinguishable from user-defined labels. For example, Kubernetes metadata labels start with kube_. For instance, cluster is replaced with kube_cluster_name.

          • Create a dashboard panel or an alert from the PromQL query you run in the PromQL Query Explore.

          • Filter data by applying the comparison operators on the label values given in the table.

          Examples for Simplifying Queries

          Given below are some of the examples of using the extended label set to simplify complex query operations.

          Memory Usage in a Kubernetes Cluster

          Query with core label set:

          avg by (agent_tag_cluster) ((sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * on(host,agent_tag_cluster) sysdig_host_info{agent_tag_cluster=~".+"}) * 100

          Query with the extended label set:

          avg by (agent_tag_cluster) (sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * 100

          CPU Usage in Containers

          Query with the core label set:

          sum by (cluster,namespace)(sysdig_container_cpu_cores_used * on (container_id) group_left(cluster,pod,namespace) kube_pod_container_info{cluster=~".+"})

          Simplified query with the extended label set:

          sum by (kube_cluster_name,kube_namespace_name)(sysdig_container_cpu_cores_used{kube_cluster_name=~".+"})

          Memory Usage in Daemonset

          Query with the core label set:

          sum by(cluster,namespace,owner_name) (sum by(pod) (label_replace(sysdig_container_memory_used_bytes * on(container_id,host_mac) group_left(label_io_kubernetes_pod_namespace,label_io_kubernetes_pod_name,label_io_kubernetes_container_name) sysdig_container_info{label_io_kubernetes_pod_namespace=~".*",cluster=~".*"},"pod","$1","label_io_kubernetes_pod_name","(.*)"))  * on(pod) group_right sum by(cluster,namespace,owner_name,pod) (kube_pod_owner{owner_kind=~"DaemonSet",owner_name=~".*",cluster=~".*",namespace=~".*"}))

          Simplified query with the extended label set:

          sum by(kube_cluster_name,kube_namespace_name,kube_daemonset_name) (sysdig_container_memory_used_bytes{kube_daemonset_name=~".*",kube_cluster_name=~".*",kube_namespace_name=~".*"})

          Pod Restarts in a Kubernetes Cluster

          Query with the core label set:

          sum by(cluster,namespace,owner_name)(changes(kube_pod_status_ready{condition="true",cluster=~$cluster,namespace=~$namespace}[$__interval]) * on(cluster,namespace,pod) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~$cluster,namespace=~$namespace})

          Simplified query with the extended label set:

          sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name)(changes(kube_pod_status_ready{condition="true",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_deployment_name=~".+"}[$__interval]))

          Containers per Image

          Query with the core label set:

          count by (owner_name,image,cluster,namespace)((sysdig_container_info{cluster=~$cluster,namespace=~$namespace})  * on(pod,namespace,cluster) group_left(owner_name) max by (pod,namespace,cluster,owner_name)(kube_pod_owner{owner_kind="Deployment",owner_name=~".+"}))

          Simplified query with the extended label set:

          count by (kube_deployment_name,image,kube_cluster_name,kube_namespace_name)(sysdig_container_info{kube_deployment_name=~".+",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace})

          Average TCP Queue per Node

          Query with the core label set:

          avg by (agent_tag_cluster,host)( sysdig_host_net_tcp_queue_len * on (host_mac) group_left(agent_tag_cluster,host) sysdig_host_info{agent_tag_cluster=~$cluster,host=~".+"})

          Simplified query with the extended label set:

          avg by (agent_tag_cluster,host_hostname) (sysdig_host_net_tcp_queue_len{agent_tag_cluster =~ $cluster})