Metrics and Label Mapping
This topic outlines the mapping between the metrics and label naming conventions in the Sysdig legacy datastore and the new Sysdig datastore.
This the multi-page printable view of this section. Click here to print.
This topic outlines the mapping between the metrics and label naming conventions in the Sysdig legacy datastore and the new Sysdig datastore.
Sysdig classic metrics such as cpu.used.percent
previously returned values from a process, container, or host depending on the query segmentation or scope. You can now use context-explicit metrics which aligns with the flat model and resource specific semantics of Prometheus naming schema.
Your existing dashboards and alerts will be automatically migrated to the new naming convention.
Sysdig Classic Metrics | Context-Specific Metrics in Prometheus Notation |
---|---|
cpu.cores.used | sysdig_container_cpu_cores_used sysdig_host_cpu_cores_used sysdig_program_cpu_cores_used |
cpu.cores.used.percent | sysdig_container_cpu_cores_used_percent sysdig_host_cpu_cores_used_percent sysdig_program_cpu_cores_used_percent |
cpu.used.percent | sysdig_container_cpu_used_percent sysdig_host_cpu_used_percent sysdig_program_cpu_used_percent |
fd.used.percent | sysdig_container_fd_used_percent sysdig_host_fd_used_percent sysdig_program_fd_used_percent |
file.bytes.in | sysdig_container_file_in_bytes sysdig_host_file_in_bytes sysdig_program_file_in_bytes |
file.bytes.out | sysdig_container_file_out_bytes sysdig_host_file_out_bytes sysdig_program_file_out_bytes |
file.bytes.total | sysdig_container_file_total_bytes sysdig_host_file_total_bytes sysdig_program_file_total_bytes |
file.error.open.count | sysdig_container_file_error_open_count sysdig_host_file_error_open_count sysdig_program_file_error_open_count |
file.error.total.count | sysdig_container_file_error_total_count sysdig_host_file_error_total_count sysdig_program_file_error_total_count |
file.iops.in | sysdig_container_file_in_iops sysdig_host_file_in_iops sysdig_program_file_in_iops |
file.iops.out | sysdig_container_file_out_iops sysdig_host_file_out_iops sysdig_program_file_out_iops |
file.iops.total | sysdig_container_file_total_iops sysdig_host_file_total_iops sysdig_program_file_total_iops |
file.open.count | sysdig_container_file_open_count sysdig_host_file_open_count sysdig_program_file_open_count |
file.time.in | sysdig_container_file_in_time sysdig_host_file_in_time sysdig_program_file_in_time |
file.time.out | sysdig_container_file_out_time sysdig_host_file_out_time sysdig_program_file_out_time |
file.time.total | sysdig_container_file_total_time sysdig_host_file_total_time sysdig_program_file_total_time |
fs.bytes.free | sysdig_container_fs_free_bytes sysdig_fs_free_bytes sysdig_host_fs_free_bytes |
fs.bytes.total | sysdig_container_fs_total_bytes sysdig_fs_total_bytes sysdig_host_fs_total_bytes |
fs.bytes.used | sysdig_container_fs_used_bytes sysdig_fs_used_bytes sysdig_host_fs_used_bytes |
fs.free.percent | sysdig_container_fs_free_percent sysdig_fs_free_percent sysdig_host_fs_free_percent |
fs.inodes.total.count | sysdig_container_fs_inodes_total_count sysdig_fs_inodes_total_count sysdig_host_fs_inodes_total_count |
fs.inodes.used.count | sysdig_container_fs_inodes_used_count sysdig_fs_inodes_used_count sysdig_host_fs_inodes_used_count |
fs.inodes.used.percent | sysdig_container_fs_inodes_used_percent sysdig_fs_inodes_used_percent sysdig_host_fs_inodes_used_percent |
fs.largest.used.percent | sysdig_container_fs_largest_used_percent sysdig_host_fs_largest_used_percent |
fs.root.used.percent | sysdig_container_fs_root_used_percent sysdig_host_fs_root_used_percent |
fs.used.percent | sysdig_container_fs_used_percent sysdig_fs_used_percent sysdig_host_fs_used_percent |
host.error.count | sysdig_container_syscall_error_count sysdig_host_syscall_error_count |
info | sysdig_agent_info sysdig_container_info sysdig_host_info |
memory.bytes.total | sysdig_host_memory_total_bytes sysdig_container_memory_used_bytes sysdig_host_memory_used_bytes sysdig_program_memory_used_bytes |
memory.bytes.virtual | sysdig_container_memory_virtual_bytes sysdig_host_memory_virtual_bytes |
memory.swap.bytes.used | sysdig_container_memory_swap_used_bytes sysdig_host_memory_swap_used_bytes |
memory.used.percent | sysdig_container_memory_used_percent sysdig_host_memory_used_percent |
net.bytes.in | sysdig_connection_net_in_bytes sysdig_container_net_in_bytes sysdig_host_net_in_bytes sysdig_program_net_in_bytes |
net.bytes.out | sysdig_connection_net_out_bytes sysdig_container_net_out_bytes sysdig_host_net_out_bytes sysdig_program_net_out_bytes |
net.bytes.total | sysdig_connection_net_total_bytes sysdig_container_net_total_bytes sysdig_host_net_total_bytes sysdig_program_net_total_bytes |
net.connection.count.in | sysdig_connection_net_connection_in_count sysdig_container_net_connection_in_count sysdig_host_net_connection_in_count sysdig_program_net_connection_in_count |
net.connection.count.out | sysdig_connection_net_connection_out_count sysdig_container_net_connection_out_count sysdig_host_net_connection_out_count sysdig_program_net_connection_out_count |
net.connection.count.total | sysdig_connection_net_connection_total_count sysdig_container_net_connection_total_count sysdig_host_net_connection_total_count sysdig_program_net_connection_total_count |
net.request.count | sysdig_connection_net_request_count sysdig_container_net_request_count sysdig_host_net_request_count sysdig_program_net_request_count |
net.error.count | sysdig_container_net_error_count sysdig_host_net_error_count sysdig_program_net_error_count |
net.request.count.in | sysdig_connection_net_request_in_count sysdig_container_net_request_in_count sysdig_host_net_request_in_count sysdig_program_net_request_in_count |
net.request.count.out | sysdig_connection_net_request_out_count sysdig_container_net_request_out_count sysdig_host_net_request_out_count sysdig_program_net_request_out_count |
net.request.time | sysdig_connection_net_request_time sysdig_container_net_request_time sysdig_host_net_request_time sysdig_program_net_request_time |
net.request.time.in | sysdig_connection_net_request_in_time sysdig_container_net_request_in_time sysdig_host_net_request_in_time sysdig_program_net_request_in_time |
net.request.time.out | sysdig_connection_net_request_out_time sysdig_container_net_request_out_time sysdig_host_net_request_out_time sysdig_program_net_request_out_time |
net.server.bytes.in | sysdig_container_net_server_in_bytes sysdig_host_net_server_in_bytes |
net.server.bytes.out | sysdig_container_net_server_out_bytes sysdig_host_net_server_out_bytes |
net.server.bytes.total | sysdig_container_net_server_total_bytes sysdig_host_net_server_total_bytes |
net.sql.error.count | sysdig_container_net_sql_error_count sysdig_host_net_sql_error_count |
net.sql.request.count | sysdig_container_net_sql_request_count sysdig_host_net_sql_request_count |
net.tcp.queue.len | sysdig_container_net_tcp_queue_len sysdig_host_net_tcp_queue_len sysdig_program_net_tcp_queue_len |
proc.count | sysdig_container_proc_count sysdig_host_proc_count sysdig_program_proc_count |
thread.count | sysdig_container_thread_count sysdig_host_thread_count sysdig_program_thread_count |
uptime | sysdig_container_up sysdig_host_up sysdig_program_up |
Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.
Entity | Type | PromQL Metric Name | Classic Metric Name | Label | Classic Label | ||
---|---|---|---|---|---|---|---|
host | info | sysdig_host_info | Not exposed |
|
| ||
sysdig_cloud_provider_info |
|
| |||||
data | sysdig_host_cpu_used_percent | cpu.used.percent |
|
| |||
sysdig_host_cpu_cores_used | cpu.cores.used | ||||||
sysdig_host_cpu_user_percent | cpu.user.percent | ||||||
sysdig_host_cpu_idle_percent | cpu.idle.percent | ||||||
sysdig_host_cpu_iowait_percent | cpu.iowait.percent | ||||||
sysdig_host_cpu_nice_percent | cpu.nice.percent | ||||||
sysdig_host_cpu_stolen_percent | cpu.stolen.percent | ||||||
sysdig_host_cpu_system_percent | cpu.system.percent | ||||||
sysdig_host_fd_used_percent | fd.used.percent | ||||||
sysdig_host_file_error_open_count | file.error.open.count | ||||||
sysdig_host_file_error_total_count | file.error.total.count | ||||||
sysdig_host_file_in_bytes | file.bytes.in | ||||||
sysdig_host_file_in_iops | file.iops.in | ||||||
sysdig_host_file_in_time | file.time.in | ||||||
sysdig_host_file_open_count | file.open.count | ||||||
sysdig_host_file_out_bytes | file.bytes.out | ||||||
sysdig_host_file_out_iops | file.iops.out | ||||||
sysdig_host_file_out_time | file.time.out | ||||||
sysdig_host_load_average_15m | load.average.15m | ||||||
sysdig_host_load_average_1m | load.average.1m | ||||||
sysdig_host_load_average_5m | load.average.5m | ||||||
sysdig_host_memory_available_bytes | memory.bytes.available | ||||||
sysdig_host_memory_total_bytes | memory.bytes.total | ||||||
sysdig_host_memory_used_bytes | memory.bytes.used | ||||||
sysdig_host_memory_swap_available_bytes | memory.swap.bytes.available | ||||||
sysdig_host_memory_swap_total_bytes | memory.swap.bytes.total | ||||||
sysdig_host_memory_swap_used_bytes | memory.swap.bytes.used | ||||||
sysdig_host_memory_virtual_bytes | memory.bytes.virtual | ||||||
sysdig_host_net_connection_in_count | net.connection.count.in | ||||||
sysdig_host_net_connection_out_count | net.connection.count.out | ||||||
sysdig_host_net_error_count | net.error.count | ||||||
sysdig_host_net_in_bytes | net.bytes.in | ||||||
sysdig_host_net_out_bytes | net.bytes.out | ||||||
sysdig_host_net_tcp_queue_len | net.tcp.queue.len | ||||||
sysdig_host_proc_count | proc.count | ||||||
sysdig_host_system_uptime | system.uptime | ||||||
sysdig_host_thread_count | thread.count | ||||||
container | info | sysdig_container_info | Not exposed | container_id | container_id | ||
container_full_id | none | ||||||
host_mac | host.mac | ||||||
container | container.name | ||||||
container_type | container.type | ||||||
image | container.image | ||||||
image_id | container.image.id | ||||||
mesos_task_id | container.mesosTaskId Only available in Mesos orchestrator. | ||||||
cluster | kubernetes.cluster.name Present only if the container is part of Kubernetes. | ||||||
pod | kubernetes.pod.name Present only if the container is part of Kubernetes | ||||||
namespace | kubernetes.namespace.name Present only if the container is part of Kubernetes. | ||||||
data | sysdig_container_cpu_used_percent | cpu.used.percent |
|
| |||
sysdig_container_cpu_cores_used | cpu.cores.used | ||||||
sysdig_container_cpu_cores_used_percent | cpu.cores.used.percent | ||||||
sysdig_container_cpu_quota_used_percent | cpu.quota.used.percent | ||||||
sysdig_container_cpu_shares | cpu.shares.count | ||||||
sysdig_container_cpu_shares_used_percent | cpu.shares.used.percent | ||||||
sysdig_container_fd_used_percent | fd.used.percent | ||||||
sysdig_container_file_error_open_count | file.error.open.count | ||||||
sysdig_container_file_error_total_count | file.error.total.count | ||||||
sysdig_container_file_in_bytes | file.bytes.in | ||||||
sysdig_container_file_in_iops | file.iops.in | ||||||
sysdig_container_file_in_time | file.time.in | ||||||
sysdig_container_file_open_count | file.open.count | ||||||
sysdig_container_file_out_bytes | file.bytes.out | ||||||
sysdig_container_file_out_iops | file.iops.out | ||||||
sysdig_container_file_out_time | file.time.out | ||||||
sysdig_container_memory_limit_bytes | memory.limit.bytes | ||||||
sysdig_container_memory_limit_used_percent | memory.limit.used.percent | ||||||
sysdig_container_memory_swap_available_bytes | memory.swap.bytes.available | ||||||
sysdig_container_memory_swap_total_bytes | memory.swap.bytes.total | ||||||
sysdig_container_memory_swap_used_bytes | memory.swap.bytes.used | ||||||
sysdig_container_memory_used_bytes | memory.bytes.used | ||||||
sysdig_container_memory_virtual_bytes | memory.bytes.virtual | ||||||
sysdig_container_net_connection_in_count | net.connection.count.in | ||||||
sysdig_container_net_connection_out_count | net.connection.count.out | ||||||
sysdig_container_net_error_count | net.error.count | ||||||
sysdig_container_net_in_bytes | net.bytes.in | ||||||
sysdig_container_net_out_bytes | net.bytes.out | ||||||
sysdig_container_net_tcp_queue_len | net.tcp.queue.len | ||||||
sysdig_container_proc_count | proc.count | ||||||
sysdig_container_swap_limit_bytes | swap.limit.bytes | ||||||
sysdig_container_thread_count | thread.count | ||||||
Process/ Program | Info | sysdig_program_info | not exposed | program | proc.name | ||
cmd_line | proc.commandLine | ||||||
host_mac | host.mac | ||||||
container_id | container.id | ||||||
container_type | container.type | ||||||
data | sysdig_program_cpu_used_percent | cpu.used.percent | host_mac | host.mac | |||
container_id | container.id | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
sysdig_program_memory_used_bytes | memory.bytes.used | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
sysdig_program_net_in_bytes | net.bytes.in | container_id | container.id | ||||
host_mac | host.mac | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
sysdig_program_net_out_bytes | net.bytes.out | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
sysdig_program_proc_count | proc.count | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
sysdig_program_thread_count | thread.count | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
program | proc.name | ||||||
cmd_line | proc.commandLine | ||||||
fs | info | sysdig_fs_info | not exposed | host_mac | host.mac | ||
container_id | container.id | ||||||
container_type | container.type | ||||||
device | fs.device | ||||||
mount_dir | fs.mountDir | ||||||
type | fs.type | ||||||
data | sysdig_fs_free_bytes | fs.bytes.free | host_mac | host.mac | |||
container_id | container.id | ||||||
container_type | container.type | ||||||
device | fs.device | ||||||
sysdig_fs_inodes_total_count | fs.inodes.total.count | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
device | fs.device | ||||||
sysdig_fs_inodes_used_count | fs.inodes.used.count | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
device | fs.device | ||||||
sysdig_fs_total_bytes | fs.bytes.total | host_mac | host.mac | ||||
container_id | container.id | ||||||
container_type | container.type | ||||||
device | fs.device | ||||||
fs.bytes.used | host_mac | host.mac | |||||
container_id | container.id | ||||||
container_type | container.type | ||||||
devide | fs.device |
Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.
For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.
Resource | Sysdig Metrics | Kubernetes State Metrics | Label | Example / More Information | |
---|---|---|---|---|---|
Pod | kubernetes.pod.containers.waiting | kube_pod_container_status_waiting |
| ||
kubernetes.pod.resourceLimits.cpuCores kubernetes.pod.resourceLimits.memBytes | kube_pod_container_resource_limits kube_pod_sysdig_resource_limits_memory_bytes kube_pod_sysdig_resource_limits_cpu_cores |
| {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"} {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"} | ||
kubernetes.pod.resourceRequests.cpuCores kubernetes.pod.resourceRequests.memBytes | kube_pod_container_resource_requests kube_pod_sysdig_resource_requests_cpu_cores kube_pod_sysdig_resource_requests_memory_bytes |
| {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"} {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"} | ||
kubernetes.pod.status.ready | kube_pod_status_ready |
| |||
kube_pod_info |
| {namespace="default",pod="pod0",host_ip="1.1.1.1",pod_ip="1.2.3.4",uid="abc-0",node="node1",created_by_kind="<none>",created_by_name="<none>",priority_class=""} | |||
kube_pod_owner |
| {namespace="default",pod="pod0",owner_kind="<none>",owner_name="<none>;",owner_is_controller="<none>"} | |||
kube_pod_labels |
| {namespace="default",pod="pod0", label_app="myApp"} | |||
kube_pod_container_info |
| {namespace="default",pod="pod0",container="container2",image="k8s.gcr.io/hyperkube2",image_id="docker://sha256:bbb",container_id="docker://cd456"} | |||
node | kubernetes.node.allocatable.cpuCores | kube_node_status_allocatable_cpu_cores |
| resource/unit have one of the values: (cpu, core); (memory, byte); (pods, integer). Sysdig currently supports only CPU, pods, and memory resources for
| |
kubernetes.node.allocatable.memBytes | kube_node_status_allocatable_memory_bytes | ||||
kubernetes.node.allocatable.pods | kube_node_status_allocatable_pods | ||||
kubernetes.node.capacity.cpuCores | kube_node_status_capacity_cpu_cores |
| |||
kubernetes.node.capacity.memBytes | kube_node_status_capacity_memory_bytes | ||||
kubernetes.node.capacity.pod | kube_node_status_capacity_pods | ||||
kubernetes.node.diskPressure | kube_node_status_condition |
| |||
kubernetes.node.memoryPressure | |||||
kubernetes.node.networkUnavailable | |||||
kubernetes.node.outOfDisk | |||||
kubernetes.node.ready | |||||
kubernetes.node.unschedulable | kube_node_spec_unschedulable |
| |||
kube_node_info |
| ||||
kube_node_labels |
| ||||
Deployment | kubernetes.deployment.replicas.available | kube_deployment_status_replicas_available |
| ||
kubernetes.deployment.replicas.desired | kube_deployment_spec_replicas | ||||
kubernetes.deployment.replicas.paused | kube_deployment_spec_paused | ||||
kubernetes.deployment.replicas.running | kube_deployment_status_replicas | ||||
kubernetes.deployment.replicas.unavailable | kube_deployment_status_replicas_unavailable | ||||
kubernetes.deployment.replicas.updated | kube_deployment_status_replicas_updated | ||||
kube_deployment_labels | |||||
job | kubernetes.job.completions | kube_job_spec_completions |
| ||
kubernetes.job.numFailed | kube_job_failed | ||||
kubernetes.job.numSucceeded | kube_job_complete | ||||
kubernetes.job.parallelism | kube_job_spec_parallelism | ||||
kube_job_status_active | |||||
kube_job_info | |||||
kube_job_owner |
| ||||
kube_job_labels |
| ||||
daemonSet | kubernetes.daemonSet.pods.desired | kube_daemonset_status_desired_number_scheduled |
| ||
kubernetes.daemonSet.pods.misscheduled | kube_daemonset_status_number_misscheduled | ||||
kubernetes.daemonSet.pods.ready | kube_daemonset_status_number_ready | ||||
kubernetes.daemonSet.pods.scheduled | kube_daemonset_status_current_number_scheduled | ||||
kube_daemonset_labels |
| ||||
replicaSet | kubernetes.replicaSet.replicas.fullyLabeled | kube_replicaset_status_fully_labeled_replicas |
| ||
kubernetes.replicaSet.replicas.ready | kube_replicaset_status_ready_replicas | ||||
kubernetes.replicaSet.replicas.running | kube_replicaset_status_replicas | ||||
kubernetes.replicaSet.replicas.desired | kube_replicaset_spec_replicas | ||||
kube_replicaset_owner |
| ||||
kube_replicaset_labels |
| ||||
statefulset | kubernetes.statefulset.replicas | kube_statefulset_replicas |
| ||
kubernetes.statefulset.status.replicas | kube_statefulset_status_replicas | ||||
kubernetes.statefulset.status.replicas.current | kube_statefulset_status_replicas_current | ||||
kubernetes.statefulset.status.replicas.ready | kube_statefulset_status_replicas_ready | ||||
kubernetes.statefulset.status.replicas.updated | kube_statefulset_status_replicas_updated | ||||
kube_statefulset_labels | |||||
hpa | kubernetes.hpa.replicas.min | kube_horizontalpodautoscaler_spec_min_replicas |
| ||
kubernetes.hpa.replicas.max | kube_horizontalpodautoscaler_spec_max_replicas | ||||
kubernetes.hpa.replicas.current | kube_horizontalpodautoscaler_status_current_replicas | ||||
kubernetes.hpa.replicas.desired | kube_horizontalpodautoscaler_status_desired_replicas | ||||
kube_horizontalpodautoscaler_labels | |||||
resourcequota | kubernetes.resourcequota.configmaps.hard kubernetes.resourcequota.configmaps.used kubernetes.resourcequota.limits.cpu.hard kubernetes.resourcequota.limits.cpu.used kubernetes.resourcequota.limits.memory.hard kubernetes.resourcequota.limits.memory.used kubernetes.resourcequota.persistentvolumeclaims.hard kubernetes.resourcequota.persistentvolumeclaims.used kubernetes.resourcequota.cpu.hard kubernetes.resourcequota.memory.hard kubernetes.resourcequota.pods.hard kubernetes.resourcequota.pods.used kubernetes.resourcequota.replicationcontrollers.hard kubernetes.resourcequota.replicationcontrollers.used kubernetes.resourcequota.requests.cpu.hard kubernetes.resourcequota.requests.cpu.used kubernetes.resourcequota.requests.memory.hard kubernetes.resourcequota.requests.memory.used kubernetes.resourcequota.requests.storage.hard kubernetes.resourcequota.requests.storage.used kubernetes.resourcequota.resourcequotas.hard kubernetes.resourcequota.resourcequotas.used kubernetes.resourcequota.secrets.hard kubernetes.resourcequota.secrets.used kubernetes.resourcequota.services.hard kubernetes.resourcequota.services.used kubernetes.resourcequota.services.loadbalancers.hard kubernetes.resourcequota.services.loadbalancers.used kubernetes.resourcequota.services.nodeports.hard kubernetes.resourcequota.services.nodeports.used | kube_resourcequota |
| ||
namespace | kube_namespace_labels |
| |||
replicationcontroller | kubernetes.replicationcontroller.replicas.desired | kube_replicationcontroller_spec_replicase |
| ||
kubernetes.replicationcontroller.replicas.running | kube_replicationcontroller_status_replicas | ||||
kube_replicationcontroller_status_fully_labeled_replicas kube_replicationcontroller_status_ready_replicas kube_replicationcontroller_status_available_replicas kube_replicationcontroller_status_observed_generation kube_replicationcontroller_metadata_generation kube_replicationcontroller_created | |||||
kube_replicationcontroller_owner |
| ||||
service | kube_service_info |
| |||
kube_service_labels |
| ||||
persistentvolume | kubernetes.persistentvolume.storage | kube_persistentvolume_capacity_bytes |
| ||
kube_persistentvolume_info |
| ||||
kube_persistentvolume_labels |
| ||||
persistentvolumeclaim | kubernetes.persistentvolumeclaim.requests.storage | kube_persistentvolumeclaim_resource_requests_storage_bytes |
| ||
kube_persistentvolumeclaim_info | |||||
kube_persistentvolumeclaim_labels |
|
Sysdig allows you to run PromQL queries smoother and faster with the extended label set. The extended label set is created by augmenting the incoming data with the rich metadata associated with your infrastructure and making it available in PromQL.
With this, you can troubleshoot a problem or building Dashboards and Alerts without the need to write complex queries. Sysdig automatically enriches your metrics with Kubernetes and application context without the need to instrument additional labels in your environment. This reduces operational complexity and cost—the enrichment takes place in Sysdig metric ingestion pipeline after time series have been sent to the backend.
Using the vector matching operation, you could run the following query and calculate the memory usage by deployment in a cluster:
sum by(cluster,namespace,owner_name) ((sysdig_container_memory_used_bytes * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~".+",namespace=~".+"})
To get the result, you need to write a query to perform a join (vector match) of various metrics, usually in the following order:
Grab a metric you need that is defined on a container level. For
example, a Prometheus metric or some of the Sysdig provided metrics,
such as sysdig_container_memory_used_byte
.
Perform a vector match on container ID with the metric
kube_pod_container_info
to get the pod metadata.
Perform a vector match on the pod, namespace, and cluster with the
kube_pod_owner
metric.
In the case of Sysdig’s extended label set for PromQL, all the metrics inherit the metadata, so that necessary container, host, and Kubernetes metadata are set on all the metrics. This simplifies the query so you can build and run it quickly.
Likewise, the above query can be simplified as follows:
sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name) (sysdig_container_memory_used_bytes{kube_cluster_name!="",kube_namespace_name!="",kube_deployment_name!=""})
The advantages of using a simplified query are:
Complex vector matching operations (the group_left and group_right operators) are no longer required. All the labels are already available on each of the metrics, and therefore, any filtering can be performed directly on the metric itself.
The metrics now will have a huge amount of labels. You can use PromQL Explorer to deal with this rich metadata.
The metadata is distinguishable from user-defined labels. For
example, Kubernetes metadata labels start with kube_
. For
instance, cluster
is replaced with kube_cluster_name
.
Create a dashboard panel or an alert from the PromQL query you run in the PromQL Query Explore.
Filter data by applying the comparison operators on the label values given in the table.
Given below are some of the examples of using the extended label set to simplify complex query operations.
Query with core label set:
avg by (agent_tag_cluster) ((sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * on(host,agent_tag_cluster) sysdig_host_info{agent_tag_cluster=~".+"}) * 100
Query with the extended label set:
avg by (agent_tag_cluster) (sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * 100
Query with the core label set:
sum by (cluster,namespace)(sysdig_container_cpu_cores_used * on (container_id) group_left(cluster,pod,namespace) kube_pod_container_info{cluster=~".+"})
Simplified query with the extended label set:
sum by (kube_cluster_name,kube_namespace_name)(sysdig_container_cpu_cores_used{kube_cluster_name=~".+"})
Query with the core label set:
sum by(cluster,namespace,owner_name) (sum by(pod) (label_replace(sysdig_container_memory_used_bytes * on(container_id,host_mac) group_left(label_io_kubernetes_pod_namespace,label_io_kubernetes_pod_name,label_io_kubernetes_container_name) sysdig_container_info{label_io_kubernetes_pod_namespace=~".*",cluster=~".*"},"pod","$1","label_io_kubernetes_pod_name","(.*)")) * on(pod) group_right sum by(cluster,namespace,owner_name,pod) (kube_pod_owner{owner_kind=~"DaemonSet",owner_name=~".*",cluster=~".*",namespace=~".*"}))
Simplified query with the extended label set:
sum by(kube_cluster_name,kube_namespace_name,kube_daemonset_name) (sysdig_container_memory_used_bytes{kube_daemonset_name=~".*",kube_cluster_name=~".*",kube_namespace_name=~".*"})
Query with the core label set:
sum by(cluster,namespace,owner_name)(changes(kube_pod_status_ready{condition="true",cluster=~$cluster,namespace=~$namespace}[$__interval]) * on(cluster,namespace,pod) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~$cluster,namespace=~$namespace})
Simplified query with the extended label set:
sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name)(changes(kube_pod_status_ready{condition="true",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_deployment_name=~".+"}[$__interval]))
Query with the core label set:
count by (owner_name,image,cluster,namespace)((sysdig_container_info{cluster=~$cluster,namespace=~$namespace}) * on(pod,namespace,cluster) group_left(owner_name) max by (pod,namespace,cluster,owner_name)(kube_pod_owner{owner_kind="Deployment",owner_name=~".+"}))
Simplified query with the extended label set:
count by (kube_deployment_name,image,kube_cluster_name,kube_namespace_name)(sysdig_container_info{kube_deployment_name=~".+",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace})
Query with the core label set:
avg by (agent_tag_cluster,host)( sysdig_host_net_tcp_queue_len * on (host_mac) group_left(agent_tag_cluster,host) sysdig_host_info{agent_tag_cluster=~$cluster,host=~".+"})
Simplified query with the extended label set:
avg by (agent_tag_cluster,host_hostname) (sysdig_host_net_tcp_queue_len{agent_tag_cluster =~ $cluster})