Metrics and Label Mapping

This topic outlines the metrics and label mapping between the Sysdig classic store and the new Sysdig datastore.

1 -

Mapping Between Classic Metrics and PromQL Metrics

Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.

Entity	Type	PromQL Metric Name	Classic Metric Name	Label	Classic Label
host	info	sysdig_host_info	Not exposed	host_mac host instance_id agent_tag_{*}	host.mac host.hostName host.instanceId agent.tag.{*}
sysdig_cloud_provider_info	host_mac provider_id account_id region availability_zone instance_type tag_{*} security_groups host_ip_public host_ip_private host_name name	host.mac cloudProvider.id cloudProvider.account.id cloudProvider.region cloudProvider.availabilityZone cloudProvider.instance.type cloudProvider.tag.{*} cloudProvider.securityGroups cloudProvider.host.ip.public cloudProvider.host.ip.private cloudProvider.host.name cloudProvider.name
data	sysdig_host_cpu_used_percent	cpu.used.percent	host_mac host	host.mac host.hostname
sysdig_host_cpu_cores_used	cpu.cores.used
sysdig_host_cpu_user_percent	cpu.user.percent
sysdig_host_cpu_idle_percent	cpu.idle.percent
sysdig_host_cpu_iowait_percent	cpu.iowait.percent
sysdig_host_cpu_nice_percent	cpu.nice.percent
sysdig_host_cpu_stolen_percent	cpu.stolen.percent
sysdig_host_cpu_system_percent	cpu.system.percent
sysdig_host_fd_used_percent	fd.used.percent
sysdig_host_file_error_open_count	file.error.open.count
sysdig_host_file_error_total_count	file.error.total.count
sysdig_host_file_in_bytes	file.bytes.in
sysdig_host_file_in_iops	file.iops.in
sysdig_host_file_in_time	file.time.in
sysdig_host_file_open_count	file.open.count
sysdig_host_file_out_bytes	file.bytes.out
sysdig_host_file_out_iops	file.iops.out
sysdig_host_file_out_time	file.time.out
sysdig_host_load_average_15m	load.average.15m
sysdig_host_load_average_1m	load.average.1m
sysdig_host_load_average_5m	load.average.5m
sysdig_host_memory_available_bytes	memory.bytes.available
sysdig_host_memory_total_bytes	memory.bytes.total
sysdig_host_memory_used_bytes	memory.bytes.used
sysdig_host_memory_swap_available_bytes	memory.swap.bytes.available
sysdig_host_memory_swap_total_bytes	memory.swap.bytes.total
sysdig_host_memory_swap_used_bytes	memory.swap.bytes.used
sysdig_host_memory_virtual_bytes	memory.bytes.virtual
sysdig_host_net_connection_in_count	net.connection.count.in
sysdig_host_net_connection_out_count	net.connection.count.out
sysdig_host_net_error_count	net.error.count
sysdig_host_net_in_bytes	net.bytes.in
sysdig_host_net_out_bytes	net.bytes.out
sysdig_host_net_tcp_queue_len	net.tcp.queue.len
sysdig_host_proc_count	proc.count
sysdig_host_system_uptime	system.uptime
sysdig_host_thread_count	thread.count
container	info	sysdig_container_info	Not exposed	container_id	container_id
container_full_id	none
host_mac	host.mac
container	container.name
container_type	container.type
image	container.image
image_id	container.image.id
mesos_task_id	container.mesosTaskId Only available in Mesos orchestrator.
cluster	kubernetes.cluster.name Present only if the container is part of Kubernetes.
pod	kubernetes.pod.name Present only if the container is part of Kubernetes
namespace	kubernetes.namespace.name Present only if the container is part of Kubernetes.
data	sysdig_container_cpu_used_percent	cpu.used.percent	host_mac container_id container_type container	host.mac container.id container.type container.name
sysdig_container_cpu_cores_used	cpu.cores.used
sysdig_container_cpu_cores_used_percent	cpu.cores.used.percent
sysdig_container_cpu_quota_used_percent	cpu.quota.used.percent
sysdig_container_cpu_shares	cpu.shares.count
sysdig_container_cpu_shares_used_percent	cpu.shares.used.percent
sysdig_container_fd_used_percent	fd.used.percent
sysdig_container_file_error_open_count	file.error.open.count
sysdig_container_file_error_total_count	file.error.total.count
sysdig_container_file_in_bytes	file.bytes.in
sysdig_container_file_in_iops	file.iops.in
sysdig_container_file_in_time	file.time.in
sysdig_container_file_open_count	file.open.count
sysdig_container_file_out_bytes	file.bytes.out
sysdig_container_file_out_iops	file.iops.out
sysdig_container_file_out_time	file.time.out
sysdig_container_memory_limit_bytes	memory.limit.bytes
sysdig_container_memory_limit_used_percent	memory.limit.used.percent
sysdig_container_memory_swap_available_bytes	memory.swap.bytes.available
sysdig_container_memory_swap_total_bytes	memory.swap.bytes.total
sysdig_container_memory_swap_used_bytes	memory.swap.bytes.used
sysdig_container_memory_used_bytes	memory.bytes.used
sysdig_container_memory_virtual_bytes	memory.bytes.virtual
sysdig_container_net_connection_in_count	net.connection.count.in
sysdig_container_net_connection_out_count	net.connection.count.out
sysdig_container_net_error_count	net.error.count
sysdig_container_net_in_bytes	net.bytes.in
sysdig_container_net_out_bytes	net.bytes.out
sysdig_container_net_tcp_queue_len	net.tcp.queue.len
sysdig_container_proc_count	proc.count
sysdig_container_swap_limit_bytes	swap.limit.bytes
sysdig_container_thread_count	thread.count
Process/ Program	Info	sysdig_program_info	not exposed	program	proc.name
cmd_line	proc.commandLine
host_mac	host.mac
container_id	container.id
container_type	container.type
data	sysdig_program_cpu_used_percent	cpu.used.percent	host_mac	host.mac
container_id	container.id
container_type	container.type
program	proc.name
cmd_line	proc.commandLine
sysdig_program_memory_used_bytes	memory.bytes.used	host_mac	host.mac
container_id	container.id
container_type	container.type
program	proc.name
cmd_line	proc.commandLine
sysdig_program_net_in_bytes	net.bytes.in	container_id	container.id
host_mac	host.mac
container_type	container.type
program	proc.name
cmd_line	proc.commandLine
sysdig_program_net_out_bytes	net.bytes.out	host_mac	host.mac
container_id	container.id
container_type	container.type
program	proc.name
cmd_line	proc.commandLine
sysdig_program_proc_count	proc.count	host_mac	host.mac
container_id	container.id
container_type	container.type
program	proc.name
cmd_line	proc.commandLine
sysdig_program_thread_count	thread.count	host_mac	host.mac

container_id	container.id
container_type	container.type
program	proc.name
cmd_line	proc.commandLine

fs	info	sysdig_fs_info	not exposed	host_mac	host.mac
	container_id	container.id
	container_type	container.type
	device	fs.device
	mount_dir	fs.mountDir
	type	fs.type
data	sysdig_fs_free_bytes	fs.bytes.free	host_mac	host.mac
container_id	container.id
container_type	container.type
device	fs.device
sysdig_fs_inodes_total_count	fs.inodes.total.count	host_mac	host.mac
container_id	container.id
container_type	container.type
device	fs.device
sysdig_fs_inodes_used_count	fs.inodes.used.count	host_mac	host.mac
container_id	container.id
container_type	container.type
device	fs.device
sysdig_fs_total_bytes	fs.bytes.total	host_mac	host.mac
container_id	container.id
container_type	container.type
device	fs.device
fs.bytes.used		host_mac	host.mac
container_id container_type	container.id container_id
devide	fs.device

2 -

Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.

For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.

Resource	Sysdig Metrics	Kubernetes State Metrics	Label	Example / More Information
Pod	kubernetes.pod.containers.waiting	kube_pod_container_status_waiting	container=<container-name> pod=<pod-name> namespace=<pod-namespace>
kubernetes.pod.resourceLimits.cpuCores kubernetes.pod.resourceLimits.memBytes	kube_pod_container_resource_limits	resource=<resource-name> unit=<resource-unit> pod=<pod-name> namespace=<pod-namespace> node=< node-name>	{namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"} {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}
kubernetes.pod.resourceRequests.cpuCores kubernetes.pod.resourceRequests.memBytes	kube_pod_container_resource_requests	resource=<resource-name> unit=<resource-unit> container=<container-name> pod=<pod-name> namespace=<pod-namespace> node=< node-name>	{namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"} {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}
kubernetes.pod.status.ready	kube_pod_status_ready	pod=<pod-name> namespace=<pod-namespace> condition=<true\|false\|unknown>
	kube_pod_info	pod=<pod-name> namespace=<pod-namespace> host_ip=<host-ip> pod_ip=<pod-ip> node=<node-name> uid=<pod-uid>	{namespace="default",pod="pod0",host_ip="1.1.1.1",pod_ip="1.2.3.4",uid="abc-0",node="node1",created_by_kind="<none>",created_by_name="<none>",priority_class=""}
	kube_pod_owner	pod=<pod-name> namespace=<pod-namespace> owner_kind=<owner kind> owner_name=<owner name>	{namespace="default",pod="pod0",owner_kind="<none>",owner_name="<none>;",owner_is_controller="<none>"}
	kube_pod_labels	pod=<pod-name> namespace=<pod-namespace> label_POD_LABEL=<POD_LABEL>	{namespace="default",pod="pod0", label_app="myApp"}
	kube_pod_container_info	pod=<pod-name> namespace=<pod-namespace> container_id=<containerid>	{namespace="default",pod="pod0",container="container2",image="k8s.gcr.io/hyperkube2",image_id="docker://sha256:bbb",container_id="docker://cd456"}
node	kubernetes.node.allocatable.cpuCores	kube_node_status_allocatable_cpu_cores	node=<node-address> resource=<resource-name> unit=<resource-unit> node=<node-address>	resource/unit have one of the values: (cpu, core); (memory, byte); (pods, integer). Sysdig currently supports only CPU, pods, and memory resources for `kube_node_status_capacity` metrics. "# HELP kube_node_status_capacity The capacity for different resources of a node. kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_1Gi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_2Mi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-master"",resource=""memory"",unit=""byte""} 4.16342016e+09 kube_node_status_capacity{node=""k8s-master"",resource=""pods"",unit=""integer""} 110 kube_node_status_capacity{node=""k8s-node1"",resource=""pods"",unit=""integer""} 110 kube_node_status_capacity{node=""k8s-node1"",resource=""cpu"",unit=""core""} 2 kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_1Gi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_2Mi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-node1"",resource=""memory"",unit=""byte""} 6.274154496e+09 kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_1Gi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_2Mi"",unit=""byte""} 0 kube_node_status_capacity{node=""k8s-node2"",resource=""memory"",unit=""byte""} 6.274154496e+09 kube_node_status_capacity{node=""k8s-node2"",resource=""pods"",unit=""integer""} 110 kube_node_status_capacity{node=""k8s-node2"",resource=""cpu"",unit=""core""} 2
kubernetes.node.allocatable.memBytes	kube_node_status_allocatable_memory_bytes
kubernetes.node.allocatable.pods	kube_node_status_allocatable_pods
kubernetes.node.capacity.cpuCores	kube_node_status_capacity_cpu_cores	node=<node-address> resource=<resource-name> unit=<resource-unit> node=<node-address>
kubernetes.node.capacity.memBytes	kube_node_status_capacity_memory_bytes
kubernetes.node.capacity.pod	kube_node_status_capacity_pods
kubernetes.node.diskPressure	kube_node_status_condition	node=<node-address condition=<node-condition> status=<true\|false\|unknown>
kubernetes.node.memoryPressure
kubernetes.node.networkUnavailable
kubernetes.node.outOfDisk
kubernetes.node.ready
kubernetes.node.unschedulable	kube_node_spec_unschedulable	node=<node-address>
	kube_node_info	node=<node-address>
	kube_node_labels	node=<node-address> label_NODE_LABEL=<NODE_LABEL>
Deployment	kubernetes.deployment.replicas.available	kube_deployment_status_replicas_available	deployment=<deployment-name> namespace=<deployment-namespace>
kubernetes.deployment.replicas.desired	kube_deployment_spec_replicas
kubernetes.deployment.replicas.paused	kube_deployment_spec_paused
kubernetes.deployment.replicas.running	kube_deployment_status_replicas
kubernetes.deployment.replicas.unavailable	kube_deployment_status_replicas_unavailable
kubernetes.deployment.replicas.updated	kube_deployment_status_replicas_updated
	kube_deployment_labels
job	kubernetes.job.completions	kube_job_spec_completions	job_name=<job-name> namespace=<job-namespace>
kubernetes.job.numFailed	kube_job_failed
kubernetes.job.numSucceeded	kube_job_complete
kubernetes.job.parallelism	kube_job_spec_parallelism
	kube_job_status_active
	kube_job_info
	kube_job_owner	job_name=<job-name> namespace=<job-namespace> owner_kind=<owner kind> owner_name=<owner name>
	kube_job_labels	job_name=<job-name> namespace=<job-namespace> label_job_label=<job_label>
daemonSet	kubernetes.daemonSet.pods.desired	kube_daemonset_status_desired_number_scheduled	daemonset=<daemonset-name> namespace=<daemonset-namespace>
kubernetes.daemonSet.pods.misscheduled	kube_daemonset_status_number_misscheduled
kubernetes.daemonSet.pods.ready	kube_daemonset_status_number_ready
kubernetes.daemonSet.pods.scheduled	kube_daemonset_status_current_number_scheduled
	kube_daemonset_labels	daemonset=<daemonset-name> namespace=<daemonset-namespace> label_daemonset_label=<daemonset_label>
replicaSet	kubernetes.replicaSet.replicas.fullyLabeled	kube_replicaset_status_fully_labeled_replicas	replicaset=<replicaset-name> namespace=<replicaset-namespace>
kubernetes.replicaSet.replicas.ready	kube_replicaset_status_ready_replicas
kubernetes.replicaSet.replicas.running	kube_replicaset_status_replicas
kubernetes.replicaSet.replicas.desired	kube_replicaset_spec_replicas
	kube_replicaset_owner	replicaset=<replicaset-name> namespace=<replicaset-namespace> owner_kind=<owner kind> owner_name=<owner name>
		kube_replicaset_labels	label_replicaset_label=<replicaset_label> replicaset=<replicaset-name> namespace=<replicaset-namespace>
statefulset	kubernetes.statefulset.replicas	kube_statefulset_replicas	statefulset=<statefulset-name> namespace=<statefulset-namespace>
kubernetes.statefulset.status.replicas	kube_statefulset_status_replicas
kubernetes.statefulset.status.replicas.current	kube_statefulset_status_replicas_current
kubernetes.statefulset.status.replicas.ready	kube_statefulset_status_replicas_ready
kubernetes.statefulset.status.replicas.updated	kube_statefulset_status_replicas_updated
	kube_statefulset_labels
hpa	kubernetes.hpa.replicas.min	kube_horizontalpodautoscaler_spec_min_replicas	hpa=<hpa-name> namespace=<hpa-namespace>
kubernetes.hpa.replicas.max	kube_horizontalpodautoscaler_spec_max_replicas
kubernetes.hpa.replicas.current	kube_horizontalpodautoscaler_status_current_replicas
kubernetes.hpa.replicas.desired	kube_horizontalpodautoscaler_status_desired_replicas
	kube_horizontalpodautoscaler_labels
resourcequota	kubernetes.resourcequota.configmaps.hard kubernetes.resourcequota.configmaps.used kubernetes.resourcequota.limits.cpu.hard kubernetes.resourcequota.limits.cpu.used kubernetes.resourcequota.limits.memory.hard kubernetes.resourcequota.limits.memory.used kubernetes.resourcequota.persistentvolumeclaims.hard kubernetes.resourcequota.persistentvolumeclaims.used kubernetes.resourcequota.cpu.hard kubernetes.resourcequota.memory.hard kubernetes.resourcequota.pods.hard kubernetes.resourcequota.pods.used kubernetes.resourcequota.replicationcontrollers.hard kubernetes.resourcequota.replicationcontrollers.used kubernetes.resourcequota.requests.cpu.hard kubernetes.resourcequota.requests.cpu.used kubernetes.resourcequota.requests.memory.hard kubernetes.resourcequota.requests.memory.used kubernetes.resourcequota.requests.storage.hard kubernetes.resourcequota.requests.storage.used kubernetes.resourcequota.resourcequotas.hard kubernetes.resourcequota.resourcequotas.used kubernetes.resourcequota.secrets.hard kubernetes.resourcequota.secrets.used kubernetes.resourcequota.services.hard kubernetes.resourcequota.services.used kubernetes.resourcequota.services.loadbalancers.hard kubernetes.resourcequota.services.loadbalancers.used kubernetes.resourcequota.services.nodeports.hard kubernetes.resourcequota.services.nodeports.used	kube_resourcequota	resourcequota=<quota-name> namespace=<namespace> resource=<ResourceName> type=<quota-type>
namespace		kube_namespace_labels	namespace=<namespace-name> label_ns_label=<ns_label>
replicationcontroller	kubernetes.replicationcontroller.replicas.desired	kube_replicationcontroller_spec_replicase	replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace>
kubernetes.replicationcontroller.replicas.running	kube_replicationcontroller_status_replicas
	kube_replicationcontroller_status_fully_labeled_replicas kube_replicationcontroller_status_ready_replicas kube_replicationcontroller_status_available_replicas kube_replicationcontroller_status_observed_generation kube_replicationcontroller_metadata_generation kube_replicationcontroller_created
	kube_replicationcontroller_owner	replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> owner_kind=<owner kind> owner_name=<owner name>
service		kube_service_info	service=<service-name> namespace=<service-namespace> cluster_ip=<service cluster ip> external_name=<service external name> load_balancer_ip=<service load balancer ip>
	kube_service_labels	service=<service-name> namespace=<service-namespace> label_service_label=<service_label>
persistentvolume	kubernetes.persistentvolume.storage	kube_persistentvolume_capacity_bytes	persistentvolume=<pv-name>
	kube_persistentvolume_info	persistentvolume=<pv-name>
	kube_persistentvolume_labels	persistentvolume=<pv-name> namespace=<label_persistentvolume_label=<persistentvolume_label>
persistentvolumeclaim	kubernetes.persistentvolumeclaim.requests.storage	kube_persistentvolumeclaim_resource_requests_storage_bytes	namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name>
	kube_persistentvolumeclaim_info
	kube_persistentvolumeclaim_labels	persistentvolumeclaim=<persistentvolumeclaim-name> namespace=<persistentvolumeclaim-namespace> label_persistentvolumeclaim_label=<persistentvolumeclaim_label>

3 -

Run PromQL Queries Faster with Extended Label Set

Sysdig allows you to run PromQL queries smoother and faster with the extended label set. The extended label set is created by augmenting the incoming data with the rich metadata associated with your infrastructure and making it available in PromQL.

With this, you can troubleshoot a problem or building Dashboards and Alerts without the need to write complex queries. Sysdig automatically enriches your metrics with Kubernetes and application context without the need to instrument additional labels in your environment. This reduces operational complexity and cost—the enrichment takes place in Sysdig metric ingestion pipeline after time series have been sent to the backend.

Calculate Memory Usage by Deployment in a Cluster

Using the vector matching operation, you could run the following query and calculate the memory usage by deployment in a cluster:

sum by(cluster,namespace,owner_name) ((sysdig_container_memory_used_bytes * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~".+",namespace=~".+"})

To get the result, you need to write a query to perform a join (vector match) of various metrics, usually in the following order:

Grab a metric you need that is defined on a container level. For example, a Prometheus metric or some of the Sysdig provided metrics, such as sysdig_container_memory_used_byte.
Perform a vector match on container ID with the metric kube_pod_container_info to get the pod metadata.
Perform a vector match on the pod, namespace, and cluster with the kube_pod_owner metric.

In the case of Sysdig’s extended label set for PromQL, all the metrics inherit the metadata, so that necessary container, host, and Kubernetes metadata are set on all the metrics. This simplifies the query so you can build and run it quickly.

Likewise, the above query can be simplified as follows:

sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name) (sysdig_container_memory_used_bytes{kube_cluster_name!="",kube_namespace_name!="",kube_deployment_name!=""})

The advantages of using a simplified query are:

Complex vector matching operations (the group_left and group_right operators) are no longer required. All the labels are already available on each of the metrics, and therefore, any filtering can be performed directly on the metric itself.
The metrics now will have a huge amount of labels. You can use PromQL Explorer to deal with this rich metadata.
The metadata is distinguishable from user-defined labels. For example, Kubernetes metadata labels start with kube_. For instance, cluster is replaced with kube_cluster_name.
Create a dashboard panel or an alert from the PromQL query you run in the PromQL Query Explore.
Filter data by applying the comparison operators on the label values given in the table.

Examples for Simplifying Queries

Given below are some of the examples of using the extended label set to simplify complex query operations.

Memory Usage in a Kubernetes Cluster

Query with core label set:

avg by (agent_tag_cluster) ((sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * on(host,agent_tag_cluster) sysdig_host_info{agent_tag_cluster=~".+"}) * 100

Query with the extended label set:

avg by (agent_tag_cluster) (sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * 100