This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

  • 1:
    • 2:
      • 3:

        Metrics and Label Mapping

        This topic outlines the metrics and label mapping between the Sysdig classic store and the new Sysdig datastore.

        1 -

        Mapping Between Classic Metrics and PromQL Metrics

        Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.

        Entity

        Type

        PromQL Metric Name

        Classic Metric Name

        Label

        Classic Label

        host

        info

        sysdig_host_info

        Not exposed

        • host_mac

        • host

        • instance_id

        • agent_tag_{*}

        • host.mac

        • host.hostName

        • host.instanceId

        • agent.tag.{*}

        sysdig_cloud_provider_info

        • host_mac

        • provider_id

        • account_id

        • region

        • availability_zone

        • instance_type

        • tag_{*}

        • security_groups

        • host_ip_public

        • host_ip_private

        • host_name

        • name

        • host.mac

        • cloudProvider.id

        • cloudProvider.account.id

        • cloudProvider.region

        • cloudProvider.availabilityZone

        • cloudProvider.instance.type

        • cloudProvider.tag.{*}

        • cloudProvider.securityGroups

        • cloudProvider.host.ip.public

        • cloudProvider.host.ip.private

        • cloudProvider.host.name

        • cloudProvider.name

        data

        sysdig_host_cpu_used_percent

        cpu.used.percent

        • host_mac

        • host

        • host.mac

        • host.hostname

        sysdig_host_cpu_cores_used

        cpu.cores.used

        sysdig_host_cpu_user_percent

        cpu.user.percent

        sysdig_host_cpu_idle_percent

        cpu.idle.percent

        sysdig_host_cpu_iowait_percent

        cpu.iowait.percent

        sysdig_host_cpu_nice_percent

        cpu.nice.percent

        sysdig_host_cpu_stolen_percent

        cpu.stolen.percent

        sysdig_host_cpu_system_percent

        cpu.system.percent

        sysdig_host_fd_used_percent

        fd.used.percent

        sysdig_host_file_error_open_count

        file.error.open.count

        sysdig_host_file_error_total_count

        file.error.total.count

        sysdig_host_file_in_bytes

        file.bytes.in

        sysdig_host_file_in_iops

        file.iops.in

        sysdig_host_file_in_time

        file.time.in

        sysdig_host_file_open_count

        file.open.count

        sysdig_host_file_out_bytes

        file.bytes.out

        sysdig_host_file_out_iops

        file.iops.out

        sysdig_host_file_out_time

        file.time.out

        sysdig_host_load_average_15m

        load.average.15m

        sysdig_host_load_average_1m

        load.average.1m

        sysdig_host_load_average_5m

        load.average.5m

        sysdig_host_memory_available_bytes

        memory.bytes.available

        sysdig_host_memory_total_bytes

        memory.bytes.total

        sysdig_host_memory_used_bytes

        memory.bytes.used

        sysdig_host_memory_swap_available_bytes

        memory.swap.bytes.available

        sysdig_host_memory_swap_total_bytes

        memory.swap.bytes.total

        sysdig_host_memory_swap_used_bytes

        memory.swap.bytes.used

        sysdig_host_memory_virtual_bytes

        memory.bytes.virtual

        sysdig_host_net_connection_in_count

        net.connection.count.in

        sysdig_host_net_connection_out_count

        net.connection.count.out

        sysdig_host_net_error_count

        net.error.count

        sysdig_host_net_in_bytes

        net.bytes.in

        sysdig_host_net_out_bytes

        net.bytes.out

        sysdig_host_net_tcp_queue_len

        net.tcp.queue.len

        sysdig_host_proc_count

        proc.count

        sysdig_host_system_uptime

        system.uptime

        sysdig_host_thread_count

        thread.count

        container

        info

        sysdig_container_info

        Not exposed

        container_id

        container_id

        container_full_id

        none

        host_mac

        host.mac

        container

        container.name

        container_type

        container.type

        image

        container.image

        image_id

        container.image.id

        mesos_task_id

        container.mesosTaskId

        Only available in Mesos orchestrator.

        cluster

        kubernetes.cluster.name

        Present only if the container is part of Kubernetes.

        pod

        kubernetes.pod.name

        Present only if the container is part of Kubernetes

        namespace

        kubernetes.namespace.name

        Present only if the container is part of Kubernetes.

        data

        sysdig_container_cpu_used_percent

        cpu.used.percent

        • host_mac

        • container_id

        • container_type

        • container

        • host.mac

        • container.id

        • container.type

        • container.name

        sysdig_container_cpu_cores_used

        cpu.cores.used

        sysdig_container_cpu_cores_used_percent

        cpu.cores.used.percent

        sysdig_container_cpu_quota_used_percent

        cpu.quota.used.percent

        sysdig_container_cpu_shares

        cpu.shares.count

        sysdig_container_cpu_shares_used_percent

        cpu.shares.used.percent

        sysdig_container_fd_used_percent

        fd.used.percent

        sysdig_container_file_error_open_count

        file.error.open.count

        sysdig_container_file_error_total_count

        file.error.total.count

        sysdig_container_file_in_bytes

        file.bytes.in

        sysdig_container_file_in_iops

        file.iops.in

        sysdig_container_file_in_time

        file.time.in

        sysdig_container_file_open_count

        file.open.count

        sysdig_container_file_out_bytes

        file.bytes.out

        sysdig_container_file_out_iops

        file.iops.out

        sysdig_container_file_out_time

        file.time.out

        sysdig_container_memory_limit_bytes

        memory.limit.bytes

        sysdig_container_memory_limit_used_percent

        memory.limit.used.percent

        sysdig_container_memory_swap_available_bytes

        memory.swap.bytes.available

        sysdig_container_memory_swap_total_bytes

        memory.swap.bytes.total

        sysdig_container_memory_swap_used_bytes

        memory.swap.bytes.used

        sysdig_container_memory_used_bytes

        memory.bytes.used

        sysdig_container_memory_virtual_bytes

        memory.bytes.virtual

        sysdig_container_net_connection_in_count

        net.connection.count.in

        sysdig_container_net_connection_out_count

        net.connection.count.out

        sysdig_container_net_error_count

        net.error.count

        sysdig_container_net_in_bytes

        net.bytes.in

        sysdig_container_net_out_bytes

        net.bytes.out

        sysdig_container_net_tcp_queue_len

        net.tcp.queue.len

        sysdig_container_proc_count

        proc.count

        sysdig_container_swap_limit_bytes

        swap.limit.bytes

        sysdig_container_thread_count

        thread.count

        Process/ Program

        Info

        sysdig_program_info

        not exposed

        program

        proc.name

        cmd_line

        proc.commandLine

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        data

        sysdig_program_cpu_used_percent

        cpu.used.percent

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        sysdig_program_memory_used_bytes

        memory.bytes.used

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        sysdig_program_net_in_bytes

        net.bytes.in

        container_id

        container.id

        host_mac

        host.mac

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        sysdig_program_net_out_bytes

        net.bytes.out

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        sysdig_program_proc_count

        proc.count

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        sysdig_program_thread_count

        thread.count

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        program

        proc.name

        cmd_line

        proc.commandLine

        fs

        info

        sysdig_fs_info

        not exposed

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        device

        fs.device

        mount_dir

        fs.mountDir

        type

        fs.type

        data

        sysdig_fs_free_bytes

        fs.bytes.free

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        device

        fs.device

        sysdig_fs_inodes_total_count

        fs.inodes.total.count

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        device

        fs.device

        sysdig_fs_inodes_used_count

        fs.inodes.used.count

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        device

        fs.device

        sysdig_fs_total_bytes

        fs.bytes.total

        host_mac

        host.mac

        container_id

        container.id

        container_type

        container.type

        device

        fs.device

        fs.bytes.used

        host_mac

        host.mac

        container_id

        container_type

        container.id

        container_id

        devide

        fs.device

        2 -

        Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

        Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.

        For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.

        Resource

        Sysdig Metrics

        Kubernetes State Metrics

        Label

        Example / More Information

        Pod

        kubernetes.pod.containers.waiting

        kube_pod_container_status_waiting

        • container=<container-name>

        • pod=<pod-name>

        • namespace=<pod-namespace>

        kubernetes.pod.resourceLimits.cpuCores

        kubernetes.pod.resourceLimits.memBytes

        kube_pod_container_resource_limits

        • resource=<resource-name>

        • unit=<resource-unit>

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • node=< node-name>

        {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"}

        {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}

        kubernetes.pod.resourceRequests.cpuCores

        kubernetes.pod.resourceRequests.memBytes

        kube_pod_container_resource_requests

        • resource=<resource-name>

        • unit=<resource-unit>

        • container=<container-name>

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • node=< node-name>

        {namespace="default",pod="pod0",container="pod1_con1",resource="cpu",unit="core"}

        {namespace="default",pod="pod0",container="pod1_con1",resource="memory",unit="byte"}

        kubernetes.pod.status.ready

        kube_pod_status_ready

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • condition=<true|false|unknown>

        kube_pod_info

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • host_ip=<host-ip>

        • pod_ip=<pod-ip>

        • node=<node-name>

        • uid=<pod-uid>

        {namespace="default",pod="pod0",host_ip="1.1.1.1",pod_ip="1.2.3.4",uid="abc-0",node="node1",created_by_kind="<none>",created_by_name="<none>",priority_class=""}

        kube_pod_owner

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • owner_kind=<owner kind>

        • owner_name=<owner name>

        {namespace="default",pod="pod0",owner_kind="<none>",owner_name="<none>;",owner_is_controller="<none>"}

        kube_pod_labels

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • label_POD_LABEL=<POD_LABEL>

        {namespace="default",pod="pod0", label_app="myApp"}

        kube_pod_container_info

        • pod=<pod-name>

        • namespace=<pod-namespace>

        • container_id=<containerid>

        {namespace="default",pod="pod0",container="container2",image="k8s.gcr.io/hyperkube2",image_id="docker://sha256:bbb",container_id="docker://cd456"}

        node

        kubernetes.node.allocatable.cpuCores

        kube_node_status_allocatable_cpu_cores

        • node=<node-address>

        • resource=<resource-name>

        • unit=<resource-unit>

        • node=<node-address>

        resource/unit have one of the values: (cpu, core); (memory, byte); (pods, integer). Sysdig currently supports only CPU, pods, and memory resources for kube_node_status_capacity metrics.

        "# HELP kube_node_status_capacity The capacity for different resources of a node.
        kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_1Gi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_2Mi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-master"",resource=""memory"",unit=""byte""} 4.16342016e+09
        kube_node_status_capacity{node=""k8s-master"",resource=""pods"",unit=""integer""} 110
        kube_node_status_capacity{node=""k8s-node1"",resource=""pods"",unit=""integer""} 110
        kube_node_status_capacity{node=""k8s-node1"",resource=""cpu"",unit=""core""} 2
        kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_1Gi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_2Mi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-node1"",resource=""memory"",unit=""byte""} 6.274154496e+09
        kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_1Gi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_2Mi"",unit=""byte""} 0
        kube_node_status_capacity{node=""k8s-node2"",resource=""memory"",unit=""byte""} 6.274154496e+09
        kube_node_status_capacity{node=""k8s-node2"",resource=""pods"",unit=""integer""} 110
        kube_node_status_capacity{node=""k8s-node2"",resource=""cpu"",unit=""core""} 2

        kubernetes.node.allocatable.memBytes

        kube_node_status_allocatable_memory_bytes

        kubernetes.node.allocatable.pods

        kube_node_status_allocatable_pods

        kubernetes.node.capacity.cpuCores

        kube_node_status_capacity_cpu_cores

        • node=<node-address>

        • resource=<resource-name>

        • unit=<resource-unit>

        • node=<node-address>

        kubernetes.node.capacity.memBytes

        kube_node_status_capacity_memory_bytes

        kubernetes.node.capacity.pod

        kube_node_status_capacity_pods

        kubernetes.node.diskPressure

        kube_node_status_condition

        • node=<node-address

        • condition=<node-condition>

        • status=<true|false|unknown>

        kubernetes.node.memoryPressure

        kubernetes.node.networkUnavailable

        kubernetes.node.outOfDisk

        kubernetes.node.ready

        kubernetes.node.unschedulable

        kube_node_spec_unschedulable

        • node=<node-address>

        kube_node_info

        • node=<node-address>

        kube_node_labels

        • node=<node-address>

        • label_NODE_LABEL=<NODE_LABEL>

        Deployment

        kubernetes.deployment.replicas.available

        kube_deployment_status_replicas_available

        • deployment=<deployment-name>

        • namespace=<deployment-namespace>

        kubernetes.deployment.replicas.desired

        kube_deployment_spec_replicas

        kubernetes.deployment.replicas.paused

        kube_deployment_spec_paused

        kubernetes.deployment.replicas.running

        kube_deployment_status_replicas

        kubernetes.deployment.replicas.unavailable

        kube_deployment_status_replicas_unavailable

        kubernetes.deployment.replicas.updated

        kube_deployment_status_replicas_updated

        kube_deployment_labels

        job

        kubernetes.job.completions

        kube_job_spec_completions

        • job_name=<job-name>

        • namespace=<job-namespace>

        kubernetes.job.numFailed

        kube_job_failed

        kubernetes.job.numSucceeded

        kube_job_complete

        kubernetes.job.parallelism

        kube_job_spec_parallelism

        kube_job_status_active

        kube_job_info

        kube_job_owner

        • job_name=<job-name>

        • namespace=<job-namespace>

        • owner_kind=<owner kind>

        • owner_name=<owner name>

        kube_job_labels

        • job_name=<job-name>

        • namespace=<job-namespace>

        • label_job_label=<job_label>

        daemonSet

        kubernetes.daemonSet.pods.desired

        kube_daemonset_status_desired_number_scheduled

        • daemonset=<daemonset-name>

        • namespace=<daemonset-namespace>

        kubernetes.daemonSet.pods.misscheduled

        kube_daemonset_status_number_misscheduled

        kubernetes.daemonSet.pods.ready

        kube_daemonset_status_number_ready

        kubernetes.daemonSet.pods.scheduled

        kube_daemonset_status_current_number_scheduled

        kube_daemonset_labels

        • daemonset=<daemonset-name>

        • namespace=<daemonset-namespace>

        • label_daemonset_label=<daemonset_label>

        replicaSet

        kubernetes.replicaSet.replicas.fullyLabeled

        kube_replicaset_status_fully_labeled_replicas

        • replicaset=<replicaset-name>

        • namespace=<replicaset-namespace>

        kubernetes.replicaSet.replicas.ready

        kube_replicaset_status_ready_replicas

        kubernetes.replicaSet.replicas.running

        kube_replicaset_status_replicas

        kubernetes.replicaSet.replicas.desired

        kube_replicaset_spec_replicas

        kube_replicaset_owner

        • replicaset=<replicaset-name>

        • namespace=<replicaset-namespace>

        • owner_kind=<owner kind>

        • owner_name=<owner name>

        kube_replicaset_labels

        • label_replicaset_label=<replicaset_label>

        • replicaset=<replicaset-name>

        • namespace=<replicaset-namespace>

        statefulset

        kubernetes.statefulset.replicas

        kube_statefulset_replicas

        • statefulset=<statefulset-name>

        • namespace=<statefulset-namespace>

        kubernetes.statefulset.status.replicas

        kube_statefulset_status_replicas

        kubernetes.statefulset.status.replicas.current

        kube_statefulset_status_replicas_current

        kubernetes.statefulset.status.replicas.ready

        kube_statefulset_status_replicas_ready

        kubernetes.statefulset.status.replicas.updated

        kube_statefulset_status_replicas_updated

        kube_statefulset_labels

        hpa

        kubernetes.hpa.replicas.min

        kube_horizontalpodautoscaler_spec_min_replicas

        • hpa=<hpa-name>

        • namespace=<hpa-namespace>

        kubernetes.hpa.replicas.max

        kube_horizontalpodautoscaler_spec_max_replicas

        kubernetes.hpa.replicas.current

        kube_horizontalpodautoscaler_status_current_replicas

        kubernetes.hpa.replicas.desired

        kube_horizontalpodautoscaler_status_desired_replicas

        kube_horizontalpodautoscaler_labels

        resourcequota

        kubernetes.resourcequota.configmaps.hard

        kubernetes.resourcequota.configmaps.used

        kubernetes.resourcequota.limits.cpu.hard

        kubernetes.resourcequota.limits.cpu.used

        kubernetes.resourcequota.limits.memory.hard

        kubernetes.resourcequota.limits.memory.used

        kubernetes.resourcequota.persistentvolumeclaims.hard

        kubernetes.resourcequota.persistentvolumeclaims.used

        kubernetes.resourcequota.cpu.hard

        kubernetes.resourcequota.memory.hard

        kubernetes.resourcequota.pods.hard

        kubernetes.resourcequota.pods.used

        kubernetes.resourcequota.replicationcontrollers.hard

        kubernetes.resourcequota.replicationcontrollers.used

        kubernetes.resourcequota.requests.cpu.hard

        kubernetes.resourcequota.requests.cpu.used

        kubernetes.resourcequota.requests.memory.hard

        kubernetes.resourcequota.requests.memory.used

        kubernetes.resourcequota.requests.storage.hard

        kubernetes.resourcequota.requests.storage.used

        kubernetes.resourcequota.resourcequotas.hard

        kubernetes.resourcequota.resourcequotas.used

        kubernetes.resourcequota.secrets.hard

        kubernetes.resourcequota.secrets.used

        kubernetes.resourcequota.services.hard

        kubernetes.resourcequota.services.used

        kubernetes.resourcequota.services.loadbalancers.hard

        kubernetes.resourcequota.services.loadbalancers.used

        kubernetes.resourcequota.services.nodeports.hard

        kubernetes.resourcequota.services.nodeports.used

        kube_resourcequota

        • resourcequota=<quota-name>

        • namespace=<namespace>

        • resource=<ResourceName>

        • type=<quota-type>

        namespace

        kube_namespace_labels

        • namespace=<namespace-name>

        • label_ns_label=<ns_label>

        replicationcontroller

        kubernetes.replicationcontroller.replicas.desired

        kube_replicationcontroller_spec_replicase

        • replicationcontroller=<replicationcontroller-name>

        • namespace=<replicationcontroller-namespace>

        kubernetes.replicationcontroller.replicas.running

        kube_replicationcontroller_status_replicas

        kube_replicationcontroller_status_fully_labeled_replicas

        kube_replicationcontroller_status_ready_replicas

        kube_replicationcontroller_status_available_replicas

        kube_replicationcontroller_status_observed_generation

        kube_replicationcontroller_metadata_generation

        kube_replicationcontroller_created

        kube_replicationcontroller_owner

        • replicationcontroller=<replicationcontroller-name>

        • namespace=<replicationcontroller-namespace>

        • owner_kind=<owner kind>

        • owner_name=<owner name>

        service

        kube_service_info

        • service=<service-name>

        • namespace=<service-namespace>

        • cluster_ip=<service cluster ip>

        • external_name=<service external name>

        • load_balancer_ip=<service load balancer ip>

        kube_service_labels

        • service=<service-name>

        • namespace=<service-namespace>

        • label_service_label=<service_label>

        persistentvolume

        kubernetes.persistentvolume.storage

        kube_persistentvolume_capacity_bytes

        • persistentvolume=<pv-name>

        kube_persistentvolume_info

        • persistentvolume=<pv-name>

        kube_persistentvolume_labels

        • persistentvolume=<pv-name>

        • namespace=<label_persistentvolume_label=<persistentvolume_label>

        persistentvolumeclaim

        kubernetes.persistentvolumeclaim.requests.storage

        kube_persistentvolumeclaim_resource_requests_storage_bytes

        • namespace=<persistentvolumeclaim-namespace>

        • persistentvolumeclaim=<persistentvolumeclaim-name>

        kube_persistentvolumeclaim_info

        kube_persistentvolumeclaim_labels

        • persistentvolumeclaim=<persistentvolumeclaim-name>

        • namespace=<persistentvolumeclaim-namespace>

        • label_persistentvolumeclaim_label=<persistentvolumeclaim_label>

        3 -

        Run PromQL Queries Faster with Extended Label Set

        Sysdig allows you to run PromQL queries smoother and faster with the extended label set. The extended label set is created by augmenting the incoming data with the rich metadata associated with your infrastructure and making it available in PromQL.

        With this, you can troubleshoot a problem or building Dashboards and Alerts without the need to write complex queries. Sysdig automatically enriches your metrics with Kubernetes and application context without the need to instrument additional labels in your environment. This reduces operational complexity and cost—the enrichment takes place in Sysdig metric ingestion pipeline after time series have been sent to the backend.

        Calculate Memory Usage by Deployment in a Cluster

        Using the vector matching operation, you could run the following query and calculate the memory usage by deployment in a cluster:

        sum by(cluster,namespace,owner_name) ((sysdig_container_memory_used_bytes * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~".+",namespace=~".+"})
        

        To get the result, you need to write a query to perform a join (vector match) of various metrics, usually in the following order:

        • Grab a metric you need that is defined on a container level. For example, a Prometheus metric or some of the Sysdig provided metrics, such as sysdig_container_memory_used_byte.

        • Perform a vector match on container ID with the metric kube_pod_container_info to get the pod metadata.

        • Perform a vector match on the pod, namespace, and cluster with the kube_pod_owner metric.

        In the case of Sysdig’s extended label set for PromQL, all the metrics inherit the metadata, so that necessary container, host, and Kubernetes metadata are set on all the metrics. This simplifies the query so you can build and run it quickly.

        Likewise, the above query can be simplified as follows:

        sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name) (sysdig_container_memory_used_bytes{kube_cluster_name!="",kube_namespace_name!="",kube_deployment_name!=""})
        

        The advantages of using a simplified query are:

        • Complex vector matching operations (the group_left and group_right operators) are no longer required. All the labels are already available on each of the metrics, and therefore, any filtering can be performed directly on the metric itself.

        • The metrics now will have a huge amount of labels. You can use PromQL Explorer to deal with this rich metadata.

        • The metadata is distinguishable from user-defined labels. For example, Kubernetes metadata labels start with kube_. For instance, cluster is replaced with kube_cluster_name.

        • Create a dashboard panel or an alert from the PromQL query you run in the PromQL Query Explore.

        • Filter data by applying the comparison operators on the label values given in the table.

        Examples for Simplifying Queries

        Given below are some of the examples of using the extended label set to simplify complex query operations.

        Memory Usage in a Kubernetes Cluster

        Query with core label set:

        avg by (agent_tag_cluster) ((sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * on(host,agent_tag_cluster) sysdig_host_info{agent_tag_cluster=~".+"}) * 100
        

        Query with the extended label set:

        avg by (agent_tag_cluster) (sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * 100
        

        CPU Usage in Containers

        Query with the core label set:

        sum by (cluster,namespace)(sysdig_container_cpu_cores_used * on (container_id) group_left(cluster,pod,namespace) kube_pod_container_info{cluster=~".+"})
        

        Simplified query with the extended label set:

        sum by (kube_cluster_name,kube_namespace_name)(sysdig_container_cpu_cores_used{kube_cluster_name=~".+"})
        

        Memory Usage in Daemonset

        Query with the core label set:

        sum by(cluster,namespace,owner_name) (sum by(pod) (label_replace(sysdig_container_memory_used_bytes * on(container_id,host_mac) group_left(label_io_kubernetes_pod_namespace,label_io_kubernetes_pod_name,label_io_kubernetes_container_name) sysdig_container_info{label_io_kubernetes_pod_namespace=~".*",cluster=~".*"},"pod","$1","label_io_kubernetes_pod_name","(.*)"))  * on(pod) group_right sum by(cluster,namespace,owner_name,pod) (kube_pod_owner{owner_kind=~"DaemonSet",owner_name=~".*",cluster=~".*",namespace=~".*"}))
        

        Simplified query with the extended label set:

        sum by(kube_cluster_name,kube_namespace_name,kube_daemonset_name) (sysdig_container_memory_used_bytes{kube_daemonset_name=~".*",kube_cluster_name=~".*",kube_namespace_name=~".*"})
        

        Pod Restarts in a Kubernetes Cluster

        Query with the core label set:

        sum by(cluster,namespace,owner_name)(changes(kube_pod_status_ready{condition="true",cluster=~$cluster,namespace=~$namespace}[$__interval]) * on(cluster,namespace,pod) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~$cluster,namespace=~$namespace})
        

        Simplified query with the extended label set:

        sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name)(changes(kube_pod_status_ready{condition="true",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_deployment_name=~".+"}[$__interval]))
        

        Containers per Image

        Query with the core label set:

        count by (owner_name,image,cluster,namespace)((sysdig_container_info{cluster=~$cluster,namespace=~$namespace})  * on(pod,namespace,cluster) group_left(owner_name) max by (pod,namespace,cluster,owner_name)(kube_pod_owner{owner_kind="Deployment",owner_name=~".+"}))
        

        Simplified query with the extended label set:

        count by (kube_deployment_name,image,kube_cluster_name,kube_namespace_name)(sysdig_container_info{kube_deployment_name=~".+",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace})
        

        Average TCP Queue per Node

        Query with the core label set:

        avg by (agent_tag_cluster,host)( sysdig_host_net_tcp_queue_len * on (host_mac) group_left(agent_tag_cluster,host) sysdig_host_info{agent_tag_cluster=~$cluster,host=~".+"})
        

        Simplified query with the extended label set:

        avg by (agent_tag_cluster,host_hostname) (sysdig_host_net_tcp_queue_len{agent_tag_cluster =~ $cluster})