Working with Metrics

• 1: Types of Metrics
• 2: Using Labels
• 3: Data Aggregation
• 4: Metric Limits
• 5: Manage Metric Scale
• 6: Metrics Library
• 6.1: Metrics and Labels Mapping
• 6.1.1: Mapping Classic Metrics with Context-Specific PromQL Metrics
• 6.1.2: Mapping Classic Metrics with PromQL Metrics
• 6.1.3: Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics
• 6.2: Metrics and Labels in Prometheus Format
• 6.2.1: Agent
• 6.2.2: Containers
• 6.2.3: Metric Labels
• 6.2.4: File
• 6.2.5: Host
• 6.2.6: JMX/JVM
• 6.2.7: Kubernetes
• 6.2.8: Network
• 6.2.9: Program
• 6.2.10: Provider
• 6.3: Metrics in Sysdig Legacy Format
• 6.3.1: Agent
• 6.3.2: Applications
• 6.3.2.1: Apache Metrics
• 6.3.2.2: Apache Kafka Metrics
• 6.3.2.2.1: Apache Kafka Consumer Metrics
• 6.3.2.2.2: Apache Kafka JMX Metrics
• 6.3.2.3: Consul Metrics
• 6.3.2.3.1: Base Consul Metrics
• 6.3.2.3.2: Consul StatsD Metrics
• 6.3.2.4: Couchbase Metrics
• 6.3.2.5: Elasticsearch Metrics
• 6.3.2.6: etcd Metrics
• 6.3.2.7: fluentd Metrics
• 6.3.2.8: Go Metrics
• 6.3.2.9: HTTP Metrics
• 6.3.2.10: HAProxy Metrics
• 6.3.2.11: Jenkins Metrics
• 6.3.2.12: Lighttpd Metrics
• 6.3.2.13: Memcached Metrics
• 6.3.2.14: Mesos/Marathon Metrics
• 6.3.2.14.1: Mesos Agent Metrics
• 6.3.2.14.2: Mesos Master Metrics
• 6.3.2.14.3: Marathon Metrics
• 6.3.2.15: MongoDB Metrics
• 6.3.2.16: MySQL Metrics
• 6.3.2.17: NGINX and NGINX Plus Metrics
• 6.3.2.17.1: NGINX Metrics
• 6.3.2.17.2: NGINX Plus Metrics
• 6.3.2.18: NTP Metrics
• 6.3.2.19: PGBouncer Metrics
• 6.3.2.20: PHP-FPM Metrics
• 6.3.2.21: PostgreSQL Metrics
• 6.3.2.22: RabbitMQ Metrics
• 6.3.2.23: Supervisord Metrics
• 6.3.2.24: TCP Metrics
• 6.3.2.25: Varnish Metrics
• 6.3.3: Benchmarks and Compliance
• 6.3.3.1: Docker/CIS Benchmarks
• 6.3.3.2: Kubernetes Benchmarks
• 6.3.4: Containers
• 6.3.5: Cloud Provider
• 6.3.5.1: AWS
• 6.3.5.1.1: Elasticache
• 6.3.5.1.2: Elastic Application Load Balancing (ALB)
• 6.3.5.1.3: Elastic Cloud Compute (EC2)
• 6.3.5.1.4: Elastic Container Service (ECS)
• 6.3.5.1.5: Elastic Load Balancing (ELB)
• 6.3.5.1.6: DynamoDB
• 6.3.5.1.7: Relational Database Service (RDS)
• 6.3.5.1.8: Simple Queue Service (SQS)
• 6.3.6: Deprecated Metrics and Labels
• 6.3.7: File
• 6.3.8: File System
• 6.3.9: Host
• 6.3.10: JVM
• 6.3.11: Prometheus Metrics Types
• 6.3.12: Kubernetes
• 6.3.12.1: Kubernetes State
• 6.3.12.2: Resource Usage
• 6.3.13: Network
• 6.3.14: Process
• 6.3.15: RedisDB Metrics
• 6.3.16: Security Policy Metrics
• 6.3.17: System
• 6.3.18: (Legacy) Heuristic and Deprecated Metrics
• 6.4: Sysdig Info Metrics
• 6.5: Troubleshooting Metrics
• 6.6: Heuristic Metrics

1 - Types of Metrics

Default Metrics

Default metrics include various kinds of metadata which Sysdig Monitor automatically knows how to label, segment, and display.

For example:

• System metrics for hosts, containers, and processes (CPU used, etc.)

• Orchestrator metrics (collected from Kubernetes, Mesos, etc.)

• Network metrics (e.g. network traffic)

• HTTP

• Platform metrics (in some cases)

Default metrics are collected mainly from two sources: syscalls and Kubernetes.

Custom Metrics

About Custom Metrics

Custom metrics generally refer to any metrics that the Sysdig Agent collects from some third-party integration. The type of infrastructure and applications integrated determine the custom metrics that the Agent collects and reports to Sysdig Monitor. The supported custom metrics are:

• Prometheus
• Monitoring Integrations
• JMX
• StatsD

Each metric comes with a set of custom labels, and additional labels can be user-created. Sysdig Monitor simply collects and reports them with minimal or no internal processing. Use the metrics_filter option in the dragent.yaml file to remove unwanted metrics or to choose the metrics to report when hosts exceed this limit. For more information on editing the dragent.yaml file, see Understanding the Agent Config Files.

Unit for Custom Metrics

Sysdig Monitor detects the default unit of custom metrics automatically with the delimiter suffix in the metrics name. For example, custom_expvar_time_seconds results in a base unit set to seconds. The supported base units are byte, percent, and time. Custom metrics name should carry one of the following delimiter suffixes in order for Sysdig Monitor to identify and configure the accurate unit type.

• second

• seconds

• byte

• bytes

• total (represents accumulating count)

• percent

Custom metrics will not be auto-detected and the unit will be incorrect unless this naming convention is followed. For instance, custom_byte_expvar will not yield the correct unit, that is MiB.

Editing the Unit Scale

You have the flexibility to change the unit scale either by editing the panel on the Dashboard or in the Explore.

Explore

From the Search Metrics and Dashboard drop-down, select the custom metrics you want to edit the unit selection for, then click More Options. Select the desired unit scale from the Metric Format drop-down and click Save.

Dashboard

Select the Dashboard Panel associated with the custom metrics you want to modify. Select the desired unit scale from the Metrics drop-down and click Save.

Display Missing Data

Data can be missing for a few different reasons:

• Problems such as faulty network connectivity in the communication channel between your infrastructure and Sysdig metrics store.

• Metrics or StatsD batch jobs are submitted sporadically.

Sysdig Monitor allows you to configure the behavior of missing data in Dashboards. Though metric type determines the default behavior, you can configure how to visualize missing data and define it at the per-query level. Use the No Data Display drop-down in the Options menu in the panel configuration, and the No Data Message text box under the Panel tab. See Create a New Panel for more information.

Consider the following guidelines:

• Use the No Data Message text box under the Panel tab to enter a custom message when no data is available to render on the panels. This custom message, which could include links in markdown format and line breaks, is shown when queries return no data and reports no errors.

• The No Data Display drop-down has only two options for the Stacked Area timechart: gap and show as zero.

• For form-based timechart panels, the default option for a metrics selection that does not contain a StatsD metric is gap.

• Adding a StatsD metric to a query in a form-based timechart panel will default the selected No Data Display type to the show as zero , which is the default option for form-based StatsD metrics. You can change this selection to any other type.

• The default display option is gap for PromQL Timechart panels.

The options for No Data Display are:

• gap: The default option for form-based timechart panel, where a query metrics selection does not contain a StatsD metric. gap is the best visualization type for most use cases because it is easy to spot indicating a problem.

  

• show as zero: The best option for StatsD metrics which are only submitted sporadically. For example, batch jobs and count of errors. This is the default display option for StatsD metrics in form-based panels.

  

  We do not recommend this option as setting zero could be misleading. For example, this setting will report the value for free disk space as 0% when the disk or host disappears, but in reality, the value is unknown.

• connect - solid: Use for measuring the value of a metric, typically a gauge, where you want to visualize the missing samples flattened.

  

  The leftmost and rightmost visible data points can be connected as Sysdig does not perform the interpolation.

• connect - dotted: Use it for measuring the value of a metric, typically a gauge, where you want to visualize the missing samples flattened.

  

  The leftmost and rightmost visible data points can be connected as Sysdig does not perform the interpolation.

2 - Using Labels

Labels are used to identify and differentiate characteristics of a metric, allowing them to be aggregated or filtered for Explore module views, dashboards, alerts, and captures. Labels can be used in different ways:

• To group infrastructure objects into logical hierarchies displayed on the Explore tab (called groupings). For more information, refer to Groupings.

• To split aggregated data into segments. For more information, refer to Segments.

There are two types of labels:

• Infrastructure labels

• Metric descriptor labels

Infrastructure Labels

Infrastructure labels are used to identify objects or entities within the infrastructure that a metric is associated with, including hosts, containers, and processes. An example label is shown below:

Sysdig Notation

kubernetes.pod.name

Prometheus Notation

kubernetes_pod_name

The table below outlines what each part of the label represents:

Infrastructure labels are obtained from the infrastructure (including from orchestrators, platforms, and the runtime processes), and Sysdig automatically builds a relationship model using the labels. This allows users to create logical hierarchical groupings to better aggregate the infrastructure objects in the Explore module.

For more information on groupings, refer to the Groupings.

Metric Descriptor Labels

Metric descriptor labels are custom descriptors or key-value pairs applied directly to metrics, obtained from integrations like StatsD, Prometheus, and JMX. Sysdig automatically collects custom metrics from these integrations, and parses the labels from them. Unlike infrastructure labels, these labels can be arbitrary, and do not necessarily map to any entity or object.

An example metric descriptor label is shown below:

website_failedRequests:20|region='Asia', customer_ID='abc'

The table below outlines what each part of the label represents:

Groupings

Groupings are hierarchical organizations of labels, allowing users to organize their infrastructure views on the Explore tab in a logical hierarchy. An example grouping is shown below:

The example above groups the infrastructure into four levels. This results in a tree view in the Explore module with four levels, with rows for each infrastructure object applicable to each level.

As each label is selected, Sysdig Monitor automatically filters out labels for the next selection that no longer fit the hierarchy, to ensure that only logical groupings are created.

The example below shows the logical hierarchy structure for Kubernetes:

• Clusters: Cluster > Namespace > Replicaset > Pod

• Namespace: Cluster > Namespace > HorizontalPodAutoscaler > Deployment > Pod

• Daemonsets : Cluster > Namespace > Daemonsets > Pod

• Services: Cluster > Namespace > Service > StatefulSet > Pod

• Job: Cluster > Namespace > Job > Pod

• ReplicationController: Cluster > Namespace > ReplicationController > Pod

The default groupings are immutable: They cannot be modified or deleted. However, you can make a copy of them that you can modify.

Unified Workload Labels

Sysdig provides the following labels to help improve your infrastructure organization and troubleshooting easier.

• kubernetes_workload_name: Displays all the Kubernetes workloads and indicates what type and name of workload resource (deployment, daemonSet, replicaSet, and so on) it is.

• kubernetes_workload_type: Indicates what type of workload resource (deployment, daemonSet, replicaSet, and so on) it is.

The availability of these labels also simplifies Groupings. You do not need different groupings for each type of deployment, instead, you have a single grouping for workloads.

The labels allow you to segment metrics, such as sysdig_host_cpu_cores_used_percent , by kubernetes_workload_name to see CPU cores usage for all the workloads, instead of having a separate query for segmenting by kubernetes_deployment_name, kubernetes_replicaSet_name , and so on.

Learn More

• Managing Groupings

Scopes

A scope is a collection of labels that are used to filter out or define the boundaries of a group of data points when creating dashboards, dashboard panels, alerts, and teams. An example scope is shown below:

In the example above, the scope is defined by two labels with operators and values defined. The table below defines each of the available operators.

The scope editor provides dynamic filtering capabilities. It restricts the scope of the selection for subsequent filters by rendering valid values that are specific to the previously selected label. Expand the list to view unfiltered suggestions. At run time, users can also supply custom values to achieve more granular filtering. The custom values are preserved. Note that changing a label higher up in the hierarchy might render the subsequent labels incompatible. For example, changing the kubernetes_namespace_name > kubernetes_deployment_name hierarchy to swarm_service_name > kubernetes_deployment_name is invalid as these entities belong to different orchestrators and cannot be logically grouped.

Dashboards and Panels

Dashboard scopes define the criteria for what metric data will be included in the dashboard’s panels. The current dashboard’s scope can be seen at the top of the dashboard:

By default, all dashboard panels abide by the scope of the overall dashboard. However, an individual panel scope can be configured for a different scope than the rest of the dashboard.

For more information on Dashboards and Panels, refer to the Dashboards documentation.

Alerts

Alert scopes are defined during the creation process, and specify what areas within the infrastructure the alert is applicable for. In the example alerts below, the first alert has a scope defined, whereas the second alert does not have a custom scope defined. If no scope is defined, the alert is applicable to the entire infrastructure.

For more information on Alerts, refer to the Alerts documentation.

Teams

A team’s scope determines the highest level of data that team members have visibility for:

• If a team’s scope is set to Host, team members can see all host-level and container-level information.

• If a team’s scope is set to Container, team members can only see container-level information.

For more information on teams and configuring team scope, refer to the Manage Teams and Roles documentation.

Segments

Aggregated data can be split into smaller sections by segmenting the data with labels. This allows for the creation of multi-series comparisons and multiple alerts. In the first image, the metric is not segmented:

In the second image, the same metric has been segmented by container_id:

Line and Area panels can display any number of segments for any given metric. The example image below displays the sysdig_connection_net_in_bytes metric segmented by both container_id and host_hostname:

For more information regarding segmentation in dashboard panels, refer to the Configure Panels documentation. For more information regarding configuring alerts, refer to the Alerts documentation.

The Meaning of n/a

Sysdig Monitor imports data related to entities such as hosts, containers, processes, and so on, and reports them in tables or panels on the Explore and Dashboards UI, as well as in events, so across the UI you see varieties of data. The term n/a can appear anywhere on the UI where some form of data is displayed.

n/a is a term that indicates data that is not available or that it does not apply to a particular instance. In Sysdig parlance, the term signifies one or more entities defined by a particular label, such as hostname or Kubernetes service, for which the label is invalid. In other words, n/a collectively represent entities whose metadata is not relevant to aggregation and filtering techniques—Grouping, Scoping, and Segmenting. For instance, a list of Kubernetes services might display the list of all the services as well as n/a that includes all the containers without the metadata describing a Kubernetes service.

You might encounter n/a sporadically in Explore UI as well as in drill-down panels or dashboards, events, and likely elsewhere on the Sysdig Monitor UI when no relevant metadata is available for that particular display. How n/a should be treated depends on the nature of your deployment. The deployment will not be affected by the entities marked n/a.

The following are some of the cases that yield n/a on the UI:

• Labels are partially available or not available. For example, a host has entities that are not associated with a monitored Kubernetes deployment, or a monitored host has an unmonitored Kubernetes deployment running.

• Labels that do not apply to the grouping criteria or at the hierarchy level. For example:

  • Containers that are not managed by Kubernetes. The containers managed by Kubernetes are identified with their  container_name labels.

  • In certain groupings by DaemonSet, Deployments render N/A and vice versa. Not all containers belong to both DaemonSet and Deployment objects concurrently. Likewise, a Kubernetes ReplicaSet grouping with the kubernetes_replicaset_name label will not show StatefulSets.

  • In a kubernetes_cluster_name > kubernetes_namespace_name > kubernetes_deployment_name  grouping, the entities without the kubernetes_cluster_name label yield n/a.

• Entities are incorrectly labeled in the infrastructure.

• Kubernetes features that are yet to be in sync with Sysdig Monitoring.

• The format is not applicable to a particular record in the database.

3 - Data Aggregation

Data Aggregation Concepts

Data Sampling

Sysdig agents collect 1-second samples and report data at 10-seconds resolution. It is the lowest resolution at which backend stores the data. In order to do so, the agent performs the downsampling from 1-second to 10-second samples.

Note: This is true for all the metrics, except Prometheus. For Prometheus metrics, data is sampled at every 1 second, but what is reported in the 10-seconds interval is the lastest value, not the downsample.

Samples are initially stored on the lowest supported resolution of 10-seconds, after which samples are being rolled up to higher downsampled timelines periodically, as new data arrives. For example, the data registered at every 10 seconds is rolled up in blocks of 1-minute interval, and the data stored in blocks of 1-minutes is being rolled up to 10-minutes blocks.

Downsampling

Downsampling refers to the process of aggregating multiple samples, on defined time interval, into set of values which can provide estimation for aggregated time ranges. In Sysdig parlance, downsampling is nothing but the data aggregation performed by the backend before exposing it as time aggregation on the UI or by the API. In effect, the data available for time aggregation during query evaluation is not the raw data, but the values that represent the estimated values for the given time range.

Reducing the amount of samples also help reduce data retention costs as well as improve query performances by reducing the amount of data loaded during query evaluation.

Downsampled data is used only for longer time ranges. If you are viewing most recent data, such as 10 minute or last 1 hour, raw data is used for evaluation.

Data Rollup

Sysdig Monitor rolls up historical data over time.

Sysdig downsampling produces data rollups of aggregated samples. In each data rollup, Sysdig calculates and records 4 values: maximum, minimum, sum, and count. These values allow for exposing the following time aggregations: max, min, sum, count, avg, rate, rateOfChange on the UI as well as by the APIs.

For example, the data collected every 10-seconds is aggregated and rolled up in blocks of 1-minute interval. From the recorded values in 1-minute rollups, data is rolled up again for a block of 10-minutes interval.

Data Resolution

Data resolution is the frequency with which the data is displayed. Sysdig Monitor supports the data resolution of 10 seconds, 1 minute, 10 minutes, 1 hour, and 1 day.

Time and Group Aggregations

There are two forms of aggregation used for metrics in Sysdig: time aggregation and group aggregation. Time aggregation is always performed before group aggregation.

Time Aggregation

Time aggregation comes into effect in two situations (that can sometimes overlap):

• Aggregation: Graphs can only render a limited number of data points. To look at a wide range of data, Sysdig Monitor aggregate granular data into larger blocks of samples for visualization as given in Data Downsampling.
• Data Rollup: Sysdig retains rollups based on each aggregation type to allow users to choose which data points to utilize when evaluating older data.

Aggregation Types

Difference Between Rate and average

• Rate and average are very similar and often provide the same result. However, the calculation of each is different.

  • If time aggregation is set to one minute, the agent is supposed to retrieve six samples (one every 10 seconds).

  • In some cases, samples may not be there, due to disconnections or other circumstances. For this example, four samples are available. If this was the case, the average would be calculated by dividing by four, while the rate would be calculated by dividing by six.

• Most metrics are sampled once for each time interval, resulting in average and rate returning the same value. However, there will be a distinction for any metrics not reported at every time interval. For example, some custom statsd metrics.

• Rate is currently referred to as timeAvg in the Sysdig Monitor API and advanced alerting language.

• By default, average is used when displaying data points for a time interval.

Time Aggregation on the UI

On the Sysdig Monitor UI, you select the time aggregation from the Metric drop-down.

Depending on the time range you have selected, how old the data is, and what the resolution is , panels display data at a granularity of 10 seconds, 1 minute, 10 minute, 1 hour, and 1 day.

The data drawn at 10-second resolution is reported every 10-second with the available aggregations (average, rate, min, max, sum) to make them available via the Sysdig Monitor UI and the API. For time series panels covering 5 minutes or less, data points are drawn at this 10-second resolution, and any time aggregation selections will have no effect.

When a panel displays an amount of time greater than 5 minutes, data points are drawn as an aggregate for an appropriate time interval. For example, for a panel covering 1 hour, each data point would reflect a 1-minute interval.

At time intervals of 1-minute and above, charts can be configured to display different aggregates for the 10-second metrics used to calculate each datapoint.

Time Aggregation and Time Range Mapping on the UI

Group Aggregation

Metrics applied to a group of items (for example, several containers, hosts, or nodes) are averaged between the members of the group by default. For example, three hosts report different CPU usage for one sample interval. The three values will be averaged, and reported on the chart as a single datapoint for that metric.

There are several different types of group aggregation:

If a chart or alert is segmented, the group aggregation settings will be utilized for both aggregations across the whole group, and aggregation within each individual segmentation.

For example, the image below shows a chart for CPU% across the infrastructure:

When segmented by proc_name, the chart shows one CPU% line for each process:

Each line provides the average value for every process with the same name. To see the difference, change the group aggregation type to sum:

The metric aggregation value showed beside the metric name is for the time aggregation. While the screenshot shows AVG, the group aggregation is set to SUM.

Aggregation Examples

The tables below provide an example of how each type of aggregation works. The first table provides the metric data, while the second displays the resulting value for each type of aggregation.

In the example below, the CPU% metric is applied to a group of servers called webserver. The first chart shows metrics using average aggregation for both time and group. The second chart shows the metrics using maximum aggregation for both time and group.

For each one minute interval, the second chart renders the highest CPU usage value found from the servers in the webserver group and from all of the samples reported during the one minute interval. This view can be useful when searching for transient spikes in metrics over long periods of time, that would otherwise be missed with average aggregation.

4 - Metric Limits

The Sysdig agent metric limit is different from the entitlement limit imposed on custom time series. Your time series entitlement could be lower than agent metric limits. For more information, see Time Series Billing.

View Metric Limits

The metric limits are automatically defined by Sysdig backend components based on your plan, agent version, and backend configuration. Metric limits are set per-category, and when aggregated the per-category limits define your overall metric limit per agent. Metric limits are global per account and the same limit will apply to each agent within a Sysdig account.

Use the Sysdig Agent Health & Status dashboard under Host Infrastructure templates to view per-category metric limits for your account, along with the current usage per host for each metric type.

Contact Sysdig Support to adjust metric limits for any category.

See the Sysdig Agent Health & Status dashboard to view the metric limits and current time series consumption for each agent.

Learn More

• Subscription
• Configure Agent Modes

5 - Manage Metric Scale

1. Include/exclude custom metrics by name filters.

   See Include/Exclude Custom Metrics.

2. Include/exclude metrics emitted by certain containers, Kubernetes annotations, or any other container label at collection time. See Prioritize/Include/Exclude Designated Containers.

3. Exclude metrics from unwanted ports. See Blacklist Ports.

6 - Metrics Library

6.1 - Metrics and Labels Mapping

6.1.1 - Mapping Classic Metrics with Context-Specific PromQL Metrics

Sysdig classic metrics such as cpu.used.percent previously returned values from a process, container, or host depending on the query segmentation or scope. You can now use context-explicit metrics which aligns with the flat model and resource specific semantics of Prometheus naming schema. Your existing dashboards and alerts will be automatically migrated to the new naming convention.

6.1.2 - Mapping Classic Metrics with PromQL Metrics

Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.

6.1.3 - Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.

For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.

"# HELP kube_node_status_capacity The capacity for different resources of a node.
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""memory"",unit=""byte""} 4.16342016e+09
kube_node_status_capacity{node=""k8s-master"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""cpu"",unit=""core""} 2
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node2"",resource=""cpu"",unit=""core""} 2

6.2 - Metrics and Labels in Prometheus Format

The Prometheus metrics library lists the metrics in Prometheus format supported by the Sysdig product suite, as well as kube state and cloud provider metrics.

The metrics listed in this section follows the statsd-compatible Sysdig naming convention. To see a mapping between Prometheus notation and Sysdig notation, see Metrics and Label Mapping.

Overview

Each metric in the dictionary has several pieces of metadata listed to provide greater context for how the metric can be used within Sysdig products. An example layout is displayed below:

Metric Name

Metric definition. For some metrics, the equation for how the value is determined is provided.

6.2.1 - Agent

sysdig_agent_info

sysdig_agent_timeseries_count_appcheck

sysdig_agent_timeseries_count_jmx

sysdig_agent_timeseries_count_prometheus

sysdig_agent_timeseries_count_statsd

6.2.2 - Containers

sysdig_container_count

sysdig_container_cpu_cgroup_used_percent

sysdig_container_cpu_cores_cgroup_limit

sysdig_container_cpu_cores_quota_limit

sysdig_container_cpu_cores_used

sysdig_container_cpu_cores_used_percent

sysdig_container_cpu_quota_used_percent

sysdig_container_cpu_shares_count

sysdig_container_cpu_shares_used_percent

sysdig_container_cpu_used_percent

sysdig_container_fd_used_percent

sysdig_container_file_error_open_count

sysdig_container_file_error_total_count

sysdig_container_file_in_bytes

sysdig_container_file_in_iops

sysdig_container_file_in_time

sysdig_container_file_open_count

sysdig_container_file_out_bytes

sysdig_container_file_out_iops

sysdig_container_file_out_time

sysdig_container_file_total_bytes

sysdig_container_file_total_iops

sysdig_container_file_total_time

sysdig_container_fs_free_bytes

sysdig_container_fs_free_percent

sysdig_container_fs_inodes_total_count

sysdig_container_fs_inodes_used_count

sysdig_container_fs_inodes_used_percent

sysdig_container_fs_largest_used_percent

sysdig_container_fs_root_used_percent

sysdig_container_fs_total_bytes

sysdig_container_fs_used_bytes

sysdig_container_fs_used_percent

sysdig_container_info

sysdig_container_memory_limit_bytes

sysdig_container_memory_limit_used_percent

sysdig_container_memory_used_bytes

sysdig_container_memory_used_percent

sysdig_container_memory_virtual_bytes

sysdig_container_net_connection_in_count

sysdig_container_net_connection_out_count

sysdig_container_net_connection_total_count

sysdig_container_net_error_count

sysdig_container_net_http_error_count

sysdig_container_net_http_request_count

sysdig_container_net_http_request_time

sysdig_container_net_http_statuscode_error_count

sysdig_container_net_http_statuscode_request_count

sysdig_container_net_http_url_error_count

sysdig_container_net_http_url_request_count

sysdig_container_net_http_url_request_time

sysdig_container_net_in_bytes

sysdig_container_net_mongodb_error_count

sysdig_container_net_mongodb_request_count

sysdig_container_net_out_bytes

sysdig_container_net_request_count

sysdig_container_net_request_in_count

sysdig_container_net_request_in_time

sysdig_container_net_request_out_count

sysdig_container_net_request_out_time

sysdig_container_net_request_time

sysdig_container_net_server_connection_in_count

sysdig_container_net_server_in_bytes

sysdig_container_net_server_out_bytes

sysdig_container_net_server_total_bytes

sysdig_container_net_sql_error_count

sysdig_container_net_sql_query_error_count

sysdig_container_net_sql_query_request_count

sysdig_container_net_sql_query_request_time

sysdig_container_net_sql_querytype_error_count

sysdig_container_net_sql_querytype_request_count

sysdig_container_net_sql_querytype_request_time

sysdig_container_net_sql_request_count

sysdig_container_net_sql_request_time

sysdig_container_net_sql_table_error_count

sysdig_container_net_sql_table_request_count

sysdig_container_net_sql_table_request_time

sysdig_container_net_tcp_queue_len

sysdig_container_net_total_bytes

sysdig_container_proc_count

sysdig_container_swap_limit_bytes

sysdig_container_swap_limit_used_percent

sysdig_container_syscall_count

sysdig_container_syscall_error_count

sysdig_container_thread_count

sysdig_container_timeseries_count_appcheck

sysdig_container_timeseries_count_jmx

sysdig_container_timeseries_count_prometheus

sysdig_container_timeseries_count_statsd

sysdig_container_up

6.2.3 - Metric Labels

_sysdig_datasource

agent_id

agent_mode

agent_version

cloud_provider_account_id

cloud_provider_availability_zone

cloud_provider_host_ip_private

cloud_provider_host_ip_public

cloud_provider_host_name

cloud_provider_id

cloud_provider_instance_type

cloud_provider_name

cloud_provider_region

cloud_provider_resource_endpoint

cloud_provider_resource_name

cloud_provider_resource_type

cloud_provider_security_groups

cloud_provider_status

container_full_id

container_id

container_image

container_image_digest

container_image_id

container_image_repo

container_image_tag

container_label_io_kubernetes_container_name

container_label_io_kubernetes_pod_name

container_label_io_kubernetes_pod_namespace

container_label_io_prometheus_path

container_label_io_prometheus_port

container_label_io_prometheus_scrape

container_name

container_type

cpu_core

ecs_cluster_name

ecs_service_name

ecs_task_family_name

file_mount

file_name

fs_device

fs_mount_dir

fs_type

host_domain

host_hostname

host_instance_id

host_ip_private

host_ip_public

host_mac

kube_cluster_id

kube_cluster_name

concurrency_policy

kube_cronjob_name

schedule

kube_daemonset_name

kube_daemonset_uid

kube_deployment_name

kube_deployment_uid

kube_hpa_name

kube_hpa_uid

kube_job_name

kube_job_owner_is_controller

kube_job_owner_kind

kube_job_owner_name

kube_job_uid

kube_namespace_name

kube_namespace_uid

kube_node_condition

kube_node_name

kube_node_resource

kube_node_status

kube_node_uid

kube_node_unit

name

claim_namespace

kube_persistentvolume_name

kube_persistentvolume_uid

access_mode

status

type

kube_persistentvolumeclaim_name