• 1:
• 1.1:
• 2:
• 2.1:
• 2.2:
• 2.3:
• 2.4:
• 3:
• 3.1:
• 3.2:
• 3.3:
• 3.4:
• 3.5:
• 3.6:
• 3.7:
• 3.8:
• 4:
• 4.1:
• 4.2:
• 4.3:
• 4.4:
• 4.5:
• 4.6:
• 4.7:
• 5:
• 5.1:
• 5.2:
• 5.2.1:
• 5.2.2:
• 5.2.3:
• 5.3:
• 5.3.1:
• 5.3.2:
• 5.3.2.1:
• 5.3.2.2:
• 5.3.3:
• 5.3.3.1:
• 5.3.3.2:
• 5.3.4:
• 5.3.5:
• 5.3.6:
• 5.3.7:
• 5.3.8:
• 5.3.9:
• 5.3.10:
• 5.3.11:
• 5.3.12:
• 5.3.13:
• 5.3.14:
• 5.3.14.1:
• 5.3.14.2:
• 5.3.14.3:
• 5.3.15:
• 5.3.16:
• 5.3.17:
• 5.3.17.1:
• 5.3.17.2:
• 5.3.18:
• 5.3.19:
• 5.3.20:
• 5.3.21:
• 5.3.22:
• 5.3.23:
• 5.3.24:
• 5.3.25:
• 5.4:
• 5.4.1:
• 5.4.2:
• 5.5:
• 5.6:
• 5.6.1:
• 5.6.1.1:
• 5.6.1.2:
• 5.6.1.3:
• 5.6.1.4:
• 5.6.1.5:
• 5.6.1.6:
• 5.6.1.7:
• 5.6.1.8:
• 5.7:
• 5.8:
• 5.9:
• 5.10:
• 5.11:
• 5.11.1:
• 5.11.2:
• 5.12:
• 5.13:
• 5.14:
• 5.15:
• 5.16:
• 6:
• 6.1:
• 6.2:
• 6.3:
• 6.4:
• 6.4.1:
• 6.4.2:
• 6.4.3:
• 6.4.4:
• 6.5:
• 6.5.1:
• 6.5.2:
• 6.5.2.1:
• 6.5.2.2:
• 6.5.2.3:
• 6.5.2.4:
• 6.5.2.5:
• 6.5.2.6:
• 6.5.3:
• 6.5.3.1:
• 6.5.3.2:
• 6.5.3.3:
• 6.5.3.4:
• 6.5.3.5:
• 6.6:
• 6.6.1:
• 6.6.2:
• 6.6.3:
• 6.6.4:
• 6.6.4.1:
• 6.6.4.2:
• 6.7:
• 6.7.1:
• 6.7.2:
• 6.7.3:
• 6.7.4:
• 6.7.5:
• 6.7.6:
• 6.7.7:
• 6.7.8:
• 6.7.9:
• 6.7.10:
• 6.7.11:
• 7:
• 7.1:
• 7.2:
• 7.3:
• 7.4:
• 7.5:
• 7.6:
• 7.7:
• 8:
• 8.1:
• 8.2:
• 8.3:
• 8.4:
• 8.5:
• 8.6:
• 8.7:
• 8.8:
• 8.9:
• 9:
• 9.1:
• 9.2:
• 10:
• 10.1:
• 10.2:
• 10.2.1:
• 10.2.2:
• 10.2.3:
• 10.2.4:
• 10.2.5:
• 10.2.5.1:
• 10.2.5.2:
• 10.2.6:
• 10.2.7:
• 10.2.8:
• 10.2.9:
• 10.3:
• 10.4:
• 10.5:
• 10.6:
• 10.6.1:
• 10.6.2:
• 10.6.3:
• 10.6.4:
• 10.6.5:
• 10.6.6:
• 10.6.7:
• 10.6.8:
• 10.6.9:
• 10.6.10:
• 10.6.11:
• 10.6.12:
• 10.6.13:
• 10.6.14:
• 10.6.15:
• 10.6.16:
• 10.6.17:
• 10.6.18:
• 10.6.19:
• 10.6.20:
• 10.6.21:
• 10.6.22:
• 10.6.23:
• 10.6.24:
• 10.6.25:
• 10.6.26:
• 10.6.27:
• 10.7:
• 10.8:
• 10.9:

1 -

Getting Started with Sysdig Monitor

Sysdig Monitor allows you to maximize the visibility of your Kubernetes environments with native Prometheus support. You can troubleshoot issues faster with Sysdig’s eBPF derived metrics, out-of-the-box dashboards, and alerts.

You can choose Sysdig Monitor for a Free Trial option to quickly connect to a single cloud account with Sysdig and start with Prometheus-compatible Kubernetes and cloud monitoring.

Once connected, the Get Started page shows a subset of the options available in the 30-day trial or Enterprise.

Get Started Page (SaaS)

The Get Started page targets the key steps to ensure users are getting the most value out of Sysdig Monitor. The page is updated with new steps as users complete tasks and Sysdig adds new features to the product.

The Get Started page also serves as a linking page for

• Documentation

• Release Notes

• The Sysdig Blog

• Self-Paced Training

• Support

Users can access the Get Started page at any time by clicking the rocketship in the side menu.

Install the Agent

Installing the agent on your infrastructure allows Sysdig to collect data for monitoring and security purposes. For more information, see Quick Install Sysdig Agent on Kubernetes.

(Optional) Connect Your Prometheus Servers

Connecting your Prometheus servers to Sysdig-managed Prometheus Service helps leverage Sysdig for scalable long-term storage of your Prometheus metrics, PromQL dashboards, centralized querying, and PromQL-based alerting. For more information, see Collect Prometheus Metrics.

Invite Your Team

Invite someone in your team to use this Sysdig Monitor account. They will be notified with an email. A user will be created for them and will be added to the default team. They are automatically assigned to the Advanced User role.

Monitor Your Kubernetes Clusters

Get a unified view of the health, risk, and capacity of your Kubernetes infrastructure in a multi- and hybrid-cloud environment. For more information, see Dashboard Templates.

Workload Status & Performance

Get deep insight into your Kubernetes workloads faster with the Workload Status & Performance Dashboard.

Pod Status & Performance

Drill down to workload pods and monitor pod-level resource usage and troubleshoot performance issues with the Pod Status & Performance Dashboard.

Cluster Capacity Planning

Verify if your cluster is sized properly for existing deployed applications, identify over-commit on resources that can lead to pod evictions, discover unused requested resources or containers without limits defined with the Cluster Capacity Planning Dashboard.

Cluster/Namespace Available Resources

Determine if your cluster has the capacity to deploy a new workload and ascertain if increasing CPU or memory requests or placing limits on an existing application is necessary with the Cluster/Namespace Available Resources Dashboard.

Pod Rightsizing & Workload Capacity Optimization

Identify resource-hogging workloads while optimizing your capacity with the Pod Rightsizing & Workload Capacity Optimization Dashboard.

Set Up Alert

Sysdig Monitor emits alerts to get proactive notification of events, anomalies, or any incident that requires attention. The alerting system provides out-of-the-box push gateways for regular email, Slack, Cloud-provider notification queues, and custom webhooks, among others. See .

Configure a Notification Channel

Alerts are used in Sysdig Monitor when Event thresholds have been crossed and can be sent over a variety of supported notification channels. Integrate Sysdig with your notification dispatchers and incident management workflows. See Set Up Notification Channels

Turn on Alerts

Turn on recommended alerts from our Alerts Library. Customize our recommendations or create your own alerts from scratch. See Alerts Library.

Monitor Your Services

Create a Dashboard

Create customized dashboards to display the most relevant views and metrics for the infrastructure in a single location. Each dashboard is comprised of a series of panels configured to display specific data in a number of different formats. See Dashboards.

Get Started with PromQL

Write PromQL queries easier with form-based querying available with Sysdig Monitor. All metrics are enriched with cloud and Kubernetes metadata avoiding complicated PromQL joins. See Using PromQL.

Monitoring Integrations

Sysdig discovers services running in infrastructure and recommends appropriate Monitoring Integrations that allow you to collect service-specific metrics. The integration bundle includes out-of-the-box dashboards and default alerts. See (Limited Availability) Configure Monitoring Integrations.

Advanced Actions

Integrate development tools:

• Use Sysdig CLI  to backup and restore, integrate with your CI/CD systems, and configure automation.

• Use the Sysdig Terraform Provider to configure Sysdig Monitor as Code.

1.1 -

Basic Onboarding

This section describes onboarding tips for Sysdig Monitor on-premises versions prior to 3.5.0.

Access the Sysdig Monitor Interface

To access the interface, the Sysdig Agent must be installed. For instructions on installing the Sysdig agent, see the Agent Installation documentation.

• The first user becomes the administrator.

• Once agent installation is complete, the Sysdig Monitor interface is available at https://app.sysdigcloud.com, if you are in the US East region. For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

  See SaaS Regions and IP Ranges and identify the correct URLs associated with your Sysdig application and region.

• Subsequent users must also have user credentials defined, either through Sysdig Monitor or through an integrated authentication tool.

  For information on creating users, see User and Team Administration.

The Sysdig Monitor Interface

The Explore module is the primary starting point for all troubleshooting with Sysdig Monitor, and the default homepage of the web interface for all users. An overview of the health of the entire infrastructure is displayed in a table, with various infrastructure components broken into a pre-configured hierarchical grouping.

The left-side menu provides access to the five primary modules within Sysdig Monitor, in addition to the user menus and Spotlight.

Manage Integrations with Sysdig Spotlight

Sysdig Spotlight provides an at a glance summary of the current state of the infrastructure and helps configure applications to be monitored.

You manage Sysdig Monitor integrations through Sysdig Spotlight, before being configured separately.

Add an Integration

To add new integrations:

1. Click the Spotlight (target) icon in the management section of the left-hand sidebar to open Sysdig Spotlight.

   

2. Click the Manage Your Integrations link.

3. Choose the integrations to add by selecting the relevant icon, or by searching for the integration, and then selecting the relevant icon. Multiple integrations can be added at once.

4. Click the Add X Integration/s button to add the integrations to Sysdig Monitor.

Integrations whose configuration is pending are marked with a warning symbol.

For configuration and supported applications, refer to the Integrations for Sysdig Monitor documentation.

Delete an Integration

To delete existing integrations:

1. From the Spotlight page, select the integrations to remove.

   Running integrations cannot be removed.

2. Click the Remove X Integration(s) button.

3. Click the Remove link to confirm the changes.

2 -

Overview

Overview leverages Sysdig’s unified data platform to monitor, secure, and troubleshoot your hosts and Kubernetes clusters and workloads.

The module provides a unified view of the health, risk, and capacity of your Kubernetes infrastructure— a single pane of glass for host machines as well as Kubernetes Clusters, Nodes, Namespaces, and Workloads across a multi- and hybrid-cloud environment. You can easily filter by any of these entities and view associated events and health data.

Overview shows metrics prioritized by event count and severity, allowing you to get to the root cause of the problem faster. Sysdig Monitor polls the infrastructure data every 10 minutes and refreshes the metrics and events on the Overview page with the system health.

Key Benefits

Overview provides the following benefits:

• Show a unified view of the health, risk, resource use, and capacity of your infrastructure environment at scale

  • Render metrics, security events, compliance CIS benchmark results, and contextual events in a single location

  • Eliminate the need for stand-alone security, monitoring, and forensics tools

  • View data on-the-fly by workload or by infrastructure

• Display contextual live event stream from alerts, Kubernetes, containers, policies, and image scanning results

• Surface entities intelligently based on event count and severity

• Drills down from Clusters to Nodes and Namespaces

• Support Infrastructure monitoring of multi- and hybrid- cloud environments

• Expose relevant information based on core operational users :

  • DevOps / Platform Ops

  • Security Analyst

  • Service Owner

Accessing the Overview User Interface

You can access and set the scope of Overview in the Sysdig Monitor UI or with the URL:

• On-Prem: https://[Sysdig URL]/#/overview

• SAAS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East is: https://app.sysdigcloud.com/#/overview

  For other regions, the format is https://<region>.app.sysdig.com/#/overview. Replace <region> with the region where your Sysdig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/#/overview.

Click Overview in the left navigation, then select one of the Kubernetes entities:

• Clusters

• Nodes

• Namespaces

• Workloads

About the Overview User Interface

The Overview interface opens to the Cluster Overview page. This section describes the major components of the interface and the navigation options.

Overview Rows

Each row represents a Kubernetes entity: a cluster, node, namespace, or workload. In the screenshot above, each row shows a Kubernetes cluster.

• Navigating rows is easy

  Click on the Overview icon in the left navigation and choose an Overview page, or drill down into the next Overview page to explore the next granular level of data. Each Overview page shows 10 rows by default and a maximum of 100 rows. Click Load More to display additional rows if there are more than 10 rows per page.

• Ability to select a specific row in an Overview page

  Each row contains the scope of the relevant entity that it is showing data for. Clicking a specific row leads to deselecting the rest of the rows (for instance, selecting staging deselects all other rows in the screenshot above) to focus on the scope of the selected entity, including the events which are scoped out by that row. Further, the Live badge will change to Paused, implying rows will not be updated regardless of the new data coming in. Pausing to focus on a single row provides a snapshot of what is going on until at the moment with the entity under purview.

• Entities are listed according to the severity and number of events detected in them, not by how new the events are

  Rows are sorted by the count and severity level of the events associated with the entity and are displayed in descending order. The items with the highest number of high severity events are shown first, followed by medium, low, and info. This organization helps to highlight events demanding immediate attention and to streamline troubleshooting efforts, in environments that may include thousands of entities.

Scope Editor

Scope Editor allows targeting down to a specific entity, such as a particular workload or namespace, from environments that may include thousands of entities. The levels of scope, determined by Kubernetes hierarchy, progresses from Workload to Cluster where Cluster being at the top level. In smaller environments, using the Scope Editor is equivalent to clicking a single row in an Overview page where no scope has been applied.

Cluster: The highest level in the hierarchy. The only scope applied to the page is Cluster. It allows you to select a specific cluster from a list of available ones.

Node: The second level in the hierarchy. The scope is determined by Cluster and Node. Selection is narrowed down to a specific node in a selected cluster.

Namespace: The third level in the hierarchy. The scope is determined by Cluster and Namespace. Selection is narrowed down to a specific namespace in a selected cluster.

Workloads: The last entity in the hierarchy. The scope is initially determined by Cluster and Namespace, then the selection is narrowed to a specific Deployment, Service, or StatefulSet. Choosing all three options are not allowed.

Time Navigation

The Overview feature is based around time. Sysdig Monitor polls the infrastructure data every 10 minutes and refreshes the metrics and events on the Overview page with the system health. You select how to view this gathered data by choosing a Preset interval and a time Range.

Presets

Presets are a way of visualizing data that Sysdig Monitor gathers every 10 minutes. A preset that is 10 minutes or less is refreshed every 30 seconds. A preset that is greater than 10 minutes is refreshed every 1 minute. Select a preset to determine the data sample to be displayed. Overview supports the following presets:

• 1 Hour: Data polled for the last one hour. This is the default value.

• 6 Hour: Data polled for the last six-hour.

• 1 Day: Data polled for the last day.

Presets work in conjunction with Range selections. Selecting a particular preset interval refreshes Range selection and reloads the Overview rows and events subsequently. For example:

10 Minutes: Resets the Range to Jul 9, 2.20 pm - Jul 9, 2.30 pm.

6 Hour: Resets the Range to Jul 9, 8.30 am - Jul 9, 2.30 pm.

1 Day: Resets the Range to Jul 8, 2.30 pm - Jul 9, 2.30 pm.

Because metrics and events are refreshed every 10 minutes on the Overview page, if you stay for more then 10 minutes on the Overview page, the data will be updated to show the newly-computed values.

Range

Range shows both date and time interval as well as the selected Presets in parenthesis. The Range indicated on the UI is determined by Presets. The time given is the closest time interval and by default, it is the current date and time preset by 1 hour. See Presets to understand how Range works with Presets.

Time Format

Overview supports UTC and PDT time formats. Use the toggle button next to Range to change the time format for the slot shown in Range. The default is PDT.

Live

The Live badge shows if the feed (Overview rows with data) is Live or Paused.

• Live: the data is continuously updating based on the 10-minute polling of the Sysdig back end. The Overview feed is normally always Live.

• Paused: When a specific row is selected, the data refresh pauses and the rows will not be updated with new data coming in.

Unified Stream of Events

The right panel of Overview provides a context-sensitive events feed.

Click an overview row to see relevant Events on the right. Each event is intelligently populated with end-to-end metadata to give context and enable troubleshooting.

Event Types

Overview renders the following event types:

• Alert: See Alerts.

• Custom: Ensure that Custom labels are enabled to view this type of events.

• Containers: Events associated with containers.

• Kubernetes: Events associated with Kubernetes infrastructure.

• Scanning: See Image Scanning.

• Policy: See Policies.

Event Statuses

Overview renders the following alert-generated event statuses:

• Triggered: The alert condition has been met and still persists.

• Resolved: A previously existed alert condition no longer persists.

• Acknowledged: The event has been acknowledged by the intended recipient.

• Un-acknowledged: The event has not been acknowledged by an intended recipient. All events are by default marked as Un-acknowledged.

• Silenced: The alert event has been silenced for a specified scope. No alert notification will be sent out to the channels during the silenced window.

General Guidelines

First-Time Usage

• If the environment is created for the first time, Sysdig Monitor fetches data and generates associated pages. The Overview feature is immediately enabled. However, wait for, at the maximum, 1 hour to see the Overview pages with the necessary data.

• Overview uses time windows in segments of 1H, 6H and 1D, and therefore wait respectively for 1H, 6H and 1D to be able to see data on the Overview pages.

• If enough data is not available for the first 1 hour, the “No Data Available” page will be presented until the first 1 hour passes.

Tuning Overview Data

Sysdig Monitor leverages a caching mechanism to fetch pre-computed data for the Overview screens.

If pre-computed data is unavailable, data fetched will be non-computed data, which must be calculated before displaying. This additional computational time adds delays. Caching is enabled for Overview but for optimum performance, you must wait for 1H, 6H, and 1D windows the first time you use Overview. After the specified time has passed, the data will be automatically be cached with every passing minute.

Enabling Overview for On-Prem Deployments

The Overview feature is not available by default on On-Prem deployments. Use the following API to enable it:

1. Get the Beta settings as follows:

   curl -X GET 'https://<Sysdig URL>/api/on-prem/settings/overviews' \
   -H 'Authorization: Bearer <GLOBAL_SUPER_ADMIN_SDC_TOKEN>' \
   -H 'X-Sysdig-Product: SDC' -k
   

   Replace <Sysdig URL> with the Sysdig URL associated with your deployment and <GLOBAL_SUPER_ADMIN_SDC_TOKEN> with the SDC token associated with your deployment.

2. Copy the payload and change the desired values in the settings.

3. Update the settings as follows:

   curl X PUT 'https://<Sysdig URL>/api/on-prem/settings/overview' \
   -H 'Authorization: Bearer <GLOBAL_SUPER_ADMIN_SDC_TOKEN>' \
   -H 'X-Sysdig-Product: SDC' \
   -d '{  "overviews": true,  "eventScopeExpansion": true}'
   

Feature Flags

• overviews: Set overviews to true to enable the backend components and the UI.

• eventScopeExpansion: Set eventScopeExpansion to true to enable scope expansion for all the Event types.

2.1 -

Clusters Data

This topic discusses the Clusters Overview page and helps you understand its gauge charts and the data displayed on them.

About Clusters Overview

In Kubernetes, a pool of nodes combine together their resources to form a more powerful machine, that is a Cluster. The Cluster Overview page provides key metrics indicating the health, risk, capacity, and compliance of each cluster. Your cluster can reside in any cloud or multi-cloud environment of your choice.

Each row in the Clusters page represents a cluster. Clusters are sorted by the severity of corresponding events in order to highlight the area that needs attention. For example, a cluster with high severity events is bubbled up to the top of the page to highlight the issue. You can further drill down to the Nodes or Namespaces Overview page for investigating at each level.

Interpret the Cluster Data

This topic gives insight into the metrics displayed on the Clusters Overview screen.

Node Ready Status

The chart shows the latest value returned by avg(min(kubernetes.node.ready)).

What Is It?

The number shows the readiness for nodes to accept pods across the entire cluster. The numeric availability indicates the percentage of time the nodes are reported as ready by Kubernetes. For example:

• 100% is displayed when 10 out of 10 nodes are ready for the entire time window, say, for the last one hour.

• 95% is displayed when 9 out of 10 nodes are ready for the entire time window and one node is ready only for 50% of the time.

The bar chart displays the trend across the selected time window, and each bar represents a time slice. For example, selecting the last 1-hour window displays 6 bars, each indicating a 10-minute time slice. Each bar represents the availability across the time slice (green) or the unavailability (red).

For instance, the following image shows an average availability of 80% across the last 1-hour, and each 10-minute time slice shows a constant availability for the same time window:

What to Expect?

Expect a constant 100% at all times.

What to Do Otherwise?

If the value is less than 100%, determine whether a node is not available at all, or one or more nodes are partially available.

• Drill down either to the Nodes screen in Overview or to the “Kubernetes Cluster Overview” in Explore to see the list of nodes and their availability.

• Check the Kubernetes Node Overview dashboard in Explore to identify the problem that Kubernetes reports.

Pods Available vs Desired

The chart shows the latest value returned by sum(avg(kubernetes.namespace.pod.available.count)) / sum(avg(kubernetes.namespace.pod.desired.count)).

What Is It?

The chart displays the ratio between available and desired pods, averaged across the selected time window, for all the pods in a given Cluster. The upper bound shows the number of desired pods in the Cluster.

For instance, the following image shows 42 desired pods are available to use:

What to Expect?

You should typically expect 100%.

If certain pods take a long time to be available you might temporarily see a value that is less than 100%. Pulling images, pod initialization, readiness probe, and so on causes such delays.

What to Do Otherwise?

Identify one or more Namespaces that have lower availability. To do so, drill down to the Namespaces screen, then drill down to the Workloads screen to identify the unavailable pods.

If the number of unavailable pods is considerably higher (the ratio is significantly low), check the status of the Nodes. A Node failure will cause several pods to become unavailable across most of the Namespaces.

Several factors could cause the pods to stuck in the Pending state:

• Pods make requests for resources that exceed what’s available across the nodes (the remaining allocatable pods).

• Pods make requests higher than the availability of every single node. For example, you have 8-core Nodes and you create a pod with a 16-core request. These pods might require reconfiguration and specific setup related to Node affinity and anti-affinity constraints.

• Namespace quota is reached before making a high resource request.

  If a quota is enforced at the Namespace level, you may hit the limit independent of the resource availability across the Nodes.

CPU Requests vs Allocatable

The chart shows the latest value returned by sum(avg(kubernetes.pod.resourceRequests.cpuCores)) / sum(avg(kubernetes.node.allocatable.cpuCores)).

What Is It?

The chart displays the ratio between CPU requests configured for all the pods in a selected Cluster and allocatable CPUs across all the nodes.

The upper bound shows the number of allocatable CPU cores across all the nodes in the Cluster.

For instance, the image below shows that out of 620 available CPU cores across all the nodes (allocatable CPUs), 71% is requested by the pods:

What to Expect?

Your resource utilization strategy determines what ratio you can expect. A healthy ratio falls between 50% and 80%.

Assuming all the nodes have the same amount of allocatable resources, a reasonable upper bound is the value of (node_count - 1) / node_count x 100. For example, the ratio will be 90% if you have 9 nodes. Having this percentage protects you against a node becoming unavailable.

What to Do Otherwise?

A lower ratio indicates under-utilized resources (and corresponding cost) in your infrastructure. A higher ratio indicates insufficient resources. As a result

• Applications cannot be scheduled to be run.

• Pods might not start and remain in a Pending/Unscheduled state.

To triage, do the following:

• Drill down to the Nodes screen to get insights into how resources are utilized across all nodes.

• Drill down to the Namespaces screen to understand how resources are requested across Namespaces.

• Drill down to Explore and refer to the following dashboards:

  • Kubernetes CPU Allocation Optimization: Evaluate whether a significant amount of resources are under-utilized in the infrastructure.

  • Kubernetes Workloads CPU Usage and Allocation: Determine whether pods are properly configured and are using resources as expected.

Can the Value Be Higher than 100%?

Currently, the ratio accounts only for scheduled pods, while pending pods are excluded from the calculation. This means pods have been scheduled to run on Nodes out of the allocatable pods. Consequently, the ratio cannot be higher than 100%.

In the case of over-commitment (pods requesting for more resources than what’s available), you can expect a higher Requests vs Allocatable ratio and a lower Pods Available vs Desired ratio. What it indicates is that most of the available resources are being used, and what’s left is not enough to schedule additional pods. Therefore, the Available vs Desired ratio for pods will decrease.

Listed below are some of the factors that could cause the pods to stuck in a Pending state:

• Pods make requests that exceed what’s available across the nodes (the remaining allocatable pods). The Requests vs Allocatable ratio is an indicator of this issue.

• Pods make requests that are higher than the availability of every single Node. For example, you have 8-core Nodes and you create a pod with a 16-core request. These pods might require reconfiguration and specific setup related to Node affinity and anti-affinity constraints.

• The Quota set at the Namespace level is reached before a request is configured. The Requests vs Allocatable ratio may not suggest the problem, but the Pods Available vs Desired ratio would decrease, especially for the specific Namespaces. See the Namespaces screen in Overview.

Memory Requests vs Allocatable

The chart shows the latest value returned by sum(avg(kubernetes.pod.resourceRequests.memBytes)) / sum(avg(kubernetes.node.allocatable.memBytes)).

What Is It?

The chart displays the ratio between memory requests configured for all the pods in the Cluster and allocatable memory available across all the Nodes.

The upper bound shows the allocatable memory available across all Nodes. The value is expressed in bytes, displayed in a specified unit.

For instance, the image below shows that out of 29.7 GiB available across all Nodes (allocatable memory), 35% is requested by the pods:

What to Expect?

Your resource utilization strategy determines what ratio you can expect. A healthy ratio falls between 50% and 80%.

Assuming all the nodes have the same amount of allocatable resources, a reasonable upper bound is the value of (node_count - 1) / node_count x 100. For example, 90% if you have 9 nodes. This ratio protects your system against a node becoming unavailable.

What to do Otherwise

A lower ratio indicates under-utilized resources (and corresponding cost) in your infrastructure. A higher ratio indicates insufficient resources. As a result

• Applications cannot be scheduled to be run.

• Pods might not start and remain in a Pending/Unscheduled state.

To troubleshoot, do the following:

• Drill down to the Nodes screen to get insights into how resources are utilized across all the Nodes.

• Drill down to the Namespaces screen to understand how resources are requested across Namespaces.

• Drill down to Explore and refer to the following dashboards:

  • Kubernetes Memory Allocation Optimization: Evaluate whether a significant amount of resources are under-utilized in the infrastructure.

  • Kubernetes Workloads Memory Usage and Allocation: Determine whether pods are properly configured and are using resources as expected.

Can the Value be Higher than 100%?

The ratio currently accounts only for scheduled pods, while pending pods are excluded from the calculation. What this implies is that pods have been scheduled to run on Nodes out of the allocatable resources available. Consequently, the ratio cannot be higher than 100%.

In the case of over-commitment (pods requesting for more resources than what’s available), expect a higher Requests vs Allocatable ratio and a lower Pods Available vs Desired ratio. What it indicates is that most of the available resources have been used and what’s left is not enough to schedule additional pods. Therefore, the Pods Available vs Desired ratio will decrease.

Listed are some of the factors that could cause your pods to stuck in a Pending state:

• Pods make requests that exceed what’s available across the nodes (the remaining allocatable pods). The Requests vs Allocatable ratio is an indicator of this issue.

• Pods make requests that are higher than the availability of every single Node. For example, you have 8-core nodes and you create a pod with a 16-core request. These pods might require configuration changes and specific setup related to node affinity and anti-affinity factors.

• The Quota set at the Namespace-level is reached before a high request is configured. The Requests vs Allocatable ratio might not suggest the problem, but the Pods Available vs Desired ratio would decrease, especially for the specific Namespaces. See the Namespaces screen in Overview.

Compliance Score

Docker: The latest value returned by avg(avg(compliance.k8s-bench.pass_pct)).

Kubernetes: The latest value returned by avg(avg(compliance.docker-bench.pass_pct)).

What Is it?

The numbers show the percentage of benchmarks that succeeded in the selected time window, respectively for Docker and Kubernetes entities.

What to Expect

If you do not have Sysdig Secure enabled, or you do not have benchmarks scheduled, then you should expect no data available.

Otherwise, the higher the score, the more compliant your infrastructure is.

What to Do Otherwise?

If the score is lower than expected, drill down to Docker Compliance Report or Kubernetes Compliance Report to see further details about benchmark checks and their results.

You may also want to use the Benchmarks / Results page in Sysdig Secure to see the history of checks.

2.2 -

Nodes Data

This topic discusses the Nodes Overview page and helps you understand its gauge charts and the data displayed on them.

About Nodes Overview

A node refers to a worker machine in Kubernetes. A physical machine or VM can represent a node. The Nodes Overview page provides key metrics indicating the health, capacity, and compliance of each node in your cluster.

Interpret the Nodes Data

This topic gives insight into the metrics displayed on the Nodes Overview page.

Node Ready Status

The chart shows the latest value returned by avg(min(kubernetes.node.ready)).

What Is It?

The number expresses the Node readiness to accept pods across the Cluster. The numeric availability indicates the percentage of time the Node is reported ready by Kubernetes. For example:

• 100% is displayed when a Node is ready for the entire time window, say, for the last one hour.

• 95% when the Node is ready for 95% of the time window, say, 57 out of 60 minutes.

The bar chart displays the trend across the selected time window, and each bar represents a time slice. For example, selecting “last 1 hour” displays 6 bars, each indicating a 10-minute time slice. Each bar shows the availability across the time slice (green) and the unavailability (red).

For instance, the image below indicates the Node has not been ready for the entire last 1-hour time window:

What to Expect?

The chart should show a constant 100% at all times.

What to Do Otherwise?

If the number is less than 100%, review the status reported by Kubernetes. Drill-down to the Kubernetes Node Overview Dashboard in Explore to see details about the Node readiness:

If the Node Ready Status has an alternating behavior, as shown in the image, the node is flapping. Flapping indicates that the kubelet is not healthy. See specific conditions reported by Kubernetes that would help determine the causes for the Node not being ready. Such conditions include network issues and memory pressure.

Pods Ready vs Allocatable

The chart reports the latest value of sum(avg(kubernetes.pod.status.ready)) / avg(avg(kubernetes.node.allocatable.pods)).

What Is It?

It is the ratio between available and allocatable pods configured on the node, averaged across the selected time window.

The upper bound shows the number of pods you can allocate in the node. See node configuration.

For instance, the image below indicates that you can allocate 110 pods in the Node (default configuration), but only 11 pods are ready:

What to Expect?

The ratio does not relate to resource utilization, but it measures the pod density on each node. The more pods you have on a single node, the more effort the kubelet has to put in order to manage the pods, the routing mechanism, and Kubernetes overall.

Given the allocatable is properly set, values lower than 80% indicate a healthy status.

What to Do Otherwise?

• Reviewing the default maximum pods configuration of the kubelet to allow more pods, especially if the CPU and memory utilization is healthy.

• Adding more nodes to allow for more pods to be scheduled.

• Reviewing kubelet process performance and Node resource utilization in general. A higher ratio indicates high pressure on the operating system and for Kubernetes itself.

CPU Requests vs Allocatable

The chart shows the latest value returned by sum(avg(kubernetes.pod.resourceRequests.cpuCores)) / sum(avg(kubernetes.node.allocatable.cpuCores)).

What Is It?

The chart shows the ratio between the number of CPU cores requested by the pods scheduled on the Node and the number of cores available to pods. The upper bound shows the CPU cores available to pods, which corresponds to the user-defined configuration for allocatable CPU.

For instance, the image below shows that the Node has 16 CPU cores available, out of which, 84% are requested by the pods scheduled on the Node:

What to Expect?

Expect a value up to 80%.

Assuming all the nodes have the same amount of allocatable resources, a reasonable upper bound is the value of (node_count - 1) / node_count x 100. For example, 90% if you have 9 nodes. Having a high ratio protects your system against a Node becoming unavailable.

What to Do Otherwise?

• A low ratio indicates the Node is underutilized. Drill up to the corresponding cluster in the Clusters page to determine whether the number of pods currently running is lower, or if the pods cannot run for other reasons.

• A high ratio indicates a potential risk of being unable to schedule additional pods on the Node.

  Drill down to the  Kubernetes Node Overview Dashboard to evaluate what Namespaces, Workloads, and pods are running. Additionally, drill up in the Clusters page to evaluate whether you are over-committing the CPU resource. You might not have enough resources to fulfill requests, and consequently, pods might not be able to run on the Node. Consider adding Nodes or replacing Nodes with additional CPU cores.

Can the Value Be Higher than 100%?

Kubernetes schedules pods on Nodes where sufficient allocatable resources are available to fulfill the pod request. This means Kubernetes does not allow having a total request higher than the allocatable. Consequently, the ratio cannot be higher than 100%.

Over-committing (pods requesting resources higher than the capacity) results in a high Requests vs Allocatable ratio and a low Pods Available vs Desired ratio at the Cluster level. What it indicates is that most of the available resources are being used, consequently, what’s available is not sufficient to schedule additional pods. Therefore, Pods Available vs Desired ratio will also decrease.

Memory Requests vs Allocatable

The chart highlights the latest value returned by sum(avg(kubernetes.pod.resourceRequests.memBytes)) / sum(avg(kubernetes.node.allocatable.memBytes)).

What Is It?

The ratio between the number of bytes of memory is requested by the pods scheduled on the node and the number of bytes of memory available.The upper bound shows the memory available to pods, which corresponds to the user-defined allocatable memory configuration.

For instance, the image below indicates the node has 62.8 GiB of memory available, out of which, 37% is requested by the pods scheduled on the Node:

What to Expect?

A healthy ratio falls under 80%.

Assuming all the nodes have the same amount of allocatable resources, a reasonable upper bound is the value of (node_count - 1) / node_count x 100. For example, the ratio is 90% if you have 9 nodes. Having a high ratio protects your system against a node becoming unavailable.

What to Do Otherwise?

• A low ratio indicates that the Node is underutilized. Drill up to the corresponding cluster in the Clusters page to determine whether the number of pods running is low, or if pods cannot run for other reasons.

• A high ratio indicates a potential risk of being unable to schedule additional pods on the node.

  • Drill down to the  Kubernetes Node Overview dashboard to evaluate what Namespaces, Workloads, and pods are running.

  • Additionally, drill up in the Clusters page to evaluate whether you are over-committing the memory resource. Consequently, you don’t have enough resources to fulfill requests, and pods might not be able to run. Consider adding nodes or replacing nodes with more memory.

Can the Value be Higher than 100%?

Kubernetes schedules pods on nodes where sufficient allocatable resources are available to fulfill the pod request. This means Kubernetes does not allow having a total request higher than the allocatable. Consequently, the ratio cannot be higher than 100%.

Over-committing (pods requesting for more resources than that are available) results in a high Requests vs Allocatable ratio at the Nodes level and a low Pods Available vs Desired ratio at the Cluster level. What it indicates is that most of the resources are being used, consequently, what’s available is not sufficient to schedule additional pods. Therefore, Pods Available vs Desired ratio will also decrease.

Network I/O

The chart shows the latest value returned by avg(avg(net.bytes.total)).

What Is It?

The sparkline shows the trend of network traffic (inbound and outbound) for a Node. The number indicates the most recent rate of restarts per second.

For reference, the sparklines show the following number of steps (sampling):

• Last hour: 6 steps, each for a 10-minute time slice

• Last 6 hours: 12 steps, each for a 20-minute time slice

• Last day: 12 steps, each for a 2-hour time slice

What to Expect?

The metric highly depends on what type of applications run on the Node. You should expect some network activity for Kubernetes related operations.

Drilling down to the Kubernetes Node Overview Dashboard in Explore will provide additional details, such as network activity across pods.

2.3 -

Namespaces Data

This topic discusses the Namespaces Overview page and helps you understand its gauge charts and the data displayed on them.

About Namespaces Overview

Namespaces are virtual clusters on a physical cluster. They provide logical separation between the teams and their environments. The Namespaces Overview page provides key metrics indicating the health, capacity, and performance of each Namespace in your cluster.

Interpret the Namespaces Data

This topic gives insight into the metrics displayed on the Namespaces Overview screen.

Pod Restarts

The chart highlights the latest value returned by avg(timeAvg(kubernetes.pod.restart.rate)).

What Is It?

The sparkline shows the trend of pod restarts rate across all the pods in a selected Namespace. The number shows the most recent rate of restarts per second.

For instance, the image shows a rate of 0.04 restarts per second for the last 2-hours, given the selected time window is one day. The trend also suggests a non-flat pattern (periodic crashes).

• Last hour: 6 steps, each for a 10-minute time slice

• Last 6 hours: 12 steps, each for a 20-minute time slice

• Last day: 12 steps, each for a 2-hour time slice

What to Expect?

Expect 0 restarts for any pod.

What to Do Otherwise?

A few restarts across the last one hour or larger time windows might not indicate a serious problem. In the event restart loop, identify the root cause as follows:

• Drill down to the Workloads page in Overview to identify the Workloads that have been stuck at a restart loop.

• Drill down to the Kubernetes Namespace Overview to see a detailed trend broken down by pods:

Pods Available vs Desired

The chart shows the latest value returned by sum(avg(kubernetes.namespace.pod.available.count)) / sum(avg(kubernetes.namespace.pod.desired.count)).

What Is It?

The chart displays the ratio between available and desired pods, averaged across the selected time window, in a given Namespace.

The upper bound shows the number of desired pods in the namespace.

For instance, the image below shows 42 desired pods that are available:

What to Expect?

Expect 100% on the chart.

If certain pods take a significant amount of time to become available due to delays (image pull time, pod initialization, readiness probe) you might temporarily see a ratio lower than 100%.

What to Do Otherwise?

• Identify one or more Workloads that have low availability by drilling down to the Workloads page.

• Once you identify the Workload, drill down to the related dashboard in Explore. For example, Kubernetes Deployment Overview to determine the trend and the state of the pods.

  For instance, in the following image, the ratio is 98% (3.93 / 4 x 100). The decline is due to an update that caused pods to be terminated and consequently to be started with a newer version.

  

CPU Used vs Requests

The chart shows the latest value returned by sum(avg(cpu.cores.used)) / sum(avg(kubernetes.pod.resourceRequests.cpuCores)).

What Is It?

The chart shows the ratio between the total CPU usage across all the pods in the Namespace and the total CPU requested by all the pods.

The upper bound shows the total CPU requested by all the pods. The value is expressed as the number of CPU cores.

For instance, the image below shows the pods in a Namespace requests for 40 CPU cores, of which only 43% is being used (about 17 cores):

What to Expect?

The value you see depends on the type of Workloads running in the Namespace.

Typically, values that fall between 80% and 120% is considered healthy. Values higher than 100% is considered healthy relatively for a short amount of time.

For applications whose resource usage is constant (such as background processes), expect the ratio to be close to 100%.

For “bursty” applications, such as an API server, expect the ratio to be less than 100%. Note that this value is averaged for the selected time window, therefore, a usage spike would be compensated by an idle period.

What to Do Otherwise?

A low usage indicates that the application is not properly running (not executing the expected functions) or the Workload configuration is not accurate (requests are too high compared to what the pods actually need).

A high usage indicates that the application is operating with a heavy load or the workload configuration is not accurate (requests are too low compared to what pods actually need).

In either case, drill down to the Workloads page to determine the workload that requires a deeper analysis.

Can the Value Be Higher than 100%?

Yes, it can.

• You can configure requests without limits, or requests lower than the limits. In either case, you are allowing the containers to use more resources than requested, typically to handle temporary overloads.

• Consider a Namespace with two Workloads with one pod each. Say, one Workload is configured to request for 1 CPU core and uses 1 CPU core (ratio of Used vs Request is 100%). The other Workload is configured without any request and uses 1 CPU core. In this example, 2 CPU cores used to 1 CPU core requested ratio at the Namespace level is 200%.

Memory Used vs Requests

The chart shows the latest value returned by sum(avg(memory.bytes.used)) / sum(avg(kubernetes.pod.resourceRequests.memBytes)).

What Is It?

The chart shows the ratio between the total memory usage across all pods of the Namespace and the total memory requested by all pods.

The upper bound shows the total memory requested by all the pods, expressed in a specified unit for bytes.

For instance, the image below shows that all the pods in the Namespace requests for 120 GiB, of which only 24% is being used (about 29 GiB):

What to Expect?

It depends on the type of Workloads you run in the Namespace. Typically, values that fall between 80% and 120% are considered healthy.

Values that are higher than 100% considered normal for a relatively short amount of time.

What to Do Otherwise?

A low usage indicates the application is not properly running (not executing the expected functions) or the workload configuration is not accurate (high requests compared to what the pods actually need).

A high usage indicates the application is operating with a high load or the Workload configuration is not accurate (Fewer requests compared to what the pods actually need).

Given the configured limits for the Workloads and the memory pressure on the nodes, if the Workloads use more memory than what’s requested they are at risk of eviction. See Exceed a Container’s Limit for more information.

In both cases, you may want to drill down to the Workloads page to determine which Workload requires a deeper analysis.

Can the Value Be Higher than 100%?

Yes, it can.

• You can configure requests without limits, or requests lower than the limits. In either case, you are allowing the containers to use more resources than requested, typically to handle temporary overloads.

• Consider a Namespace with two Workloads with one pod each. Say, one Workload is configured to request for 1 GiB of memory and uses 1 GiB (Used vs Request ratio is 100%). The other Workload is configured without any request and uses 1 GiB. In this example, 2 GiB of Memory Used to1 GiB Requested ratio at the Namespace level is 200%.

Network I/O

The chart shows the latest value returned by avg(avg(net.bytes.total)).

What Is It?

The sparkline shows the trend of network traffic (inbound and outbound) for all the pods in the Namespace. The number shows the most recent rate, expressed in restarts per second.

For reference, the sparklines show the following number of steps (sampling):

• Last hour: 6 steps, each for a 10-minute time slice

• Last 6 hours: 12 steps, each for a 30-minute time slice

• Last day: 12 steps, each for a 2-hour time slice

What to Expect?

The type of applications run in the Namespace determine the metrics. Drilling down to the Kubernetes Namespace Overview Dashboard in Explore provides additional details, such as network activity across pods.

2.4 -

Workloads Data

This topic discusses the Workloads Overview page and helps you understand its gauge charts and the data displayed on them.

About Workloads Overview

Workloads, in Kubernetes terminology, refers to your containerized applications. Workloads comprise of Deployments, Statefulsets, and Daemonsets within a Namespace.

In a Cluster, worker nodes run your application workloads, whereas the master node provides the core Kubernetes services and orchestration for application workloads. The Workloads Overview page provides the key metrics indicating health, capacity, and compliance.

Interpret the Workloads Data

This topic gives insight into the metrics displayed on the Workloads Overview page.

Pod Restarts

The chart displays the latest value returned by sum(timeAvg(kubernetes.pod.restart.rate)).

What Is It?

The sparkline shows the trend of Pod Restarts rate across all the pods in a selected Workload. The number shows the most recent rate, expressed in Restarts per Second.

For instance, the image below shows the trend for the last hour. The number indicates that the rate of pod restarts is less than 0.01 for the last 10 minutes.

For reference, the sparklines show the following number of steps (sampling):

• Last hour: 6 steps, each for a 10-minute time slice.

• Last 6 hours: 12 steps, each for a 20-minute time slice.

• Last day: 12 steps, each for a 2-hour time slice.

What to Expect?

A healthy pod will have 0 restarts at any given time.

What to Do Otherwise?

In most cases, fewer restarts in the last hour (or larger time windows) do not indicate a serious problem. Drill down to the Kubernetes Overview Dashboard related to the Workload in Explore. For example, Kubernetes StatefulSet Overview provides a detailed trend broken down by pods.

In this example, the number of restarts is constant (roughly every 5 minutes) and no pods are ready. This might indicate a crash loop back-off .

Pods Available vs Desired

The chart shows the latest value of returned by sum(avg(kubernetes.deployment.replicas.available)) / sum(avg(kubernetes.deployment.replicas.desired)).

What Is It?

The chart displays the ratio between available and desired pods, averaged across the selected time window, for all the pods in a given Workload.

The upper bound shows the number of desired pods in the Workload.

For instance, the image below shows all the 42 desired pods are available.

What to Expect?

You should typically expect 100%.

If certain pods take a significant amount of time to become available (image pull time, pod initialization, readiness probe), then you may temporarily see a ratio lower than 100%.

What to Do Otherwise?

Determine the Workloads that have low availability by drilling down to the related Dashboard in Explore. For example, the Kubernetes Deployment Overview helps understand the trend and the state of the pods.

For instance, the image above shows that the ratio is 98% (3.93 / 4 x 100). The slight decline is due to an update that caused pods to be terminated and consequently to be started with a newer version.

CPU Used vs Requests

The chart shows the latest value returned by sum(avg(cpu.cores.used)) / sum(avg(kubernetes.pod.resourceRequests.cpuCores)).

What Is It?

The chart shows the ratio between the total CPU usage across all pods of a selected Workload and the total CPU requested by all the pods.

The upper bound shows the total CPU requested by all the pods. The value denotes the number of CPU cores.

In this image, the pods in the Workload requests for 40 CPU cores, of which 43% is actually used (about 17 cores).

What to Expect?

It depends on the type of workload.

For applications (background processes) whose resource usage is constant, expect the ratio to be around 100%.

For “bursty” applications, such as an API server, expect the ratio to be lower than 100%. Note that the value is averaged for the selected time window, therefore, a usage spike would be compensated by an idle period.

Generally, values between 80% and 120% are considered normal. Values that are higher than 100% deemed normal if it’s observed only for a relatively short time.

What to Do Otherwise?

• A low usage indicates that the application is not properly running (not executing the expected functions) or the Workload configuration is not accurate (requests are too high compared to what the pods actually need).

• A high usage indicates that the load is high for applications or the Workload configuration is not accurate (low requests compared to what the pods actually need).

In either case, drill down to the Kubernetes Overview Dashboard corresponding to the Workload in Explore. For example, the Kubernetes Deployment Overview Dashboard provides insight into resource usage and configuration.

Can the Value Be Higher than 100%?

Yes, it can.

• Configuring CPU requests without limits or requests lower than limits is permissible. In these cases, you are allowing the containers to use more resources than requested, typically to handle temporary overloads.

• Consider a Workload with two containers. Say, one container is configured to request for 1 CPU core and uses 1 CPU core (Used vs Request ratio is 100%). The other is configured without any request and uses 1 CPU core. In this example, the 2 CPU core Used to 1 CPU core Requested ratio is 200% at the Workload level.

What Does “No Data” Mean?

If the Workload is configured with no requests and limits, then the Usage vs Requests ratio cannot be computed. In this case, the chart will show “no data”. Drill down to the Dashboard in Explore to evaluate the actual usage.

You must always configure requests. Setting requests helps to detect Workloads that require reconfiguration.

Memory Used vs Requests

The chart shows the latest value returned by sum(avg(memory.bytes.used)) / sum(avg(kubernetes.pod.resourceRequests.memBytes)).

What Is It?

The chart shows the ratio between the total memory usage across all the pods in a Workload and the total memory requested by the Workload.

The upper bound shows the total memory requested by all the pods, expressed in the specified unit of bytes.

For instance, the image shows that the pods in the selected Workload requested for 120 GiB, of which 24% is actually used (about 29 GiB).

What to Expect?

The type of Workload determines the ratio. Values between 80% and 120% are considered normal. Values that are higher than 100% is deemed normal if it’s observed only for a relatively short time.

What to Do Otherwise?

A low memory usage indicates that the application is not properly running (not executing the expected functions) or the Workload configuration is not accurate (requests are too high compared to what the pods actually need).

A high memory usage indicates that the load is higher for applications or the Workload configuration is not accurate (low requests compared to what the pods actually need).

In either case, drill down to the Workloads page to determine the Workload that requires a deeper analysis.

Can the Value Be Higher than 100%?

Yes, it can.

• Configuring memory requests without limits or requests lower than limits is permissible. In these cases, you are allowing the containers to use more resources than requested, typically to handle temporary overloads.

• Consider a Workload with two containers. Say, one container is configured to request for 1 GiB of memory and uses 1 GiB (Used vs Request ratio is 100%), while the other is configured without any request and uses 1 GiB of memory. In this example, the 2 GiB of memory used to 1 GiB requested ratio is 200% at the Workload level.

What Does “No Data” Mean?

If the Workload is configured with no memory requests and limits, then the Usage vs Requests ratio cannot be computed. In this case, the chart will show “no data”. Drill down to the Dashboard in Explore to evaluate the actual usage.

You must configure requests. It helps to detect Workloads that require reconfiguration.

Network I/O

The chart shows the latest value returned by avg(avg(net.bytes.total)).

What Is It?

The sparkline shows the trend of network traffic (inbound and outbound) for the Workload. The number shows the most recent rate, expressed in bytes per second in a specific unit.

For reference, the sparklines show the following number of steps (sampling):

• Last hour: 6 steps, each for a 10-minute time slice

• Last 6 hours: 12 steps, each for a 30-minute time slice

• Last day: 12 steps, each for a 2-hour time slice

What to Expect?

The type of application runs in the Workload determines the metrics. Drill down to the Kubernetes Overview Dashboard corresponding to the Workload in Explore. For example, the Kubernetes Deployment Overview Dashboard provides additional details, such as network activity across pods.

3 -

Explore

The Sysdig Monitor web interface centers around the Explore module, where you perform the majority of infrastructure monitoring operations. Explore provides you the ability to view and troubleshoot key metrics and entities of your infrastructure stack. Sysdig Monitor automatically discovers your stack and presents pre-built views in Explore. You can drill down to any layers of your infrastructure hierarchy and view granular-level data. Grouping controls how entities are organized in Explore. Grouping is fully customizable by logical layers, such as containers, Kubernetes clusters, or services.

In addition to the Explore interface, Sysdig provides a PromQL Query Explorer and PromQL Library. They help you understand metrics and corresponding labels and values clearly, to create queries faster, and to build Dashboard and Alerts easily.

Learn more about using Explore in the following sections:

• Navigate the Explore Interface

• PromQL Query Explorer

• PromQL Library

• Explore Interface

• Groupings Editor

• Visualize Metrics Using the Topology View

• Time Windows

• Explore Workflows

3.1 -

Explore Interface

The sections below outline the key areas of the interface and detail basic navigation steps.

The C-Frame Structure

The image below provides a complete view of the Explore UI in the Sysdig Monitor c-frame style interface:

There are several key areas highlighted in the image above:

• Product Selector: This allows you to switch between Sysdig products.

• Grouping: Groupings are hierarchical organizations of tags, allowing users to organize their infrastructure views using the Grouping Wizard in a logical hierarchy. For more information on groupings, refer to Grouping, Scoping, and Segmenting Metrics.

  

• Modules: Quick links for each of the main Sysdig Monitor modules: Explore, Dashboards, Alerts, Events, and Captures.

• PromQL Query Explorer: Run PromQL queries to build your infrastructure views and get an in-depth insight into what’s going on. See PromQL Query Explorer.

• Management: Quick links for Sysdig Spotlight, help material, and the user profile configuration settings.

• Drill-Down: This allows you to explore deep down the infrastructure stack and retrieve all the components in a certain category in a single organized element.

• Search Metrics and Dashboards: Helps you select desired metrics or dashboards.

• Time Navigation: Helps you customize the time window used for displaying data

• Key Page Actions: Quick links to create events, alerts, and dashboards.

3.2 -

Navigate the Explore Interface

This section helps you navigate the Explore menu in the Sysdig Monitor UI.

Switch Groupings

Sysdig Monitor detects and collects the metrics associated with your infrastructure once the agent is deployed in your environment. Use the Explore UI to search, group, and troubleshoot your infrastructure components.

To switch between available data sources:

1. On the Explore tab, click the My Groupings drop-down menu:

2. Select the desired grouping from the drop-down list.

   

Groupings Editor

The Groupings Editor helps you create and manage your infrastructure groupings.

Use Drill-Down Menu

Sysdig Monitor users can drill down into the infrastructure by using the numerous dashboards and metrics available for display in the Explore UI. These displays can be found by selecting an infrastructure object, and opening the drill-down menu.

Sysdig Monitor only displays the metrics and dashboards that are relevant to the selected infrastructure object.

Metrics

Sysdig Monitor users can view specific metrics for an infrastructure object by navigating the drill-down menu:

1. On the Explore tab, open the drill-down menu.

2. Navigate to Search Metrics and Dashboard.

   

3. Select the desired metrics.

   The metric will now be presented on the Explore UI, until the user navigates away from it.

   The scope of the metric, when viewed via the drill-down menu, is set to the infrastructure object that you have selected.

Troubleshooting Views

The drill-down menu displays all the default dashboard templates relevant to the selected infrastructure object. These Troubleshooting Views are broken into the following sections:

The scope of the Troubleshooting View, when viewed via the drill-down menu, is set to the infrastructure object that you have selected from the drill-down.

• Application Dashboards

• AWS CloudWatch Dashboards

• Compliance & Security Dashboards

• Containers Dashboards

• Hosts Infrastructure Dashboards

• Kubernetes Dashboards

• Marathon Dashboards

• Mesos Dashboards

• Troubleshooting Dashboards

To navigate to the Troubleshooting Views:

1. On the Explore tab, select an infrastructure object.

2. Open the drill-down menu and select the desired infrastructure element

3. Navigate to Search Metrics and Dashboard.

   

4. Select the desired troubleshooting view.

   The selected dashboard will now be presented on the screen, until you navigate away from it.

Pin and Unpin the Drill-Down Menu

1. On the Explore tab, select an infrastructure object.

2. Open the drill-down menu.

3. Click Pin Menu to pin the menu to the Explore tab.

   

   To unpin the menu, click Unpin Menu at the bottom of the menu.

3.3 -

PromQL Query Explorer

Use the PromQL Query Explorer to run PromQL queries and build infrastructure views. It allows you

• Write PromQL queries faster by automatically identifying the common labels and labels among different metrics.

  See Run PromQL Queries Faster with Extended Label Set.

• Query metrics by leveraging advanced functions, operators, and boolean logic.

• Interactively modify the PromQL results by using visual label filtering.

• Use label filtering to visualize the common labels between metrics, which is key when combining multiple metrics.

About the PromQL Explorer UI

The main components of the PromQL Query Explorer UI include widgets, time navigation, and dashboard and time series panel.

You’ll find PromQL Explore under the Explore tab on the Sysdig Monitor UI.

PromQL Query

The PromQL field supports manually building PromQL queries. You can manually enter simple or complex PromQL queries and build dashboards and create alerts. The PromQL Query Explorer allows running up to 5 queries simultaneously. With the query field, you can do the following:

• Explore metrics and labels available in your infrastructure.

  For example, calculate the number of bytes received in a selected host:

  sysdig_host_net_total_bytes{host_mac="0a:e2:e8:b4:6c:1a"}
  

  Calculate the number of bytes received in all the hosts except one:

  sysdig_host_net_total_bytes{host_mac!="0a:a3:4b:3e:db:a2"}
  

  Compare current data with historical data:

  sysdig_host_net_total_bytes offset 7d
  

• Use arithmetic operators to perform calculations on one or more metrics or labels.

  For example, calculate the rate of incoming bytes and convert it to bits:

  rate(sysdig_host_net_total_bytes[5m]) * 8
  

• Build complex PromQL queries.

  For example, return summary ingress traffic across all the network interfaces grouped by instances

  sum(rate(sysdig_host_net_total_bytes[5m])) by (container_id)
  

Label Filtering

Label filtering to automatically identify common labels between queries for vector matching. In the given example, you can see that A and B metrics have only the host_mac label in common.

You can also filter by using the relational operators available in the time series table. Simply click the operator for it to be automatically applied to the queries. Run the queries again to visualize the metrics.

Filtering simultaneously applies to all the queries in the PromQL Query Explorer.

Widgets

PromQL Query Explorer supports only time series (Timechart). You can run advanced (PromQL) queries and build dashboard panels. PromQL Explorer does not support building form-based queries.

Time Navigation

PromQL Query Explorer is designed around time. After a query has been executed, Sysdig Monitor polls the infrastructure data every 10 seconds and refreshes the metrics on the Dashboard panel. You select how to view this gathered data by choosing a Preset interval and a time Range. For more information, see Time Navigation.

Legend

The legend is positioned on the upper right corner of the panel. Each query will have associated legends listed in the same execution order.

Build a Query

1. On the Explore tab, click PromQL Query.

   

2. Enter a PromQL query manually.

   sysdig_host_cpu_used_percent
   

   Click Add Query to run multiple queries. You can run up to 5 queries at once.

   sysdig_container_cpu_used_percent
   

3. Click Run Query or press command+Enter.

   A dashboard will appear on the screen. You can either Copy to a Dashboard or Create an Alert.

   

Copy to a Dashboard

1. Run a PromQL query.

2. Click Create > Create a Dashboard Panel.

3. Either select an existing Dashboard or enter the Dashboard name to copy to a new Dashboard.

4. Click Copy and Open.

   

   The new Dashboard panel with the given title will open to the Dashboard tab.

   

   You might want to continue with the Dashboard operations as given in Dashboards.

Create an Alert

1. Run a PromQL query.

2. Click Create > Create Alert.

3. If you have multiple queries, select the query you want to create the alert for.

   A new PromQL Alert page for the selected query appears on the screen.

   Continue with PromQL Alerts.

Remove a Query

Click the three dots next to the query field to remove the query.

Toggle Query Results

Click the respective query buttons, for example, A or B, to show or hide query results.

3.4 -

PromQL Library

PromQL is a powerful language to query metrics, but it could be challenging for beginners. To ease the learning curve of PromQL, Sysdig provides a set of curated examples, called PromQL Library. It helps you perform complex queries against your metrics with one click and get insight into your infrastructure problems which was not previously possible with Sysdig querying. For example, identify containers > 90% limit and counting pods per namespace, and so on.

You have the following categories currently to experiment with PromQL:

• Kubernetes

• Infrastructure

• Troubleshooting

• PromQL 101

Access PromQL Library

1. Log in to Sysdig Monitor.

2. Click Explore from the left navigation pane.

3. On the Explore tab, click PromQL Library.

   

   The tab opens to a list of PromQL examples.

Use PromQL Library

Click Try me to open PromQL Query Explore. A visualization corresponding to the query will be displayed. You can do the following with the query:

• Create a dashboard panel

• Create an alert

See PromQL Query Explorer for more information.

To copy a query, click the copy icon next to the query.

Filter PromQL Queries

Automatic tag filtering identifies common tags in the given examples. You can use the following to filter queries:

• Visual label filtering: Simply click the desired color-coded label to filter queries based on tags.

• Text search: Use the Text Search bar on the top-left navigation pane.

• Label search: Use the Label drop-down list on the top-left navigation pane.

• Filter using categories: Use the All Categories checkboxes.

3.5 -

Groupings Editor

Groupings are hierarchical organizations of labels, allowing you to organize your infrastructure views on the Explore UI in a logical hierarchy.

An example grouping is shown below:

The example above groups the infrastructure into four levels. This results in a tree view in the Groupings Editor with four levels, with rows for each infrastructure object applicable to each level.

As each label is selected, Sysdig Monitor automatically filters out labels for the next selection that no longer fit the hierarchy, to ensure that only logical groupings are created.

Sysdig Monitor automatically organizes all the configured groupings that are inapplicable to the current infrastructure under Inapplicable Groupings.

Manage Groupings

You can perform the following operations using the Groupings Editor:

• Search existing groupings

• Create a new grouping

• Edit an existing grouping

• Rename a groupings

• Share a grouping with the active team

Search for a Grouping

1. Do one of the following:

   • From Explore, click the Groupings drop-down. Search for the desired grouping.

     

     Either select the desired grouping, or search for it by scrolling down the list or by using the search bar, and then select it.

   • Click Manage Groupings and open the Groupings Editor.

     

     Either select the desired grouping, or search for it by scrolling down the list or by using the search bar, and then select it.

Create a New Grouping

1. In the Explore tab, click the Groupings drop-down, then click Manage Groupings.

2. Open the Groupings Editor.

3. Click Add.

   

   The New Groupings page is displayed.

4. Enter the following information:

   • Groupings Name: Set an appropriate name to identify the grouping that you are creating.

   • Shared with Team: Select if you want to share the grouping with the active team that you are part of.

   • Hierarchy: Determine the hierarchical representation of the grouping by choosing a top-level label and subsequent ones. Repeat adding the labels until there are no further layers available in the infrastructure label hierarchy.

     You can search for the label by entering the first few characters in the Select label drop-down or scrolling down. As you add labels, the preview displays associated components in your infrastructure.

5. Check the preview to ensure that the label selection is correct.

6. Click Save&Apply.

Rename a Grouping

Renaming is allowed only for groupings that are owned by you. To rename a shared grouping, create a copy of it and edit the name.

1. Do one of the following in Explore:

   • Click the Groupings drill-down. Search for the desired grouping. Click the Edit button next to the grouping.

   • Click the Groupings drill-down and click Manage Groupings.

2. Open the Groupings Editor.

3. Either select the desired grouping, or search for it by scrolling down the list or by using the search bar, and then select it.

4. Click Edit.

   

   The edit window is displayed on the screen.

5. Specify the new grouping name, then click Save& Apply to save the changes.

Share a Grouping with Your Active Team

Custom groupings are owned by you, and therefore you can share them with all the members of your active team. To share a default grouping, create a custom grouping and use the Shared with Team option in the Grouping Editor.

1. Click the Groupings drill-down and click Manage Groupings.

   The Grouping Editor screen appears.

2. Highlight the relevant grouping and click Edit.

3. Click Shared with Team.

4. Click Save &Apply to save the changes.

To share a default grouping, create a custom grouping and then use the Shared with Team option in the Grouping Editor.

3.6 -

Time Windows

By default, Sysdig Monitor displays information in Live mode. This means that dashboards, panels, and the Explore table will be automatically updated with new data as time passes, and will display the most recent data available for the configured time window.

The time window navigation bar provides users with quick links to common time windows, as well as the ability to configure a custom time period in order to review historical data.

As shown in the image above, the navigation bar provides a number of pieces of information:

• The state of the data (Live or Past).

• The current time window.

• The configured timezone.

In addition, the navigation bar provides:

• Quick links for common time windows (one second, one minute, ten minutes, one hour, six hours, one day, and two weeks).

• A custom time window configuration option.

• A pause/play button to exit Live mode and freeze the data to a time window, and to return to Live mode.

• Step back/forward buttons to jump through a frozen time window to review historical data.

• Zoom in/out buttons to increase/decrease the time window.

Configure a Custom Time Period

The Time Navigation dropdown panel can be used to configure a specific time range. To configure a manual range:

1. On the Explore tab or the Dashboards tab, click the Custom link in the time navigation bar.

2. Configure the start and end points, and click the Adjust Time button to save the changes.

   

Some limitations apply to custom time windows. Refer to the Time Window Limitations section for more information.

Time Window Limitations

Some time window configurations may not be available in certain situations. In these instances, a modification to the time window is automatically applied, and a warning notification will be displayed:

There are two main reasons for a time window being unavailable. Both relate to data granularity and specificity:

• The time window specifies the granularity of data that has expired and is no longer available. For example, a time window specifying a one-hour time range from six months ago would not be available, resulting in the time window being modified to a time range of at least one day.

• The time window specifies a granularity of data that is too high given the size of the window, as a graph can only handle a certain number of data points. For example, a multi-hour time range would contain too many datapoints at one-minute granularity, and would automatically be modified to 10-minute granularity.

3.7 -

Visualize Metrics Using the Topology View

Topology View provides an interactive, animated interface to visualize how different components in your system interact with each other in real-time. The interface by default renders a selected host’s top processes and their interaction with processes on remote hosts or host groups. Entities in any valid logical grouping can be visually represented with Topology View. The interaction is depicted as nodes and links. Links connect nodes. Nodes and links radially expand from the left.

• Nodes: The entities participating in network communication. A node could be a process, a container, a host, or any label identified by Sysdig Agent. For example: kubernetes.pod.name.

  The limit imposed on the maximum number of nodes that can be visualized is as follows:

  • host groups (hosts in a region): 20

  • hosts: 20

  • containers: 20

  • processes: 10

  This limit applies to the entities at any node level.

• Links: The network connection between nodes.

Hosts and their child processes (host.hostName > proc.name) serve as the default grouping for the Topology View. Scaling a Topology View is limited by the number of processes and connections. Sysdig Monitor creates the Topology View by identifying network endpoints (IP addresses) derived from system call data.

Topology View in the Explore tab provides pre-defined dashboards to represent CPU Usage, Network``Traffic, and Response Time metrics, and as such, they are not configurable. It serves as a template diagnostic tool that provides bottom-up inter-process connection metrics as graphs. For a detailed description on each default metric type supported by the Topology View, see Topology Dashboards.

About Topology View

The UI allows you to zoom in, zoom out, fit, pan, and reset the display, and expand and collapse the nodes.

Zoom into an entity by clicking the associated + icon and view the top processes within the selected entity. You can expand the hosts to see individual processes running inside and corresponding metric value. The links indicating network connections are rendered between entities in the selected hierarchy and, when zoomed, between processes and entities. Use the mouse scroll wheel to zoom the content, then left-click and drag to move the map components within the window.

Topology View uses legends and color schemes. This section explains what they mean.

• The line width (thickness) of the links expresses metric values relative to other connections in the system. Scaling line width is relative to the metric values associated with each link. Topology View dynamically adjusts the legend as the data recorded for each connection changes.

  • The thin black line(—): Lower metric values.

  • The thick black line(—): Higher metric values.

  • Dashed lines(—): A previously existed connection. For example, an active connection between node A and B, rendered as a solid line, could dissolve after a data update if that connection does not exist anymore in the new data set. Instead of removing it from the graph, Sysdig Monitor depicts the inactive connection as a dotted line until the next data update. If the connection returns in a subsequent data update, it is rendered again as a solid line. If the connection does not exist, the line will be removed.

• The color scheme is determined by color coding for the customizable threshold. The nodes are identified by five colors:

  • 

    Green indicates the node is healthy.

  • 

    Yellow indicates the node is in a warning state.

  • 

    Red indicates the node is in a critical state.

  • 

    Grey indicates the node state is unknown. An unknown node typically signifies a node where no Sysdig Agent installed.

  • 

    Light Blue indicates no threshold is defined for the metrics.

Configure Topology View

You can navigate to the Topology View from the Explore or Dashboard menus.

Access from Explore

The types of Topology View you can see on Explore are pre-defined dashboard templates that contain only a single panel, and as such, they can’t be edited.

1. Click Explore.

2. On the Explore tab, click the Switch Data Source drop-down menu.

3. Select Sysdig Agents.

4. In the My Groupings drop-down, select the desired data source for which you want a topology view.

5. From the Search Metrics and Dashboard drill-down, select Topology.

   

6. Select one of the three pre-defined Dashboards.

   The Topology View for the selected metrics will appear on the screen.

3.8 -

Explore Workflows

While every user has unique needs from Sysdig Monitor, there are three main workflows that you can follow when building out the interface and monitoring your infrastructure.

Workflow One

This workflow assumes that an alert has not been triggered yet.

Start with Explore , identify a problem area, then drill-down into the data. This workflow is the most basic approach, as it begins with a user monitoring the overall infrastructure, rather than with a specific alert notification. The workflow tends to follow the following steps:

1. Organize the infrastructure with groupings.

2. Define key signals with alerts and dashboards to detect a problem.

3. Identify a problem area, and drill down into the data using dashboards, metrics, and by adjusting groupings and scope as necessary.

Workflow Two

Start with an event notification, and begin troubleshooting. This workflow begins with an already configured alert and event being triggered. Unlike workflow one, this workflow assumes that pre-determined data boundaries have already been set:

1. Explore the event by adjusting time windows, scope, and segmentation.

2. Identify the exact area of concern within the infrastructure.

3. Drill down into the data to troubleshoot the issue.

Workflow Three

Customize default dashboard panels to troubleshoot a potential issue. This workflow assumes that an issue has been identified within one of the default dashboards, but alerts have not been set up for the problem area.

1. Copy the displayed panel to a new dashboard.

2. Create an alert based on the dashboard panel.

3. Configure a Sysdig Capture on demand.

4 -

Metrics

Metrics are quantitative values or measures that can be grouped/divided by labels. Sysdig Monitor metrics are divided into two groups: default metrics (out-of-the-box metrics concerning the system, orchestrator, and network infrastructure), and custom metrics(JMX, StatsD, and multiple other integrated application metrics).

Sysdig automatically collects all types of metrics, and auto-labels them. Custom metrics can also have custom (user-defined) labels.

Out-of-the box, when an agent has been deployed on a host, Sysdig Monitor automatically begins collecting and reporting on a wide array of metrics. The sections below describe how those metrics are conceptualized within the system.

Learn more about the metrics types and the data aggregation techniques supported by Sysdig Monitor in the following sections:

• Understanding Default, Custom, and Missing Metrics

• Heuristic and Deprecated Metrics

• Grouping, Scoping, and Segmenting Metrics

• Data Aggregation

• Data Retention

• Manage Metric Scale

4.1 -

Grouping, Scoping, and Segmenting Metrics

Data aggregation and filtering in Sysdig Monitor are done through the use of assigned labels. The sections below explain how labels work, the ways they can be used, and how to work with groupings, scopes, and segments.

Labels

Labels are used to identify and differentiate characteristics of a metric, allowing them to be aggregated or filtered for Explore module views, dashboards, alerts, and captures. Labels can be used in different ways:

• To group infrastructure objects into logical hierarchies displayed on the Explore tab (called groupings). For more information, refer to Groupings .

• To split aggregated data into segments. For more information, refer to Segments.

There are two types of labels:

• Infrastructure labels

• Metric descriptor labels

Infrastructure Labels

Infrastructure labels are used to identify objects or entities within the infrastructure that a metric is associated with, including hosts, containers, and processes. An example label is shown below:

kubernetes.pod.name

The table below outlines what each part of the label represents:

Infrastructure labels are obtained from the infrastructure (including from orchestrators, platforms, and the runtime processes), and Sysdig automatically builds a relationship model using the labels. This allows users to create logical hierarchical groupings to better aggregate the infrastructure objects in the Explore module.

For more information on groupings, refer to the Groupings.

Metric Descriptor Labels

Metric descriptor labels are custom descriptors or key-value pairs applied directly to metrics, obtained from integrations like StatsD, Prometheus, and JMX. Sysdig automatically collects custom metrics from these integrations, and parses the labels from them. Unlike infrastructure labels, these labels can be arbitrary, and do not necessarily map to any entity or object.

An example metric descriptor label is shown below:

website_failedRequests:20|region=‘Asia’, customer_ID=‘abc’

The table below outlines what each part of the label represents:

Groupings

Groupings are hierarchical organizations of labels, allowing users to organize their infrastructure views on the Explore tab in a logical hierarchy. An example grouping is shown below:

The example above groups the infrastructure into four levels. This results in a tree view in the Explore module with four levels, with rows for each infrastructure object applicable to each level.

As each label is selected, Sysdig Monitor automatically filters out labels for the next selection that no longer fit the hierarchy, to ensure that only logical groupings are created.

The example below shows the logical hierarchy structure for Kubernetes:

• Clusters: Cluster > Namespace > Replicaset > Pod

• Namespace: Cluster > Namespace > HorizontalPodAutoscaler > Deployment > Pod

• Daemonsets : Cluster > Namespace > Daemonsets > Pod

• Services: Cluster > Namespace > Service > StatefulSet > Pod

• Job: Cluster > Namespace > Job > Pod

• ReplicationController: Cluster > Namespace > ReplicationController > Pod

The default groupings are immutable: They cannot be modified or deleted. However, you can make a copy of them that you can modify.

Unified Workload Labels

Sysdig provides the following labels to help improve your infrastructure organization and troubleshooting easier.

• kubernetes.workload.name: Displays all the Kubernetes workloads and indicates what type and name of workload resource (deployment, daemonSet, replicaSet, and so on) it is.

• kubernetes.workload.type: Indicates what type of workload resource (deployment, daemonSet, replicaSet, and so on) it is.

The availability of these labels also simplifies Groupings. You do not need different groupings for each type of deployment, instead, you have a single grouping for workloads.

The labels allow you to segment metrics, such as cpu.used.percent , by kubernetes.workload.name to see CPU usage for all the workloads, instead of having a separate query for segmenting by kubernetes.deployment.name, kubernetes.replicaSet.name , and so on.

Learn More

• Groupings Editor

Scopes

A scope is a collection of labels that are used to filter out or define the boundaries of a group of data points when creating dashboards, dashboard panels, alerts, and teams. An example scope is shown below:

In the example above, the scope is defined by two labels with operators and values defined. The table below defines each of the available operators.

The scope editor provides dynamic filtering capabilities. It restricts the scope of the selection for subsequent filters by rendering valid values that are specific to the previously selected label. Expand the list to view unfiltered suggestions. At run time, users can also supply custom values to achieve more granular filtering. The custom values are preserved. Note that changing a label higher up in the hierarchy might render the subsequent labels incompatible. For example, changing the kubernetes.namespace.name > kubernetes.deployment.name hierarchy to swarm.service.name > kubernetes.deployment.name is invalid as these entities belong to different orchestrators and cannot be logically grouped.

Dashboards and Panels

Dashboard scopes define the criteria for what metric data will be included in the dashboard’s panels. The current dashboard’s scope can be seen at the top of the dashboard:

By default, all dashboard panels abide by the scope of the overall dashboard. However, an individual panel scope can be configured for a different scope than the rest of the dashboard.

For more information on Dashboards and Panels, refer to the Dashboards documentation.

Alerts

Alert scopes are defined during the creation process, and specify what areas within the infrastructure the alert is applicable for. In the example alerts below, the first alert has a scope defined, whereas the second alert does not have a custom scope defined. If no scope is defined, the alert is applicable to the entire infrastructure.

For more information on Alerts, refer to the Alerts documentation.

Teams

A team’s scope determines the highest level of data that team members have visibility for:

• If a team’s scope is set to Host, team members can see all host-level and container-level information.

• If a team’s scope is set to Container, team members can only see container-level information.

For more information on teams and configuring team scope, refer to the Manage Teams and Roles documentation.

Segments

Aggregated data can be split into smaller sections by segmenting the data with labels. This allows for the creation of multi-series comparisons and multiple alerts. In the first image, the metric is not segmented:

In the second image, the same metric has been segmented by container.id:

Line and Area panels can display up to five different segments for any given metric. The example image below displays the net.byte.in metric segmented by both container.id and net.http.url:

For more information regarding segmentation in dashboard panels, refer to the Configure Panels documentation. For more information regarding configuring alerts, refer to the Alerts documentation.

The Meaning of n/a

Sysdig Monitor imports data related to entities such as hosts, containers, processes, and so on, and reports them in tables or panels on the Explore and Dashboards UI, as well as in events, so across the UI you see varieties of data. The term n/a can appear anywhere on the UI where some form of data is displayed.

n/a is a term that indicates data that is not available or that it does not apply to a particular instance. In Sysdig parlance, the term signifies one or more entities defined by a particular label, such as hostname or Kubernetes service, for which the label is invalid. In other words, n/a collectively represent entities whose metadata is not relevant to aggregation and filtering techniques—Grouping, Scoping, and Segmenting. For instance, a list of Kubernetes services might display the list of all the services as well as n/a that includes all the containers without the metadata describing a Kubernetes service.

You might encounter n/a sporadically in Explore UI as well as in drill-down panels or dashboards, events, and likely elsewhere on the Sysdig Monitor UI when no relevant metadata is available for that particular display. How n/a should be treated depends on the nature of your deployment. The deployment will not be affected by the entities marked n/a.

The following are some of the cases that yield n/a on the UI:

• Labels are partially available or not available. For example, a host has entities that are not associated with a monitored Kubernetes deployment, or a monitored host has an unmonitored Kubernetes deployment running.

• Labels that do not apply to the grouping criteria or at the hierarchy level. For example:

  • Containers that are not managed by Kubernetes. The containers managed by Kubernetes are identified with their  container.name labels.

  • In certain groupings by DaemonSet, Deployments render N/A and vice versa. Not all containers belong to both DaemonSet and Deployment objects concurrently. Likewise, a Kubernetes ReplicaSet grouping with the  kubernetes.replicaset.name label will not show StatefulSets.

  • In a kubernetes.cluster.name > kubernetes.namespace.name > kubernetes.deployment.name  grouping, the entities without the kubernetes.cluster.name label yield n/a.

• Entities are incorrectly labeled in the infrastructure.

• Kubernetes features that are yet to be in sync with Sysdig Monitoring.

• The format is not applicable to a particular record in the database.

4.2 -

Understanding Default, Custom, and Missing Metrics

Default Metrics

Default metrics include various kinds of metadata which Sysdig Monitor automatically knows how to label, segment, and display.

For example:

• System metrics for hosts, containers, and processes (CPU used, etc.)

• Orchestrator metrics (collected from Kubernetes, Mesos, etc.)

• Network metrics (e.g. network traffic)

• HTTP

• Platform metrics (in some cases)

Default metrics are collected mainly from two sources: syscalls and Kubernetes.

Custom Metrics

About Custom Metrics

Custom metrics generally refer to any metrics that the Sysdig Agent collects from some third-party integration. The type of infrastructure and applications integrated determine the custom metrics that the Agent collects and reports to Sysdig Monitor. The supported custom metrics are:

• Prometheus

• JMX

• StatsD

• App Checks

Each metric comes with a set of custom labels, and additional labels can be user-created. Sysdig Monitor simply collects and reports them with minimal or no internal processing. The limit currently enforced is 3000 metrics per host. Use the metrics_filter option in the dragent.yaml file to remove unwanted metrics or to choose the metrics to report when hosts exceed this limit. For more information on editing the dragent.yaml file, see Understanding the Agent Config Files.

Unit for Custom Metrics

Sysdig Monitor detects the default unit of custom metrics automatically with the delimiter suffix in the metrics name. For example, custom_expvar_time_seconds results in a base unit set to seconds. The supported base units are byte, percent, and time. Custom metrics name should carry one of the following delimiter suffixes in order for Sysdig Monitor to identify and configure the accurate unit type.

• second

• seconds

• byte

• bytes

• total (represents accumulating count)

• percent

Custom metrics will not be auto-detected and the unit will be incorrect unless this naming convention is followed. For instance, custom_byte_expvar will not yield the correct unit, that is MiB.

Editing the Unit Scale

You have the flexibility to change the unit scale either by editing the panel on the Dashboard or in the Explore.

Explore

From the Search Metrics and Dashboard drop-down, select the custom metrics you want to edit the unit selection for, then click More Options. Select the desired unit scale from the Metric Format drop-down and click Save.

Dashboard

Select the Dashboard Panel associated with the custom metrics you want to modify. Select the desired unit scale from the Metrics drop-down and click Save.

Display Missing Data

Data can be missing for a few different reasons:

• Problems such as faulty network connectivity in the communication channel between your infrastructure and Sysdig metrics store.

• Metrics or StatsD batch jobs are submitted sporadically.

Sysdig Monitor allows you to configure the behavior of missing data in Dashboards. Though metric type determines the default behavior, you can configure how to visualize missing data and define it at the per-query level. Use the No Data Display drop-down in the Options menu in the panel configuration. See Create a New Panel for more information.

Consider the following guidelines:

• The No Data Display drop-down has only two options for the Stacked Area timechart: gap and show as zero.

• For the Number panel, the No Data Display option allows entering a custom no data text.

• For form-based timechart panels, the default option for a metrics selection that does not contain a StatsD metric is gap.

• Adding a StatsD metric to a query in a form-based timechart panel will default the selected No Data Display type to the show as zero , which is the default option for form-based StatsD metrics. You can change this selection to any other type.

• The default display option is gap for PromQL Timechart panels.

The options for No Data Display are:

• gap: The default option for form-based timechart panel, where a query metrics selection does not contain a StatsD metric. gap is the best visualization type for most use cases because it is easy to spot indicating a problem.

  

• show as zero: The best option for StatsD metrics which are only submitted sporadically. For example, batch jobs and count of errors. This is the default display option for StatsD metrics in form-based panels.

  

  We do not recommend this option as setting zero could be misleading. For example, this setting will report the value for free disk space as 0% when the disk or host disappears, but in reality, the value is unknown.

  Prometheus best practices recommend avoiding missing metrics.

• connect - solid: Use for measuring the value of a metric, typically a gauge, where you want to visualize the missing samples flattened.

  

  The leftmost and rightmost visible data points can be connected as Sysdig does not perform the interpolation.

• connect - dotted: Use it for measuring the value of a metric, typically a gauge, where you want to visualize the missing samples flattened.

  

  The leftmost and rightmost visible data points can be connected as Sysdig does not perform the interpolation.

4.3 -

Prometheus Metrics Types

Sysdig Monitor transforms Prometheus metrics into usable, actionable entries in two ways:

Calculated Metrics

The Prometheus metrics that are scraped by the Sysdig agent and transformed into the traditional StatsD model are called calculated metrics. In calculated metrics, the delta is stored with the previous value. This delta is what Sysdig uses on the classic backend for metrics analyzing and visualization. While generating the calculated metrics, the gauge metrics are kept as they are, but the counter metrics are transformed.

Prometheus calculated metrics cannot be used in PromQL.

The Histogram and Summary metrics are transformed into a different format called Prometheus histogram and summary metrics respectively. The transformations include:

• All of the quantiles are transformed into a different metric, with the quantile added as a suffix.

• The count and sum of these summary metrics are exposed as different metrics with names slightly changed. _ (underscore) in the name is replaced with a period .. For more information, see Mapping Between Classic Metrics and PromQL Metrics.

Raw Metrics

In Sysdig parlance, the Prometheus metrics that are scraped (by the Sysdig agent), collected, sent, stored, visualized, and presented exactly as Prometheus exposes them are called raw metrics. Raw metrics are used with PromQL.

Sysdig counter is a StatsD type counter, where the difference in value is kept, but not the raw value of the counter, whereas Prometheus raw metrics are counters that are always monotonically increasing. A rate function needs to be applied on Prometheus raw metrics to make sense of it.

Time Aggregations Over Prometheus Metrics

The following time aggregations are supported for both the metric types:

• Average: Returns an average of a set of data points, keeping all the labels.

• Maximum and Minimum: Returns a maximal or minimal value, keeping all the labels.

• Sum: Returns a sum of the values of data points, keeping all the labels.

• Rate (timeAvg): Returns a sum of changes to the counter across data points in a given time period and divides by time, keeping all the labels as they are. For Prometheus raw metrics, timeAvg is calculated by taking the difference and dividing it by time.

Prometheus Calculated Metrics

Prometheus calculated metrics are treated as gauges by Sysdig, and there the following time aggregations are available:

• Average

• Sum

• Minimum

• Maximum

Rate (timeAvg) is not available because they are not applicable to gauge metrics.

Prometheus Raw Metrics

For the gauge type, the following types are available:

• Average

• Minimum

• Maximum

For the counter type, the following types are available:

• Rate: Calculates the first derivative of the counter (change over time).

• Sum: Calculates a complete change of the counter over a period of time.

4.4 -

Heuristic and Deprecated Metrics

Heuristic Metrics

Various network-related metrics reported by Sysdig, including response times, are calculated at the kernel level by measuring latency between systems calls. In an effort to ensure Sysdig remains the trusted source of infrastructure insights, moving forward we will be labeling some network related metrics as heuristic and are tagging with the symbol in the application.

Additional heuristic metric details are listed below:

Deprecated Metrics:

Based on low usage patterns, Sysdig has decided to deprecate the following metrics on August 1, 2018. Users will continue to have the ability to collect similar data using Prometheus, or another method of code instrumentation (i.e. StatsD or JMX for Java applications).

The table below shows the current metrics and options for similar functionality.

4.5 -

Manage Metric Scale

Sysdig provides several knobs for managing metric scale.

There are three primary ways in which you could include/exclude metrics, should you encounter unwanted metrics limits.

1. Include/exclude custom metrics by name filters.

   See Include/Exclude Custom Metrics.

2. Include/exclude metrics emitted by certain containers, Kubernetes annotations, or any other container label at collection time.

   See Prioritize/Include/Exclude Designated Containers.

3. Exclude metrics from unwanted ports.

   See Blacklist Ports.

4.6 -

Data Aggregation

Sysdig Monitor allows users to adjust the aggregation settings when graphing or creating alerts for a metric, informing how Sysdig rolls up the available data samples in order to create the chart or evaluate the alert. There are two forms of aggregation used for metrics in Sysdig: time aggregation and group aggregation.

Time Aggregation

Time aggregation comes into effect in two overlapping situations:

• Charts can only render a limited number of data points. To look at a wide range of data, Sysdig Monitor may need to aggregate granular data into larger samples for visualization.

• Sysdig Monitor rolls up historical data over time.

  Sysdig retains rollups based on each aggregation type, to allow users to choose which data points to utilize when evaluating older data.

Sysdig agents collect 1-second samples and report data at 10-second resolution. The data is stored and reported every 10-second with the available aggregations (average, rate, min, max, sum) to make them available via the Sysdig Monitor UI and the API. For time series charts covering five minutes or less, data points are drawn at this 10-second resolution, and any time aggregation selections will have no effect. When an amount of time greater than five minutes is displayed, data points are drawn as an aggregate for an appropriate time interval. For example, for a chart covering one hour, each data point would reflect a one minute interval.

At time intervals of one minute and above, charts can be configured to display different aggregates for the 10-second metrics used to calculate each datapoint.

In the example images below, the kubernetes.deployment.replicas.available metrics first uses the average for time aggregation:

Then uses the sum for time aggregation:

• Rate and average are very similar and often provide the same result. However, the calculation of each is different.

  • If time aggregation is set to one minute, the agent is supposed to retrieve six samples (one every 10 seconds).

  • In some cases, samples may not be there, due to disconnections or other circumstances. For this example, four samples are available. If this was the case, the average would be calculated by dividing by four, while the rate would be calculated by dividing by six.

• Most metrics are sampled once for each time interval, resulting in average and rate returning the same value. However, there will be a distinction for any metrics not reported at every time interval. For example, some custom statsd metrics.

• Rate is currently referred to as timeAvg in the Sysdig Monitor API and advanced alerting language.

• By default, average is used when displaying data points for a time interval.

Group Aggregation

Metrics applied to a group of items (for example, several containers, hosts, or nodes) are averaged between the members of the group by default. For example, three hosts report different CPU usage for one sample interval. The three values will be averaged, and reported on the chart as a single datapoint for that metric.

There are several different types of group aggregation:

If a chart or alert is segmented, the group aggregation settings will be utilized for both aggregations across the whole group, and aggregation within each individual segmentation.

For example, the image below shows a chart for CPU% across the infrastructure:

When segmented by proc.name, the chart shows one CPU% line for each process:

Each line provides the average value for every process with the same name. To see the difference, change the group aggregation type to sum:

The metric aggregation value showed beside the metric name is for the time aggregation. While the screenshot shows AVG, the group aggregation is set to SUM.

Aggregation Examples

The tables below provide an example of how each type of aggregation works. The first table provides the metric data, while the second displays the resulting value for each type of aggregation.

In the example below, the CPU% metric is applied to a group of servers called webserver. The first chart shows metrics using average aggregation for both time and group. The second chart shows the metrics using maximum aggregation for both time and group.

For each one minute interval, the second chart renders the highest CPU usage value found from the servers in the webserver group and from all of the samples reported during the one minute interval. This view can be useful when searching for transient spikes in metrics over long periods of time, that would otherwise be missed with average aggregation.

4.7 -

Metric Limits

Sysdig ensures that you see the most relevant metric information relevant to your monitored environment. To achieve this, limits are enforced on the number of metrics that the datastore can store. Different limits apply to different metric types and agent versions.

Enterprise

The metric limits are automatically set by the Sysdig backend components based on your plan, agent version, and backend configuration. The default limits are provided below:

The custom metrics limit of 10,000 does not include the agent metrics that are provided out-of-the-box, such as host, container, and Kube State Metrics.

View Metric Limits

Use the Sysdig Agent Health & Status dashboard under Host Infrastructure templates to view current usage per host for each metric type.

The metric limits are exposed to the UI through the following agent metrics.

Learn More

• Subscription

• Configure Agent Modes

5 -

Metrics Dictionary

The Sysdig metrics dictionary lists all current default metrics supported by the Sysdig product suite, as well as kube state and cloud provider metrics. The Metrics Dictionary is a living document and is updated as new metrics are added to the product.

Overview

Each metric in the dictionary has several pieces of metadata listed to provide greater context for how the metric can be used within Sysdig products. An example layout is displayed below:

Metric Name

Metric definition. For some metrics, the equation for how the value is determined is provided.

5.1 -

Agent

dragent.analyzer

dragent is the main process in the agent that collects and collates data from multiple sources, including syscall events from the kernel in order to generate metrics. The analyzer module that runs in the dragent process does much of the work involved in generating metrics. These internal metrics are used to troubleshoot the health of the analyzer component.

Sysdig Monitor provides the following analyzer metrics:

5.2 -

Metrics and Label Mapping

This topic outlines the metrics and label mapping between the Sysdig classic store and the new Sysdig datastore.

• Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

• Mapping Between Classic Metrics and PromQL Metrics

• Run PromQL Queries Faster with Extended Label Set

5.2.1 -

Mapping Between Classic Metrics and PromQL Metrics

Starting SaaS v 3.2.6, Sysdig classic metrics and labels have been renamed to be aligned with Prometheus naming convention. For example, Sysdig classic metrics have a dot-oriented hierarchy, whereas Prometheus has label-based metric organization. The table below helps you identify the Prometheus metrics and labels and the corresponding ones in the Sysdig classic system.

5.2.2 -

Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics

Prometheus metrics, in Kubernetes parlance, are nothing but Kube State Metrics. These metrics are available in Sysdig PromQL and can be mapped to existing Sysdig Kubernetes metrics.

For descriptions on Kubernetes State Metrics, see Kubernetes State Metrics.

"# HELP kube_node_status_capacity The capacity for different resources of a node.
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-master"",resource=""memory"",unit=""byte""} 4.16342016e+09
kube_node_status_capacity{node=""k8s-master"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node1"",resource=""cpu"",unit=""core""} 2
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node1"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_1Gi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""hugepages_2Mi"",unit=""byte""} 0
kube_node_status_capacity{node=""k8s-node2"",resource=""memory"",unit=""byte""} 6.274154496e+09
kube_node_status_capacity{node=""k8s-node2"",resource=""pods"",unit=""integer""} 110
kube_node_status_capacity{node=""k8s-node2"",resource=""cpu"",unit=""core""} 2

5.2.3 -

Run PromQL Queries Faster with Extended Label Set

Sysdig allows you to run PromQL queries smoother and faster with the extended label set. The extended label set is created by augmenting the incoming data with the rich metadata associated with your infrastructure and making it available in PromQL.

With this, you can troubleshoot a problem or building Dashboards and Alerts without the need to write complex queries. Sysdig automatically enriches your metrics with Kubernetes and application context without the need to instrument additional labels in your environment. This reduces operational complexity and cost—the enrichment takes place in Sysdig metric ingestion pipeline after time series have been sent to the backend.

Calculate Memory Usage by Deployment in a Cluster

Using the vector matching operation, you could run the following query and calculate the memory usage by deployment in a cluster:

sum by(cluster,namespace,owner_name) ((sysdig_container_memory_used_bytes * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~".+",namespace=~".+"})

To get the result, you need to write a query to perform a join (vector match) of various metrics, usually in the following order:

• Grab a metric you need that is defined on a container level. For example, a Prometheus metric or some of the Sysdig provided metrics, such as sysdig_container_memory_used_byte.

• Perform a vector match on container ID with the metric kube_pod_container_info to get the pod metadata.

• Perform a vector match on the pod, namespace, and cluster with the kube_pod_owner metric.

In the case of Sysdig’s extended label set for PromQL, all the metrics inherit the metadata, so that necessary container, host, and Kubernetes metadata are set on all the metrics. This simplifies the query so you can build and run it quickly.

Likewise, the above query can be simplified as follows:

sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name) (sysdig_container_memory_used_bytes{kube_cluster_name!="",kube_namespace_name!="",kube_deployment_name!=""})

The advantages of using a simplified query are:

• Complex vector matching operations (the group_left and group_right operators) are no longer required. All the labels are already available on each of the metrics, and therefore, any filtering can be performed directly on the metric itself.

• The metrics now will have a huge amount of labels. You can use PromQL Explorer to deal with this rich metadata.

• The metadata is distinguishable from user-defined labels. For example, Kubernetes metadata labels start with kube_. For instance, cluster is replaced with kube_cluster_name.

• Create a dashboard panel or an alert from the PromQL query you run in the PromQL Query Explore.

• Filter data by applying the comparison operators on the label values given in the table.

Examples for Simplifying Queries

Given below are some of the examples of using the extended label set to simplify complex query operations.

Memory Usage in a Kubernetes Cluster

Query with core label set:

avg by (agent_tag_cluster) ((sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * on(host,agent_tag_cluster) sysdig_host_info{agent_tag_cluster=~".+"}) * 100

Query with the extended label set:

avg by (agent_tag_cluster) (sysdig_host_memory_used_bytes/sysdig_host_memory_total_bytes) * 100

CPU Usage in Containers

Query with the core label set:

sum by (cluster,namespace)(sysdig_container_cpu_cores_used * on (container_id) group_left(cluster,pod,namespace) kube_pod_container_info{cluster=~".+"})

Simplified query with the extended label set:

sum by (kube_cluster_name,kube_namespace_name)(sysdig_container_cpu_cores_used{kube_cluster_name=~".+"})

Memory Usage in Daemonset

Query with the core label set:

sum by(cluster,namespace,owner_name) (sum by(pod) (label_replace(sysdig_container_memory_used_bytes * on(container_id,host_mac) group_left(label_io_kubernetes_pod_namespace,label_io_kubernetes_pod_name,label_io_kubernetes_container_name) sysdig_container_info{label_io_kubernetes_pod_namespace=~".*",cluster=~".*"},"pod","$1","label_io_kubernetes_pod_name","(.*)"))  * on(pod) group_right sum by(cluster,namespace,owner_name,pod) (kube_pod_owner{owner_kind=~"DaemonSet",owner_name=~".*",cluster=~".*",namespace=~".*"}))

Simplified query with the extended label set:

sum by(kube_cluster_name,kube_namespace_name,kube_daemonset_name) (sysdig_container_memory_used_bytes{kube_daemonset_name=~".*",kube_cluster_name=~".*",kube_namespace_name=~".*"})

Pod Restarts in a Kubernetes Cluster

Query with the core label set:

sum by(cluster,namespace,owner_name)(changes(kube_pod_status_ready{condition="true",cluster=~$cluster,namespace=~$namespace}[$__interval]) * on(cluster,namespace,pod) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~".+",cluster=~$cluster,namespace=~$namespace})

Simplified query with the extended label set:

sum by (kube_cluster_name,kube_namespace_name,kube_deployment_name)(changes(kube_pod_status_ready{condition="true",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_deployment_name=~".+"}[$__interval]))

Containers per Image

Query with the core label set:

count by (owner_name,image,cluster,namespace)((sysdig_container_info{cluster=~$cluster,namespace=~$namespace})  * on(pod,namespace,cluster) group_left(owner_name) max by (pod,namespace,cluster,owner_name)(kube_pod_owner{owner_kind="Deployment",owner_name=~".+"}))

Simplified query with the extended label set:

count by (kube_deployment_name,image,kube_cluster_name,kube_namespace_name)(sysdig_container_info{kube_deployment_name=~".+",kube_cluster_name=~$cluster,kube_namespace_name=~$namespace})

Average TCP Queue per Node

Query with the core label set:

avg by (agent_tag_cluster,host)( sysdig_host_net_tcp_queue_len * on (host_mac) group_left(agent_tag_cluster,host) sysdig_host_info{agent_tag_cluster=~$cluster,host=~".+"})

Simplified query with the extended label set:

avg by (agent_tag_cluster,host_hostname) (sysdig_host_net_tcp_queue_len{agent_tag_cluster =~ $cluster})

5.3 -

Applications

The metrics in this section are collected from either default or customized agent configurations for integrated applications. See also: Integrate Applications (Default App Checks).

Contents

• Apache Metrics

• Apache Kafka Metrics

• Consul Metrics

• Couchbase Metrics

• Elasticsearch Metrics

• etcd Metrics

• fluentd Metrics

• Go Metrics

• HAProxy Metrics

• HTTP Metrics

• Jenkins Metrics

• Lighttpd Metrics

• Memcached Metrics

• Mesos/Marathon Metrics

• MongoDB Metrics

• MySQL Metrics

• NGINX and NGINX Plus Metrics

• NTP Metrics

• PGBouncer Metrics

• PHP-FPM Metrics

• PostgreSQL Metrics

• RabbitMQ Metrics

• RedisDB Metrics

• Supervisord Metrics

• TCP Metrics

• Varnish Metrics

5.3.1 -

Apache Metrics

See also: Apache integration information.

apache.conns_async_closing

The number of asynchronous closing connections.

apache.conns_async_keep_alive

The number of asynchronous keep-alive connections.

apache.conns_async_writing

The number of asynchronous write connections.

apache.conns_total

The total number of connections handled.

apache.net.bytes

The total number of bytes served.

apache.net.bytes_per_s

The number of bytes served per second.

apache.net.hits

The total number of requests performed.

apache.net.request_per_s

The number of requests performed per second.

apache.performance.busy_workers

The number of workers currently serving requests.

apache.performance.cpu_load

The percentage of CPU used.

apache.performance.idle_workers

The number of idle workers in the instance.

apache.performance.uptime

The amount of time the server has been running in seconds.

5.3.2 -

Apache Kafka Metrics

Contents

• Apache Kafka Consumer Metrics

• Apache Kafka JMX Metrics

5.3.2.1 -

Apache Kafka Consumer Metrics

See also: Apache Kafka integration information.

kafka.broker_offset

The current message offset value on the broker.

kafka.consumer_lag

The lag in messages between the consumer and the broker.

kafka.consumer_offset

The current message offset value on the consumer.

5.3.2.2 -

Apache Kafka JMX Metrics

See also: Apache Kafka integration information.

The kafka.consumer.* and kafka.producer.* metrics are only available with JMX customization as documented in Integrate JMX Metrics from Java Virtual Machines.

kafka.consumer.bytes_consumed

The average number of bytes consumed for a specific topic per second.

kafka.consumer.bytes_in

The rate of bytes coming in to the consumer.

kafka.consumer.delayed_requests

The number of delayed consumer requests.

kafka.consumer.expires_per_second

The rate of delayed consumer request expiration.

kafka.consumer.fetch_rate

The minimum rate at which the consumer sends fetch requests to a broker.

kafka.consumer.fetch_size_avg

The average number of bytes fetched for a specific topic per request.

kafka.consumer.fetch_size_max

The maximum number of bytes fetched for a specific topic per request.

kafka.consumer.kafka_commits

The rate of offset commits to Kafka.

kafka.consumer.max_lag

The maximum consumer lag.

kafka.consumer.messages_in

The rate of consumer message consumption.

kafka.consumer.records_consumed

The average number of records consumed per second for a specific topic.

kafka.consumer.records_per_request_avg

The average number of records in each request for a specific topic.

kafka.consumer.zookeeper_commits

The rate of offset commits to ZooKeeper.

kafka.expires_sec

The rate of delayed producer request expiration.

kafka.follower.expires_per_second

The rate of request expiration on followers.

kafka.log.flush_rate

The log flush rate.

kafka.messages_in

The incoming message rate.

kafka.net.bytes_in

The incoming byte rate.

kafka.net.bytes_out

The outgoing byte rate.

kafka.net.bytes_rejected

The rejected byte rate.

kafka.producer.available_buffer_bytes

The total amount of buffer memory, including unallocated buffer memory and memory in the free list, that is not being used.

kafka.producer.batch_size_avg

The average number of bytes sent per partition per-request.

kafka.producer.batch_size_max

The maximum number of bytes sent per partition per-request.

kafka.producer.buffer_bytes_total

The maximum amount of buffer memory the client can use.

kafka.producer.bufferpool_wait_time

The fraction of time an appender waits for space allocation.

kafka.producer.bytes_out

The rate of bytes going out for the producer.

kafka.producer.compression_rate

The average compression rate of record batches for a topic.

kafka.producer.compression_rate_avg

The average compression rate of record batches.

kafka.producer.delayed_requests

The number of producer requests delayed.

kafka.producer.expires_per_seconds

The rate of producer request expiration.

kafka.producer.io_wait

The producer I/O wait time.

kafka.producer.message_rate

The producer message rate.

kafka.producer.metadata_age

The age of the current producer metadata being used, in seconds.

kafka.producer.record_error_rate

The average number of retried record sends for a topic per second.

kafka.producer.record_queue_time_avg

The average time that record batches spent in the record accumulator, in milliseconds.

kafka.producer.record_queue_time_max

The maximum amount of time record batches can spend in the record accumulator, in milliseconds.

kafka.producer.record_retry_rate

The average number of retried record sends for a topic per second.

kafka.producer.record_send_rate

The average number of records sent per second for a topic.

kafka.producer.record_size_avg

The average record size.

kafka.producer.record_size_max

The maximum record size.

kafka.producer.records_per_request

The average number of records sent per second.

kafka.producer.request_latency_avg

The average request latency of the producer.

kafka.producer.request_latency_max

The maximum request latency in milliseconds.

kafka.producer.request_rate

The number of producer requests per second.

kafka.producer.requests_in_flight

The current number of in-flight requests awaiting a response

kafka.producer.response_rate

The number of producer responses per second.

kafka.producer.throttle_time_avg

The average time in a request was throttled by a broker, in milliseconds.

kafka.producer.throttle_time_max

The maximum time in a request was throttled by a broker, in milliseconds.

kafka.producer.waiting_threads

The number of user threads blocked waiting for buffer memory to enqueue their records.

kafka.replication.isr_expands

The rate of replicas joining the ISR pool.

kafka.replication.isr_shrinks

The rate of replicas leaving the ISR pool.

kafka.replication.leader_elections

The leader election rate.

kafka.replication.unclean_leader_elections

The unclean leader election rate.

kafka.replication.under_replicated_partitions

The number of unreplicated partitions.

kafka.request.fetch.failed

The number of client fetch request failures.

kafka.request.fetch.failed_per_second

The rate of client fetch request failures per second.

kafka.request.fetch.time.99percentile

The time for fetch requests for the 99th percentile.

kafka.request.fetch.time.avg

The average time per fetch request.

kafka.request.handler.avg.idle.pct

The average fraction of time the request handler threads are idle.

kafka.request.metadata.time.99percentile

The time for metadata requests for 99th percentile.

kafka.request.metadata.time.avg

The average time for a metadata request.

kafka.request.offsets.time.99percentile

The time for offset requests for the 99th percentile.

kafka.request.offsets.time.avg

The average time for an offset request.

kafka.request.produce.failed

The number of failed produce requests.

kafka.request.produce.failed_per_second

The rate of failed produce requests per second.

kafka.request.produce.time.99percentile

The time for produce requests for the 99th percentile.

kafka.request.produce.time.avg

The average time for a produce request.

kafka.request.update_metadata.time.99percentile

The time for update metadata requests for the 99th percentile

kafka.request.update_metadata.time.avg

The average time for a request to update metadata.

5.3.3 -

Consul Metrics

Contents

• Base Consul Metrics

• Consul StatsD Metrics

5.3.3.1 -

Base Consul Metrics

For related information, see Consul integration.

consul.catalog.nodes_critical

Number of nodes with service status `critical` from those registered.

consul.catalog.nodes_passing

Number of nodes with service status `passing` from those registered.

consul.catalog.nodes_up

Number of nodes.

consul.catalog.nodes_warning

Number of nodes with service status `warning` from those registered.

consul.catalog.services_critical

Total critical services on nodes.

consul.catalog.services_passing

Total passing services on nodes.

consul.catalog.services_up

Total services registered on nodes.

consul.catalog.services_warning

Total warning services on nodes.

consul.catalog.total_nodes

Number of nodes registered in the consul cluster.

consul.net.node.latency.max

Maximum latency from this node to all others.

consul.net.node.latency.median

Median latency from this node to all others.

consul.net.node.latency.min

Minimum latency from this node to all others.

consul.net.node.latency.p25

p25 latency from this node to all others.

consul.net.node.latency.p75

p75 latency from this node to all others.

consul.net.node.latency.p90

p90 latency from this node to all others.

consul.net.node.latency.p95

p95 latency from this node to all others.

consul.net.node.latency.p99

p99 latency from this node to all others.

consul.peers

Number of peers in the peer set.

5.3.3.2 -

Consul StatsD Metrics

For related information, see Consul integration.

consul.memberlist.msg.suspect

Number of times an agent suspects another as failed while probing during gossip protocol.

consul.raft.apply

Number of raft transactions occurring.

consul.raft.commitTime.95percentile

The p95 time it takes to commit a new entry to the raft log on the leader.

consul.raft.commitTime.avg

The average time it takes to commit a new entry to the raft log on the leader.

consul.raft.commitTime.count

The number of samples of raft.commitTime

consul.raft.commitTime.max

The max time it takes to commit a new entry to the raft log on the leader.

consul.raft.commitTime.median

The median time it takes to commit a new entry to the raft log on the leader.

consul.raft.leader.dispatchLog.95percentile

The p95 time it takes for the leader to write log entries to disk.

consul.raft.leader.dispatchLog.avg

The average time it takes for the leader to write log entries to disk.

consul.raft.leader.dispatchLog.count

The number of samples of raft.leader.dispatchLog.

consul.raft.leader.dispatchLog.max

The max time it takes for the leader to write log entries to disk.

consul.raft.leader.dispatchLog.median

The median time it takes for the leader to write log entries to disk.

consul.raft.leader.lastContact.95percentile

P95 time elapsed since the leader was last able to check its lease with followers.

consul.raft.leader.lastContact.avg

Average time elapsed since the leader was last able to check its lease with followers.

consul.raft.leader.lastContact.count

The number of samples of raft.leader.lastContact.

consul.raft.leader.lastContact.max

Max time elapsed since the leader was last able to check its lease with followers.

consul.raft.leader.lastContact.median

Median time elapsed since the leader was last able to check its lease with followers.

consul.raft.state.candidate

The number of initiated leader elections.

consul.raft.state.leader

Number of completed leader elections.

consul.runtime.alloc_bytes

Current bytes allocated by the Consul process.

consul.runtime.free_count

Cumulative count of heap objects freed.

consul.runtime.heap_objects

Number of objects allocated on the heap.

consul.runtime.malloc_count

Cumulative count of heap objects allocated.

consul.runtime.num_goroutines

Number of running goroutines.

consul.runtime.sys_bytes

Total size of the virtual address space reserved by the Go runtime.

consul.runtime.total_gc_pause_ns

Cumulative nanoseconds in GC stop-the-world pauses since Consul started.

consul.runtime.total_gc_runs

Number of completed GC cycles.

consul.serf.events

Incremented when an agent processes a serf event.

consul.serf.member.flap

Number of times an agent is marked dead and then quickly recovers.

consul.serf.member.join

Incremented when an agent processes a join event.

5.3.4 -

Couchbase Metrics

See also: Couchbase integration information.

couchbase.by_bucket.avg_bg_wait_time

The average background wait time.

couchbase.by_bucket.avg_disk_commit_time

The average disk commit time.

couchbase.by_bucket.avg_disk_update_time

The average disk update time.

couchbase.by_bucket.bg_wait_total

The total background wait time.

couchbase.by_bucket.bytes_read

The number of bytes read.

couchbase.by_bucket.bytes_written

The number of bytes written.

couchbase.by_bucket.cas_badval

The number of compare and swap bad values.

couchbase.by_bucket.cas_hits

The number of compare and swap hits.

couchbase.by_bucket.cas_misses

The number of compare and swap misses.

couchbase.by_bucket.cmd_get

The number of compare and swap gets.

couchbase.by_bucket.cmd_set

The number of compare and swap sets.

couchbase.by_bucket.couch_docs_actual_disk_size

The size of the couchbase docs on disk.

couchbase.by_bucket.couch_docs_data_size

The data size of the couchbase docs.

couchbase.by_bucket.couch_docs_disk_size

Couch docs total size in bytes.

couchbase.by_bucket.couch_docs_fragmentation

The percentage of couchbase docs fragmentation.

couchbase.by_bucket.couch_spatial_data_size

The size of object data for spatial views.

couchbase.by_bucket.couch_spatial_disk_size

The amount of disk space occupied by spatial views.

couchbase.by_bucket.couch_spatial_ops

Spatial operations.

couchbase.by_bucket.couch_total_disk_size

The total disk size for couchbase.

couchbase.by_bucket.couch_views_data_size

The size of object data for views.

couchbase.by_bucket.couch_views_disk_size

The amount of disk space occupied by views.

couchbase.by_bucket.couch_views_fragmentation

The view fragmentation.

couchbase.by_bucket.couch_views_ops

View operations.

couchbase.by_bucket.cpu_idle_ms

CPU idle milliseconds.

couchbase.by_bucket.cpu_utilization_rate

CPU utilization percentage.

couchbase.by_bucket.curr_connections

Current bucket connections.

couchbase.by_bucket.curr_items

Number of active items in memory.

couchbase.by_bucket.curr_items_tot

Total number of items.

couchbase.by_bucket.decr_hits

Decrement hits.

couchbase.by_bucket.decr_misses

Decrement misses.

couchbase.by_bucket.delete_hits

Delete hits.

couchbase.by_bucket.delete_misses

Delete misses.

couchbase.by_bucket.disk_commit_count

Disk commits.

couchbase.by_bucket.disk_update_count

Disk updates.

couchbase.by_bucket.disk_write_queue

Disk write queue depth.

couchbase.by_bucket.ep_bg_fetched

Disk reads per second.

couchbase.by_bucket.ep_cache_miss_rate

Cache miss rate.

couchbase.by_bucket.ep_cache_miss_ratio

Cache miss ratio.

couchbase.by_bucket.ep_dcp_2i_backoff

Number of backoffs for indexes DCP connections.

couchbase.by_bucket.ep_dcp_2i_count

Number of indexes DCP connections.

couchbase.by_bucket.ep_dcp_2i_items_remaining

Number of indexes items remaining to be sent.

couchbase.by_bucket.ep_dcp_2i_items_sent

Number of indexes items sent.

couchbase.by_bucket.ep_dcp_2i_producer_count

Number of indexes producers

couchbase.by_bucket.ep_dcp_2i_total_bytes

Number bytes per second being sent for indexes DCP connections.

couchbase.by_bucket.ep_dcp_fts_backoff

Number of backoffs for fts DCP connections.

couchbase.by_bucket.ep_dcp_fts_count

Number of fts DCP connections.

couchbase.by_bucket.ep_dcp_fts_items_remaining

Number of fts items remaining to be sent.

couchbase.by_bucket.ep_dcp_fts_items_sent

Number of fts items sent.

couchbase.by_bucket.ep_dcp_fts_producer_count

Number of fts producers.

couchbase.by_bucket.ep_dcp_fts_total_bytes

Number bytes per second being sent for fts DCP connections.

couchbase.by_bucket.ep_dcp_other_backoff

Number of backoffs for other DCP connections.

couchbase.by_bucket.ep_dcp_other_count

Number of other DCP connections.

couchbase.by_bucket.ep_dcp_other_items_remaining

Number of other items remaining to be sent.

couchbase.by_bucket.ep_dcp_other_items_sent

Number of other items sent.

couchbase.by_bucket.ep_dcp_other_producer_count

Number of other producers.

couchbase.by_bucket.ep_dcp_other_total_bytes

Number bytes per second being sent for other DCP connections.

couchbase.by_bucket.ep_dcp_replica_backoff

Number of backoffs for replica DCP connections.

couchbase.by_bucket.ep_dcp_replica_count

Number of replica DCP connections.

couchbase.by_bucket.ep_dcp_replica_items_remaining

Number of replica items remaining to be sent.

couchbase.by_bucket.ep_dcp_replica_items_sent

Number of replica items sent.

couchbase.by_bucket.ep_dcp_replica_producer_count

Number of replica producers.

couchbase.by_bucket.ep_dcp_replica_total_bytes

Number bytes per second being sent for replica DCP connections.

couchbase.by_bucket.ep_dcp_views_backoff

Number of backoffs for views DCP connections.

couchbase.by_bucket.ep_dcp_views_count

Number of views DCP connections.

couchbase.by_bucket.ep_dcp_views_items_remaining

Number of views items remaining to be sent.

couchbase.by_bucket.ep_dcp_views_items_sent

Number of views items sent.

couchbase.by_bucket.ep_dcp_views_producer_count

Number of views producers.

couchbase.by_bucket.ep_dcp_views_total_bytes

Number bytes per second being sent for views DCP connections.

couchbase.by_bucket.ep_dcp_xdcr_backoff

Number of backoffs for xdcr DCP connections.

couchbase.by_bucket.ep_dcp_xdcr_count

Number of xdcr DCP connections.

couchbase.by_bucket.ep_dcp_xdcr_items_remaining

Number of xdcr items remaining to be sent.

couchbase.by_bucket.ep_dcp_xdcr_items_sent

Number of xdcr items sent.

couchbase.by_bucket.ep_dcp_xdcr_producer_count

Number of xdcr producers.

couchbase.by_bucket.ep_dcp_xdcr_total_bytes

Number bytes per second being sent for xdcr DCP connections.

couchbase.by_bucket.ep_diskqueue_drain

Total Drained items on disk queue.

couchbase.by_bucket.ep_diskqueue_fill

Total enqueued items on disk queue.

couchbase.by_bucket.ep_diskqueue_items

Total number of items waiting to be written to disk.

couchbase.by_bucket.ep_flusher_todo

Number of items currently being written.

couchbase.by_bucket.ep_item_commit_failed

Number of times a transaction failed to commit due to storage errors.

couchbase.by_bucket.ep_kv_size

Total amount of user data cached in RAM in this bucket.

couchbase.by_bucket.ep_max_size

The maximum amount of memory this bucket can use.

couchbase.by_bucket.ep_mem_high_wat

Memory usage high water mark for auto-evictions.

couchbase.by_bucket.ep_mem_low_wat

Memory usage low water mark for auto-evictions.

couchbase.by_bucket.ep_meta_data_memory

Total amount of item metadata consuming RAM in this bucket.

couchbase.by_bucket.ep_num_non_resident

Number of non-resident items.

couchbase.by_bucket.ep_num_ops_del_meta

Number of delete operations per second for this bucket as the target for XDCR.

couchbase.by_bucket.ep_num_ops_del_ret_meta

Number of delRetMeta operations per second for this bucket as the target for XDCR.

couchbase.by_bucket.ep_num_ops_get_meta

Number of read operations per second for this bucket as the target for XDCR.

couchbase.by_bucket.ep_num_ops_set_meta

Number of set operations per second for this bucket as the target for XDCR.

couchbase.by_bucket.ep_num_ops_set_ret_meta

Number of setRetMeta operations per second for this bucket as the target for XDCR.

couchbase.by_bucket.ep_num_value_ejects

Number of times item values got ejected from memory to disk.\

couchbase.by_bucket.ep_oom_errors

Number of times unrecoverable OOMs happened while processing operations.

couchbase.by_bucket.ep_ops_create

Create operations.

couchbase.by_bucket.ep_ops_update

Update operations.

couchbase.by_bucket.ep_overhead

Extra memory used by transient data like persistence queues or checkpoints.

couchbase.by_bucket.ep_queue_size

Number of items queued for storage.

couchbase.by_bucket.ep_resident_items_rate

Number of resident items.

couchbase.by_bucket.ep_tap_replica_queue_drain

Total drained items in the replica queue.

couchbase.by_bucket.ep_tap_total_queue_drain

Total drained items in the queue.

couchbase.by_bucket.ep_tap_total_queue_fill

Total enqueued items in the queue.

couchbase.by_bucket.ep_tap_total_total_backlog_size

Number of remaining items for replication.

couchbase.by_bucket.ep_tmp_oom_errors

Number of times recoverable OOMs happened while processing operations.

couchbase.by_bucket.ep_vb_total

Total number of vBuckets for this bucket.

couchbase.by_bucket.evictions

Number of evictions

couchbase.by_bucket.get_hits

Number of get hits

couchbase.by_bucket.get_misses

Number of get misses.

couchbase.by_bucket.hibernated_requests

Number of streaming requests now idle.

couchbase.by_bucket.hibernated_waked

Rate of streaming request wakeups.

couchbase.by_bucket.hit_ratio

Hit ratio.

couchbase.by_bucket.incr_hits

Number of increment hits.

couchbase.by_bucket.incr_misses

Number of increment misses.

couchbase.by_bucket.mem_actual_free

Free memory.

couchbase.by_bucket.mem_actual_used

Used memory.

couchbase.by_bucket.mem_free

Free memory.

couchbase.by_bucket.mem_total

Total available memory.

couchbase.by_bucket.mem_used (deprecated)

Engine’s total memory usage.

couchbase.by_bucket.mem_used_sys

System memory usage.

couchbase.by_bucket.misses

Total number of misses.

couchbase.by_bucket.ops

Total number of operations.

couchbase.by_bucket.page_faults

Number of page faults.

couchbase.by_bucket.replication_docs_rep_queue

couchbase.by_bucket.replication_meta_latency_aggr

couchbase.by_bucket.rest_requests

Number of HTTP requests.

couchbase.by_bucket.swap_total

Total amount of swap available.

couchbase.by_bucket.swap_used

Amount of swap used.

couchbase.by_bucket.vb_active_eject

Number of items per second being ejected to disk from active vBuckets.

couchbase.by_bucket.vb_active_itm_memory

Amount of active user data cached in RAM in this bucket.

couchbase.by_bucket.vb_active_meta_data_memory

Amount of active item metadata consuming RAM in this bucket.

couchbase.by_bucket.vb_active_num

Number of active items.

couchbase.by_bucket.vb_active_num_non_resident

Number of non resident vBuckets in the active state for this bucket.

couchbase.by_bucket.vb_active_ops_create

New items per second being inserted into active vBuckets in this bucket.

couchbase.by_bucket.vb_active_ops_update

Number of items updated on active vBucket per second for this bucket.

couchbase.by_bucket.vb_active_queue_age

Sum of disk queue item age in milliseconds.

couchbase.by_bucket.vb_active_queue_drain

Total drained items in the queue.

couchbase.by_bucket.vb_active_queue_fill

Number of active items per second being put on the active item disk queue.

couchbase.by_bucket.vb_active_queue_size

Number of active items in the queue.

couchbase.by_bucket.vb_active_resident_items_ratio

Number of resident items.

couchbase.by_bucket.vb_avg_active_queue_age

Average age in seconds of active items in the active item queue.

couchbase.by_bucket.vb_avg_pending_queue_age

Average age in seconds of pending items in the pending item queue.

couchbase.by_bucket.vb_avg_replica_queue_age

Average age in seconds of replica items in the replica item queue.

couchbase.by_bucket.vb_avg_total_queue_age

Average age of items in the queue.

couchbase.by_bucket.vb_pending_curr_items

Number of items in pending vBuckets.

couchbase.by_bucket.vb_pending_eject

Number of items per second being ejected to disk from pending vBuckets.

couchbase.by_bucket.vb_pending_itm_memory

Amount of pending user data cached in RAM in this bucket.

couchbase.by_bucket.vb_pending_meta_data_memory

Amount of pending item metadata consuming RAM in this bucket.

couchbase.by_bucket.vb_pending_num

Number of pending items.

couchbase.by_bucket.vb_pending_num_non_resident

Number of non resident vBuckets in the pending state for this bucket.

couchbase.by_bucket.vb_pending_ops_create

Number of pending create operations.

couchbase.by_bucket.vb_pending_ops_update

Number of items updated on pending vBucket per second for this bucket.

couchbase.by_bucket.vb_pending_queue_age

Sum of disk pending queue item age in milliseconds.

couchbase.by_bucket.vb_pending_queue_drain

Total drained pending items in the queue.

couchbase.by_bucket.vb_pending_queue_fill

Total enqueued pending items on disk queue.

couchbase.by_bucket.vb_pending_queue_size

Number of pending items in the queue.

couchbase.by_bucket.vb_pending_resident_items_ratio

Number of resident pending items.

couchbase.by_bucket.vb_replica_curr_items

Number of in memory items.

couchbase.by_bucket.vb_replica_eject

Number of items per second being ejected to disk from replica vBuckets.

couchbase.by_bucket.vb_replica_itm_memory

Amount of replica user data cached in RAM in this bucket.

couchbase.by_bucket.vb_replica_meta_data_memory

Total metadata memory.

couchbase.by_bucket.vb_replica_num

Number of replica vBuckets.

couchbase.by_bucket.vb_replica_num_non_resident

Number of non resident vBuckets in the replica state for this bucket.

couchbase.by_bucket.vb_replica_ops_create

Number of replica create operations.

couchbase.by_bucket.vb_replica_ops_update

Number of items updated on replica vBucket per second for this bucket.

couchbase.by_bucket.vb_replica_queue_age

Sum of disk replica queue item age in milliseconds.

couchbase.by_bucket.vb_replica_queue_drain

Total drained replica items in the queue.

couchbase.by_bucket.vb_replica_queue_fill

Total enqueued replica items on disk queue.

couchbase.by_bucket.vb_replica_queue_size

Replica items in disk queue.

couchbase.by_bucket.vb_replica_resident_items_ratio

Number of resident replica items.

couchbase.by_bucket.vb_total_queue_age

Sum of disk queue item age in milliseconds.

couchbase.by_bucket.xdc_ops

Number of cross-datacenter replication operations.

couchbase.by_node.couch_docs_actual_disk_size

Couch docs total size on disk in bytes.

couchbase.by_node.couch_docs_data_size

Couch docs data size in bytes.

couchbase.by_node.couch_views_actual_disk_size

Couch views total size on disk in bytes.

couchbase.by_node.couch_views_data_size

Couch views data size on disk in bytes.

couchbase.by_node.curr_items

Number of active items in memory.

couchbase.by_node.curr_items_tot

Total number of items.

couchbase.by_node.vb_replica_curr_items

Number of in memory items.

couchbase.hdd.free

Free hard disk space.

couchbase.hdd.quota_total

Hard disk quota.

couchbase.hdd.total

Total hard disk space.

couchbase.hdd.used

Used hard disk space.

couchbase.hdd.used_by_data

Hard disk used for data.

couchbase.query.cores

couchbase.query.cpu_sys_percent

couchbase.query.cpu_user_percent

couchbase.query.gc_num

couchbase.query.gc_pause_percent

couchbase.query.gc_pause_time

couchbase.query.memory_system

couchbase.query.memory_total

couchbase.query.memory_usage

couchbase.query.request_active_count

couchbase.query.request_completed_count

couchbase.query.request_per_sec_15min

couchbase.query.request_per_sec_1min

couchbase.query.request_per_sec_5min

couchbase.query.request_prepared_percent

couchbase.query.request_time_80percentile

couchbase.query.request_time_95percentile

couchbase.query.request_time_99percentile

couchbase.query.request_time_mean

couchbase.query.request_time_median

couchbase.query.total_threads

couchbase.ram.quota_total

RAM quota.

couchbase.ram.total

The total RAM available.

couchbase.ram.used

The amount of RAM in use.

couchbase.ram.used_by_data

The amount of RAM used for data.

5.3.5 -

Elasticsearch Metrics

See also: Elasticsearch integration information.

elasticsearch.active_primary_shards

The number of active primary shards in the cluster.

elasticsearch.active_shards

The number of active shards in the cluster.

elasticsearch.breakers.fielddata.estimated_size_in_bytes

The estimated size in bytes of the field data circuit breaker.

elasticsearch.breakers.fielddata.overhead

The constant multiplier for byte estimations of the field data circuit breaker.

elasticsearch.breakers.fielddata.tripped

The number of times the field data circuit breaker has tripped.

elasticsearch.breakers.parent.estimated_size_in_bytes

The estimated size in bytes of the parent circuit breaker.

elasticsearch.breakers.parent.overhead

The constant multiplier for byte estimations of the parent circuit breaker.

elasticsearch.breakers.parent.tripped

The number of times the parent circuit breaker has tripped.

elasticsearch.breakers.request.estimated_size_in_bytes

The estimated size in bytes of the request circuit breaker.

elasticsearch.breakers.request.overhead

The constant multiplier for byte estimations of the request circuit breaker.

elasticsearch.breakers.request.tripped

The number of times the request circuit breaker has tripped.

elasticsearch.breakers.inflight_requests.tripped

The number of times the inflight circuit breaker has tripped.

elasticsearch.breakers.inflight_requests.overhead

The constant multiplier for byte estimations of the inflight circuit breaker.

elasticsearch.breakers.inflight_requests.estimated_size_in_bytes

The estimated size in bytes of the inflight circuit breaker.

elasticsearch.cache.field.evictions

The total number of evictions from the field data cache.

elasticsearch.cache.field.size

The size of the field cache.

elasticsearch.cache.filter.count

The number of items in the filter cache.

elasticsearch.cache.filter.evictions

The total number of evictions from the filter cache.

elasticsearch.cache.filter.size

The size of the filter cache.

elasticsearch.cluster_status

The elasticsearch cluster health as a number: red = 0, yellow = 1, green = 2

elasticsearch.docs.count

The total number of documents in the cluster across all shards.

elasticsearch.docs.deleted

The total number of documents deleted from the cluster across all shards.

elasticsearch.fielddata.evictions

The total number of evictions from the fielddata cache.

elasticsearch.fielddata.size

The size of the fielddata cache.

elasticsearch.flush.total

The total number of index flushes to disk since start.

elasticsearch.flush.total.time

The total time spent flushing the index to disk.

elasticsearch.fs.total.available_in_bytes

The total number of bytes available to this Java virtual machine on this file store.

elasticsearch.fs.total.disk_io_op

The total I/O operations on the file store.

elasticsearch.fs.total.disk_io_size_in_bytes

Total bytes used for all I/O operations on the file store.

elasticsearch.fs.total.disk_read_size_in_bytes

The total bytes read from the file store.

elasticsearch.fs.total.disk_reads

The total number of reads from the file store.

elasticsearch.fs.total.disk_write_size_in_bytes

The total bytes written to the file store.

elasticsearch.fs.total.disk_writes

The total number of writes to the file store.

elasticsearch.fs.total.free_in_bytes

The total number of unallocated bytes in the file store.

elasticsearch.fs.total.total_in_bytes

The total size in bytes of the file store.

elasticsearch.get.current

The number of get requests currently running.

elasticsearch.get.exists.time

The total time spent on get requests where the document existed.

elasticsearch.get.exists.total

The total number of get requests where the document existed.

elasticsearch.get.missing.time

The total time spent on get requests where the document was missing.

elasticsearch.get.missing.total

The total number of get requests where the document was missing.

elasticsearch.get.time

The total time spent on get requests.

elasticsearch.get.total

The total number of get requests.

elasticsearch.http.current_open

The number of current open HTTP connections.

elasticsearch.http.total_opened

The total number of opened HTTP connections.

elasticsearch.id_cache.size

The size of the id cache

elasticsearch.indexing.delete.current

The number of documents currently being deleted from an index.

elasticsearch.indexing.delete.time

The total time spent deleting documents from an index.

elasticsearch.indexing.delete.total

The total number of documents deleted from an index.

elasticsearch.indexing.index.current

The number of documents currently being indexed to an index.

elasticsearch.indexing.index.time

The total time spent indexing documents to an index.

elasticsearch.indexing.index.total

The total number of documents indexed to an index.

elasticsearch.indices.count

The number of indices in the cluster.

elasticsearch.indices.indexing.index_failed

The number of failed indexing operations.

elasticsearch.indices.indexing.throttle_time

The total time indexing waited due to throttling.

elasticsearch.indices.query_cache.evictions

The number of query cache evictions.

elasticsearch.indices.query_cache.hit_count

The number of query cache hits.

elasticsearch.indices.query_cache.memory_size_in_bytes

The memory used by the query cache.

elasticsearch.indices.query_cache.miss_count

The number of query cache misses.

elasticsearch.indices.recovery.current_as_source

The number of ongoing recoveries for which a shard serves as a source.

elasticsearch.indices.recovery.current_as_target

The number of ongoing recoveries for which a shard serves as a target.

elasticsearch.indices.recovery.throttle_time

The total time recoveries waited due to throttling.

elasticsearch.indices.request_cache.evictions

The number of request cache evictions.

elasticsearch.indices.request_cache.hit_count

The number of request cache hits.

elasticsearch.indices.request_cache.memory_size_in_bytes

The memory used by the request cache.

elasticsearch.indices.request_cache.miss_count

The number of request cache misses.

elasticsearch.indices.segments.count

The number of segments in an index shard.

elasticsearch.indices.segments.doc_values_memory_in_bytes

The memory used by doc values.

elasticsearch.indices.segments.fixed_bit_set_memory_in_bytes

The memory used by fixed bit set.

elasticsearch.indices.segments.index_writer_max_memory_in_bytes

The maximum memory used by the index writer.

elasticsearch.indices.segments.index_writer_memory_in_bytes

The memory used by the index writer.

elasticsearch.indices.segments.memory_in_bytes

The memory used by index segments.

elasticsearch.indices.segments.norms_memory_in_bytes

The memory used by norms.

elasticsearch.indices.segments.stored_fields_memory_in_bytes

The memory used by stored fields.

elasticsearch.indices.segments.term_vectors_memory_in_bytes

The memory used by term vectors.

elasticsearch.indices.segments.terms_memory_in_bytes

The memory used by terms.

elasticsearch.indices.segments.version_map_memory_in_bytes

The memory used by the segment version map.

elasticsearch.indices.translog.operations

The number of operations in the transaction log.

elasticsearch.indices.translog.size_in_bytes

The size of the transaction log.

elasticsearch.initializing_shards

The number of shards that are currently initializing.

elasticsearch.merges.current

The number of currently active segment merges.

elasticsearch.merges.current.docs

The number of documents across segments currently being merged.

elasticsearch.merges.current.size

The size of the segments currently being merged.

elasticsearch.merges.total

The total number of segment merges.

elasticsearch.merges.total.docs

The total number of documents across all merged segments.

elasticsearch.merges.total.size

The total size of all merged segments.

elasticsearch.merges.total.time

The total time spent on segment merging.

elasticsearch.number_of_data_nodes

The number of data nodes in the cluster.

elasticsearch.number_of_nodes

The total number of nodes in the cluster.

elasticsearch.pending_tasks_priority_high

The number of high priority pending tasks.

elasticsearch.pending_tasks_priority_urgent

The number of urgent priority pending tasks.

elasticsearch.pending_tasks_time_in_queue

The average time spent by tasks in the queue.

elasticsearch.pending_tasks_total

The total number of pending tasks.

elasticsearch.process.open_fd

The number of opened file descriptors associated with the current process, or -1 if not supported.

elasticsearch.refresh.total

The total number of index refreshes.

elasticsearch.refresh.total.time

The total time spent on index refreshes.

elasticsearch.relocating_shards

The number of shards that are relocating from one node to another.

elasticsearch.search.fetch.current

The number of search fetches currently running.

elasticsearch.search.fetch.open_contexts

The number of active searches.

elasticsearch.search.fetch.time

The total time spent on the search fetch.

elasticsearch.search.fetch.total

The total number of search fetches.

elasticsearch.search.query.current

The number of currently active queries.

elasticsearch.search.query.time

The total time spent on queries.

elasticsearch.search.query.total

The total number of queries.

elasticsearch.store.size

The total size in bytes of the store.

elasticsearch.thread_pool.bulk.active

The number of active threads in the bulk pool.

elasticsearch.thread_pool.bulk.queue

The number of queued threads in the bulk pool.

elasticsearch.thread_pool.bulk.threads

The total number of threads in the bulk pool.

elasticsearch.thread_pool.bulk.rejected

The number of rejected threads in the bulk pool.

elasticsearch.thread_pool.fetch_shard_started.active

The number of active threads in the fetch shard started pool.

elasticsearch.thread_pool.fetch_shard_started.threads

The total number of threads in the fetch shard started pool.

elasticsearch.thread_pool.fetch_shard_started.queue

The number of queued threads in the fetch shard started pool.

elasticsearch.thread_pool.fetch_shard_started.rejected

The number of rejected threads in the fetch shard started pool.

elasticsearch.thread_pool.fetch_shard_store.active

The number of active threads in the fetch shard store pool.

elasticsearch.thread_pool.fetch_shard_store.threads

The total number of threads in the fetch shard store pool.

elasticsearch.thread_pool.fetch_shard_store.queue

The number of queued threads in the fetch shard store pool.

elasticsearch.thread_pool.fetch_shard_store.rejected

The number of rejected threads in the fetch shard store pool.

elasticsearch.thread_pool.flush.active

The number of active threads in the flush queue.

elasticsearch.thread_pool.flush.queue

The number of queued threads in the flush pool.

elasticsearch.thread_pool.flush.threads

The total number of threads in the flush pool.

elasticsearch.thread_pool.flush.rejected

The number of rejected threads in the flush pool.

elasticsearch.thread_pool.force_merge.active

The number of active threads for force merge operations.

elasticsearch.thread_pool.force_merge.threads

The total number of threads for force merge operations.

elasticsearch.thread_pool.force_merge.queue

The number of queued threads for force merge operations.

elasticsearch.thread_pool.force_merge.rejected

The number of rejected threads for force merge operations.

elasticsearch.thread_pool.generic.active

The number of active threads in the generic pool.

elasticsearch.thread_pool.generic.queue

The number of queued threads in the generic pool.

elasticsearch.thread_pool.generic.threads

The total number of threads in the generic pool.

elasticsearch.thread_pool.generic.rejected

The number of rejected threads in the generic pool.

elasticsearch.thread_pool.get.active

The number of active threads in the get pool.

elasticsearch.thread_pool.get.queue

The number of queued threads in the get pool.

elasticsearch.thread_pool.get.threads

The total number of threads in the get pool.

elasticsearch.thread_pool.get.rejected

The number of rejected threads in the get pool.

elasticsearch.thread_pool.index.active

The number of active threads in the index pool.

elasticsearch.thread_pool.index.queue

The number of queued threads in the index pool.

elasticsearch.thread_pool.index.threads

The total number of threads in the index pool.

elasticsearch.thread_pool.index.rejected

The number of rejected threads in the index pool.

elasticsearch.thread_pool.listener.active

The number of active threads in the listener pool.

elasticsearch.thread_pool.listener.queue

The number of queued threads in the listener pool.

elasticsearch.thread_pool.listener.threads

The total number of threads in the listener pool.

elasticsearch.thread_pool.listener.rejected

The number of rejected threads in the listener pool.

elasticsearch.thread_pool.management.active

The number of active threads in the management pool.

elasticsearch.thread_pool.management.queue

The number of queued threads in the management pool.

elasticsearch.thread_pool.management.threads

The total number of threads in the management pool.

elasticsearch.thread_pool.management.rejected

The number of rejected threads in the management pool.

elasticsearch.thread_pool.merge.active

The number of active threads in the merge pool.

elasticsearch.thread_pool.merge.queue

The number of queued threads in the merge pool.

elasticsearch.thread_pool.merge.threads

The total number of threads in the merge pool.

elasticsearch.thread_pool.merge.rejected

The number of rejected threads in the merge pool.

elasticsearch.thread_pool.percolate.active

The number of active threads in the percolate pool.

elasticsearch.thread_pool.percolate.queue

The number of queued threads in the percolate pool.

elasticsearch.thread_pool.percolate.threads

The total number of threads in the percolate pool.

elasticsearch.thread_pool.percolate.rejected

The number of rejected threads in the percolate pool.

elasticsearch.thread_pool.refresh.active

The number of active threads in the refresh pool.

elasticsearch.thread_pool.refresh.queue

The number of queued threads in the refresh pool.

elasticsearch.thread_pool.refresh.threads

The total number of threads in the refresh pool.

elasticsearch.thread_pool.refresh.rejected

The number of rejected threads in the refresh pool.

elasticsearch.thread_pool.search.active

The number of active threads in the search pool.

elasticsearch.thread_pool.search.queue

The number of queued threads in the search pool.

elasticsearch.thread_pool.search.threads

The total number of threads in the search pool.

elasticsearch.thread_pool.search.rejected

The number of rejected threads in the search pool.

elasticsearch.thread_pool.snapshot.active

The number of active threads in the snapshot pool.

elasticsearch.thread_pool.snapshot.queue

The number of queued threads in the snapshot pool.

elasticsearch.thread_pool.snapshot.threads

The total number of threads in the snapshot pool.

elasticsearch.thread_pool.snapshot.rejected

The number of rejected threads in the snapshot pool.

elasticsearch.thread_pool.write.active

The number of active threads in the write pool.

elasticsearch.thread_pool.write.queue

The number of queued threads in the write pool.

elasticsearch.thread_pool.write.threads

The total number of threads in the write pool.

elasticsearch.thread_pool.write.rejected

The number of rejected threads in the write pool.

elasticsearch.transport.rx_count

The total number of packets received in cluster communication.

elasticsearch.transport.rx_size

The total size of data received in cluster communication.

elasticsearch.transport.server_open

The number of connections opened for cluster communication.

elasticsearch.transport.tx_count

The total number of packets sent in cluster communication.

elasticsearch.transport.tx_size

The total size of data sent in cluster communication.

elasticsearch.unassigned_shards

The number of shards that are unassigned to a node.

elasticsearch.delayed_unassigned_shards

The number of shards whose allocation has been delayed.

jvm.gc.collection_count

The total number of garbage collections run by the JVM.

jvm.gc.collection_time

The total time spent on garbage collection in the JVM.

jvm.gc.collectors.old.collection_time

The total time spent in major GCs in the JVM that collect old generation objects.

jvm.gc.collectors.old.count

The total count of major GCs in the JVM that collect old generation objects.

jvm.gc.collectors.young.collection_time

The total time spent in minor GCs in the JVM that collects young generation objects.

jvm.gc.collectors.young.count

The total count of minor GCs in the JVM that collects young generation objects.

jvm.gc.concurrent_mark_sweep.collection_time

The total time spent on “concurrent mark & sweep” GCs in the JVM.

jvm.gc.concurrent_mark_sweep.count

The total count of “concurrent mark & sweep” GCs in the JVM.

jvm.gc.par_new.collection_time

The total time spent on “parallel new” GCs in the JVM.

jvm.gc.par_new.count

The total count of “parallel new” GCs in the JVM.

jvm.mem.heap_committed

The amount of memory guaranteed to be available to the JVM heap.

jvm.mem.heap_in_use

The amount of memory currently used by the JVM heap as a value between 0 and 1.

jvm.mem.heap_max

The maximum amount of memory that can be used by the JVM heap.

jvm.mem.heap_used

The amount of memory in bytes currently used by the JVM heap.

jvm.mem.non_heap_committed

The amount of memory guaranteed to be available to JVM non-heap.

jvm.mem.non_heap_used

The amount of memory in bytes currently used by the JVM non-heap.

jvm.mem.pools.young.used

The amount of memory in bytes currently used by the Young Generation heap region.

jvm.mem.pools.young.max

The maximum amount of memory that can be used by the Young Generation heap region.

jvm.mem.pools.old.used

The amount of memory in bytes currently used by the Old Generation heap region.

jvm.mem.pools.old.max

The maximum amount of memory that can be used by the Old Generation heap region.

jvm.mem.pools.survivor.used

The amount of memory in bytes currently used by the Survivor Space.

jvm.mem.pools.survivor.max

The maximum amount of memory that can be used by the Survivor Space.

jvm.threads.count

The number of active threads in the JVM.

jvm.threads.peak_count

The peak number of threads used by the JVM.

elasticsearch.index.health

The status of the index.

elasticsearch.index.docs.count

The number of documents in the index.

elasticsearch.index.docs.deleted

The number of deleted documents in the index.

elasticsearch.index.primary_shards

The number of primary shards in the index.

elasticsearch.index.replica_shards

The number of replica shards in the index.

elasticsearch.index.primary_store_size

The store size of primary shards in the index.

elasticsearch.index.store_size

The store size of primary and replica shards in the index.

5.3.6 -

etcd Metrics

For related information, see etcd integration.

etcd.leader.counts.fail

Rate of failed Raft RPC requests.

etcd.leader.counts.success

Rate of successful Raft RPC requests.

etcd.leader.latency.avg

Average latency to each peer in the cluster.

etcd.leader.latency.current

Current latency to each peer in the cluster.

etcd.leader.latency.max

Maximum latency to each peer in the cluster.

etcd.leader.latency.min

Minimum latency to each peer in the cluster.

etcd.leader.latency.stddev

Standard deviation latency to each peer in the cluster.

etcd.self.recv.appendrequest.count

Rate of append requests this node has processed.

etcd.self.recv.bandwidthrate

Rate of bytes received.

etcd.self.recv.pkgrate

Rate of packets received.

etcd.self.send.appendrequest.count

Rate of append requests this node has sent.

etcd.self.send.bandwidthrate

Rate of bytes sent.

etcd.self.send.pkgrate

Rate of packets sent.

etcd.store.compareanddelete.fail

Rate of compare and delete requests failure.

etcd.store.compareanddelete.success

Rate of compare and delete requests success.

etcd.store.compareandswap.fail

Rate of compare and swap requests failure.

etcd.store.compareandswap.success

Rate of compare and swap requests success.

etcd.store.create.fail

Rate of failed create requests.

etcd.store.create.success

Rate of successful create requests.

etcd.store.delete.fail

Rate of failed delete requests.

etcd.store.delete.success

Rate of successful delete requests.

etcd.store.expire.count

Rate of expired keys.

etcd.store.gets.fail

Rate of failed get requests.

etcd.store.gets.success

Rate of successful get requests.

etcd.store.sets.fail

Rate of failed set requests.

etcd.store.sets.success

Rate of successful set requests.

etcd.store.update.fail

Rate of failed update requests.

etcd.store.update.success

Rate of successful update requests.

etcd.store.watchers

Rate of watchers.

5.3.7 -

fluentd Metrics

For related information, see fluentd integration.

fluentd.buffer_queue_length

The length of the plugin buffer queue for this plugin.

fluentd.buffer_total_queued_size

The size of the buffer queue for this plugin.

fluentd.retry_count

The number of retries for this plugin.

5.3.8 -

Go Metrics

See also: Go integration information.

go_expvar.memstats.alloc

The number of bytes allocated and not yet freed.

go_expvar.memstats.frees

The number of free bytes.

go_expvar.memstats.heap_alloc

go_expvar.memstats.heap_idle

The number of bytes in idle spans.

go_expvar.memstats.heap_inuse

The number of bytes in non-idle spans.

go_expvar.memstats.heap_objects

The total number of allocated objects.

go_expvar.memstats.heap_released

The number of bytes released to the OS.

go_expvar.memstats.heap_sys

The number of bytes obtained from the system.

go_expvar.memstats.lookups

The number of pointer lookups.

go_expvar.memstats.mallocs

The number of mallocs.

go_expvar.memstats.num_gc

The number of garbage collections.

go_expvar.memstats.pause_ns.avg

The average of recent GC pause durations.

go_expvar.memstats.pause_ns.count

The number of submitted GC pause durations.

go_expvar.memstats.pause_ns.max

The max GC pause duration.

go_expvar.memstats.pause_ns.median

The median GC pause duration.

go_expvar.memstats.pause_total_ns

The total GC pause duration over the lifetime of process.

go_expvar.memstats.total_alloc

The bytes allocated (even if freed).

5.3.9 -

HTTP Metrics

See HTTP integration.

http.ssl.days_left

The number of days until the SSL certificate expires.

network.http.response_time

The response time of a HTTP request to a specified URL.

5.3.10 -

HAProxy Metrics

See also: HAProxy integration information.

haproxy.backend_hosts

The number of backend hosts.

haproxy.backend.bytes.in_rate

The rate of bytes in on backend hosts.

haproxy.backend.bytes.out_rate

The rate of bytes out on backend hosts.

haproxy.backend.connect.time

The average connect time over the last 1024 requests.

haproxy.backend.denied.req_rate

The number of requests denied due to security concerns.

haproxy.backend.denied.resp_rate

The number of responses denied due to security concerns.

haproxy.backend.errors.con_rate

The rate of requests that encountered an error trying to connect to a backend server.

haproxy.backend.errors.resp_rate

The rate of responses aborted due to error.

haproxy.backend.queue.current

The number of requests without an assigned backend.

haproxy.backend.queue.time

The average queue time over the last 1024 requests.

haproxy.backend.response.1xx

The backend HTTP responses with 1xx code.

haproxy.backend.response.2xx

The backend HTTP responses with 2xx code.

haproxy.backend.response.3xx

The backend HTTP responses with 3xx code.

haproxy.backend.response.4xx

The backend HTTP responses with 4xx code.

haproxy.backend.response.5xx

The backend HTTP responses with 5xx code.

haproxy.backend.response.other

The backend HTTP responses with another code (protocol error).

haproxy.backend.response.time

The average response time over the last 1024 requests (0 for TCP).

haproxy.backend.session.current

The number of active backend sessions.

haproxy.backend.session.limit

The configured backend session limit.

haproxy.backend.session.pct

The percentage of sessions in use. The formula used for this metric is backend.session.current / backend.session.limit * 100.

haproxy.backend.session.rate

The number of backend sessions created per second.

haproxy.backend.session.time

The average total session time over the last 1024 requests.

haproxy.backend.uptime

The number of seconds since the last UP<->DOWN transition.

haproxy.backend.warnings.redis_rate

The number of times a request was redispatched to another server.

haproxy.backend.warnings.retr_rate

The number of times a connection to a server was retried.

haproxy.count_per_status

The number of hosts by status (UP/DOWN/NOLB/MAINT).

haproxy.frontend.bytes.in_rate

The rate of bytes in on frontend hosts.

haproxy.frontend.bytes.out_rate

The rate of bytes out on frontend hosts.

haproxy.frontend.denied.req_rate

The number of requests denied due to security concerns.

haproxy.frontend.denied.resp_rate

The number of responses denied due to security concerns.

haproxy.frontend.errors.req_rate

The rate of request errors.

haproxy.frontend.requests.rate

The number of HTTP requests per second.

haproxy.frontend.response.1xx

The frontend HTTP responses with 1xx code.

haproxy.frontend.response.2xx

The frontend HTTP responses with 2xx code.

haproxy.frontend.response.3xx

The frontend HTTP responses with 3xx code.

haproxy.frontend.response.4xx

The frontend HTTP responses with 4xx code.

haproxy.frontend.response.5xx

The frontend HTTP responses with 5xx code.

haproxy.frontend.response.other

The frontend HTTP responses with another code (protocol error).

haproxy.frontend.session.current

The number of active frontend sessions.

haproxy.frontend.session.limit

The configured backend session limit.

haproxy.frontend.session.pct

The percentage of sessions in use. The formula used for this metric is frontend.session.current / frontend.session.limit * 100.

haproxy.frontend.session.rate

The number of frontend sessions created per second.

Agent 9.6.0 Additional HAProxy Metrics

• haproxy.backend.requests.tot_rate

  Rate of total number of HTTP requests

• haproxy.frontend.connections.rate

  Number of connections per second

• haproxy.frontend.connections.tot_rate

  Rate of total number of connections

• haproxy.frontend.requests.intercepted

  Number of intercepted requests per second

• haproxy.frontend.requests.tot_rate

  Rate of total number of HTTP requests

5.3.11 -

Jenkins Metrics

See also: Jenkins integration information.

jenkins.job.duration

The duration of a job, measured in seconds.

jenkins.job.success

The status of a successful job.

jenkins.job.failure

The status of a failed job.

5.3.12 -

Lighttpd Metrics

See also: Lighttpd integration information.

lighttpd.net.bytes

The total number of bytes sent and received.

lighttpd.net.bytes_per_s

The number of bytes sent and received per second.

lighttpd.net.hits

The total number of hits since the start.

lighttpd.net.request_per_s

The number of requests per second.

lighttpd.performance.busy_servers

The number of active connections.

lighttpd.performance.idle_server

The number of idle connections.

lighttpd.performance.uptime

The amount of time the server has been up and running.

5.3.13 -

Memcached Metrics

See also: Memcached integration information.

memcache.avg_item_size

The average size of an item.

memcache.bytes

The current number of bytes used by this server to store items.

memcache.bytes_read_rate

The rate of bytes read from the network by this server.

memcache.bytes_written_rate

The rate of bytes written to the network by this server.

memcache.cas_badval_rate

The rate at which keys are compared and swapped where the comparison (original) value did not match the supplied value.

memcache.cas_hits_rate

The rate at which keys are compared and swapped and found present.

memcache.cas_misses_rate

The rate at which keys are compared and swapped and not found present.

memcache.cmd_flush_rate

The rate of flush_all commands.

memcache.cmd_get_rate

The rate of get commands.

memcache.cmd_set_rate

The rate of set commands.

memcache.connection_structures

The number of connection structures allocated by the server.

memcache.curr_connections

The number of open connections to this server.

memcache.curr_items

The current number of items stored by the server.

memcache.delete_hits_rate

The rate at which delete commands result in items being removed.

memcache.delete_misses_rate

The rate at which delete commands result in no items being removed.

memcache.evictions_rate

The rate at which valid items are removed from cache to free memory for new items.

memcache.fill_percent

The amount of memory being used by the server for storing items as a percentage of the max allowed.

memcache.get_hit_percent

The percentage of requested keys that are found present since the start of the Memcached server.

memcache.get_hits_rate

The rate at which keys are requested and found present.

memcache.get_misses_rate

The rate at which keys are requested and not found.

memcache.items.age

The age of the oldest item in the LRU.

memcache.items.crawler_reclaimed_rate

The rate at which items freed by the LRU Crawler.

memcache.items.direct_reclaims_rate

The rate at which worker threads had to directly pull LRU tails to find memory for a new item.

memcache.items.evicted_nonzero_rate

The rate at which nonzero items which had an explicit expire time set had to be evicted from the LRU before expiring.

memcache.items.evicted_rate

The rate st which items had to be evicted from the LRU before expiring.

memcache.items.evicted_time

The number of seconds since the last access for the most recent item evicted from this class.

memcache.items.evicted_unfetched_rate

The rate at which valid items evicted from the LRU which were never touched after being set.

memcache.items.expired_unfetched_rate

The rate at which expired items reclaimed from the LRU which were never touched after being set.

memcache.items.lrutail_reflocked_rate

The rate at which items found to be refcount locked in the LRU tail.

memcache.items.moves_to_cold_rate

The rate at which items were moved from HOT or WARM into COLD.

memcache.items.moves_to_warm_rate

The rate at which items were moved from COLD to WARM.

memcache.items.moves_within_lru_rate

The rate at which active items were bumped within HOT or WARM.

memcache.items.number

The number of items presently stored in this slab class.

memcache.items.number_cold

The number of items presently stored in the COLD LRU.

memcache.items.number_hot

The number of items presently stored in the HOT LRU.

memcache.items.number_noexp

The number of items presently stored in the NOEXP class.

memcache.items.number_warm

The number of items presently stored in the WARM LRU.

memcache.items.outofmemory_rate

The rate at which the underlying slab class was unable to store a new item.

memcache.items.reclaimed_rate

The rate at which entries were stored using memory from an expired entry.

memcache.items.tailrepairs_rate

The rate at which Memcached self-healed a slab with a refcount leak.

memcache.limit_maxbytes

The number of bytes this server is allowed to use for storage.

memcache.listen_disabled_num_rate

The rate at which the server has reached the max connection limit.

memcache.pointer_size

The default size of pointers on the host OS (generally 32 or 64).

memcache.rusage_system_rate

The fraction of user time the CPU spent executing this server process.

memcache.rusage_user_rate

The fraction of time the CPU spent executing kernel code on behalf of this server process.

memcache.slabs.active_slabs

The total number of slab classes allocated.

memcache.slabs.cas_badval_rate

The rate at which CAS commands failed to modify a value due to a bad CAS ID.

memcache.slabs.cas_hits_rate

The rate at which CAS commands modified this slab class.

memcache.slabs.chunk_size

The amount of space each chunk uses.

memcache.slabs.chunks_per_page

The number of chunks that exist within one page.

memcache.slabs.cmd_set_rate

The rate at which set requests stored data in this slab class.

memcache.slabs.decr_hits_rate

The rate at which decrs commands modified this slab class.

memcache.slabs.delete_hits_rate

The rate at which delete commands succeeded in this slab class.

memcache.slabs.free_chunks

The number of chunks not yet allocated to items or freed via delete.

memcache.slabs.free_chunks_end

The number of free chunks at the end of the last allocated page.

memcache.slabs.get_hits_rate

The rate at which get requests were serviced by this slab class.

memcache.slabs.incr_hits_rate

The rate at which incrs commands modified this slab class.

memcache.slabs.mem_requested

The number of bytes requested to be stored in this slab.

memcache.slabs.total_chunks

The total number of chunks allocated to the slab class.

memcache.slabs.total_malloced

The total amount of memory allocated to slab pages.

memcache.slabs.total_pages

The total number of pages allocated to the slab class.

memcache.slabs.touch_hits_rate

The rate of touches serviced by this slab class.

memcache.slabs.used_chunks

The number of chunks that have been allocated to items.

memcache.slabs.used_chunks_rate

The rate at which chunks have been allocated to items.

memcache.threads

The number of threads used by the current Memcached server process.

memcache.total_connections_rate

The rate at which connections to this server are opened.

memcache.total_items

The total number of items stored by this server since it started.

memcache.uptime

The number of seconds this server has been running.

5.3.14 -

Mesos/Marathon Metrics

Contents

• Marathon Metrics

• Mesos Master Metrics

• Mesos Agent Metrics

5.3.14.1 -

Mesos Agent Metrics

See also: Mesos/Marathon integration information.

mesos.slave.cpus_percent

The percentage of CPUs allocated to the slave.

mesos.slave.cpus_total

The total number of CPUs.

mesos.slave.cpus_used

The number of CPUs allocated to the slave.

mesos.slave.disk_percent

The percentage of disk space allocated to the slave.

mesos.slave.disk_total

The total disk space available.

mesos.slave.disk_used

The amount of disk space allocated to the slave.

mesos.slave.executors_registering

The number of executors registering.

mesos.slave.executors_running

The number of executors currently running.

mesos.slave.executors_terminated

The number of terminated executors.

mesos.slave.executors_terminating

The number of terminating executors.

mesos.slave.frameworks_active

The number of active frameworks.

mesos.slave.invalid_framework_messages

The number of invalid framework messages.

mesos.slave.invalid_status_updates

The number of invalid status updates.

mesos.slave.mem_percent

The percentage of memory allocated to the slave.

mesos.slave.mem_total

The total memory available.

mesos.slave.mem_used

The amount of memory allocated to the slave.

mesos.slave.recovery_errors

The number of errors encountered during slave recovery.

mesos.slave.tasks_failed

The number of failed tasks.

mesos.slave.tasks_finished

The number of finished tasks.

mesos.slave.tasks_killed

The number of killed tasks.

mesos.slave.tasks_lost

The number of lost tasks.

mesos.slave.tasks_running

The number of running tasks.

mesos.slave.tasks_staging

The number of staging tasks.

mesos.slave.tasks_starting

The number of starting tasks.

mesos.slave.valid_framework_messages

The number of valid framework messages.

mesos.slave.valid_status_updates

The number of valid status updates.

mesos.state.task.cpu

The task CPU.

mesos.state.task.disk

The disk space available for the task.

mesos.state.task.mem

The amount of memory used by the task.

mesos.stats.registered

Defines whether this slave is registered with a master.

mesos.stats.system.cpus_total

The total number of CPUs available.

mesos.stats.system.load_1min

The average load for the last minute.

mesos.stats.system.load_5min

The average load for the last five minutes.

mesos.stats.system.load_15min

The average load for the last 15 minutes.

mesos.stats.system.mem_free_bytes

The amount of free memory.

mesos.stats.system.mem_total_bytes

The total amount of memory.

mesos.stats.uptime_secs

The current uptime for the slave.

5.3.14.2 -

Mesos Master Metrics

See also: Mesos/Marathon integration information.

mesos.cluster.cpus_percent

The percentage of CPUs allocated to the cluster.

mesos.cluster.cpus_total

The total number of CPUs.

mesos.cluster.cpus_used

The number of CPUs used by the cluster.

mesos.cluster.disk_percent

The percentage of disk space allocated to the cluster.

mesos.cluster.disk_total

The total amount of disk space.

mesos.cluster.disk_used

The amount of disk space used by the cluster.

mesos.cluster.dropped_messages

The number of dropped messages.

mesos.cluster.event_queue_dispatches

The number of dispatches in the event queue.

mesos.cluster.event_queue_http_requests

The number of HTTP requests in the event queue.

mesos.cluster.event_queue_messages

The number of messages in the event queue.

mesos.cluster.frameworks_active

The number of active frameworks.

mesos.cluster.frameworks_connected

The number of connected frameworks.

mesos.cluster.frameworks_disconnected

The number of disconnected frameworks.

mesos.cluster.frameworks_inactive

The number of inactive frameworks.

mesos.cluster.gpus_total

The total number of GPUs.

mesos.cluster.invalid_framework_to_executor_messages

The number of invalid messages between the framework and the executor.

mesos.cluster.invalid_status_update_acknowledgements

The number of invalid status update acknowledgements.

mesos.cluster.invalid_status_updates

The number of invalid framework messages.

mesos.cluster.mem_percent

The percentage of memory allocated to the cluster.

mesos.cluster.mem_total

The total amount of memory available.

mesos.cluster.mem_used

The amount of memory the cluster is using.

mesos.cluster.outstanding_offers

The number of outstanding resource offers.

mesos.cluster.slave_registrations

The number of slaves able to rejoin the cluster after a disconnect.

mesos.cluster.slave_removals

The number of slaves that have been removed for any reason, including maintenance.

mesos.cluster.slave_reregistrations

The number of slaves that have re-registered.

mesos.cluster.slave_shutdowns_canceled

The number of slave shutdowns processes that have been cancelled.

mesos.cluster.slave_shutdowns_scheduled

The number of slaves that have failed health checks and are scheduled for removal.

mesos.cluster.slaves_active

The number of active slaves.

mesos.cluster.slaves_connected

The number of connected slaves.

mesos.cluster.slaves_disconnected

The number of disconnected slaves.

mesos.cluster.slaves_inactive

The number of inactive slaves.

mesos.cluster.tasks_error

The number of cluster tasks that resulted in an error.

mesos.cluster.tasks_failed

The number of failed cluster tasks.

mesos.cluster.tasks_finished

The number of completed cluster tasks.

mesos.cluster.tasks_killed

The number of killed cluster tasks.

mesos.cluster.tasks_lost

The number of lost cluster tasks.

mesos.cluster.tasks_running

The number of cluster tasks currently running.

mesos.cluster.tasks_staging

The number of cluster tasks currently staging.

mesos.cluster.tasks_starting

The number of cluster tasks starting.

mesos.cluster.valid_framework_to_executor_messages

The number of valid framework messages.

mesos.cluster.valid_status_update_acknowledgements

The number of valid status update acknowledgements.

mesos.cluster.valid_status_updates

The number of valid status updates.

mesos.framework.cpu

The CPU of the Mesos framework.

mesos.framework.disk

The total disk space of the Mesos framework, measured in mebibytes.

mesos.framework.mem

The total memory of the Mesos framework, measured in mebibytes.

mesos.registrar.queued_operations

The number of queued operations.

mesos.registrar.registry_size_bytes

The size of the Mesos registry in bytes.

mesos.registrar.state_fetch_ms

The Mesos registry’s read latency, in bytes.

mesos.registrar.state_store_ms

The Mesos registry’s write latency, in bytes.

mesos.registrar.state_store_ms.count

The Mesos registry’s write count, in bytes.

mesos.registrar.state_store_ms.max

The maximum write latency for the registry, in milliseconds.

mesos.registrar.state_store_ms.min

The minimum write latency for the registry, in miliseconds.

mesos.registrar.state_store_ms.p50

The median registry write latency, in milliseconds.

mesos.registrar.state_store_ms.p90

The 90th percentile registry write latency, in milliseconds.

mesos.registrar.state_store_ms.p95

The 95th percentile registry write latency, in milliseconds.

mesos.registrar.state_store_ms.p99

The 99th percentile registry write latency, in milliseconds.

mesos.registrar.state_store_ms.p999

The 99.9th percentile registry write latency, in milliseconds.

mesos.registrar.state_store_ms.p9999

The 99.99th percentile registry write latency, in milliseconds.

mesos.role.cpu

The CPU capacity of the configured role.

mesos.role.disk

The total disk space available to the Mesos role, in mebibytes.

mesos.role.mem

The total memory available to the Mesos role, in mebibytes.

mesos.stats.elected

Defines whether this is the elected master or not.

mesos.stats.system.cpus_total

The total number of CPUs in the system.

mesos.stats.system.load_1min

The average load for the last minute.

mesos.stats.system.load_5min

The average load for the last five minutes.

mesos.stats.system.load_15min

The average load for the last fifteen minutes.

mesos.stats.system.mem_free_bytes

The total amount of free system memory, in bytes.

mesos.stats.system.mem_total_bytes

The total cluster memory in bytes.

mesos.stats.uptime_secs

The current uptime of the cluster.

5.3.14.3 -

Marathon Metrics

See also: Mesos/Marathon integration information.

marathon.apps

The total number of applications.

marathon.backoffFactor

The multiplication factor for the delay between each consecutive failed task. This value is multiplied by the value of marathon.backoffSeconds each time the task fails until the maximum delay is reached, or the task succeeds.

marathon.backoffSeconds

The period of time between attempts to run a failed task. This value is multiplied by marathon.backoffFactor for each consecutive task failure, until either the task succeeds or the maximum delay is reached.

marathon.cpus

The number of CPUs configured for each application instance.

marathon.disk

The amount of disk space configured for each application instance.

marathon.instances

The number of instances of a specific application.

marathon.mem

The total amount of configured memory for each instance of a specific application.

marathon.tasksRunning

The number of tasks running for a specific application.

marathon.tasksStaged

The number of tasks staged for a specific application.

5.3.15 -

MongoDB Metrics

See also: MongoDB integration information.

Metrics Introduced with Agent v9.7.0

The following metrics are supported by Sysdig Agent v9.7.0 and above.

mongodb.asserts.msgps

Number of message assertions raised per second.

mongodb.asserts.regularps

Number of regular assertions raised per second.

mongodb.asserts.rolloversps

Number of times that the rollover counters roll over per second. The counters rollover to zero every 2^30 assertions.

mongodb.asserts.userps

Number of user assertions raised per second.

mongodb.asserts.warningps

Number of warnings raised per second.

mongodb.backgroundflushing.average_ms

Average time for each flush to disk.

mongodb.backgroundflushing.flushesps

Number of times the database has flushed all writes to disk.

mongodb.backgroundflushing.last_ms

Amount of time that the last flush operation took to complete.

mongodb.backgroundflushing.total_ms

Total number of time that the `mongod` processes have spent writing (i.e. flushing) data to disk.

mongodb.connections.available

Number of unused available incoming connections the database can provide.

mongodb.connections.current

Number of connections to the database server from clients.

mongodb.connections.totalcreated

Total number of connections created.

mongodb.cursors.timedout

Total number of cursors that have timed out since the server process started.

mongodb.cursors.totalopen

Number of cursors that MongoDB is maintaining for clients

mongodb.dbs

Total number of existing databases

mongodb.dur.commits

Number of transactions written to the journal during the last journal group commit interval.

mongodb.dur.commitsinwritelock

Count of the commits that occurred while a write lock was held.

mongodb.dur.compression

Compression ratio of the data written to the journal.

mongodb.dur.earlycommits

Number of times MongoDB requested a commit before the scheduled journal group commit interval.

mongodb.dur.journaledmb

Amount of data written to journal during the last journal group commit interval.

mongodb.dur.timems.commits

Amount of time spent for commits.

mongodb.dur.timems.commitsinwritelock

Amount of time spent for commits that occurred while a write lock was held.

mongodb.dur.timems.dt

Amount of time over which MongoDB collected the `dur.timeMS` data.

mongodb.dur.timems.preplogbuffer

Amount of time spent preparing to write to the journal.

mongodb.dur.timems.remapprivateview

Amount of time spent remapping copy-on-write memory mapped views.

mongodb.dur.timems.writetodatafiles

Amount of time spent writing to data files after journaling.

mongodb.dur.timems.writetojournal

Amount of time spent writing to the journal

mongodb.dur.writetodatafilesmb

Amount of data written from journal to the data files during the last journal group commit interval.

mongodb.extra_info.page_faultsps

Number of page faults per second that require disk operations.

mongodb.fsynclocked

Number of fsynclocked performed on a mongo instance.

mongodb.globallock.activeclients.readers

Count of the active client connections performing read operations.

mongodb.globallock.activeclients.total

Total number of active client connections to the database.

mongodb.globallock.activeclients.writers

Count of active client connections performing write operations.

mongodb.globallock.currentqueue.readers

Number of operations that are currently queued and waiting for the read lock.

mongodb.globallock.currentqueue.total

Total number of operations queued waiting for the lock.

mongodb.globallock.currentqueue.writers

Number of operations that are currently queued and waiting for the write lock.

mongodb.globallock.locktime

Time since the database last started that the globalLock has been held.

mongodb.globallock.ratio

Ratio of the time that the globalLock has been held to the total time since it was created.

mongodb.globallock.totaltime

Time since the database last started and created the global lock.

mongodb.indexcounters.accessesps

Number of times that operations have accessed indexes per second.

mongodb.indexcounters.hitsps

Number of times per second that an index has been accessed and mongod is able to return the index from memory.

mongodb.indexcounters.missesps

Number of times per second that an operation attempted to access an index that was not in memory.

mongodb.indexcounters.missratio

Ratio of index hits to misses.

mongodb.indexcounters.resetsps

Number of times per second the index counters have been reset.

mongodb.locks.collection.acquirecount.exclusiveps

Number of times the collection lock type was acquired in the Exclusive (X) mode.

mongodb.locks.collection.acquirecount.intent_exclusiveps

Number of times the collection lock type was acquired in the Intent Exclusive (IX) mode.

mongodb.locks.collection.acquirecount.intent_sharedps

Number of times the collection lock type was acquired in the Intent Shared (IS) mode.

mongodb.locks.collection.acquirecount.sharedps

Number of times the collection lock type was acquired in the Shared (S) mode.

mongodb.locks.collection.acquirewaitcount.exclusiveps

Number of times the collection lock type acquisition in the Exclusive (X) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.collection.acquirewaitcount.sharedps

Number of times the collection lock type acquisition in the Shared (S) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.collection.timeacquiringmicros.exclusiveps

Wait time for the collection lock type acquisitions in the Exclusive (X) mode.

mongodb.locks.collection.timeacquiringmicros.sharedps

Wait time for the collection lock type acquisitions in the Shared (S) mode.

mongodb.locks.database.acquirecount.exclusiveps

Number of times the database lock type was acquired in the Exclusive (X) mode.

mongodb.locks.database.acquirecount.intent_exclusiveps

Number of times the database lock type was acquired in the Intent Exclusive (IX) mode.

mongodb.locks.database.acquirecount.intent_sharedps

Number of times the database lock type was acquired in the Intent Shared (IS) mode.

mongodb.locks.database.acquirecount.sharedps

Number of times the database lock type was acquired in the Shared (S) mode.

mongodb.locks.database.acquirewaitcount.exclusiveps

Number of times the database lock type acquisition in the Exclusive (X) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.database.acquirewaitcount.intent_exclusiveps

Number of times the database lock type acquisition in the Intent Exclusive (IX) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.database.acquirewaitcount.intent_sharedps

Number of times the database lock type acquisition in the Intent Shared (IS) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.database.acquirewaitcount.sharedps

Number of times the database lock type acquisition in the Shared (S) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.database.timeacquiringmicros.exclusiveps

Wait time for the database lock type acquisitions in the Exclusive (X) mode.

mongodb.locks.database.timeacquiringmicros.intent_exclusiveps

Wait time for the database lock type acquisitions in the Intent Exclusive (IX) mode.

mongodb.locks.database.timeacquiringmicros.intent_sharedps

Wait time for the database lock type acquisitions in the Intent Shared (IS) mode.

mongodb.locks.database.timeacquiringmicros.sharedps

Wait time for the database lock type acquisitions in the Shared (S) mode.

mongodb.locks.global.acquirecount.exclusiveps

Number of times the global lock type was acquired in the Exclusive (X) mode.

mongodb.locks.global.acquirecount.intent_exclusiveps

Number of times the global lock type was acquired in the Intent Exclusive (IX) mode.

mongodb.locks.global.acquirecount.intent_sharedps

Number of times the global lock type was acquired in the Intent Shared (IS) mode.

mongodb.locks.global.acquirecount.sharedps

Number of times the global lock type was acquired in the Shared (S) mode.

mongodb.locks.global.acquirewaitcount.exclusiveps

Number of times the global lock type acquisition in the Exclusive (X) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.global.acquirewaitcount.intent_exclusiveps

Number of times the global lock type acquisition in the Intent Exclusive (IX) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.global.acquirewaitcount.intent_sharedps

Number of times the global lock type acquisition in the Intent Shared (IS) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.global.acquirewaitcount.sharedps

Number of times the global lock type acquisition in the Shared (S) mode encountered waits because the locks were held in a conflicting mode.

mongodb.locks.global.timeacquiringmicros.exclusiveps

Wait time for the global lock type acquisitions in the Exclusive (X) mode.

mongodb.locks.global.timeacquiringmicros.intent_exclusiveps

Wait time for the global lock type acquisitions in the Intent Exclusive (IX) mode.

mongodb.locks.global.timeacquiringmicros.intent_sharedps

Wait time for the global lock type acquisitions in the Intent Shared (IS) mode.

mongodb.locks.global.timeacquiringmicros.sharedps

Wait time for the global lock type acquisitions in the Shared (S) mode.

mongodb.locks.metadata.acquirecount.exclusiveps