Sysdig Documentation

Understanding Default, Custom, and Missing Metrics

Default Metrics

Default metrics include various kinds of metadata which Sysdig Monitor automatically knows how to label, segment, and display.

For example:

  • System metrics for hosts, containers, and processes (CPU used, etc.)

  • Orchestrator metrics (collected from Kubernetes, Mesos, etc.)

  • Network metrics (e.g. network traffic)

  • HTTP

  • Platform metrics (in some cases)

Default metrics are collected mainly from two sources: syscalls and Kubernetes.

Custom Metrics

About Custom Metrics

Custom metrics generally refer to any metrics that the Sysdig Agent collects from some third-party integration. The type of infrastructure and applications integrated determine the custom metrics that the Agent collects and reports to Sysdig Monitor. The supported custom metrics are:

Each metric comes with a set of custom labels, and additional labels can be user-created. Sysdig Monitor simply collects and reports them with minimal or no internal processing. The limit currently enforced is 3000 metrics per host. Use the metrics_filter option in the dragent.yaml file to remove unwanted metrics or to choose the metrics to report when hosts exceed this limit. For more information on editing the dragent.yaml file, see Understanding the Agent Config Files.

Unit for Custom Metrics

Sysdig Monitor detects the default unit of custom metrics automatically with the delimiter suffix in the metrics name. For example, custom_expvar_time_seconds results in a base unit set to seconds. The supported base units are byte, percent, and time. Custom metrics name should carry one of the following delimiter suffixes in order for Sysdig Monitor to identify and configure the accurate unit type.

  • second

  • seconds

  • byte

  • bytes

  • total (represents accumulating count)

  • percent

Custom metrics will not be auto-detected and the unit will be incorrect unless this naming convention is followed. For instance, custom_byte_expvar will not yield the correct unit, that is MiB.

Editing the Unit Scale

You have the flexibility to change the unit scale either by editing the panel on the Dashboard or in the Explore.

Explore

From the Search Metrics and Dashboard drop-down, select the custom metrics you want to edit the unit selection for, then click More Options. Select the desired unit scale from the Metric Format drop-down and click Save.

373653858.png

Dashboard

Select the Dashboard Panel associated with the custom metrics you want to modify. Select the desired unit scale from the Metrics drop-down and click Save.

373653862.png

Missing Metrics and the Custom Metrics Lifecycle

Because custom metrics and their labels can grow to millions of entries, they are indexed by the system, and a "metrics lifecycle" is imposed both on them and their labels. This improves performance and prevents cluttering the system with a short-lived container or metric names.

Sysdig Monitor maintains an index of all the custom metrics and the custom labels that have reported data in the last 14 days. Each indexed metric/label is available when configuring new or existing artifacts, i.e. dashboards, scopes, alerts, and so on.

When the system does not receive any data for a metric/label for more than 14 days, or a metric/label is found that has never reported data (for example, it was never added to the metric index), it is removed from the index if present, and marked as missing.

Note

Missing metrics are different from instances where a panel/dashboard states that "Metric data is not available". A missing metric/label is still available for data queries and charts. Existing artifacts are not affected.

However, it is not possible to configure new artifacts with a missing metric/label, or to update existing artifacts unless the missing metrics/labels are removed from the configuration. If new data is received for a missing metric/label, we put it back into the index and the above limitations are removed.

What You See in the UI:

If you configured a dashboard three months ago and a metric in it is now missing, when you go to the dashboard and update it in the dashboard settings, a warning appears below the metric stating that the metric is missing.

To be able to update the dashboard, you must remove all missing metrics from it.

This model applies to custom metrics and custom labels, as they appear in:

  • Dashboards

  • Alerts

  • Team scopes

  • Explore table