Sysdig Documentation

Grouping, Scoping, and Segmenting Metrics

Data aggregation and filtering in Sysdig Monitor are done through the use of assigned labels. The sections below explain how labels work, the ways they can be used, and how to work with groupings, scopes, and segments.

Labels

Labels are used to identify and differentiate characteristics of a metric, allowing them to be aggregated or filtered for Explore module views, dashboards, alerts, and captures. Labels can be used in three different ways:

  • To group infrastructure objects into logical hierarchies displayed on the Explore tab (called groupings). For more information, refer to the Groupings section below.

  • To filter out data. For more information, refer to the Scopes section below.

  • To split aggregated data into segments. For more information, refer to the Segments section below.

373555749.png

There are two types of labels:

  • Infrastructure labels

  • Metric descriptor labels

Infrastructure Labels

Infrastructure labels are used to identify objects or entities within the infrastructure that a metric is associated with, including hosts, containers, and processes. An example label is shown below:

kubernetes.pod.name

The table below outlines what each part of the label represents:

Example Label Component

Description

kubernetes

The infrastructure type.

pod

The object.

name

The label key.

Infrastructure labels are obtained from the infrastructure (including from orchestrators, platforms, and the runtime processes), and Sysdig automatically builds a relationship model using the labels. This allows users to create logical hierarchical groupings to better aggregate the infrastructure objects in the Explore module.

For more information on groupings, refer to the Groupings.

Metric Descriptor Labels

Metric descriptor labels are custom descriptors or key-value pairs applied directly to metrics, obtained from integrations like StatsD, Prometheus, and JMX. Sysdig automatically collects custom metrics from these integrations, and parses the labels from them. Unlike infrastructure labels, these labels can be arbitrary, and do not necessarily map to any entity or object.

Note

Metric descriptor labels can only be used for segmenting, not grouping or scoping.

An example metric descriptor label is shown below:

website_failedRequests:20|region=‘Asia’, customer_ID=‘abc’

The table below outlines what each part of the label represents:

Example Label Component

Description

website_failedRequests

The metric name.

20

The metric value.

region='Asia', customer_ID='abc'

The metric descriptor labels. Multiple key-value pairs can be assigned using a comma separated list.

Warning

Sysdig recommends not using labels to store dimensions with high cardinalities (numerous different label values), such as user IDs, email addresses, URLs, or other unbounded sets of values. Each unique key-value label pair represents a new time series, which can dramatically increase the amount of data stored.

Groupings

Groupings are hierarchical organizations of labels, allowing users to organize their infrastructure views on the Explore tab in a logical hierarchy. An example grouping is shown below:

373555728.png

The example above groups the infrastructure into four levels. This results in a tree view in the Explore module with four levels, with rows for each infrastructure object applicable to each level.

As each label is selected, Sysdig Monitor automatically filters out labels for the next selection that no longer fit the hierarchy, to ensure that only logical groupings are created.

Note

In earlier Sysdig Monitor releases, users could create arbitrary hierarchies, which led to inaccurate data.

The example below shows the logical hierarchy structure for Kubernetes:

384434369.png

The new default grouping has been introduced in v2.4.0 and is available to the new teams you create. The default groupings are immutable: They cannot be modified or deleted. However, you can make a copy of them that you can modify.

Search for a Grouping

To search for, or switch to, an existing grouping:

  1. From the Explore module, open the Groupings drop-down list.

    373555714.png

    Note

    Sysdig Monitor automatically hides configured groupings that are inapplicable to the current infrastructure. To review these groupings, click the Show X Inapplicable Grouping/s link.

  2. Either select the desired grouping or search for it by scrolling down the list or using the search bar, then select it.

Create a New Grouping or Edit an Existing Grouping

To create a new grouping, or edit an existing grouping:

  1. Optional: Click the Close icon on each unnecessary label in the current grouping.

    Note

    Groupings must have at least one label; when there is only one label, the Close icon is hidden.

  2. On the Explore tab, open the first level grouping drop-down menu.

    373555707.png
  3. Select the desired top-level label, or search for it by scrolling or using the search bar, then select it.

  4. Optional: To add additional grouping levels, click the Add Grouping (plus) button beside the previous grouping, and repeat step 3.

    Repeat until there are no further layers available in the infrastructure label hierarchy.

  5. Open the Groupings drop-down list.

  6. Select the Save Current Grouping option.

  7. Define a name for the grouping, and click the Save (tick) icon to save the grouping.

Rename a Grouping

To rename a grouping:

  1. From the Explore module, open the Groupings drop-down list.

  2. Highlight the relevant grouping and click the Show Grouping actions button.

  3. Select the Rename (pencil) icon:

    Note

    The default groupings cannot be renamed.

    373555679.png
  4. Define the new name, and click the Save (tick) icon to save the changes.

Share a Grouping with the Active Team

Custom groupings can be shared to all users in the active team:

  1. From the Explore module, open the Groupings drop-down list.

  2. Highlight the relevant grouping and click the Show Grouping actions button.

  3. Click the Share Grouping (person) icon:

    Note

    The default groupings cannot be shared.

    373555672.png
  4. Toggle the Share with Team switch to make the grouping available for all users in the active team.

Copy a Grouping to a Different Team

Groupings cannot be shared between different teams, but they can be copied from the current team to one or more other teams:

  1. From the Explore module, open the Groupings drop-down list.

  2. Highlight the relevant grouping and click the Show Grouping actions button:

    373555700.png
  3. Click the Copy Grouping (paper) icon:

    Note

    The default groupings cannot be copied.

    373555693.png
  4. Open the Copy to drop-down menu, and select the team/s to copy the grouping to:

    373555686.png
  5. Optional: Rename the grouping as necessary.

  6. Click the Send copy button to save the new grouping.

Delete a Grouping

To delete a grouping:

  1. From the Explore module, open the Groupings drop-down list.

  2. Highlight the relevant grouping and click the Show Grouping actions button.

  3. Select the Delete Grouping (trash) icon:

    373555665.png
  4. Click the Yes, delete grouping button to confirm the change.

Scopes

A scope is a collection of labels that are used to filter out or define the boundaries of a group of data points when creating dashboards, dashboard panels, alerts, and teams. An example scope is shown below:

373555658.png

In the example above, the scope is defined by two labels with operators and values defined. The table below defines each of the available operators.

Operator

Description

is

The value matches the defined label value exactly.

is not

The value does not match the defined label value exactly.

in

The value is among the comma separated values entered.

not in

The value is not among the comma separated values entered.

contains

The label value contains the defined value.

does not contain

The label value does not contain the defined value.

The scope editor provides dynamic filtering capabilities. It restricts the scope of the selection for subsequent filters by rendering valid values that are specific to the previously selected label. Expand the list to view unfiltered suggestions. At run time, users can also supply custom values to achieve more granular filtering. The custom values are preserved. Note that changing a label higher up in the hierarchy might render the subsequent labels incompatible. For example, changing the kubernetes.namespace.name > kubernetes.deployment.name hierarchy to swarm.service.name > kubernetes.deployment.name is invalid as these entities belong to different orchestrators and cannot be logically grouped.

The Explore Table

Each row of the Explore table has a defined scope, based on the current grouping:

373555651.png

In this example, the grouping defines three levels: host.hostName, container.id, and container.image. The tree structure reflects the logical hierarchy created by the grouping, and each row's scope is defined by its place in the hierarchy, as shown below:

373555644.png

The example above shows that the selected row's scope is:

host.hostName = 'ip-10-0-1-51.us-west-1.compute.internal' and container.id = 'slave'

Selecting this row displays the default dashboard available for this scope. Additional dashboards are listed in the drill-down menu.

Note

The drill-down menu only displays applicable dashboards and metrics for the current scope. Dashboards and metrics that are not applicable are hidden, until a scope is selected where they are relevant. For more information, refer to the Dashboards section of the documentation.

Dashboards and Panels

Dashboard scopes define the criteria for what metric data will be included in the dashboard's panels. The current dashboard's scope can be seen at the top of the dashboard:

373555637.png

By default, all dashboard panels abide by the scope of the overall dashboard. However, an individual panel scope can be configured for a different scope than the rest of the dashboard.

For more information on Dashboards and Panels, refer to the Dashboards documentation.

Alerts

Alert scopes are defined during the creation process, and specify what areas within the infrastructure the alert is applicable for. In the example alerts below, the first alert has a scope defined, whereas the second alert does not have a custom scope defined. If no scope is defined, the alert is applicable to the entire infrastructure.

373555630.png

For more information on Alerts, refer to the Alerts documentation.Alerts

Teams

A team's scope determines the highest level of data that team members have visibility for:

  • If a team's scope is set to Host, team members can see all host-level and container-level information.

  • If a team's scope is set to Container, team members can only see container-level information.

Note

A team's scope only applies to that team. Users that are members of multiple teams may have different visibility depending on which team is active.

For more information on teams and configuring team scope, refer to the Manage Teams and Roles documentation.

Segments

Aggregated data can be split into smaller sections by segmenting the data with labels. This allows for the creation of multi-series comparisons and multiple alerts. In the first image, the metric is not segmented:

373555623.png

In the second image, the same metric has been segmented by container.id:

373555616.png

Line and Area panels can display up to five different segments for any given metric. The example image below displays the net.byte.in metric segmented by both container.id and net.http.url:

373555609.png

For more information regarding segmentation in dashboard panels, refer to the Configure Panels documentation. For more information regarding configuring alerts, refer to the Alerts documentation.Alerts

The Meaning of n/a

Sysdig Monitor imports data related to entities such as hosts, containers, processes, and so on, and reports them in tables or panels on the Explore and Dashboards UI, as well as in events, so across the UI you see varieties of data. The term n/a can appear anywhere on the UI where some form of data is displayed.

n/a is a term that indicates data that is not available or that it does not apply to a particular instance. In Sysdig parlance, the term signifies one or more entities defined by a particular label, such as hostname or Kubernetes service, for which the label is invalid. In other words, n/a collectively represent entities whose metadata is not relevant to aggregation and filtering techniques—Grouping, Scoping, and Segmenting. For instance, a list of Kubernetes services might display the list of all the services as well as n/a that includes all the containers without the metadata describing a Kubernetes service.

You might encounter n/a sporadically in Explore table as well as in drill-down panels or dashboards, events, and likely elsewhere on the Sysdig Monitor UI when no relevant metadata is available for that particular display. How n/a should be treated depends on the nature of your deployment. The deployment will not be affected by the entities marked n/a.

not_applicable.png

The following are some of the cases that yield n/a on the UI:

  • Labels are partially available or not available. For example, a host has entities that are not associated with a monitored Kubernetes deployment, or a monitored host has an unmonitored Kubernetes deployment running.

  • Labels that do not apply to the grouping criteria or at the hierarchy level. For example:

    • Containers that are not managed by Kubernetes. The containers managed by Kubernetes are identified with their  container.name labels.

    • In certain groupings by DaemonSet, Deployments render N/A and vice versa. Not all containers belong to both DaemonSet and Deployment objects concurrently. Likewise, a Kubernetes ReplicaSet grouping with the  kubernetes.replicaset.name label will not show StatefulSets.

    • In a kubernetes.cluster.name > kubernetes.namespace.name > kubernetes.deployment.name  grouping, the entities without the kubernetes.cluster.name label yield n/a.

  • Entities are incorrectly labeled in the infrastructure.

  • Kubernetes features that are yet to be in sync with Sysdig Monitoring.

  • The format is not applicable to a particular record in the database.