Prioritize Designated Containers

To get the most out of Sysdig Monitor, you may want to customize the way in which container data is prioritized and reported. This topic helps you understand the default behavior and sorting rules, and to implement custom behavior when and where you need it. By prioritizing containers, you can reduce agent and backend load by not monitoring unnecessary containers, or if encountering backend limits for containers, you can filter to ensure that the important containers are always reported.

Overview

By default, a Sysdig agent will collect metrics from all containers it detects in an environment. When reporting to the Monitor interface, it uses default sorting behavior to prioritize what container information to display first.

Understand Default Behavior

Out of the box, it chooses the containers with the highest

• CPU

• Memory

• File IO

• Net IO

and allocates approximately 1/4 of the total limit to each stat type.

Understand Simple Container Filtering

You can use the use_container_filter parameter in the agent configuration file, label specific containers, and set include/exclude rules to push those containers to the top of the reporting hierarchy.

This is an effective sorting tool when:

• You can manually mark each container with an include or exclude label

• The number of includes is small (say, less than 100)

In this case, the containers that explicitly match the include rules will take top priority.

Understand Smart Container Reporting

In some enterprises, the number of containers is too high to tag with simple filtering rules, and the include_all group is too large to ensure that the most-desired containers are consistently reported. In this case, you can append another parameter, smart_container_reporting, to the configuration file.

This is an effective sorting tool when:

• The number of containers is large and you can’t or won’t mark each one with include/exclude tags, AND

• There are certain containers you would like to always prioritize

This helps ensure that even when there are thousands of containers in an environment, the most-desired containers are consistently reported.

Understand Sysdig Aggregated Container

The sysdig_aggregated parameter is automatically activated when smart container reporting is enabled, to capture the most-desired metrics from the containers that were excluded by smart filtering and report them under a single entity. It appears like any other container in the Sysdig Monitor UI, with the name “sysdig_aggregated.”

Sysdig_aggregated can report on a wide array of metrics; see Sysdig_aggregated Container Metrics. However, because this is not a regular container, certain limitations apply:

• container\_id and container\_image do not exist.

• The aggregated container cannot be segmented by certain metrics that are excluded, such as process.

• Some default dashboards associated with the aggregated container may have some empty graphs.

Use Simple Container Filtering

By default, the filtering feature is turned off. It can be enabled by adding the following line to the agent configuration:

• use_container_filter: true

When enabled, the agent will follow include/exclude filtering rules based on:

• container image

• container name

• container label

• Kubernetes annotation or label

The default behavior in default.dragent.yaml excludes based on a container label (com.sysdig.report) and a Kubernetes pod annotation (.sysdig.com/report ).

Container Condition Parameters and Rules

Parameters

The condition parameters are described in the following table:

Rules

Once enabled by setting use_container_filter: true , the agent will follow filtering rules from the container_filter section.

• Each rule is an include or exclude rule which can contain one or more conditions.

• The first matching rule in the list will determine if the container is included or excluded.

• The conditions consist of a key name and a value. If the given key for a container matches the value, the rule will be matched.

• If a rule contains multiple conditions they all need to match for the rule to be considered a match.

Default Configuraton

The default configuration file contains the following for container filters:

use_container_filter: false

container_filter:
  - include:
      container.label.com.sysdig.report: true
  - exclude:
      container.label.com.sysdig.report: false
  - include:
      kubernetes.pod.annotation.sysdig.com/report: true
  - exclude:
      kubernetes.pod.annotation.sysdig.com/report: false
  - include:
        all

Note that it excludes via a container.label and by a kubernetes.pod.annotation.

The examples on this page show how to edit in the dragent.yaml file directly. Convert the examples to Docker or Helm commands, if applicable for your situation.

Enable Container Filtering

Option 1: Use the Default Configuration

To enable container filtering using the default configuration in default.dragent.yaml (above), follow the steps below.

1. Apply Labels and Annotations to Designated Containers

To set up, decide which containers should be excluded from automatic monitoring.

Apply the container label .com.sysdig.report and/or the Kubernetes pod annotation sysdig.com/report to the designated containers.

2. Edit the Agent Configuration

Add the following line to dragent.yaml to turn on the default functionality:

use_container_filter: true

Option 2: Define Your Own Rules

You can also edit dragent.yaml to apply your own container filtering rules.

1. Designate Containers

To set up, decide which containers should be excluded from automatic monitoring.

Note the image, name, label, or Kubernetes pod information as appropriate, and build your rule set accordingly.

2. Edit the Agent Configuration

For example:

use_container_filter: true

container_filter:
  - include:
      container.name: my-app
  - include:
      container.label.com.sysdig.report: true
  - exclude:
      kubernetes.namespace.name: kube-system
      container.image: "gcr.io*"
  - include:
      all

The above example shows a container_filter with 3 include rules and 1 exclude rule.

• If the container name is “my-app” it will be included.

• Likewise, if the container has a label with the key “com.sysdig.report” and with the value “true”.

• If neither of those rules is true, and the container is part of a Kubernetes hierarchy within the “kube-system” namespace and the container image starts with “gcr.io”, it will be excluded.

• The last rule includes all, so any containers not matching an earlier rule will be monitored and metrics for them will be sent to the backend.

Use Smart Container Reporting

As of Sysdig agent version 0.91, you can add another parameter to the config file: smart_container_reporting = true

This enables several new prioritization checks:

• container_filter (you would enable and set include/exclude rules, as described above)

• container age

• high stats

• legacy patterns

The sort is modified with the following rules in priority order:

1. User-specified containers come before others

2. Containers reported previously should be reported before those which have never been reported

3. Containers with higher usage by each of the 4 default stats should come before those with lower usage

Enable Smart Container Reporting

1. Set up any simple container filtering rules you need, following either Option 1 or Option 2, above.

2. Edit the agent configuration:

   smart_container_reporting: true
   

3. This turns on both smart_container_reporting and sysdig_aggregated. The changes will be visible in the Sysdig Monitor UI.

   See Sysdig_aggregated Container Metrics.

Logging

When the log level is set to DEBUG, the following messages may be found in the logs:

See Change the Agent Log Level.