Filter Custom Metrics

For Sysdig’s recommended methods for dropping metrics, see How to Reduce Metrics Consumption. The recommended methods involve editing the prometheus.yaml file, while the methods outlined on this page involve using metric_filter. The latter is less desirable because it requires an agent restart before the changes take effect.

You can filter custom metrics in the following ways:

To filter custom metrics, you can:

  • Use the configurable patterns.

  • Use log and identify the custom metrics that are exceeding limits.

After you identify those key custom metrics that must be received, use the new include and exclude filtering parameters to make sure you receive them before the metrics limit is hit.

Include and Exclude Metrics

Here is an example configuration entry that would be put into the agent configuration file, /opt/draios/etc/dragent.yaml.

metrics_filter:
  - include: test.*
  - exclude: test.*
  - include: haproxy.backend.*
  - exclude: haproxy.*
  - exclude: redis.*
  - exclude: '*'

Given the config entry above, this is the action for these metrics:

test.\* : send

haproxy.backend.request: send

haproxy.frontend.bytes : drop

redis.keys : drop

If no metric matches any of the above include filters, they will be dropped with the last - exclude: '*'

When the agent reads the metrics, they are filtered according to the configured filters and the filtering rule order. The first rule that matches is applied. Therefore, in this example, the inclusion item for test.\* will be applied, and the second exclude rule for the same exact metric entry will be ignored.

Log Accepted and Dropped Metrics

Logging is disabled by default. To enable logging and see which metrics are accepted or dropped, add the following configuration entry into the dragent.yaml file:

metrics_excess_log: true

Once enabled, logging occurs at INFO level every 30 seconds and lasts for 10 seconds. Logs are stored in /opt/draios/logs/draios.log and are formatted like this:

+/-[type] [metric included/excluded]: metric.name (filter: +/-[metric.filter])

+ indicates included and - indicates excluded.

The first + or -, followed by type provides an easy way to quickly scan the list of metrics and spot which are included or excluded .

The second entry specifies the metric type: statsd, app\_check, service\_check, or jmx.

A third entry spells out whether included or excluded, followed by the metric name. Finally, the last entry, in parentheses, gives information about the filter applied and its effect.

With this example filter rule set:

metrics_filter:
  - include: mongo.statsd.net*
  - exclude: mongo.statsd.*

You might see the following INFO level log entries (timestamps stripped):

-[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.*])
+[statsd] metric included: mongo.statsd.netIn (filter: +[mongo.statsd.net*])