Include/Exclude Custom Metrics

It is possible to filter custom metrics in the following ways:

  • Ability to include/exclude custom metrics using configurable patterns,

  • Ability to log which custom metrics are exceeding limits

After you identify those key custom metrics that must be received, use the new ‘include’ and ’exclude’ filtering parameters to make sure you receive them before the metrics limit is hit.

Filter Metrics Example

Here is an example configuration entry that would be put into the agent config file: (/opt/draios/etc/dragent.yaml)

  - include: test.*
  - exclude: test.*
  - include: haproxy.backend.*
  - exclude: haproxy.*
  - exclude: redis.*

Given the config entry above, this is the action for these metrics:

test.* → send

haproxy.backend.request → send

haproxy.frontend.bytes → drop

redis.keys → drop

The semantic is: whenever the agent is reading metrics, they are filtered according to configured filters and the filtering rule order - the first rule that matches will be applied. Thus since the inclusion item for test.* was listed first it will be followed and that second ’exclude’ rule for the same exact metric entry will be ignored.

Logging Accepted/Dropped Metrics

Logging is disabled by default. You can enable logging to see which metrics are accepted or dropped by adding the following configuration entry into the dragent.yaml config file:

metrics_excess_log: true

When logging of excess metrics is enabled, logging occurs at INFO-level, every 30 seconds and lasts for 10 seconds. The entries that can be seen in /opt/draios/logs/draios.log will be formatted like this:

+/-[type] [metric included/excluded]: (filter: +/-[metric.filter])

The first ‘+’ or ‘-’, followed by ’type’ provides an easy way to quickly scan the list of metrics and spot which are included or excluded (’+’ means “included”, ‘-’ means “excluded”).

The second entry specifies metric type (“statsd”, “app_check”, “service_check”, or “jmx”).

A third entry spells out whether “included” or “excluded”, followed by the metric name. Finally, inside the last entry (in parentheses), there is information about filter applied and its effect (’+’ or ‘-’, meaning “include” or “exclude”).

With this example filter rule set:

  - include:*
  - exclude: mongo.statsd.*

We might see the following INFO-level log entries (timestamps stripped):

-[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.*])
+[statsd] metric included: mongo.statsd.netIn (filter: +[*])

Last modified September 23, 2022