Percentage of Change Alerts

Percentage of Change Alerts let you receive alerts when your metrics change considerably over time. Instead of setting a fixed threshold for when a metric goes above or below a certain value, use Percentage of Change Alerts to be notified when the metric changes by a certain percentage. This is ideal for environments with multiple regions where traffic and usage variations occur. By setting up a Percentage of Change alert, you can detect changes relative to the baseline, rather than just using a static threshold.

When to Use Percentage of Change Alerts

• A historical baseline can be established. For example, steady network traffic
• Metric is relatively steady. For example, database disk usage
• An abrupt metric spike. For example, request latency

When Not to Use Percentage of Change Alerts

• Metric is expected to fluctuate significantly over time
• Metric is noisy and unreliable
• Metric has a low baseline

Define a Percentage of Change Alert

To create a Percentage of Change Alert:

1. Log in to Sysdig Monitor.

2. Select Alerts.

3. Select New Alert > Percentage of Change.

Percentage of Change Alerts compare a shorter time aggregation to a longer one and trigger an alert if the difference between the two time aggregations exceeds a threshold defined by you.

For example, if you want to be alerted on an increase in database latency, you can configure a Percentage of Change Alerts on the relevant metric, such as database_query_duration_seconds, comparing the last 5 minutes with the last 1 hour. If the change in the metric between the two time time aggregations exceeds the custom-defined threshold, the Percentage of Change Alert will trigger and you will receive an alert notification.

Percentage of Change Alerts Resolution

Unlike alerts with static thresholds which resolve when the threshold is no longer met, Percentage of Change Alerts resolve when difference between the shorter interval and the longer interval no longer violate the user-defined threshold.

For instance, the following graph shows database latency that increases significantly after 3:00 and remains high. Using a static threshold of 5s for a Threshold Alert would result in an alert that remains triggered.

On the other hand, setting a threshold of 50% for a Percentage of Change Alert would result in an alert that triggers for the initial spike and then resolves. This is because the difference between the last 5 minutes and the last 1 hour is no longer significant.

Prevent Automatic Alert Resolution

In order to prevent an incident from being automatically closed when a Percentage of Change Alert no longer violates the threshold, you can configure an alert to not send alert resolutions to the notification channel when an alert resolves. This can help prevent confusion in the on-call process as an alert resolution does not necessarily mean that an incident has been resolved.

For more information, see Notify when Resolved.